2024-06-14T22:17:22.053 INFO:root:teuthology version: 0.0.1.dev316+gc1b06c2 2024-06-14T22:17:22.053 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954 --owner scheduled_yuriw@teuthology --name yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi --verbose --description upgrade:quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} -- /home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954/orig.config.yaml 2024-06-14T22:17:22.105 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-14T22:17:22.201 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954 branch: wip-yuri5-testing-2024-06-08-0902-reef description: upgrade:quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} email: yweinste@redhat.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps - echo wait for minority of mons to upgrade - while ! ceph mon versions | grep $sha1 ; do sleep 2 ; done - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for majority of mons to upgrade - 'while ! ceph mon versions | grep $sha1 | egrep '': [23]'' ; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for all mons to upgrade - 'while ! ceph mon versions | grep $sha1 | grep '': 3'' ; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for half of osds to upgrade - 'while ! ceph osd versions | grep $sha1 | egrep '': [45678]''; do sleep 2 ; done' - ceph orch upgrade pause - ceph orch ps - ceph orch ps - ceph versions first-half-tasks: - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' first_in_suite: false job_id: '7755954' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run basic import/export cli tests for rbd ' name: yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-06-08-0902-reef ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-reef-drop-centos8 sha1: 906f73e382c3916fe5507b8227cc56e63a20794f owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - ceph orch upgrade resume - sleep 60 - echo wait for upgrade to complete - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' seed: 5916 sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: quincy clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 111/120000 suite: upgrade:quincy-x suite_branch: wip-reef-drop-centos8 suite_path: /home/teuthworker/src/github.com_cbodley_ceph_906f73e382c3916fe5507b8227cc56e63a20794f/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: 906f73e382c3916fe5507b8227cc56e63a20794f targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHoYdELwPy+YsIoOm6aQSmH5kuvE3rpmMQ8M1fy06by0j1PqgrkMUxIXY5yorIB9KGUMtX3ckdEh+7IVO8Bg0KQ= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEKU43GMZdaCs9TjY9PGgo/sOEnYGjsbG0DKjjqc3wf4BS0nWJxWJ+yygayFOzmAA8iJD3wGeD/F+Xoo60hFyU8= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: branch: quincy exclude_packages: - ceph-volume - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c1b06c2b5d3a02a3583265a115f09e052855bcf2 timestamp: 2024-06-14_18:07:03 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-14T22:17:22.201 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_906f73e382c3916fe5507b8227cc56e63a20794f/qa; will attempt to use it 2024-06-14T22:17:22.202 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_906f73e382c3916fe5507b8227cc56e63a20794f/qa/tasks 2024-06-14T22:17:22.202 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-14T22:17:22.203 INFO:teuthology.task.internal:Checking packages... 2024-06-14T22:17:22.223 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6f4b09560ed399c3f139402a02f389c2fc8bb2cc' 2024-06-14T22:17:22.223 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T22:17:22.223 INFO:teuthology.packaging:ref: None 2024-06-14T22:17:22.223 INFO:teuthology.packaging:tag: None 2024-06-14T22:17:22.223 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-06-08-0902-reef 2024-06-14T22:17:22.223 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:17:22.223 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri5-testing-2024-06-08-0902-reef 2024-06-14T22:17:22.367 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1443.g6f4b0956 2024-06-14T22:17:22.368 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-14T22:17:22.375 INFO:teuthology.task.internal:no buildpackages task found 2024-06-14T22:17:22.375 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-14T22:17:22.423 INFO:teuthology.task.internal:Saving configuration 2024-06-14T22:17:22.442 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-14T22:17:22.457 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-14T22:17:22.481 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-14 22:12:25.594883', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHoYdELwPy+YsIoOm6aQSmH5kuvE3rpmMQ8M1fy06by0j1PqgrkMUxIXY5yorIB9KGUMtX3ckdEh+7IVO8Bg0KQ='} 2024-06-14T22:17:22.505 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-14 22:12:25.596740', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEKU43GMZdaCs9TjY9PGgo/sOEnYGjsbG0DKjjqc3wf4BS0nWJxWJ+yygayFOzmAA8iJD3wGeD/F+Xoo60hFyU8='} 2024-06-14T22:17:22.505 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-14T22:17:22.511 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-06-14T22:17:22.512 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-06-14T22:17:22.512 INFO:teuthology.run_tasks:Running task console_log... 2024-06-14T22:17:22.591 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc99b31bac0>, signals=[15]) 2024-06-14T22:17:22.591 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-14T22:17:22.595 INFO:teuthology.task.internal:Opening connections... 2024-06-14T22:17:22.595 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-06-14T22:17:22.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:17:22.672 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-06-14T22:17:22.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:17:22.742 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-14T22:17:22.749 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-06-14T22:17:22.767 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-06-14T22:17:22.767 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-06-14T22:17:22.823 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-06-14T22:17:22.823 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-14T22:17:22.824 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-14T22:17:22.825 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-06-14T22:17:22.843 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-06-14T22:17:22.860 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-06-14T22:17:22.860 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-06-14T22:17:22.917 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-14T22:17:22.918 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-14T22:17:22.919 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-06-14T22:17:22.943 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-14T22:17:22.952 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-14T22:17:22.990 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-14T22:17:22.990 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-06-14T22:17:22.992 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-06-14T22:17:23.009 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-14T22:17:23.014 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-14T22:17:23.014 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-06-14T22:17:23.049 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-06-14T22:17:23.167 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-14T22:17:23.222 INFO:teuthology.run_tasks:Running task kernel... 2024-06-14T22:17:23.237 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-14T22:17:23.237 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-14T22:17:23.237 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-14T22:17:23.238 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-14T22:17:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:17:23.254 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-06-14T22:17:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:17:23.255 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-06-14T22:17:23.310 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-437.el9.x86_64 2024-06-14T22:17:23.311 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-437.el9.x86_64 2024-06-14T22:17:23.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-06-14T22:17:23.313 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-437.el9.x86_64 2024-06-14T22:17:23.313 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.14.0-437.el9.x86_64 2024-06-14T22:17:23.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-06-14T22:17:25.662 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-06-14T22:17:26.343 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.1 MB 00:01 2024-06-14T22:17:27.496 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-06-14T22:17:32.001 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 20 MB 00:04 2024-06-14T22:17:33.755 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.3 MB 00:02 2024-06-14T22:17:35.639 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 17 kB 00:00 2024-06-14T22:17:36.610 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-06-14T22:17:37.344 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.3 MB 00:01 2024-06-14T22:17:39.361 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 17 kB 00:00 2024-06-14T22:17:40.315 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-06-14T22:17:41.749 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-06-14T22:17:43.764 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-14T22:17:43.764 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-14T22:17:43.856 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:17:43.871 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:17:43.871 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-06-14T22:17:43.871 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:17:43.871 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:17:43.871 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-14T22:17:43.871 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-14T22:17:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-14T22:17:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-14T22:17:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-14T22:17:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:17:43.872 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:17:43.872 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:17:43.872 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-06-14T22:17:43.872 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:17:43.873 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 81 M 2024-06-14T22:17:43.873 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 124 M 2024-06-14T22:17:43.873 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:17:44.159 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 6.7 MB/s | 989 kB 00:00 2024-06-14T22:17:44.418 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-06-14T22:17:44.801 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-core-5.14.0-457.el9.x86_6 45 MB/s | 29 MB 00:00 2024-06-14T22:17:45.101 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 32 MB/s | 35 MB 00:01 2024-06-14T22:17:45.101 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:17:45.102 INFO:teuthology.orchestra.run.smithi086.stdout:Total 66 MB/s | 81 MB 00:01 2024-06-14T22:17:45.595 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-06-14T22:17:45.801 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:17:46.134 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:17:46.135 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:17:47.093 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:17:47.093 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:17:47.660 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-14T22:17:47.660 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-14T22:17:47.755 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:17:47.763 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:17:47.764 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-06-14T22:17:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:17:47.764 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 81 M 2024-06-14T22:17:47.764 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 124 M 2024-06-14T22:17:47.764 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:17:48.197 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 2.4 MB/s | 989 kB 00:00 2024-06-14T22:17:49.495 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:17:50.223 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-14T22:17:50.318 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T22:17:52.971 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T22:17:53.145 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T22:17:56.480 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 1.8 MB/s | 16 MB 00:08 2024-06-14T22:17:57.855 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T22:17:57.992 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:02.725 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:04.604 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): kernel-modules-core-5.14.0-457.el9.x86_6 1.8 MB/s | 29 MB 00:16 2024-06-14T22:18:06.887 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 1.8 MB/s | 35 MB 00:19 2024-06-14T22:18:06.887 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:18:06.887 INFO:teuthology.orchestra.run.smithi137.stdout:Total 4.2 MB/s | 81 MB 00:19 2024-06-14T22:18:07.585 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:18:07.918 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:18:07.919 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:18:08.861 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:18:08.861 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:18:11.284 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:18:11.948 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-14T22:18:12.037 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T22:18:14.682 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T22:18:14.840 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T22:18:19.563 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T22:18:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:24.432 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:32.325 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:32.365 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-14T22:18:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T22:18:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T22:18:33.506 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:33.506 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:18:33.506 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:18:33.506 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-14T22:18:33.506 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-14T22:18:33.506 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-14T22:18:33.506 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-14T22:18:33.506 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:18:33.506 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:18:33.866 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-06-14T22:18:33.928 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-14T22:18:33.929 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-14T22:18:33.929 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-14T22:18:33.929 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-14T22:18:33.929 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-14T22:18:33.930 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-14T22:18:33.930 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-06-14T22:18:33.930 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-06-14T22:18:33.981 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-14T22:18:33.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-14T22:18:33.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-06-14T22:18:34.012 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-06-14T22:18:34.012 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-06-14T22:18:34.012 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-06-14T22:18:34.012 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2024-06-14T22:18:34.012 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-06-14T22:18:34.012 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-14T22:18:34.012 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-06-14T22:18:34.012 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-14T22:18:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-14T22:18:34.014 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-06-14T22:18:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:18:34.014 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-06-14T22:18:34.014 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-14T22:18:34.015 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-457.el9.x86_64 2024-06-14T22:18:34.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-14T22:18:34.593 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-06-14T22:18:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-14T22:18:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-06-14T22:18:36.092 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-06-14T22:18:36.107 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.lSYTGHsAC1 2024-06-14T22:18:36.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lSYTGHsAC1 2024-06-14T22:18:36.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.lSYTGHsAC1 2024-06-14T22:18:36.306 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.lSYTGHsAC1 is 6KB 2024-06-14T22:18:36.326 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.lSYTGHsAC1 2024-06-14T22:18:36.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-06-14T22:18:36.406 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-14T22:18:36.406 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-14T22:18:36.407 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-14T22:18:36.407 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-14T22:18:36.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-14T22:18:36.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-06-14T22:18:53.767 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:53.807 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:54.085 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:54.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-14T22:18:54.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T22:18:54.086 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T22:18:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T22:18:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:18:54.840 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:18:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-14T22:18:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-14T22:18:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-14T22:18:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-14T22:18:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:18:54.840 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:18:55.208 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-06-14T22:18:55.283 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-14T22:18:55.283 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-14T22:18:55.283 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-14T22:18:55.284 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-14T22:18:55.284 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-14T22:18:55.284 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-06-14T22:18:55.284 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-06-14T22:18:55.284 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-06-14T22:18:55.336 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-14T22:18:55.337 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-14T22:18:55.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-06-14T22:18:55.366 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-06-14T22:18:55.366 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-06-14T22:18:55.366 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-06-14T22:18:55.366 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 68.el9 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8264254 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-06-14T22:18:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:18:55.368 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-14T22:18:55.368 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-14T22:18:55.368 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-14T22:18:55.368 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-06-14T22:18:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:18:55.368 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-06-14T22:18:55.368 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-14T22:18:55.369 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 5.14.0-457.el9.x86_64 2024-06-14T22:18:55.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-14T22:18:55.961 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-06-14T22:18:57.387 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-14T22:18:57.420 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-06-14T22:18:57.422 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-06-14T22:18:57.438 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.zv5hPSx0QF 2024-06-14T22:18:57.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zv5hPSx0QF 2024-06-14T22:18:57.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.zv5hPSx0QF 2024-06-14T22:18:57.641 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.zv5hPSx0QF is 6KB 2024-06-14T22:18:57.661 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.zv5hPSx0QF 2024-06-14T22:18:57.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-06-14T22:18:57.737 INFO:teuthology.orchestra.run.smithi137.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-14T22:18:57.737 INFO:teuthology.orchestra.run.smithi137.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-14T22:18:57.737 INFO:teuthology.orchestra.run.smithi137.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-14T22:18:57.738 INFO:teuthology.orchestra.run.smithi137.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-14T22:18:57.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-14T22:18:57.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-06-14T22:19:06.567 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-14T22:19:06.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-14T22:19:06.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:19:25.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-06-14T22:19:27.894 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-06-14T22:19:27.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-06-14T22:19:27.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:19:34.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-14T22:19:34.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:19:46.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-06-14T22:19:55.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-06-14T22:19:55.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:19:58.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-06-14T22:20:10.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-06-14T22:20:10.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:20:28.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-06-14T22:20:34.099 DEBUG:teuthology.orchestra.remote:timed out 2024-06-14T22:20:43.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-06-14T22:20:43.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:20:44.194 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-06-14T22:20:44.532 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-06-14T22:20:44.533 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-457.el9.x86_64"... 2024-06-14T22:20:44.533 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-06-14T22:20:44.547 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-457.el9.x86_64 2024-06-14T22:20:44.547 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-14T22:20:44.547 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-14T22:20:44.548 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-14T22:20:45.548 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-06-14T22:20:45.549 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-14T22:20:45.619 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-06-14T22:20:45.657 DEBUG:teuthology.parallel:result is None 2024-06-14T22:20:46.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-14T22:20:46.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:20:46.356 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-06-14T22:20:46.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-14T22:20:46.712 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-457.el9.x86_64"... 2024-06-14T22:20:46.713 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-06-14T22:20:46.729 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-457.el9.x86_64 2024-06-14T22:20:46.729 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-14T22:20:46.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-14T22:20:46.729 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-14T22:20:47.730 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-14T22:20:47.731 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-14T22:20:47.810 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-06-14T22:20:47.849 DEBUG:teuthology.parallel:result is None 2024-06-14T22:20:47.849 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-14T22:20:47.856 INFO:teuthology.task.internal:Creating test directory... 2024-06-14T22:20:47.856 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-14T22:20:47.859 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-14T22:20:47.876 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-14T22:20:47.892 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-14T22:20:47.925 INFO:teuthology.task.internal:Creating archive directory... 2024-06-14T22:20:47.926 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-14T22:20:47.928 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-14T22:20:47.990 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-14T22:20:47.996 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-14T22:20:47.997 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-14T22:20:47.999 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-14T22:20:48.028 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-14T22:20:48.031 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-14T22:20:48.038 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-14T22:20:48.042 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-14T22:20:48.044 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-14T22:20:48.050 INFO:teuthology.task.internal:Configuring sudo... 2024-06-14T22:20:48.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-14T22:20:48.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-14T22:20:48.111 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-14T22:20:48.119 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-14T22:20:48.120 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-14T22:20:48.153 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-14T22:20:48.168 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-14T22:20:48.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-14T22:20:48.296 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-14T22:20:48.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-14T22:20:48.419 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:20:48.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-14T22:20:48.483 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-14T22:20:48.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-14T22:20:48.642 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-14T22:20:48.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-14T22:20:48.774 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:20:48.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-14T22:20:48.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-06-14T22:20:48.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-06-14T22:20:48.887 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-14T22:20:48.971 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-14T22:20:49.428 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-14T22:20:49.439 INFO:teuthology.task.internal:Starting timer... 2024-06-14T22:20:49.439 INFO:teuthology.run_tasks:Running task pcp... 2024-06-14T22:20:49.485 INFO:teuthology.run_tasks:Running task selinux... 2024-06-14T22:20:49.515 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-06-14T22:20:49.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-06-14T22:20:49.563 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-06-14T22:20:49.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-06-14T22:20:49.928 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-06-14T22:20:49.930 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-14T22:20:49.930 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-06-14T22:20:49.984 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-06-14T22:20:49.984 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-06-14T22:20:50.019 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-06-14T22:20:50.020 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-06-14T22:20:50.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-14T22:20:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:20:50.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-14T22:20:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:20:50.085 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-14T22:20:50.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-06-14T22:20:50.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-06-14T22:20:50.159 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-14T22:20:50.169 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-14T22:20:50.172 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-14T22:20:50.298 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-14T22:20:50.308 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-14T22:20:50.309 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-14T22:25:06.210 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-06-14T22:25:06.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-14T22:25:06.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:25:06.293 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-06-14T22:25:06.378 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-06-14T22:25:06.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-06-14T22:25:06.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T22:25:06.454 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-06-14T22:25:06.538 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-06-14T22:25:06.538 INFO:teuthology.run_tasks:Running task clock... 2024-06-14T22:25:06.553 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-14T22:25:06.553 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-14T22:25:06.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-14T22:25:06.557 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-14T22:25:06.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-14T22:25:06.589 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-14T22:25:06.605 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-14T22:25:06.608 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-14T22:25:06.622 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-14T22:25:06.631 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-06-14T22:25:06.643 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-06-14T22:25:06.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-06-14T22:25:06.658 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-06-14T22:25:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-14T22:25:06.672 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-14T22:25:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-14T22:25:06.685 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-14T22:25:06.727 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-06-14T22:25:06.731 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-14T22:25:06.731 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-06-14T22:25:06.731 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T22:25:06.731 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T22:25:06.732 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T22:25:06.732 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T22:25:06.741 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-06-14T22:25:06.745 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-14T22:25:06.745 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-06-14T22:25:06.745 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T22:25:06.746 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T22:25:06.746 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T22:25:06.746 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T22:25:06.746 INFO:teuthology.run_tasks:Running task pexec... 2024-06-14T22:25:06.794 INFO:teuthology.task.pexec:Executing custom commands... 2024-06-14T22:25:06.794 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi086.front.sepia.ceph.com 2024-06-14T22:25:06.794 DEBUG:teuthology.orchestra.run.smithi086:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-06-14T22:25:06.795 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi137.front.sepia.ceph.com 2024-06-14T22:25:06.795 DEBUG:teuthology.orchestra.run.smithi137:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-06-14T22:25:07.395 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:06 ago on Fri 14 Jun 2024 10:24:01 PM UTC. 2024-06-14T22:25:07.398 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:06 ago on Fri 14 Jun 2024 10:24:01 PM UTC. 2024-06-14T22:25:07.515 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:25:07.520 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:25:07.521 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:25:07.522 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-06-14T22:25:07.522 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:25:07.522 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:25:07.522 INFO:teuthology.orchestra.run.smithi137.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-06-14T22:25:07.522 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:07.522 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:25:07.522 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:25:07.522 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-06-14T22:25:07.522 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:07.523 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.1 M 2024-06-14T22:25:07.523 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 9.8 M 2024-06-14T22:25:07.523 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:25:07.526 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:25:07.526 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-06-14T22:25:07.526 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:25:07.526 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:25:07.526 INFO:teuthology.orchestra.run.smithi086.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-06-14T22:25:07.526 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:07.526 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:25:07.527 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:25:07.527 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-14T22:25:07.527 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:07.527 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.1 M 2024-06-14T22:25:07.527 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 9.8 M 2024-06-14T22:25:07.527 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:25:07.892 INFO:teuthology.orchestra.run.smithi086.stdout:runc-1.1.12-2.el9.x86_64.rpm 11 MB/s | 3.1 MB 00:00 2024-06-14T22:25:07.892 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:25:07.892 INFO:teuthology.orchestra.run.smithi086.stdout:Total 8.6 MB/s | 3.1 MB 00:00 2024-06-14T22:25:07.928 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:25:07.934 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:25:07.934 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:25:07.965 INFO:teuthology.orchestra.run.smithi137.stdout:runc-1.1.12-2.el9.x86_64.rpm 11 MB/s | 3.1 MB 00:00 2024-06-14T22:25:07.966 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:25:07.966 INFO:teuthology.orchestra.run.smithi137.stdout:Total 7.1 MB/s | 3.1 MB 00:00 2024-06-14T22:25:08.002 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:25:08.008 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:25:08.008 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:25:08.016 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:25:08.017 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:25:08.094 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:25:08.095 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:25:08.241 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:25:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:25:08.334 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-06-14T22:25:08.409 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-06-14T22:25:08.908 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-06-14T22:25:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-06-14T22:25:09.166 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-06-14T22:25:09.166 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:09.166 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:25:09.166 INFO:teuthology.orchestra.run.smithi086.stdout: runc-4:1.1.12-2.el9.x86_64 2024-06-14T22:25:09.166 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:09.166 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:25:09.222 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-06-14T22:25:09.222 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:09.222 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:25:09.222 INFO:teuthology.orchestra.run.smithi137.stdout: runc-4:1.1.12-2.el9.x86_64 2024-06-14T22:25:09.222 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:09.222 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:25:09.451 DEBUG:teuthology.parallel:result is None 2024-06-14T22:25:09.485 DEBUG:teuthology.parallel:result is None 2024-06-14T22:25:09.486 INFO:teuthology.run_tasks:Running task install... 2024-06-14T22:25:09.499 DEBUG:teuthology.task.install:project ceph 2024-06-14T22:25:09.499 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'}} 2024-06-14T22:25:09.499 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'} 2024-06-14T22:25:09.499 INFO:teuthology.task.install:Using flavor: default 2024-06-14T22:25:09.504 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-14T22:25:09.504 INFO:teuthology.task.install:extra packages: [] 2024-06-14T22:25:09.504 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'tag': None, 'wait_for_package': False} 2024-06-14T22:25:09.505 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T22:25:09.505 INFO:teuthology.packaging:ref: None 2024-06-14T22:25:09.505 INFO:teuthology.packaging:tag: None 2024-06-14T22:25:09.505 INFO:teuthology.packaging:branch: quincy 2024-06-14T22:25:09.505 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:25:09.505 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-14T22:25:09.507 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'tag': None, 'wait_for_package': False} 2024-06-14T22:25:09.507 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T22:25:09.507 INFO:teuthology.packaging:ref: None 2024-06-14T22:25:09.508 INFO:teuthology.packaging:tag: None 2024-06-14T22:25:09.508 INFO:teuthology.packaging:branch: quincy 2024-06-14T22:25:09.508 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:25:09.508 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-14T22:25:09.740 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/ 2024-06-14T22:25:09.740 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1008.ge47ff4b8 2024-06-14T22:25:09.823 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/ 2024-06-14T22:25:09.824 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1008.ge47ff4b8 2024-06-14T22:25:09.856 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-14T22:25:09.856 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:25:09.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-14T22:25:09.883 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-14T22:25:09.883 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T22:25:09.883 INFO:teuthology.packaging:ref: None 2024-06-14T22:25:09.883 INFO:teuthology.packaging:tag: None 2024-06-14T22:25:09.883 INFO:teuthology.packaging:branch: quincy 2024-06-14T22:25:09.883 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:25:09.883 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-14T22:25:09.930 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-14T22:25:09.930 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:25:09.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-14T22:25:09.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-14T22:25:09.956 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-14T22:25:09.956 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T22:25:09.956 INFO:teuthology.packaging:ref: None 2024-06-14T22:25:09.956 INFO:teuthology.packaging:tag: None 2024-06-14T22:25:09.956 INFO:teuthology.packaging:branch: quincy 2024-06-14T22:25:09.956 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:25:09.957 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-14T22:25:10.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-14T22:25:10.024 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-14T22:25:10.090 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-06-14T22:25:10.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-06-14T22:25:10.097 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-14T22:25:10.167 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-06-14T22:25:10.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-06-14T22:25:10.322 INFO:teuthology.orchestra.run.smithi086.stdout:45 files removed 2024-06-14T22:25:10.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-06-14T22:25:10.404 INFO:teuthology.orchestra.run.smithi137.stdout:45 files removed 2024-06-14T22:25:10.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-06-14T22:25:10.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 219 kB/s | 77 kB 00:00 2024-06-14T22:25:11.121 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 187 kB/s | 77 kB 00:00 2024-06-14T22:25:11.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-06-14T22:25:11.480 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-06-14T22:25:11.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 5.7 kB/s | 1.8 kB 00:00 2024-06-14T22:25:11.839 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 5.7 kB/s | 1.8 kB 00:00 2024-06-14T22:25:12.797 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.1 MB 00:00 2024-06-14T22:25:13.031 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.1 MB 00:01 2024-06-14T22:25:15.867 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-06-14T22:25:16.215 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 20 MB 00:02 2024-06-14T22:25:19.949 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.3 MB 00:00 2024-06-14T22:25:21.626 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 17 kB 00:00 2024-06-14T22:25:21.925 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.3 MB 00:02 2024-06-14T22:25:22.562 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-06-14T22:25:23.601 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 17 kB 00:00 2024-06-14T22:25:24.838 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-06-14T22:25:27.843 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-06-14T22:25:30.128 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-06-14T22:25:30.275 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:25:30.277 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 14 M 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.1 M 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 5.0 M 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 22 M 2024-06-14T22:25:30.278 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 638 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 457 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 4.9 M 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-14T22:25:30.279 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 45 k 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 118 k 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 156 k 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 320 k 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 292 k 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 100 k 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout:Install 22 Packages 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-06-14T22:25:30.280 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:30.281 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 58 M 2024-06-14T22:25:30.281 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:25:31.646 INFO:teuthology.orchestra.run.smithi137.stdout:(1/24): ceph-base-17.2.7-1008.ge47ff4b8.el9.x86 4.1 MB/s | 5.0 MB 00:01 2024-06-14T22:25:31.686 INFO:teuthology.orchestra.run.smithi137.stdout:(2/24): ceph-selinux-17.2.7-1008.ge47ff4b8.el9. 608 kB/s | 25 kB 00:00 2024-06-14T22:25:31.811 INFO:teuthology.orchestra.run.smithi137.stdout:(3/24): libcephfs2-17.2.7-1008.ge47ff4b8.el9.x8 5.0 MB/s | 638 kB 00:00 2024-06-14T22:25:31.903 INFO:teuthology.orchestra.run.smithi137.stdout:(4/24): libradosstriper1-17.2.7-1008.ge47ff4b8. 4.9 MB/s | 457 kB 00:00 2024-06-14T22:25:32.362 INFO:teuthology.orchestra.run.smithi137.stdout:(5/24): ceph-radosgw-17.2.7-1008.ge47ff4b8.el9. 7.3 MB/s | 14 MB 00:01 2024-06-14T22:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout:(6/24): python3-ceph-argparse-17.2.7-1008.ge47f 911 kB/s | 45 kB 00:00 2024-06-14T22:25:32.512 INFO:teuthology.orchestra.run.smithi137.stdout:(7/24): librgw2-17.2.7-1008.ge47ff4b8.el9.x86_6 8.1 MB/s | 4.9 MB 00:00 2024-06-14T22:25:32.537 INFO:teuthology.orchestra.run.smithi137.stdout:(8/24): python3-ceph-common-17.2.7-1008.ge47ff4 939 kB/s | 118 kB 00:00 2024-06-14T22:25:32.563 INFO:teuthology.orchestra.run.smithi137.stdout:(9/24): python3-cephfs-17.2.7-1008.ge47ff4b8.el 3.0 MB/s | 156 kB 00:00 2024-06-14T22:25:32.576 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:25:32.578 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:25:32.578 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-14T22:25:32.578 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 14 M 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.1 M 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 5.0 M 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 22 M 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 638 k 2024-06-14T22:25:32.579 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 457 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 4.9 M 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 45 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 118 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 156 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-14T22:25:32.580 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 320 k 2024-06-14T22:25:32.581 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 292 k 2024-06-14T22:25:32.581 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 100 k 2024-06-14T22:25:32.581 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-14T22:25:32.581 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:32.581 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:25:32.581 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:25:32.581 INFO:teuthology.orchestra.run.smithi086.stdout:Install 22 Packages 2024-06-14T22:25:32.581 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-06-14T22:25:32.581 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:32.582 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 58 M 2024-06-14T22:25:32.582 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:25:32.596 INFO:teuthology.orchestra.run.smithi137.stdout:(10/24): python3-rados-17.2.7-1008.ge47ff4b8.el 5.3 MB/s | 320 kB 00:00 2024-06-14T22:25:32.647 INFO:teuthology.orchestra.run.smithi137.stdout:(11/24): python3-rbd-17.2.7-1008.ge47ff4b8.el9. 3.4 MB/s | 292 kB 00:00 2024-06-14T22:25:32.674 INFO:teuthology.orchestra.run.smithi137.stdout:(12/24): python3-rgw-17.2.7-1008.ge47ff4b8.el9. 1.3 MB/s | 100 kB 00:00 2024-06-14T22:25:32.806 INFO:teuthology.orchestra.run.smithi137.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2024-06-14T22:25:32.831 INFO:teuthology.orchestra.run.smithi137.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 871 kB/s | 160 kB 00:00 2024-06-14T22:25:32.923 INFO:teuthology.orchestra.run.smithi137.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-06-14T22:25:32.957 INFO:teuthology.orchestra.run.smithi137.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-06-14T22:25:32.982 INFO:teuthology.orchestra.run.smithi137.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 721 kB/s | 42 kB 00:00 2024-06-14T22:25:33.083 INFO:teuthology.orchestra.run.smithi137.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.4 MB/s | 309 kB 00:00 2024-06-14T22:25:33.108 INFO:teuthology.orchestra.run.smithi137.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-06-14T22:25:33.124 INFO:teuthology.orchestra.run.smithi137.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 341 kB/s | 49 kB 00:00 2024-06-14T22:25:33.175 INFO:teuthology.orchestra.run.smithi137.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-06-14T22:25:33.504 INFO:teuthology.orchestra.run.smithi086.stdout:(1/24): ceph-base-17.2.7-1008.ge47ff4b8.el9.x86 8.0 MB/s | 5.0 MB 00:00 2024-06-14T22:25:33.562 INFO:teuthology.orchestra.run.smithi086.stdout:(2/24): ceph-selinux-17.2.7-1008.ge47ff4b8.el9. 432 kB/s | 25 kB 00:00 2024-06-14T22:25:33.646 INFO:teuthology.orchestra.run.smithi086.stdout:(3/24): libcephfs2-17.2.7-1008.ge47ff4b8.el9.x8 7.5 MB/s | 638 kB 00:00 2024-06-14T22:25:33.721 INFO:teuthology.orchestra.run.smithi086.stdout:(4/24): libradosstriper1-17.2.7-1008.ge47ff4b8. 5.9 MB/s | 457 kB 00:00 2024-06-14T22:25:33.800 INFO:teuthology.orchestra.run.smithi137.stdout:(22/24): librados2-17.2.7-1008.ge47ff4b8.el9.x8 4.6 MB/s | 3.1 MB 00:00 2024-06-14T22:25:34.076 INFO:teuthology.orchestra.run.smithi137.stdout:(23/24): librbd1-17.2.7-1008.ge47ff4b8.el9.x86_ 3.3 MB/s | 2.9 MB 00:00 2024-06-14T22:25:34.113 INFO:teuthology.orchestra.run.smithi086.stdout:(5/24): librgw2-17.2.7-1008.ge47ff4b8.el9.x86_6 13 MB/s | 4.9 MB 00:00 2024-06-14T22:25:34.164 INFO:teuthology.orchestra.run.smithi086.stdout:(6/24): python3-ceph-argparse-17.2.7-1008.ge47f 902 kB/s | 45 kB 00:00 2024-06-14T22:25:34.214 INFO:teuthology.orchestra.run.smithi086.stdout:(7/24): python3-ceph-common-17.2.7-1008.ge47ff4 2.3 MB/s | 118 kB 00:00 2024-06-14T22:25:34.264 INFO:teuthology.orchestra.run.smithi086.stdout:(8/24): python3-cephfs-17.2.7-1008.ge47ff4b8.el 3.0 MB/s | 156 kB 00:00 2024-06-14T22:25:34.315 INFO:teuthology.orchestra.run.smithi086.stdout:(9/24): python3-rados-17.2.7-1008.ge47ff4b8.el9 6.2 MB/s | 320 kB 00:00 2024-06-14T22:25:34.365 INFO:teuthology.orchestra.run.smithi086.stdout:(10/24): python3-rbd-17.2.7-1008.ge47ff4b8.el9. 5.7 MB/s | 292 kB 00:00 2024-06-14T22:25:34.415 INFO:teuthology.orchestra.run.smithi086.stdout:(11/24): python3-rgw-17.2.7-1008.ge47ff4b8.el9. 2.0 MB/s | 100 kB 00:00 2024-06-14T22:25:34.534 INFO:teuthology.orchestra.run.smithi137.stdout:(24/24): ceph-common-17.2.7-1008.ge47ff4b8.el9. 5.5 MB/s | 22 MB 00:04 2024-06-14T22:25:34.539 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:25:34.539 INFO:teuthology.orchestra.run.smithi137.stdout:Total 14 MB/s | 58 MB 00:04 2024-06-14T22:25:34.574 INFO:teuthology.orchestra.run.smithi086.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-06-14T22:25:34.602 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:25:34.624 INFO:teuthology.orchestra.run.smithi086.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-06-14T22:25:34.636 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:25:34.636 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:25:34.724 INFO:teuthology.orchestra.run.smithi086.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-06-14T22:25:34.774 INFO:teuthology.orchestra.run.smithi086.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-06-14T22:25:34.825 INFO:teuthology.orchestra.run.smithi086.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-06-14T22:25:34.892 INFO:teuthology.orchestra.run.smithi086.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.5 MB/s | 309 kB 00:00 2024-06-14T22:25:34.917 INFO:teuthology.orchestra.run.smithi086.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-14T22:25:34.934 INFO:teuthology.orchestra.run.smithi086.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 4.0 MB/s | 67 kB 00:00 2024-06-14T22:25:34.993 INFO:teuthology.orchestra.run.smithi086.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-06-14T22:25:35.162 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:25:35.162 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:25:35.209 INFO:teuthology.orchestra.run.smithi086.stdout:(21/24): librados2-17.2.7-1008.ge47ff4b8.el9.x8 14 MB/s | 3.1 MB 00:00 2024-06-14T22:25:35.410 INFO:teuthology.orchestra.run.smithi086.stdout:(22/24): librbd1-17.2.7-1008.ge47ff4b8.el9.x86_ 15 MB/s | 2.9 MB 00:00 2024-06-14T22:25:36.127 INFO:teuthology.orchestra.run.smithi086.stdout:(23/24): ceph-radosgw-17.2.7-1008.ge47ff4b8.el9 4.3 MB/s | 14 MB 00:03 2024-06-14T22:25:36.336 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:25:36.731 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-14T22:25:36.795 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T22:25:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T22:25:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-14T22:25:37.004 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T22:25:37.100 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T22:25:37.147 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-14T22:25:37.184 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-14T22:25:37.614 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-14T22:25:37.649 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-14T22:25:37.712 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-14T22:25:37.740 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-14T22:25:37.783 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 10/26 2024-06-14T22:25:37.820 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-14T22:25:37.899 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-14T22:25:37.927 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-14T22:25:37.978 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-14T22:25:38.029 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-14T22:25:38.060 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-14T22:25:38.101 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-14T22:25:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-14T22:25:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-14T22:25:38.512 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-14T22:25:38.577 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 19/26 2024-06-14T22:25:38.617 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 20/26 2024-06-14T22:25:40.026 INFO:teuthology.orchestra.run.smithi086.stdout:(24/24): ceph-common-17.2.7-1008.ge47ff4b8.el9. 3.1 MB/s | 22 MB 00:07 2024-06-14T22:25:40.031 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:25:40.031 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.7 MB/s | 58 MB 00:07 2024-06-14T22:25:40.094 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:25:40.129 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:25:40.129 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:25:40.661 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:25:40.662 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:25:41.092 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T22:25:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T22:25:41.666 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T22:25:41.717 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-14T22:25:41.801 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-14T22:25:41.801 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-14T22:25:41.801 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-14T22:25:41.801 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:41.827 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T22:25:41.849 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:25:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-14T22:25:42.314 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T22:25:42.404 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T22:25:42.492 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-14T22:25:42.539 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T22:25:42.636 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T22:25:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-14T22:25:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-14T22:25:43.142 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-14T22:25:43.175 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-14T22:25:43.238 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-14T22:25:43.266 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-14T22:25:43.309 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 10/26 2024-06-14T22:25:43.346 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-14T22:25:43.408 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-14T22:25:43.436 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-14T22:25:43.485 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-14T22:25:43.525 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-14T22:25:43.578 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-14T22:25:43.610 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-14T22:25:43.901 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-14T22:25:43.940 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-14T22:25:44.031 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-14T22:25:44.103 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 19/26 2024-06-14T22:25:44.134 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 20/26 2024-06-14T22:25:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T22:25:46.648 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T22:25:47.176 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T22:25:47.210 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-14T22:25:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-14T22:25:47.309 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-14T22:25:47.309 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-14T22:25:47.309 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:47.344 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T22:25:51.332 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T22:25:51.332 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-06-14T22:25:51.332 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-06-14T22:25:51.332 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-06-14T22:25:51.332 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-06-14T22:25:51.332 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-06-14T22:25:51.332 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-06-14T22:25:51.332 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-06-14T22:25:51.333 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:52.588 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-14T22:25:52.683 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-14T22:25:52.684 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:25:52.684 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-14T22:25:52.684 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-14T22:25:52.684 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-14T22:25:52.684 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:52.685 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-14T22:25:52.783 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-14T22:25:52.783 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T22:25:53.995 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T22:25:53.995 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/26 2024-06-14T22:25:53.995 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T22:25:53.995 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-14T22:25:53.995 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T22:25:53.995 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 6/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 7/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 8/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 9/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 10/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 13/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-14T22:25:53.996 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-14T22:25:53.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-14T22:25:53.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-14T22:25:53.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-14T22:25:53.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-14T22:25:53.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T22:25:53.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-14T22:25:53.997 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 25/26 2024-06-14T22:25:54.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T22:25:54.745 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:54.745 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-06-14T22:25:54.745 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-14T22:25:54.746 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-14T22:25:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:54.748 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:25:55.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-06-14T22:25:55.695 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:28 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:25:55.820 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:25:55.820 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 48 M 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 163 k 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-06-14T22:25:55.821 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:25:55.822 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 48 M 2024-06-14T22:25:55.822 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 192 M 2024-06-14T22:25:55.822 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:25:56.270 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): libcephsqlite-17.2.7-1008.ge47ff4b8.el9. 437 kB/s | 163 kB 00:00 2024-06-14T22:25:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T22:25:56.672 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-06-14T22:25:56.672 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-06-14T22:25:56.672 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-06-14T22:25:56.672 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-06-14T22:25:56.672 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-06-14T22:25:56.672 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-06-14T22:25:56.672 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-06-14T22:25:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:56.803 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 335 kB/s | 304 kB 00:00 2024-06-14T22:25:56.953 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 93 kB/s | 64 kB 00:00 2024-06-14T22:25:57.973 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-14T22:25:58.051 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-14T22:25:58.051 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:25:58.051 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-14T22:25:58.051 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-14T22:25:58.051 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-14T22:25:58.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:58.053 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-14T22:25:58.142 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-14T22:25:58.142 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T22:25:58.512 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): ceph-test-17.2.7-1008.ge47ff4b8.el9.x86_ 18 MB/s | 48 MB 00:02 2024-06-14T22:25:58.514 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:25:58.514 INFO:teuthology.orchestra.run.smithi137.stdout:Total 18 MB/s | 48 MB 00:02 2024-06-14T22:25:58.541 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:25:58.561 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:25:58.561 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:25:58.925 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:25:58.925 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:25:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 6/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 7/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 8/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 9/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 10/26 2024-06-14T22:25:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 13/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-14T22:25:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 25/26 2024-06-14T22:25:59.439 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:25:59.537 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-14T22:25:59.588 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-14T22:25:59.614 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-14T22:25:59.981 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T22:25:59.981 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:59.981 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-06-14T22:25:59.981 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-14T22:25:59.982 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:25:59.983 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-14T22:25:59.984 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:25:59.984 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:00.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-06-14T22:26:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:30 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:01.036 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 48 M 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 163 k 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-14T22:26:01.037 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:26:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-06-14T22:26:01.038 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 48 M 2024-06-14T22:26:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 192 M 2024-06-14T22:26:01.039 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:26:01.488 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): libcephsqlite-17.2.7-1008.ge47ff4b8.el9. 431 kB/s | 163 kB 00:00 2024-06-14T22:26:01.688 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 526 kB/s | 304 kB 00:00 2024-06-14T22:26:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 177 kB/s | 64 kB 00:00 2024-06-14T22:26:04.014 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): ceph-test-17.2.7-1008.ge47ff4b8.el9.x86_ 16 MB/s | 48 MB 00:02 2024-06-14T22:26:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:Total 16 MB/s | 48 MB 00:02 2024-06-14T22:26:04.044 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:26:04.063 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:26:04.063 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:26:04.083 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-14T22:26:04.170 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-14T22:26:04.427 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:26:04.428 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:26:04.954 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:26:05.028 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-14T22:26:05.075 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-14T22:26:05.099 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-14T22:26:06.340 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-14T22:26:06.340 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/4 2024-06-14T22:26:06.340 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/4 2024-06-14T22:26:06.340 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-14T22:26:06.609 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-14T22:26:06.609 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:06.610 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:06.610 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:06.610 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:06.610 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-14T22:26:06.610 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-14T22:26:06.610 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:06.610 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:06.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-06-14T22:26:07.474 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:40 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:07.603 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:07.606 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:07.606 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-06-14T22:26:07.606 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:07.606 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:07.606 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 6.5 k 2024-06-14T22:26:07.606 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-06-14T22:26:07.606 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.1 M 2024-06-14T22:26:07.606 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 1.4 M 2024-06-14T22:26:07.606 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 240 k 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.9 M 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 15 M 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-14T22:26:07.607 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-14T22:26:07.608 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout:Install 33 Packages 2024-06-14T22:26:07.609 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:07.610 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 27 M 2024-06-14T22:26:07.610 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 96 M 2024-06-14T22:26:07.610 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:08.546 INFO:teuthology.orchestra.run.smithi137.stdout:(1/33): ceph-17.2.7-1008.ge47ff4b8.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-06-14T22:26:08.830 INFO:teuthology.orchestra.run.smithi137.stdout:(2/33): ceph-mds-17.2.7-1008.ge47ff4b8.el9.x86_ 4.6 MB/s | 2.1 MB 00:00 2024-06-14T22:26:08.872 INFO:teuthology.orchestra.run.smithi137.stdout:(3/33): ceph-mgr-17.2.7-1008.ge47ff4b8.el9.x86_ 2.9 MB/s | 1.4 MB 00:00 2024-06-14T22:26:08.997 INFO:teuthology.orchestra.run.smithi137.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1008.ge47f 1.9 MB/s | 240 kB 00:00 2024-06-14T22:26:09.056 INFO:teuthology.orchestra.run.smithi137.stdout:(5/33): ceph-mon-17.2.7-1008.ge47ff4b8.el9.x86_ 7.7 MB/s | 3.9 MB 00:00 2024-06-14T22:26:09.492 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-14T22:26:09.563 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-14T22:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout:(6/33): ceph-osd-17.2.7-1008.ge47ff4b8.el9.x86_ 20 MB/s | 15 MB 00:00 2024-06-14T22:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 366 kB/s | 253 kB 00:00 2024-06-14T22:26:09.714 INFO:teuthology.orchestra.run.smithi137.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 800 kB/s | 106 kB 00:00 2024-06-14T22:26:09.832 INFO:teuthology.orchestra.run.smithi137.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 955 kB/s | 135 kB 00:00 2024-06-14T22:26:09.882 INFO:teuthology.orchestra.run.smithi137.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-06-14T22:26:09.966 INFO:teuthology.orchestra.run.smithi137.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-06-14T22:26:09.990 INFO:teuthology.orchestra.run.smithi137.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-06-14T22:26:10.582 INFO:teuthology.orchestra.run.smithi137.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 56 kB/s | 35 kB 00:00 2024-06-14T22:26:10.649 INFO:teuthology.orchestra.run.smithi137.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 77 kB/s | 51 kB 00:00 2024-06-14T22:26:10.708 INFO:teuthology.orchestra.run.smithi137.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 208 kB/s | 172 kB 00:00 2024-06-14T22:26:10.758 INFO:teuthology.orchestra.run.smithi137.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 402 kB/s | 43 kB 00:00 2024-06-14T22:26:10.800 INFO:teuthology.orchestra.run.smithi137.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 8.9 MB/s | 358 kB 00:00 2024-06-14T22:26:10.825 INFO:teuthology.orchestra.run.smithi137.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-06-14T22:26:10.851 INFO:teuthology.orchestra.run.smithi137.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-06-14T22:26:10.901 INFO:teuthology.orchestra.run.smithi137.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-06-14T22:26:10.926 INFO:teuthology.orchestra.run.smithi137.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 450 kB/s | 23 kB 00:00 2024-06-14T22:26:10.952 INFO:teuthology.orchestra.run.smithi137.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-06-14T22:26:10.985 INFO:teuthology.orchestra.run.smithi137.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-06-14T22:26:11.010 INFO:teuthology.orchestra.run.smithi137.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 792 kB/s | 46 kB 00:00 2024-06-14T22:26:11.035 INFO:teuthology.orchestra.run.smithi137.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-06-14T22:26:11.061 INFO:teuthology.orchestra.run.smithi137.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-06-14T22:26:11.086 INFO:teuthology.orchestra.run.smithi137.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-06-14T22:26:11.112 INFO:teuthology.orchestra.run.smithi137.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-06-14T22:26:11.137 INFO:teuthology.orchestra.run.smithi137.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-06-14T22:26:11.162 INFO:teuthology.orchestra.run.smithi137.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-06-14T22:26:11.187 INFO:teuthology.orchestra.run.smithi137.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-06-14T22:26:11.213 INFO:teuthology.orchestra.run.smithi137.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-06-14T22:26:11.263 INFO:teuthology.orchestra.run.smithi137.stdout:(33/33): python3-toml-0.10.2-6.el9.noarch.rpm 61 kB/s | 42 kB 00:00 2024-06-14T22:26:11.270 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:11.270 INFO:teuthology.orchestra.run.smithi137.stdout:Total 7.4 MB/s | 27 MB 00:03 2024-06-14T22:26:11.384 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:11.421 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:11.421 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:11.784 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:11.784 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:11.844 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-14T22:26:11.845 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/4 2024-06-14T22:26:11.845 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/4 2024-06-14T22:26:11.845 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-14T22:26:12.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-14T22:26:12.142 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:12.142 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:26:12.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:12.142 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:12.142 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-14T22:26:12.142 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-14T22:26:12.143 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:12.143 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:12.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-06-14T22:26:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-14T22:26:13.006 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-14T22:26:13.036 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:43 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:13.039 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-14T22:26:13.129 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-14T22:26:13.167 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:13.169 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:13.169 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-14T22:26:13.169 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:13.169 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:26:13.169 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 6.5 k 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.1 M 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 1.4 M 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 240 k 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.9 M 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 15 M 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-14T22:26:13.170 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-14T22:26:13.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-14T22:26:13.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections 2024-06-14T22:26:13.171 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-14T22:26:13.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-14T22:26:13.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-14T22:26:13.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-14T22:26:13.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-14T22:26:13.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-14T22:26:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-14T22:26:13.173 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-14T22:26:13.173 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-14T22:26:13.173 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:13.173 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:26:13.173 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:13.173 INFO:teuthology.orchestra.run.smithi086.stdout:Install 33 Packages 2024-06-14T22:26:13.173 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:13.174 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 27 M 2024-06-14T22:26:13.174 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 96 M 2024-06-14T22:26:13.174 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:26:13.208 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-14T22:26:13.250 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-14T22:26:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-14T22:26:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-14T22:26:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-14T22:26:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-14T22:26:13.432 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-14T22:26:13.475 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-14T22:26:13.509 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-14T22:26:13.547 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-14T22:26:13.620 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-14T22:26:13.758 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-14T22:26:13.838 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-14T22:26:13.891 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-14T22:26:13.930 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-14T22:26:13.979 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-14T22:26:13.998 INFO:teuthology.orchestra.run.smithi086.stdout:(1/33): ceph-17.2.7-1008.ge47ff4b8.el9.x86_64.r 27 kB/s | 6.5 kB 00:00 2024-06-14T22:26:14.034 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-14T22:26:14.132 INFO:teuthology.orchestra.run.smithi086.stdout:(2/33): ceph-mds-17.2.7-1008.ge47ff4b8.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-06-14T22:26:14.237 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-14T22:26:14.286 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-14T22:26:14.364 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-14T22:26:14.382 INFO:teuthology.orchestra.run.smithi086.stdout:(3/33): ceph-mgr-17.2.7-1008.ge47ff4b8.el9.x86_ 2.3 MB/s | 1.4 MB 00:00 2024-06-14T22:26:14.452 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-14T22:26:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-14T22:26:14.515 INFO:teuthology.orchestra.run.smithi086.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1008.ge47f 1.8 MB/s | 240 kB 00:00 2024-06-14T22:26:14.616 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-14T22:26:14.764 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 28/33 2024-06-14T22:26:14.787 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-14T22:26:14.812 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-14T22:26:14.812 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:14.812 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-14T22:26:14.813 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-14T22:26:14.813 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-14T22:26:14.813 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:14.883 INFO:teuthology.orchestra.run.smithi086.stdout:(5/33): ceph-osd-17.2.7-1008.ge47ff4b8.el9.x86_ 20 MB/s | 15 MB 00:00 2024-06-14T22:26:15.233 INFO:teuthology.orchestra.run.smithi086.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 353 kB/s | 253 kB 00:00 2024-06-14T22:26:15.433 INFO:teuthology.orchestra.run.smithi086.stdout:(7/33): ceph-mon-17.2.7-1008.ge47ff4b8.el9.x86_ 2.7 MB/s | 3.9 MB 00:01 2024-06-14T22:26:15.466 INFO:teuthology.orchestra.run.smithi086.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 2.1 MB/s | 1.2 MB 00:00 2024-06-14T22:26:15.492 INFO:teuthology.orchestra.run.smithi086.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 412 kB/s | 106 kB 00:00 2024-06-14T22:26:15.542 INFO:teuthology.orchestra.run.smithi086.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 1.2 MB/s | 135 kB 00:00 2024-06-14T22:26:15.600 INFO:teuthology.orchestra.run.smithi086.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-06-14T22:26:15.626 INFO:teuthology.orchestra.run.smithi086.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.6 MB/s | 215 kB 00:00 2024-06-14T22:26:15.868 INFO:teuthology.orchestra.run.smithi086.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-06-14T22:26:15.968 INFO:teuthology.orchestra.run.smithi086.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-06-14T22:26:16.035 INFO:teuthology.orchestra.run.smithi086.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 653 kB/s | 43 kB 00:00 2024-06-14T22:26:16.060 INFO:teuthology.orchestra.run.smithi086.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 217 kB/s | 42 kB 00:00 2024-06-14T22:26:16.110 INFO:teuthology.orchestra.run.smithi086.stdout:(17/33): python3-mako-1.1.4-6.el9.noarch.rpm 303 kB/s | 172 kB 00:00 2024-06-14T22:26:16.122 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-14T22:26:16.135 INFO:teuthology.orchestra.run.smithi086.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-06-14T22:26:16.149 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-14T22:26:16.149 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:16.149 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-14T22:26:16.149 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-14T22:26:16.149 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-14T22:26:16.150 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:16.186 INFO:teuthology.orchestra.run.smithi086.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 721 kB/s | 18 kB 00:00 2024-06-14T22:26:16.211 INFO:teuthology.orchestra.run.smithi086.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 109 kB/s | 11 kB 00:00 2024-06-14T22:26:16.236 INFO:teuthology.orchestra.run.smithi086.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 485 kB/s | 23 kB 00:00 2024-06-14T22:26:16.261 INFO:teuthology.orchestra.run.smithi086.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 781 kB/s | 19 kB 00:00 2024-06-14T22:26:16.287 INFO:teuthology.orchestra.run.smithi086.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-06-14T22:26:16.312 INFO:teuthology.orchestra.run.smithi086.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 914 kB/s | 46 kB 00:00 2024-06-14T22:26:16.337 INFO:teuthology.orchestra.run.smithi086.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-06-14T22:26:16.363 INFO:teuthology.orchestra.run.smithi086.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-06-14T22:26:16.388 INFO:teuthology.orchestra.run.smithi086.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-06-14T22:26:16.413 INFO:teuthology.orchestra.run.smithi086.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-14T22:26:16.439 INFO:teuthology.orchestra.run.smithi086.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-06-14T22:26:16.464 INFO:teuthology.orchestra.run.smithi086.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-14T22:26:16.489 INFO:teuthology.orchestra.run.smithi086.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-06-14T22:26:16.514 INFO:teuthology.orchestra.run.smithi086.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-06-14T22:26:16.522 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-14T22:26:16.546 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-14T22:26:16.546 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:16.546 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-14T22:26:16.546 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-14T22:26:16.546 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-14T22:26:16.546 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:16.548 INFO:teuthology.orchestra.run.smithi086.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-06-14T22:26:16.552 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:16.553 INFO:teuthology.orchestra.run.smithi086.stdout:Total 8.0 MB/s | 27 MB 00:03 2024-06-14T22:26:16.667 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:26:16.704 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:26:16.704 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:26:16.757 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-14T22:26:16.780 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-14T22:26:16.781 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:16.781 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-14T22:26:16.781 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-14T22:26:16.781 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-14T22:26:16.781 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:16.845 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-14T22:26:17.069 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:26:17.070 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:26:18.211 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:26:18.277 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-14T22:26:18.316 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-14T22:26:18.357 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-14T22:26:18.425 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-14T22:26:18.425 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 6/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-14T22:26:18.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-14T22:26:18.427 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-14T22:26:18.428 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-14T22:26:18.428 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-14T22:26:18.428 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-14T22:26:18.429 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-14T22:26:18.429 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-14T22:26:18.429 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-14T22:26:18.429 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-14T22:26:18.449 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-14T22:26:18.527 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-14T22:26:18.568 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-14T22:26:18.603 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-14T22:26:18.633 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-14T22:26:18.667 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-14T22:26:18.701 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-14T22:26:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-14T22:26:18.776 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-14T22:26:18.811 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-14T22:26:18.851 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-14T22:26:18.923 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-14T22:26:19.078 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-14T22:26:19.166 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-14T22:26:19.218 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-14T22:26:19.252 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-14T22:26:19.252 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:19.252 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:19.252 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-14T22:26:19.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-14T22:26:19.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-14T22:26:19.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-14T22:26:19.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-14T22:26:19.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-14T22:26:19.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-14T22:26:19.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-14T22:26:19.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-14T22:26:19.255 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:19.255 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:19.257 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-14T22:26:19.308 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-14T22:26:19.364 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-14T22:26:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-14T22:26:19.613 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-14T22:26:19.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-06-14T22:26:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-14T22:26:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-14T22:26:19.829 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-14T22:26:19.945 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-14T22:26:20.093 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 28/33 2024-06-14T22:26:20.123 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-14T22:26:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-14T22:26:20.148 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:20.149 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-14T22:26:20.149 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-14T22:26:20.149 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-14T22:26:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:20.217 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:53 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:20.301 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:26:20.344 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:20.345 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-06-14T22:26:20.345 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:20.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-06-14T22:26:20.978 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:53 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:21.105 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:21.106 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:21.106 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-06-14T22:26:21.106 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:21.106 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:21.106 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 82 k 2024-06-14T22:26:21.107 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:21.107 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:21.107 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:21.107 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-06-14T22:26:21.107 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:21.107 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 82 k 2024-06-14T22:26:21.107 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 372 k 2024-06-14T22:26:21.107 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:21.422 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-17.2.7-1008.ge47ff4b8.el9.noarch.rpm 260 kB/s | 82 kB 00:00 2024-06-14T22:26:21.426 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:21.426 INFO:teuthology.orchestra.run.smithi137.stdout:Total 259 kB/s | 82 kB 00:00 2024-06-14T22:26:21.426 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:21.428 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:21.429 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:21.430 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:21.431 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:21.500 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-14T22:26:21.501 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:21.530 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-14T22:26:21.530 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:21.530 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-14T22:26:21.530 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-14T22:26:21.530 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-14T22:26:21.530 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:21.925 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-14T22:26:21.950 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-14T22:26:21.950 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:21.950 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-14T22:26:21.950 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-14T22:26:21.950 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-14T22:26:21.950 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:21.972 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T22:26:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T22:26:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-14T22:26:22.175 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-14T22:26:22.175 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:22.175 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-14T22:26:22.175 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-14T22:26:22.175 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-14T22:26:22.176 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:22.239 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-14T22:26:22.408 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T22:26:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T22:26:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:22.667 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:22.667 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:22.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-06-14T22:26:23.449 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:56 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:23.576 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:23.577 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-06-14T22:26:23.577 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:23.577 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 137 k 2024-06-14T22:26:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-06-14T22:26:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 137 k 2024-06-14T22:26:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 379 k 2024-06-14T22:26:23.578 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/33 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/33 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/33 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/33 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/33 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 6/33 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-14T22:26:23.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-14T22:26:23.766 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-14T22:26:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-14T22:26:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-14T22:26:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-14T22:26:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-14T22:26:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-14T22:26:23.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-14T22:26:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-14T22:26:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-14T22:26:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-14T22:26:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-14T22:26:23.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-14T22:26:23.929 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-17.2.7-1008.ge47ff4 392 kB/s | 137 kB 00:00 2024-06-14T22:26:23.929 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:23.929 INFO:teuthology.orchestra.run.smithi137.stdout:Total 390 kB/s | 137 kB 00:00 2024-06-14T22:26:23.930 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:23.937 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:23.937 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:23.993 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:23.994 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:24.133 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:24.205 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T22:26:24.230 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T22:26:24.230 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:24.230 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-14T22:26:24.230 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:24.635 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-14T22:26:24.636 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-14T22:26:24.637 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:24.638 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:25.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-06-14T22:26:25.193 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T22:26:25.193 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:25.193 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:25.193 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:25.193 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:25.194 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:25.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-06-14T22:26:25.592 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:55 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:25.676 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:26:25.718 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:25.719 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-14T22:26:25.719 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:25.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-06-14T22:26:25.994 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:58 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:26.080 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:26:26.123 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:26.123 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-06-14T22:26:26.124 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:26.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-06-14T22:26:26.351 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:56 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:26.475 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:26.476 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:26.476 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-14T22:26:26.476 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:26.476 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:26:26.476 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 82 k 2024-06-14T22:26:26.476 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:26.476 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:26:26.476 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:26.476 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-14T22:26:26.477 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:26.477 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 82 k 2024-06-14T22:26:26.477 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 372 k 2024-06-14T22:26:26.477 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:26:26.692 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-17.2.7-1008.ge47ff4b8.el9.noarch.rpm 381 kB/s | 82 kB 00:00 2024-06-14T22:26:26.693 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:26.693 INFO:teuthology.orchestra.run.smithi086.stdout:Total 379 kB/s | 82 kB 00:00 2024-06-14T22:26:26.693 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:26:26.698 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:26:26.698 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:26:26.700 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:26:26.700 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:26:26.750 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:59 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:26.773 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:26:26.877 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:26.878 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:26.878 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-06-14T22:26:26.878 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:26.878 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 3.4 M 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 24 k 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 15 k 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:26.879 INFO:teuthology.orchestra.run.smithi137.stdout:Install 7 Packages 2024-06-14T22:26:26.880 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:26.880 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.7 M 2024-06-14T22:26:26.880 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 80 M 2024-06-14T22:26:26.880 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:27.215 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T22:26:27.249 INFO:teuthology.orchestra.run.smithi137.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1008.ge47f 73 kB/s | 15 kB 00:00 2024-06-14T22:26:27.299 INFO:teuthology.orchestra.run.smithi137.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1008.ge47 98 kB/s | 24 kB 00:00 2024-06-14T22:26:27.312 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T22:26:27.349 INFO:teuthology.orchestra.run.smithi137.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 90 kB/s | 9.0 kB 00:00 2024-06-14T22:26:27.374 INFO:teuthology.orchestra.run.smithi137.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-06-14T22:26:27.399 INFO:teuthology.orchestra.run.smithi137.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 417 kB/s | 41 kB 00:00 2024-06-14T22:26:27.424 INFO:teuthology.orchestra.run.smithi137.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-06-14T22:26:27.642 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T22:26:27.683 INFO:teuthology.orchestra.run.smithi137.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1008.ge47ff4b8 5.3 MB/s | 3.4 MB 00:00 2024-06-14T22:26:27.687 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:27.687 INFO:teuthology.orchestra.run.smithi137.stdout:Total 4.5 MB/s | 3.7 MB 00:00 2024-06-14T22:26:27.687 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:27.695 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:27.695 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:27.757 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:27.757 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:27.882 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T22:26:27.882 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:27.883 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:26:27.883 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:27.883 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:27.883 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:28.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-06-14T22:26:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-14T22:26:28.219 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-14T22:26:28.265 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-14T22:26:28.297 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-14T22:26:28.326 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 5/7 2024-06-14T22:26:28.679 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:58 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:28.809 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:28.810 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:28.810 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-06-14T22:26:28.810 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:28.810 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:26:28.810 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 137 k 2024-06-14T22:26:28.810 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:28.810 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:26:28.810 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:28.811 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-14T22:26:28.811 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:28.811 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 137 k 2024-06-14T22:26:28.811 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 379 k 2024-06-14T22:26:28.811 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:26:29.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-17.2.7-1008.ge47ff4 599 kB/s | 137 kB 00:00 2024-06-14T22:26:29.041 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:29.041 INFO:teuthology.orchestra.run.smithi086.stdout:Total 595 kB/s | 137 kB 00:00 2024-06-14T22:26:29.041 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:26:29.049 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:26:29.049 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:26:29.105 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:26:29.105 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:26:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:26:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 6/7 2024-06-14T22:26:29.325 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T22:26:29.350 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T22:26:29.350 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:26:29.350 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-14T22:26:29.351 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:29.371 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-14T22:26:30.292 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T22:26:30.293 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:30.293 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:26:30.293 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:30.293 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:30.293 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:30.419 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-14T22:26:30.419 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 1/7 2024-06-14T22:26:30.419 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 2/7 2024-06-14T22:26:30.419 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 3/7 2024-06-14T22:26:30.419 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-14T22:26:30.419 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-14T22:26:30.419 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-14T22:26:30.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-06-14T22:26:30.690 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-14T22:26:30.690 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:30.690 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:30.690 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:30.690 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:30.690 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:30.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-14T22:26:30.690 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-14T22:26:30.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-14T22:26:30.691 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-14T22:26:30.691 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:30.691 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:30.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-14T22:26:31.091 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:01 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:31.178 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:26:31.222 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:31.223 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-14T22:26:31.223 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:31.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-06-14T22:26:31.444 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:04 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:31.568 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:31.569 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-06-14T22:26:31.569 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-06-14T22:26:31.569 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-06-14T22:26:31.569 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:31.569 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 7.4 M 2024-06-14T22:26:31.569 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-06-14T22:26:31.569 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-14T22:26:31.570 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:31.571 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:31.571 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-06-14T22:26:31.571 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-06-14T22:26:31.571 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:31.571 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 38 M 2024-06-14T22:26:31.571 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 209 M 2024-06-14T22:26:31.571 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:31.866 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:01 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:31.967 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 960 kB/s | 195 kB 00:00 2024-06-14T22:26:31.993 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:31.993 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:31.993 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-14T22:26:31.993 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:31.993 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 3.4 M 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 24 k 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 15 k 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:31.994 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:26:31.995 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:31.995 INFO:teuthology.orchestra.run.smithi086.stdout:Install 7 Packages 2024-06-14T22:26:31.995 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:31.995 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.7 M 2024-06-14T22:26:31.995 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 80 M 2024-06-14T22:26:31.996 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:26:32.025 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-06-14T22:26:32.042 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 434 kB/s | 33 kB 00:00 2024-06-14T22:26:32.076 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 545 kB/s | 18 kB 00:00 2024-06-14T22:26:32.110 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 35 kB 00:00 2024-06-14T22:26:32.226 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-06-14T22:26:32.245 INFO:teuthology.orchestra.run.smithi086.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1008.ge47 120 kB/s | 24 kB 00:00 2024-06-14T22:26:32.268 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 5.8 MB/s | 244 kB 00:00 2024-06-14T22:26:32.303 INFO:teuthology.orchestra.run.smithi086.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 155 kB/s | 9.0 kB 00:00 2024-06-14T22:26:32.328 INFO:teuthology.orchestra.run.smithi086.stdout:(3/7): python3-jwt-2.4.0-1.el9.noarch.rpm 1.6 MB/s | 41 kB 00:00 2024-06-14T22:26:32.335 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-06-14T22:26:32.353 INFO:teuthology.orchestra.run.smithi086.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.2 MB/s | 31 kB 00:00 2024-06-14T22:26:32.377 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-06-14T22:26:32.379 INFO:teuthology.orchestra.run.smithi086.stdout:(5/7): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-06-14T22:26:32.412 INFO:teuthology.orchestra.run.smithi086.stdout:(6/7): ceph-prometheus-alerts-17.2.7-1008.ge47f 39 kB/s | 15 kB 00:00 2024-06-14T22:26:32.453 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-06-14T22:26:32.564 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-1 9.2 MB/s | 7.4 MB 00:00 2024-06-14T22:26:32.828 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-06-14T22:26:32.829 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:32.829 INFO:teuthology.orchestra.run.smithi137.stdout:Total 30 MB/s | 38 MB 00:01 2024-06-14T22:26:32.862 INFO:teuthology.orchestra.run.smithi086.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1008.ge47ff4b8 4.1 MB/s | 3.4 MB 00:00 2024-06-14T22:26:32.866 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:32.866 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.2 MB/s | 3.7 MB 00:00 2024-06-14T22:26:32.866 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:26:32.874 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:26:32.874 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:26:32.936 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:26:32.936 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:26:33.180 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:33.206 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:33.207 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:26:33.389 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-14T22:26:33.423 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-14T22:26:33.469 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-14T22:26:33.502 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-14T22:26:33.531 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 5/7 2024-06-14T22:26:33.708 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:33.708 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:34.461 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:34.527 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 6/7 2024-06-14T22:26:34.536 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-14T22:26:34.568 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-14T22:26:34.576 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-14T22:26:34.675 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-14T22:26:34.719 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-14T22:26:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-14T22:26:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-14T22:26:35.021 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-14T22:26:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-14T22:26:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 1/7 2024-06-14T22:26:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 2/7 2024-06-14T22:26:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 3/7 2024-06-14T22:26:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-14T22:26:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-14T22:26:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-14T22:26:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-14T22:26:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-14T22:26:35.799 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:35.800 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:36.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-14T22:26:36.570 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:06 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:36.693 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:36.694 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2024-06-14T22:26:36.694 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-14T22:26:36.694 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2024-06-14T22:26:36.694 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:26:36.694 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 7.4 M 2024-06-14T22:26:36.694 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-14T22:26:36.694 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:26:36.695 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2024-06-14T22:26:36.696 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-06-14T22:26:36.696 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:36.696 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 38 M 2024-06-14T22:26:36.696 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 209 M 2024-06-14T22:26:36.696 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:26:37.040 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-14T22:26:37.896 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 292 kB/s | 195 kB 00:00 2024-06-14T22:26:38.005 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-06-14T22:26:38.021 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 262 kB/s | 33 kB 00:00 2024-06-14T22:26:38.067 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-14T22:26:38.092 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-14T22:26:38.105 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 218 kB/s | 18 kB 00:00 2024-06-14T22:26:38.197 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-10 7.6 MB/s | 7.4 MB 00:00 2024-06-14T22:26:38.222 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 300 kB/s | 35 kB 00:00 2024-06-14T22:26:38.340 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-06-14T22:26:38.940 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.2 MB/s | 3.0 MB 00:00 2024-06-14T22:26:39.082 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-06-14T22:26:39.357 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.0 MB/s | 5.1 MB 00:01 2024-06-14T22:26:39.732 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.1 MB/s | 4.7 MB 00:01 2024-06-14T22:26:40.528 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-14T22:26:40.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 1/12 2024-06-14T22:26:40.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-14T22:26:40.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-14T22:26:40.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-14T22:26:40.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-14T22:26:40.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-14T22:26:40.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-14T22:26:40.528 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-14T22:26:40.529 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-14T22:26:40.529 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-14T22:26:40.529 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-14T22:26:40.925 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-14T22:26:40.925 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-14T22:26:40.926 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-14T22:26:40.927 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-14T22:26:40.927 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-14T22:26:40.927 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:40.927 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:41.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-06-14T22:26:41.852 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:14 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:41.979 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:41.980 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 47 k 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-14T22:26:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-14T22:26:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-14T22:26:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-14T22:26:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-14T22:26:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-14T22:26:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:41.982 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:41.982 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:41.982 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-06-14T22:26:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:41.983 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.2 M 2024-06-14T22:26:41.983 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-06-14T22:26:41.983 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:42.348 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 181 kB/s | 26 kB 00:00 2024-06-14T22:26:42.374 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 19 kB 00:00 2024-06-14T22:26:42.399 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): ceph-mgr-rook-17.2.7-1008.ge47ff4b8.el9 247 kB/s | 47 kB 00:00 2024-06-14T22:26:42.483 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-06-14T22:26:42.516 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-06-14T22:26:42.541 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-06-14T22:26:42.633 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 275 kB/s | 32 kB 00:00 2024-06-14T22:26:42.658 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-06-14T22:26:42.684 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 105 kB/s | 15 kB 00:00 2024-06-14T22:26:42.734 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-06-14T22:26:42.759 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-google-auth-2.29.0-2.el9.noarc 1.6 MB/s | 205 kB 00:00 2024-06-14T22:26:42.785 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-06-14T22:26:42.826 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-06-14T22:26:42.829 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:42.829 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-06-14T22:26:42.891 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:42.906 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:42.906 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:42.932 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.2 MB/s | 16 MB 00:03 2024-06-14T22:26:42.934 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:42.934 INFO:teuthology.orchestra.run.smithi086.stdout:Total 6.1 MB/s | 38 MB 00:06 2024-06-14T22:26:42.991 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:42.991 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:43.285 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:26:43.312 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:26:43.313 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:26:43.492 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:43.667 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-14T22:26:43.719 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-14T22:26:43.783 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-14T22:26:43.818 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:26:43.819 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:26:43.820 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-14T22:26:43.862 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-14T22:26:43.956 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-14T22:26:44.065 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-14T22:26:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-14T22:26:44.577 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:26:44.664 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-14T22:26:44.705 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-14T22:26:44.806 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-14T22:26:44.836 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-14T22:26:44.856 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-14T22:26:44.885 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-14T22:26:44.919 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-14T22:26:44.965 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-14T22:26:44.980 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-14T22:26:45.043 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-14T22:26:45.072 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-14T22:26:45.160 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-14T22:26:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-14T22:26:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-14T22:26:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-14T22:26:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-14T22:26:45.926 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-14T22:26:46.114 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-14T22:26:46.114 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-14T22:26:46.115 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-14T22:26:46.116 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-14T22:26:46.116 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-14T22:26:46.116 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:46.116 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:46.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-06-14T22:26:46.966 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:19 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:47.091 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:47.092 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:47.092 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-06-14T22:26:47.092 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:47.092 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:47.092 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 123 k 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-06-14T22:26:47.093 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:47.094 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 7.0 M 2024-06-14T22:26:47.094 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 33 M 2024-06-14T22:26:47.094 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:47.148 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-14T22:26:47.738 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-06-14T22:26:47.805 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-06-14T22:26:47.830 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-14T22:26:47.858 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-06-14T22:26:47.897 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): ceph-mgr-cephadm-17.2.7-1008.ge47ff4b8.e 362 kB/s | 123 kB 00:00 2024-06-14T22:26:48.195 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-14T22:26:48.230 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-14T22:26:49.389 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.3 MB/s | 6.0 MB 00:01 2024-06-14T22:26:49.391 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:49.391 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.0 MB/s | 7.0 MB 00:02 2024-06-14T22:26:49.461 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:49.472 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:49.472 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:49.569 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:49.569 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:49.871 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:50.030 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-14T22:26:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-14T22:26:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-14T22:26:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-14T22:26:50.749 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-14T22:26:50.778 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-14T22:26:50.808 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 1/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-14T22:26:50.809 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-14T22:26:51.211 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-14T22:26:51.211 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:51.211 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:26:51.211 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:51.211 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-14T22:26:51.211 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-14T22:26:51.211 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-14T22:26:51.211 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-14T22:26:51.212 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-14T22:26:51.212 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-14T22:26:51.212 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-14T22:26:51.212 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-14T22:26:51.212 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-14T22:26:51.212 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-14T22:26:51.212 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-14T22:26:51.212 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:51.212 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-14T22:26:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/6 2024-06-14T22:26:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-14T22:26:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-14T22:26:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-14T22:26:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-14T22:26:51.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-06-14T22:26:51.811 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-14T22:26:51.812 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:51.812 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:51.812 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:51.812 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-14T22:26:51.812 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-14T22:26:51.813 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-14T22:26:51.813 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-14T22:26:51.813 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-14T22:26:51.813 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:51.813 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:52.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-06-14T22:26:52.202 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:22 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:52.338 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:52.339 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:52.339 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-14T22:26:52.339 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:52.339 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:26:52.339 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 47 k 2024-06-14T22:26:52.339 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-14T22:26:52.340 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:52.341 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:26:52.341 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:52.341 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-06-14T22:26:52.341 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:52.341 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.2 M 2024-06-14T22:26:52.341 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-06-14T22:26:52.341 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:26:52.616 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:25 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:52.742 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:52.743 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:52.743 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 812 k 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 892 k 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 2.6 M 2024-06-14T22:26:52.744 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:52.935 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 272 kB/s | 26 kB 00:00 2024-06-14T22:26:52.952 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 173 kB/s | 19 kB 00:00 2024-06-14T22:26:52.974 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 723 kB/s | 80 kB 00:00 2024-06-14T22:26:53.019 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.3 MB/s | 222 kB 00:00 2024-06-14T22:26:53.044 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-06-14T22:26:53.069 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 5.4 MB/s | 279 kB 00:00 2024-06-14T22:26:53.086 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.2 MB/s | 54 kB 00:00 2024-06-14T22:26:53.174 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-17.2.7-1008.ge47ff4b8.el9.x86_ 2.5 MB/s | 812 kB 00:00 2024-06-14T22:26:53.175 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:53.175 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.0 MB/s | 892 kB 00:00 2024-06-14T22:26:53.186 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:53.187 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): ceph-mgr-rook-17.2.7-1008.ge47ff4b8.el9 137 kB/s | 47 kB 00:00 2024-06-14T22:26:53.203 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-06-14T22:26:53.207 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:53.207 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:53.245 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 94 kB/s | 15 kB 00:00 2024-06-14T22:26:53.262 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 4.0 MB/s | 59 kB 00:00 2024-06-14T22:26:53.284 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:53.284 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:53.287 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-06-14T22:26:53.312 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-google-auth-2.29.0-2.el9.noarc 1.6 MB/s | 205 kB 00:00 2024-06-14T22:26:53.354 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.0 MB/s | 1.0 MB 00:00 2024-06-14T22:26:53.358 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:53.358 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-06-14T22:26:53.421 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:26:53.436 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:26:53.436 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:26:53.465 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:53.519 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:26:53.520 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:26:53.594 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-14T22:26:53.664 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-14T22:26:54.019 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:26:54.191 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-14T22:26:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-14T22:26:54.310 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-14T22:26:54.348 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-14T22:26:54.350 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-14T22:26:54.351 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/2 2024-06-14T22:26:54.389 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-14T22:26:54.484 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-14T22:26:54.586 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-14T22:26:54.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-14T22:26:54.613 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:54.613 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:54.613 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-14T22:26:54.613 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:54.613 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:54.633 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-14T22:26:54.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-06-14T22:26:55.340 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-14T22:26:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-14T22:26:55.422 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-14T22:26:55.462 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:28 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:55.467 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-14T22:26:55.491 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-14T22:26:55.589 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:55.590 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:55.590 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-06-14T22:26:55.590 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:55.591 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:55.591 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 126 k 2024-06-14T22:26:55.591 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:55.591 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:55.591 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:55.591 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-06-14T22:26:55.591 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:55.591 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 126 k 2024-06-14T22:26:55.591 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 448 k 2024-06-14T22:26:55.592 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:55.895 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-17.2.7-1008.ge47ff4b8.el9.x86_64 415 kB/s | 126 kB 00:00 2024-06-14T22:26:55.896 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:55.896 INFO:teuthology.orchestra.run.smithi137.stdout:Total 413 kB/s | 126 kB 00:00 2024-06-14T22:26:55.896 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:55.902 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:55.902 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:55.963 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:55.963 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:56.113 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:56.197 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-14T22:26:56.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/13 2024-06-14T22:26:56.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-14T22:26:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-14T22:26:56.210 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:26:56.512 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-14T22:26:56.513 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-14T22:26:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-14T22:26:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-14T22:26:56.514 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:56.514 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:26:56.665 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:26:56.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-06-14T22:26:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:26:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:56.945 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:56.945 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:57.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-06-14T22:26:57.342 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:27 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:26:57.468 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:26:57.469 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:57.469 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-14T22:26:57.469 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 123 k 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-06-14T22:26:57.470 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:26:57.471 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 7.0 M 2024-06-14T22:26:57.471 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 33 M 2024-06-14T22:26:57.471 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:26:57.797 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:30 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:57.882 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:26:57.926 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:57.927 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-06-14T22:26:57.927 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:57.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-06-14T22:26:58.231 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-06-14T22:26:58.297 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-06-14T22:26:58.322 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-14T22:26:58.356 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.5 MB/s | 51 kB 00:00 2024-06-14T22:26:58.381 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): ceph-mgr-cephadm-17.2.7-1008.ge47ff4b8.e 339 kB/s | 123 kB 00:00 2024-06-14T22:26:58.532 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-06-14T22:26:58.533 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:58.534 INFO:teuthology.orchestra.run.smithi086.stdout:Total 6.6 MB/s | 7.0 MB 00:01 2024-06-14T22:26:58.560 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:31 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:26:58.604 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:26:58.615 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:26:58.615 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:26:58.685 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:26:58.686 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:58.686 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-06-14T22:26:58.686 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:58.686 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:26:58.686 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-14T22:26:58.686 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:58.686 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:26:58.686 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:26:58.687 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-06-14T22:26:58.687 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:58.687 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 25 k 2024-06-14T22:26:58.687 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 110 k 2024-06-14T22:26:58.687 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:26:58.711 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:26:58.711 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:26:58.849 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-17.2.7-1008.ge47ff4b8.el9.x86_6 154 kB/s | 25 kB 00:00 2024-06-14T22:26:58.850 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:26:58.850 INFO:teuthology.orchestra.run.smithi137.stdout:Total 152 kB/s | 25 kB 00:00 2024-06-14T22:26:58.850 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:26:58.853 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:26:58.853 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:26:58.855 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:26:58.855 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:26:58.934 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:26:59.007 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:26:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:26:59.174 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-14T22:26:59.225 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-14T22:26:59.446 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:26:59.667 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:26:59.667 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:59.668 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:26:59.668 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:26:59.668 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:26:59.668 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:26:59.740 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-14T22:26:59.827 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-14T22:26:59.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-06-14T22:26:59.883 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-14T22:26:59.920 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-14T22:27:00.407 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:33 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:27:00.490 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:00.531 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:27:00.532 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-06-14T22:27:00.532 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:27:00.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-06-14T22:27:00.647 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-14T22:27:00.647 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/6 2024-06-14T22:27:00.648 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-14T22:27:00.648 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-14T22:27:00.648 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-14T22:27:00.648 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-14T22:27:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-14T22:27:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:27:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T22:27:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-14T22:27:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-14T22:27:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-14T22:27:00.903 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-14T22:27:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-14T22:27:00.904 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:01.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-06-14T22:27:01.166 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:34 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:27:01.249 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:01.290 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:27:01.291 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-06-14T22:27:01.292 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:27:01.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-06-14T22:27:01.721 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:31 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:01.845 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-06-14T22:27:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:27:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 812 k 2024-06-14T22:27:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-06-14T22:27:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-14T22:27:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:27:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-06-14T22:27:01.846 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 892 k 2024-06-14T22:27:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.6 M 2024-06-14T22:27:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:27:01.931 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:34 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:27:02.015 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:02.058 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:27:02.059 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-06-14T22:27:02.059 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:27:02.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-06-14T22:27:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 314 kB/s | 80 kB 00:00 2024-06-14T22:27:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-17.2.7-1008.ge47ff4b8.el9.x86_ 2.5 MB/s | 812 kB 00:00 2024-06-14T22:27:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.0 MB/s | 892 kB 00:00 2024-06-14T22:27:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:27:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:27:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:27:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:27:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:27:02.580 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:27:02.701 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:35 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:27:02.715 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-14T22:27:02.785 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:02.790 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-14T22:27:02.826 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:27:02.827 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-06-14T22:27:02.827 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:27:02.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-06-14T22:27:03.443 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-14T22:27:03.444 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/2 2024-06-14T22:27:03.477 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:36 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:27:03.561 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:03.602 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:27:03.603 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-06-14T22:27:03.603 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:27:03.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-06-14T22:27:03.695 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-14T22:27:03.696 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:27:03.696 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-14T22:27:03.696 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:03.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-06-14T22:27:04.254 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:37 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:27:04.338 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:04.379 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:27:04.380 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-06-14T22:27:04.380 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:27:04.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-06-14T22:27:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:34 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 126 k 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 126 k 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 448 k 2024-06-14T22:27:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:27:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-17.2.7-1008.ge47ff4b8.el9.x86_64 557 kB/s | 126 kB 00:00 2024-06-14T22:27:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:Total 552 kB/s | 126 kB 00:00 2024-06-14T22:27:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:27:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:27:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:27:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:27:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:27:05.011 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:38 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:27:05.135 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:27:05.136 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:27:05.136 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:27:05.136 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-06-14T22:27:05.137 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:27:05.137 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:27:05.137 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 82 k 2024-06-14T22:27:05.137 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:05.137 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:27:05.137 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:27:05.137 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-06-14T22:27:05.137 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:05.138 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 82 k 2024-06-14T22:27:05.138 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 222 k 2024-06-14T22:27:05.138 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:27:05.297 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:05.452 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 261 kB/s | 82 kB 00:00 2024-06-14T22:27:05.453 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:05.453 INFO:teuthology.orchestra.run.smithi137.stdout:Total 260 kB/s | 82 kB 00:00 2024-06-14T22:27:05.453 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:27:05.460 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:27:05.460 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:27:05.521 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:27:05.521 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:27:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:27:05.692 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:05.934 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:05.934 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:05.934 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:27:05.935 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:27:05.935 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:05.935 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:06.143 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:06.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-06-14T22:27:06.393 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:06.393 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:06.393 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:27:06.393 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:27:06.393 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:06.393 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:27:06.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-06-14T22:27:06.795 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:36 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:06.880 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:06.924 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:06.925 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-14T22:27:06.925 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:06.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-06-14T22:27:07.217 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:40 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:27:07.347 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:27:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:27:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-06-14T22:27:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:27:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:27:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-14T22:27:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:27:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:27:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-06-14T22:27:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.9 M 2024-06-14T22:27:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 12 M 2024-06-14T22:27:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:27:07.569 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:37 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:07.693 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:07.693 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 25 k 2024-06-14T22:27:07.694 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 110 k 2024-06-14T22:27:07.695 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:27:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 7.1 MB/s | 2.9 MB 00:00 2024-06-14T22:27:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:Total 7.1 MB/s | 2.9 MB 00:00 2024-06-14T22:27:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:27:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:27:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:27:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:27:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:27:07.941 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-17.2.7-1008.ge47ff4b8.el9.x86_6 101 kB/s | 25 kB 00:00 2024-06-14T22:27:07.941 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:07.942 INFO:teuthology.orchestra.run.smithi086.stdout:Total 101 kB/s | 25 kB 00:00 2024-06-14T22:27:07.942 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:27:07.946 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:27:07.946 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:27:07.947 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:27:07.947 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:27:08.023 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:27:08.128 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:27:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:08.356 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:27:08.356 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-14T22:27:08.356 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-14T22:27:08.356 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-14T22:27:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:08.496 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:08.718 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:08.719 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:08.719 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:27:08.719 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:27:08.719 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:08.719 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:08.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-06-14T22:27:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:09.377 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:27:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:27:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:09.377 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:27:09.463 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:39 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:09.546 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:09.587 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:09.589 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-14T22:27:09.590 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:09.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-06-14T22:27:09.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-06-14T22:27:10.218 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:43 ago on Fri 14 Jun 2024 10:25:27 PM UTC. 2024-06-14T22:27:10.235 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:40 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:10.318 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:10.346 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-06-14T22:27:10.346 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 165 k 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 165 k 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 477 k 2024-06-14T22:27:10.347 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-06-14T22:27:10.360 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:10.361 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-14T22:27:10.361 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:10.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-06-14T22:27:10.580 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 711 kB/s | 165 kB 00:00 2024-06-14T22:27:10.581 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:10.581 INFO:teuthology.orchestra.run.smithi137.stdout:Total 706 kB/s | 165 kB 00:00 2024-06-14T22:27:10.581 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-06-14T22:27:10.588 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-06-14T22:27:10.588 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-06-14T22:27:10.650 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-06-14T22:27:10.650 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-06-14T22:27:10.806 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-06-14T22:27:10.910 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:11.034 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:41 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:11.119 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:11.161 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:11.162 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-14T22:27:11.162 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:11.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-06-14T22:27:11.333 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:11.600 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:11.600 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:11.600 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-06-14T22:27:11.601 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:27:11.601 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:27:11.601 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-06-14T22:27:11.810 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:41 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:11.851 DEBUG:teuthology.parallel:result is None 2024-06-14T22:27:11.894 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:11.936 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:11.937 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-14T22:27:11.937 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:12.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-06-14T22:27:12.579 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:42 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:12.663 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:12.704 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:12.705 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-14T22:27:12.705 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:12.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-06-14T22:27:13.358 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:43 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:13.442 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T22:27:13.484 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:13.485 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-06-14T22:27:13.485 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:13.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-06-14T22:27:14.128 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:44 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:14.252 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:14.252 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 82 k 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 82 k 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 222 k 2024-06-14T22:27:14.253 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:27:14.569 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 260 kB/s | 82 kB 00:00 2024-06-14T22:27:14.569 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:14.569 INFO:teuthology.orchestra.run.smithi086.stdout:Total 259 kB/s | 82 kB 00:00 2024-06-14T22:27:14.569 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:27:14.576 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:27:14.576 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:27:14.637 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:27:14.638 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:27:14.777 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:27:14.890 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:15.261 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:15.503 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:15.503 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:15.503 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:27:15.503 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:27:15.503 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:15.503 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:15.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-06-14T22:27:16.327 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:46 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:16.455 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.9 M 2024-06-14T22:27:16.457 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 12 M 2024-06-14T22:27:16.458 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:27:17.046 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 5.0 MB/s | 2.9 MB 00:00 2024-06-14T22:27:17.047 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:17.047 INFO:teuthology.orchestra.run.smithi086.stdout:Total 5.0 MB/s | 2.9 MB 00:00 2024-06-14T22:27:17.048 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:27:17.056 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:27:17.056 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:27:17.135 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:27:17.136 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:27:17.545 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:27:17.605 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:17.630 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:17.630 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T22:27:17.630 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-14T22:27:17.630 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-14T22:27:17.630 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-14T22:27:17.630 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:18.594 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:18.594 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:18.594 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:27:18.594 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:27:18.594 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:18.594 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:18.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-06-14T22:27:19.446 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:49 ago on Fri 14 Jun 2024 10:25:30 PM UTC. 2024-06-14T22:27:19.574 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-06-14T22:27:19.575 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-06-14T22:27:19.575 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:19.575 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-06-14T22:27:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 165 k 2024-06-14T22:27:19.575 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:19.575 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-06-14T22:27:19.576 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-06-14T22:27:19.576 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-06-14T22:27:19.576 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:19.576 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 165 k 2024-06-14T22:27:19.576 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 477 k 2024-06-14T22:27:19.576 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-06-14T22:27:19.806 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 719 kB/s | 165 kB 00:00 2024-06-14T22:27:19.806 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:19.807 INFO:teuthology.orchestra.run.smithi086.stdout:Total 714 kB/s | 165 kB 00:00 2024-06-14T22:27:19.807 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-06-14T22:27:19.814 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-06-14T22:27:19.814 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-06-14T22:27:19.878 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-06-14T22:27:19.878 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-06-14T22:27:20.032 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-06-14T22:27:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:20.532 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T22:27:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:20.793 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-06-14T22:27:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T22:27:20.793 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:27:20.793 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-06-14T22:27:21.038 DEBUG:teuthology.parallel:result is None 2024-06-14T22:27:21.039 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T22:27:21.039 INFO:teuthology.packaging:ref: None 2024-06-14T22:27:21.039 INFO:teuthology.packaging:tag: None 2024-06-14T22:27:21.039 INFO:teuthology.packaging:branch: quincy 2024-06-14T22:27:21.039 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:27:21.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-14T22:27:21.274 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-14T22:27:21.295 INFO:teuthology.orchestra.run.smithi086.stdout:17.2.7-1008.ge47ff4b8.el9 2024-06-14T22:27:21.296 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1008.ge47ff4b8.el9 2024-06-14T22:27:21.296 INFO:teuthology.task.install:The correct ceph version 17.2.7-1008.ge47ff4b8 is installed. 2024-06-14T22:27:21.297 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T22:27:21.297 INFO:teuthology.packaging:ref: None 2024-06-14T22:27:21.297 INFO:teuthology.packaging:tag: None 2024-06-14T22:27:21.297 INFO:teuthology.packaging:branch: quincy 2024-06-14T22:27:21.298 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:27:21.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-14T22:27:21.519 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-14T22:27:21.545 INFO:teuthology.orchestra.run.smithi137.stdout:17.2.7-1008.ge47ff4b8.el9 2024-06-14T22:27:21.545 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1008.ge47ff4b8.el9 2024-06-14T22:27:21.545 INFO:teuthology.task.install:The correct ceph version 17.2.7-1008.ge47ff4b8 is installed. 2024-06-14T22:27:21.547 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-14T22:27:21.547 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:27:21.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-14T22:27:21.573 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:27:21.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-14T22:27:21.616 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-14T22:27:21.616 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:27:21.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-06-14T22:27:21.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-14T22:27:21.707 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:27:21.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-06-14T22:27:21.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-14T22:27:21.804 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-14T22:27:21.804 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:27:21.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-14T22:27:21.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-14T22:27:21.896 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:27:21.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-14T22:27:21.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-14T22:27:21.992 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-14T22:27:21.992 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:27:21.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-06-14T22:27:22.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-14T22:27:22.080 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:27:22.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-06-14T22:27:22.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-14T22:27:22.170 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-14T22:27:22.255 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'} 2024-06-14T22:27:22.255 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-06-14T22:27:22.255 INFO:tasks.cephadm:Cluster fsid is 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:27:22.255 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-14T22:27:22.256 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.86', 'mon.c': '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]', 'mon.b': '172.21.15.137'} 2024-06-14T22:27:22.256 INFO:tasks.cephadm:First mon is mon.a on smithi086 2024-06-14T22:27:22.256 INFO:tasks.cephadm:First mgr is y 2024-06-14T22:27:22.256 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-14T22:27:22.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-06-14T22:27:22.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo hostname $(hostname -s) 2024-06-14T22:27:22.306 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-06-14T22:27:22.307 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-14T22:27:22.678 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 14 22:27 /home/ubuntu/cephtest/cephadm 2024-06-14T22:27:22.679 DEBUG:teuthology.orchestra.run.smithi137:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-14T22:27:22.926 INFO:teuthology.orchestra.run.smithi137.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 14 22:27 /home/ubuntu/cephtest/cephadm 2024-06-14T22:27:22.927 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-14T22:27:22.943 DEBUG:teuthology.orchestra.run.smithi137:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-14T22:27:22.968 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-06-14T22:27:22.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-14T22:27:22.985 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-14T22:27:23.880 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-14T22:27:23.986 INFO:teuthology.orchestra.run.smithi137.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-14T22:27:50.030 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T22:27:50.030 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)", 2024-06-14T22:27:50.030 INFO:teuthology.orchestra.run.smithi137.stdout: "image_id": "37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95", 2024-06-14T22:27:50.030 INFO:teuthology.orchestra.run.smithi137.stdout: "repo_digests": [ 2024-06-14T22:27:50.031 INFO:teuthology.orchestra.run.smithi137.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580" 2024-06-14T22:27:50.031 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:27:50.031 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T22:27:53.937 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:27:53.937 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)", 2024-06-14T22:27:53.937 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95", 2024-06-14T22:27:53.937 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-06-14T22:27:53.937 INFO:teuthology.orchestra.run.smithi086.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580" 2024-06-14T22:27:53.937 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:27:53.937 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:27:53.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-06-14T22:27:54.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph 2024-06-14T22:27:54.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-06-14T22:27:54.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /etc/ceph 2024-06-14T22:27:54.105 INFO:tasks.cephadm:Writing seed config... 2024-06-14T22:27:54.106 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-14T22:27:54.106 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-14T22:27:54.107 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-14T22:27:54.107 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-14T22:27:54.107 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-06-14T22:27:54.107 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-06-14T22:27:54.107 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-14T22:27:54.107 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-14T22:27:54.107 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-14T22:27:54.107 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-14T22:27:54.107 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-14T22:27:54.108 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:27:54.108 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-14T22:27:54.123 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 44b2848a-2a9d-11ef-bc9d-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-06-14T22:27:54.123 DEBUG:teuthology.orchestra.run.smithi086:mon.a> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a.service 2024-06-14T22:27:54.166 DEBUG:teuthology.orchestra.run.smithi086:mgr.y> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y.service 2024-06-14T22:27:54.209 INFO:tasks.cephadm:Bootstrapping... 2024-06-14T22:27:54.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.86 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-14T22:27:54.413 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-06-14T22:27:54.413 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '44b2848a-2a9d-11ef-bc9d-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.86', '--skip-admin-label'] 2024-06-14T22:27:54.438 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 5.1.0 2024-06-14T22:27:54.439 INFO:teuthology.orchestra.run.smithi086.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-14T22:27:54.439 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying podman|docker is present... 2024-06-14T22:27:54.462 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 5.1.0 2024-06-14T22:27:54.462 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying lvm2 is present... 2024-06-14T22:27:54.462 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying time synchronization is in place... 2024-06-14T22:27:54.470 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-14T22:27:54.470 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-14T22:27:54.477 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-14T22:27:54.477 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-14T22:27:54.484 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout enabled 2024-06-14T22:27:54.491 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout active 2024-06-14T22:27:54.491 INFO:teuthology.orchestra.run.smithi086.stdout:Unit chronyd.service is enabled and running 2024-06-14T22:27:54.491 INFO:teuthology.orchestra.run.smithi086.stdout:Repeating the final host check... 2024-06-14T22:27:54.514 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 5.1.0 2024-06-14T22:27:54.514 INFO:teuthology.orchestra.run.smithi086.stdout:podman (/bin/podman) version 5.1.0 is present 2024-06-14T22:27:54.514 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl is present 2024-06-14T22:27:54.514 INFO:teuthology.orchestra.run.smithi086.stdout:lvcreate is present 2024-06-14T22:27:54.521 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-14T22:27:54.521 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-14T22:27:54.528 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-14T22:27:54.528 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-14T22:27:54.535 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout enabled 2024-06-14T22:27:54.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout active 2024-06-14T22:27:54.542 INFO:teuthology.orchestra.run.smithi086.stdout:Unit chronyd.service is enabled and running 2024-06-14T22:27:54.542 INFO:teuthology.orchestra.run.smithi086.stdout:Host looks OK 2024-06-14T22:27:54.542 INFO:teuthology.orchestra.run.smithi086.stdout:Cluster fsid: 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:27:54.542 INFO:teuthology.orchestra.run.smithi086.stdout:Acquiring lock 139827774342048 on /run/cephadm/44b2848a-2a9d-11ef-bc9d-c7b262605968.lock 2024-06-14T22:27:54.543 INFO:teuthology.orchestra.run.smithi086.stdout:Lock 139827774342048 acquired on /run/cephadm/44b2848a-2a9d-11ef-bc9d-c7b262605968.lock 2024-06-14T22:27:54.543 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying IP 172.21.15.86 port 3300 ... 2024-06-14T22:27:54.543 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying IP 172.21.15.86 port 6789 ... 2024-06-14T22:27:54.543 INFO:teuthology.orchestra.run.smithi086.stdout:Base mon IP(s) is [172.21.15.86:3300, 172.21.15.86:6789], mon addrv is [v2:172.21.15.86:3300,v1:172.21.15.86:6789] 2024-06-14T22:27:54.546 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.86 metric 100 2024-06-14T22:27:54.547 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.86 metric 100 2024-06-14T22:27:54.549 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-14T22:27:54.549 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-14T22:27:54.550 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1757sec hoplimit 64 pref medium 2024-06-14T22:27:54.552 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-14T22:27:54.552 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-14T22:27:54.552 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-14T22:27:54.552 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-14T22:27:54.552 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d809/64 scope link 2024-06-14T22:27:54.552 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-14T22:27:54.553 INFO:teuthology.orchestra.run.smithi086.stdout:Mon IP `172.21.15.86` is in CIDR network `172.21.0.0/20` 2024-06-14T22:27:54.553 INFO:teuthology.orchestra.run.smithi086.stdout:Mon IP `172.21.15.86` is in CIDR network `172.21.0.0/20` 2024-06-14T22:27:54.553 INFO:teuthology.orchestra.run.smithi086.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-14T22:27:54.554 INFO:teuthology.orchestra.run.smithi086.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-14T22:27:54.554 INFO:teuthology.orchestra.run.smithi086.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-14T22:27:55.028 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 2024-06-14T22:27:55.029 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-14T22:27:55.029 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Getting image source signatures 2024-06-14T22:27:55.029 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Copying blob sha256:ae1bd4835d962ece7feffb132e988e26ca074ae51fc4b41b3d8d10a6caf0861f 2024-06-14T22:27:55.029 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Copying blob sha256:9065cd298533077f0006921bb12bd331d24f6c140cc8fa972c2e6c4e382038d1 2024-06-14T22:27:55.029 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Copying config sha256:37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 2024-06-14T22:27:55.029 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-14T22:27:55.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph: stdout ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable) 2024-06-14T22:27:55.898 INFO:teuthology.orchestra.run.smithi086.stdout:Ceph version: ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable) 2024-06-14T22:27:55.899 INFO:teuthology.orchestra.run.smithi086.stdout:Extracting ceph user uid/gid from container image... 2024-06-14T22:27:56.912 INFO:teuthology.orchestra.run.smithi086.stdout:stat: stdout 167 167 2024-06-14T22:27:56.912 INFO:teuthology.orchestra.run.smithi086.stdout:Creating initial keys... 2024-06-14T22:27:57.842 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph-authtool: stdout AQDtw2xmJuBmDhAAcryZqqR3S/URdTJasiSSAA== 2024-06-14T22:27:58.856 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph-authtool: stdout AQDuw2xmULHIChAAsLBEfIZVnz181i+o7Vwl5Q== 2024-06-14T22:27:59.752 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph-authtool: stdout AQDvw2xmnR+dCxAAEcHnXIUNfx7XwTGtbX2gGQ== 2024-06-14T22:27:59.753 INFO:teuthology.orchestra.run.smithi086.stdout:Creating initial monmap... 2024-06-14T22:28:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-14T22:28:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-06-14T22:28:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:28:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-14T22:28:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool for a [v2:172.21.15.86:3300,v1:172.21.15.86:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-14T22:28:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:setting min_mon_release = octopus 2024-06-14T22:28:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: set fsid to 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:28:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-14T22:28:00.810 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:28:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:Creating mon... 2024-06-14T22:28:01.739 INFO:teuthology.orchestra.run.smithi086.stdout:create mon.a on 2024-06-14T22:28:01.936 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-14T22:28:02.082 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-14T22:28:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968.target. 2024-06-14T22:28:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968.target. 2024-06-14T22:28:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a 2024-06-14T22:28:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to reset failed state of unit ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a.service: Unit ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a.service not loaded. 2024-06-14T22:28:02.550 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968.target.wants/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a.service → /etc/systemd/system/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@.service. 2024-06-14T22:28:03.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:02 smithi086 ceph-mon[29416]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-14T22:28:03.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:03 smithi086 bash[29378]: 5f3d906b3d04373984965e6e1fb324463a750d5154baf6e98b8d73a8eeb4c586 2024-06-14T22:28:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-14T22:28:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-06-14T22:28:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-14T22:28:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-14T22:28:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-06-14T22:28:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to enable service . firewalld.service is not available 2024-06-14T22:28:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mon to start... 2024-06-14T22:28:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mon... 2024-06-14T22:28:03.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:03 smithi086 systemd[1]: Started Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:28:04.008 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout cluster: 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout id: 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout services: 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.594251s) 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout data: 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout pgs: 2024-06-14T22:28:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:mon is available 2024-06-14T22:28:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:Assimilating anything we can from ceph.conf... 2024-06-14T22:28:04.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:03 smithi086 ceph-mon[29416]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-14T22:28:04.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:03 smithi086 ceph-mon[29416]: monmap e1: 1 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:28:04.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:03 smithi086 ceph-mon[29416]: fsmap 2024-06-14T22:28:04.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:03 smithi086 ceph-mon[29416]: osdmap e1: 0 total, 0 up, 0 in 2024-06-14T22:28:04.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:03 smithi086 ceph-mon[29416]: mgrmap e1: no daemons active 2024-06-14T22:28:04.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:03 smithi086 ceph-mon[29416]: from='client.? 172.21.15.86:0/709358612' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-14T22:28:05.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:04 smithi086 ceph-mon[29416]: from='client.? 172.21.15.86:0/3979799125' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-14T22:28:05.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:04 smithi086 ceph-mon[29416]: from='client.? 172.21.15.86:0/3979799125' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-14T22:28:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [global] 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout fsid = 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.86:3300,v1:172.21.15.86:6789] 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [osd] 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-14T22:28:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-14T22:28:05.065 INFO:teuthology.orchestra.run.smithi086.stdout:Generating new minimal ceph.conf... 2024-06-14T22:28:06.102 INFO:teuthology.orchestra.run.smithi086.stdout:Restarting the monitor... 2024-06-14T22:28:06.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:05 smithi086 ceph-mon[29416]: from='client.? 172.21.15.86:0/1958375918' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:28:06.372 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:06 smithi086 systemd[1]: Stopping Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:28:06.636 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[29393]: 2024-06-14T22:28:06.377+0000 7f3f056e3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T22:28:06.637 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[29393]: 2024-06-14T22:28:06.377+0000 7f3f056e3640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-14T22:28:06.637 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:06 smithi086 podman[29736]: 2024-06-14 22:28:06.419046272 +0000 UTC m=+0.148059139 container died 5f3d906b3d04373984965e6e1fb324463a750d5154baf6e98b8d73a8eeb4c586 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612) 2024-06-14T22:28:06.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:06 smithi086 podman[29736]: 2024-06-14 22:28:06.741173279 +0000 UTC m=+0.470186143 container cleanup 5f3d906b3d04373984965e6e1fb324463a750d5154baf6e98b8d73a8eeb4c586 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-14T22:28:06.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:06 smithi086 bash[29736]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a 2024-06-14T22:28:07.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:06 smithi086 podman[29752]: 2024-06-14 22:28:06.942092576 +0000 UTC m=+0.516400105 container remove 5f3d906b3d04373984965e6e1fb324463a750d5154baf6e98b8d73a8eeb4c586 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-14T22:28:07.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-14T22:28:07.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 systemd[1]: Stopped Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:28:07.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 systemd[1]: Starting Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:28:07.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 podman[29851]: 2024-06-14 22:28:07.275830213 +0000 UTC m=+0.016843702 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph:quincy 2024-06-14T22:28:07.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 podman[29851]: 2024-06-14 22:28:07.376116212 +0000 UTC m=+0.117129688 container create 00d4288edc6f73fc20c932a25a48c22be87315f4bb97313ae8e9b3b0faeac61d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:28:07.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 podman[29851]: 2024-06-14 22:28:07.570301211 +0000 UTC m=+0.311314696 container init 00d4288edc6f73fc20c932a25a48c22be87315f4bb97313ae8e9b3b0faeac61d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-14T22:28:07.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 podman[29851]: 2024-06-14 22:28:07.575656819 +0000 UTC m=+0.316670296 container start 00d4288edc6f73fc20c932a25a48c22be87315f4bb97313ae8e9b3b0faeac61d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-06-14T22:28:07.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: set uid:gid to 167:167 (ceph:ceph) 2024-06-14T22:28:07.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable), process ceph-mon, pid 7 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: pidfile_write: ignore empty --pid-file 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: load: jerasure load: lrc 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: RocksDB version: 6.15.5 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Compile date Jun 14 2024 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: DB SUMMARY 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: DB Session ID: BOYSTB7OKO020D6SE0A1 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: CURRENT file: CURRENT 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: IDENTITY file: IDENTITY 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83718 ; 2024-06-14T22:28:07.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.error_if_exists: 0 2024-06-14T22:28:07.641 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.create_if_missing: 0 2024-06-14T22:28:07.641 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.paranoid_checks: 1 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.env: 0x56370c8c9440 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.fs: Posix File System 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.info_log: 0x56370ebe6f40 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_file_opening_threads: 16 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.statistics: (nil) 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.use_fsync: 0 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_log_file_size: 0 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.keep_log_file_num: 1000 2024-06-14T22:28:07.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.recycle_log_file_num: 0 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.allow_fallocate: 1 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.allow_mmap_reads: 0 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.allow_mmap_writes: 0 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.use_direct_reads: 0 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.create_missing_column_families: 0 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.db_log_dir: 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-14T22:28:07.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.advise_random_on_open: 1 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.db_write_buffer_size: 0 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.write_buffer_manager: 0x56370eca47b0 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.rate_limiter: (nil) 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.wal_recovery_mode: 2 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.enable_thread_tracking: 0 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.enable_pipelined_write: 0 2024-06-14T22:28:07.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.unordered_write: 0 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.row_cache: None 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.wal_filter: None 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.allow_ingest_behind: 0 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.preserve_deletes: 0 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.two_write_queues: 0 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.manual_wal_flush: 0 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.atomic_flush: 0 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-14T22:28:07.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.log_readahead_size: 0 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.best_efforts_recovery: 0 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.allow_data_in_errors: 0 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.db_host_id: __hostname__ 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_background_jobs: 2 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_background_compactions: -1 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_subcompactions: 1 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-14T22:28:07.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_total_wal_size: 0 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_open_files: -1 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bytes_per_sync: 0 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_readahead_size: 0 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_background_flushes: -1 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Compression algorithms supported: 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: kZSTD supported: 0 2024-06-14T22:28:07.647 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: kXpressCompression supported: 0 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: kBZip2Compression supported: 0 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: kLZ4Compression supported: 1 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: kZlibCompression supported: 1 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: kLZ4HCCompression supported: 1 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: kSnappyCompression supported: 1 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.merge_operator: 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_filter: None 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_filter_factory: None 2024-06-14T22:28:07.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.sst_partitioner_factory: None 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56370eb8da68) 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: index_type: 0 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: data_block_index_type: 0 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: index_shortening: 1 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: hash_index_allow_collision: 1 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: checksum: 1 2024-06-14T22:28:07.649 INFO:journalctl@ceph.mon.a.smithi086.stdout: no_block_cache: 0 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache: 0x56370ebc3350 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache_options: 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: capacity : 536870912 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: num_shard_bits : 4 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: strict_capacity_limit : 0 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache_compressed: (nil) 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: persistent_cache: (nil) 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_size: 4096 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_size_deviation: 10 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_restart_interval: 16 2024-06-14T22:28:07.650 INFO:journalctl@ceph.mon.a.smithi086.stdout: index_block_restart_interval: 1 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: metadata_block_size: 4096 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: partition_filters: 0 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: use_delta_encoding: 1 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: whole_key_filtering: 1 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: verify_compression: 0 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: format_version: 4 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: enable_index_compression: 1 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_align: 0 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.write_buffer_size: 33554432 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_write_buffer_number: 2 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compression: NoCompression 2024-06-14T22:28:07.651 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bottommost_compression: Disabled 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.prefix_extractor: nullptr 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.num_levels: 7 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-14T22:28:07.652 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compression_opts.level: 32767 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compression_opts.strategy: 0 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compression_opts.enabled: false 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.target_file_size_base: 67108864 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-14T22:28:07.653 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.arena_block_size: 4194304 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-14T22:28:07.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.disable_auto_compactions: 0 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.table_properties_collectors: 2024-06-14T22:28:07.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.inplace_update_support: 0 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.bloom_locality: 0 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.max_successive_merges: 0 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.paranoid_file_checks: 0 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.force_consistency_checks: 1 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.report_bg_io_stats: 0 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.ttl: 2592000 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.enable_blob_files: false 2024-06-14T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.min_blob_size: 0 2024-06-14T22:28:07.657 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.blob_file_size: 268435456 2024-06-14T22:28:07.657 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-14T22:28:07.657 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-14T22:28:07.657 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-14T22:28:07.657 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404087598607, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404087599322, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80265, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78533, "index_size": 220, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10136, "raw_average_key_size": 47, "raw_value_size": 72740, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1718404087, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "0a7968b0-5381-4231-8672-4adf5c5c9dde", "db_session_id": "BOYSTB7OKO020D6SE0A1"}} 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404087599632, "job": 1, "event": "recovery_finished"} 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56370ebf6700 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: DB pointer 0x56370ebc6000 2024-06-14T22:28:07.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** DB Stats ** 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-14T22:28:07.659 INFO:journalctl@ceph.mon.a.smithi086.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: L0 2/0 80.14 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.8 0.00 0.00 1 0.000 0 0 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: Sum 2/0 80.14 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.8 0.00 0.00 1 0.000 0 0 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.8 0.00 0.00 1 0.000 0 0 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.8 0.00 0.00 1 0.000 0 0 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-14T22:28:07.660 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative compaction: 0.00 GB write, 25.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval compaction: 0.00 GB write, 25.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: L0 2/0 80.14 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.8 0.00 0.00 1 0.000 0 0 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: Sum 2/0 80.14 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.8 0.00 0.00 1 0.000 0 0 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-14T22:28:07.661 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.8 0.00 0.00 1 0.000 0 0 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative compaction: 0.00 GB write, 25.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:28:07.662 INFO:journalctl@ceph.mon.a.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: starting mon.a rank 0 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mon.a@-1(???) e1 preinit fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mon.a@-1(???).mds e1 new map 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mon.a@-1(???).mds e1 print_map 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout: e1 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout: legacy client fscid: -1 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout: No filesystems configured 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-14T22:28:07.663 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-14T22:28:07.664 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-14T22:28:07.664 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-14T22:28:07.664 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 bash[29851]: 00d4288edc6f73fc20c932a25a48c22be87315f4bb97313ae8e9b3b0faeac61d 2024-06-14T22:28:07.664 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 systemd[1]: Started Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:28:07.664 INFO:teuthology.orchestra.run.smithi086.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-14T22:28:07.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-14T22:28:07.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: monmap e1: 1 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:28:07.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: fsmap 2024-06-14T22:28:07.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: osdmap e1: 0 total, 0 up, 0 in 2024-06-14T22:28:07.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:07 smithi086 ceph-mon[29889]: mgrmap e1: no daemons active 2024-06-14T22:28:08.564 INFO:teuthology.orchestra.run.smithi086.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-14T22:28:08.564 INFO:teuthology.orchestra.run.smithi086.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-14T22:28:08.564 INFO:teuthology.orchestra.run.smithi086.stdout:Creating mgr... 2024-06-14T22:28:08.565 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying port 9283 ... 2024-06-14T22:28:08.741 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y 2024-06-14T22:28:08.741 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to reset failed state of unit ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y.service: Unit ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y.service not loaded. 2024-06-14T22:28:08.885 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968.target.wants/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@.service. 2024-06-14T22:28:09.079 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:08 smithi086 systemd[1]: Starting Ceph mgr.y for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:28:09.422 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:09 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2067474366' entity='client.admin' 2024-06-14T22:28:09.423 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 podman[30127]: 2024-06-14 22:28:09.027282931 +0000 UTC m=+0.016607963 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph:quincy 2024-06-14T22:28:09.423 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 podman[30127]: 2024-06-14 22:28:09.151959535 +0000 UTC m=+0.141284606 container create 3ed18014b7a87431fed59503ac4e324f5e6ddd075b6260868b45d9f503ba8241 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux ) 2024-06-14T22:28:09.423 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 podman[30127]: 2024-06-14 22:28:09.337281808 +0000 UTC m=+0.326606852 container init 3ed18014b7a87431fed59503ac4e324f5e6ddd075b6260868b45d9f503ba8241 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-06-14T22:28:09.423 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 podman[30127]: 2024-06-14 22:28:09.342454498 +0000 UTC m=+0.331779524 container start 3ed18014b7a87431fed59503ac4e324f5e6ddd075b6260868b45d9f503ba8241 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, ceph=True) 2024-06-14T22:28:09.423 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 bash[30127]: 3ed18014b7a87431fed59503ac4e324f5e6ddd075b6260868b45d9f503ba8241 2024-06-14T22:28:09.437 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-14T22:28:09.437 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-06-14T22:28:09.445 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-14T22:28:09.445 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-14T22:28:09.445 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-06-14T22:28:09.445 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to enable service . firewalld.service is not available 2024-06-14T22:28:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-14T22:28:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-06-14T22:28:09.462 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-14T22:28:09.462 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-14T22:28:09.462 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-06-14T22:28:09.462 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-14T22:28:09.462 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr to start... 2024-06-14T22:28:09.462 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr... 2024-06-14T22:28:09.693 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 systemd[1]: Started Ceph mgr.y for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:28:09.693 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:09.460+0000 7f52baa6aec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:28:09.693 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:09.588+0000 7f52baa6aec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:28:09.693 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:09.655+0000 7f52baa6aec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:28:09.968 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:09.912+0000 7f52baa6aec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:28:09.968 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:09.973+0000 7f52baa6aec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:28:10.285 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2951444137' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-14T22:28:10.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:10.290+0000 7f52baa6aec0 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:28:10.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:10.290+0000 7f52baa6aec0 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:28:10.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:28:10.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from .module import Module 2024-06-14T22:28:10.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:28:10.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:28:10.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:28:10.580 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:10.580 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:10.291+0000 7f52baa6aec0 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:28:10.580 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:10.291+0000 7f52baa6aec0 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:28:10.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:10.637 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsid": "44b2848a-2a9d-11ef-bc9d-c7b262605968", 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "health": { 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 0 2024-06-14T22:28:10.638 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "a" 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-14T22:28:10.639 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-14T22:28:10.640 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-14T22:28:10.641 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "restful" 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modified": "2024-06-14T22:28:02.984785+0000", 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-14T22:28:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:mgr not available, waiting (1/15)... 2024-06-14T22:28:11.252 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:10.903+0000 7f52baa6aec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:28:11.252 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:11.014+0000 7f52baa6aec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:28:11.252 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:11.069+0000 7f52baa6aec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:28:11.253 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:11.257+0000 7f52baa6aec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:28:11.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:11.317+0000 7f52baa6aec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:28:11.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:11.382+0000 7f52baa6aec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:28:11.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:11.505+0000 7f52baa6aec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:28:12.199 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:11.915+0000 7f52baa6aec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:28:12.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:12.204+0000 7f52baa6aec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:28:12.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:12.269+0000 7f52baa6aec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:28:12.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:12.396+0000 7f52baa6aec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:28:12.899 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:12.650+0000 7f52baa6aec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:28:12.899 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:12.714+0000 7f52baa6aec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:28:12.899 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:12.904+0000 7f52baa6aec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:28:13.155 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:12.969+0000 7f52baa6aec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:28:13.155 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:13.056+0000 7f52baa6aec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:28:13.155 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:13.117+0000 7f52baa6aec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:28:13.467 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:13 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4128620675' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-14T22:28:13.735 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:13.662+0000 7f52baa6aec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:28:13.742 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsid": "44b2848a-2a9d-11ef-bc9d-c7b262605968", 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "health": { 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 0 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-14T22:28:13.743 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "a" 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-14T22:28:13.744 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-14T22:28:13.745 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "restful" 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:13.746 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-14T22:28:13.747 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:13.747 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modified": "2024-06-14T22:28:02.984785+0000", 2024-06-14T22:28:13.747 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-14T22:28:13.747 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:13.747 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-14T22:28:13.747 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-14T22:28:13.747 INFO:teuthology.orchestra.run.smithi086.stdout:mgr not available, waiting (2/15)... 2024-06-14T22:28:14.079 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:13.802+0000 7f52baa6aec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:28:14.079 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:13.802+0000 7f52baa6aec0 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:28:14.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: Activating manager daemon y 2024-06-14T22:28:14.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: mgrmap e2: y(active, starting, since 0.00292539s) 2024-06-14T22:28:14.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:28:14.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:28:14.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:28:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:28:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:28:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: Manager daemon y is now available 2024-06-14T22:28:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:28:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:28:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' 2024-06-14T22:28:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' 2024-06-14T22:28:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:14 smithi086 ceph-mon[29889]: from='mgr.14100 172.21.15.86:0/3855784076' entity='mgr.y' 2024-06-14T22:28:16.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:15 smithi086 ceph-mon[29889]: mgrmap e3: y(active, since 1.00593s) 2024-06-14T22:28:17.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:16 smithi086 ceph-mon[29889]: mgrmap e4: y(active, since 2s) 2024-06-14T22:28:17.016 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:16 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1507128447' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-14T22:28:17.020 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:17.020 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-14T22:28:17.020 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsid": "44b2848a-2a9d-11ef-bc9d-c7b262605968", 2024-06-14T22:28:17.020 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "health": { 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 0 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "a" 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:17.021 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-14T22:28:17.022 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-14T22:28:17.025 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:17.025 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-14T22:28:17.025 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-14T22:28:17.025 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-14T22:28:17.026 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "restful" 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-06-14T22:28:17.027 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-14T22:28:17.028 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:17.028 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-14T22:28:17.028 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-14T22:28:17.028 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modified": "2024-06-14T22:28:02.984785+0000", 2024-06-14T22:28:17.028 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-14T22:28:17.028 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-06-14T22:28:17.028 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-14T22:28:17.028 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-14T22:28:17.028 INFO:teuthology.orchestra.run.smithi086.stdout:mgr is available 2024-06-14T22:28:17.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:17 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2517619050' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [global] 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout fsid = 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-14T22:28:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-14T22:28:18.268 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-06-14T22:28:18.268 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [osd] 2024-06-14T22:28:18.268 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-14T22:28:18.268 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-14T22:28:18.268 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling cephadm module... 2024-06-14T22:28:19.205 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:18 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3060226142' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-14T22:28:19.206 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:18 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ignoring --setuser ceph since I am not root 2024-06-14T22:28:19.206 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:18 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ignoring --setgroup ceph since I am not root 2024-06-14T22:28:19.206 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:19.082+0000 7f4ba7527ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:28:19.206 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:19.209+0000 7f4ba7527ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:28:19.551 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:19.276+0000 7f4ba7527ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:28:19.551 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:19.523+0000 7f4ba7527ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:28:19.829 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:19.582+0000 7f4ba7527ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:28:20.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3060226142' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-14T22:28:20.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-mon[29889]: mgrmap e5: y(active, since 5s) 2024-06-14T22:28:20.329 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:19.884+0000 7f4ba7527ec0 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:28:20.329 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:19.884+0000 7f4ba7527ec0 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:28:20.329 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:28:20.329 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from .module import Module 2024-06-14T22:28:20.329 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:28:20.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:28:20.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:28:20.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:20.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:19.884+0000 7f4ba7527ec0 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:28:20.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:19.884+0000 7f4ba7527ec0 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:28:20.659 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:20 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:20.495+0000 7f4ba7527ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:28:20.660 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:20 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:20.608+0000 7f4ba7527ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:28:20.660 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:20 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:20.664+0000 7f4ba7527ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:28:20.914 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:20 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:20.859+0000 7f4ba7527ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:28:20.915 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:20 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:20.919+0000 7f4ba7527ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:28:21.329 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:20 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:20.986+0000 7f4ba7527ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:28:21.329 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:21.115+0000 7f4ba7527ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:28:21.829 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:21.542+0000 7f4ba7527ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:28:22.287 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:21.839+0000 7f4ba7527ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:28:22.287 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:21.904+0000 7f4ba7527ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:28:22.287 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:22.033+0000 7f4ba7527ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:28:22.287 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:22.292+0000 7f4ba7527ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:28:22.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:22.354+0000 7f4ba7527ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:28:22.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:22.539+0000 7f4ba7527ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:28:22.979 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:22.603+0000 7f4ba7527ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:28:22.979 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:22.686+0000 7f4ba7527ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:28:22.979 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:22.744+0000 7f4ba7527ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:28:23.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:23.302+0000 7f4ba7527ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:28:23.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:23.450+0000 7f4ba7527ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:23.450+0000 7f4ba7527ec0 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: Active manager daemon y restarted 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: Activating manager daemon y 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: osdmap e2: 0 total, 0 up, 0 in 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: mgrmap e6: y(active, starting, since 0.00303113s) 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:28:23.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: Manager daemon y is now available 2024-06-14T22:28:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:28:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:28:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:28:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:28:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:23 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:28:24.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:24 smithi086 ceph-mon[29889]: Found migration_current of "None". Setting to last migration. 2024-06-14T22:28:24.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:24 smithi086 ceph-mon[29889]: mgrmap e7: y(active, since 1.00481s) 2024-06-14T22:28:26.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:25.622+0000 7f4b599b3640 -1 log_channel(cephadm) log [ERR] : [14/Jun/2024:22:28:25] ENGINE Error in HTTPServer.serve 2024-06-14T22:28:26.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: Traceback (most recent call last): 2024-06-14T22:28:26.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:28:26.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self._connections.run(self.expiration_interval) 2024-06-14T22:28:26.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:28:26.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self._run(expiration_interval) 2024-06-14T22:28:26.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:28:26.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:28:26.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: s = self.context.wrap_socket( 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: return self.sslsocket_class._create( 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self.do_handshake() 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self._sslobj.do_handshake() 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:28:26.010 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:26.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:26 smithi086 ceph-mon[29889]: [14/Jun/2024:22:28:25] ENGINE Bus STARTING 2024-06-14T22:28:26.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:26 smithi086 ceph-mon[29889]: mgrmap e8: y(active, since 2s) 2024-06-14T22:28:26.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:26 smithi086 ceph-mon[29889]: [14/Jun/2024:22:28:25] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:28:26.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:26 smithi086 ceph-mon[29889]: [14/Jun/2024:22:28:25] ENGINE Bus STARTED 2024-06-14T22:28:26.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:26 smithi086 ceph-mon[29889]: [14/Jun/2024:22:28:25] ENGINE Error in HTTPServer.serve 2024-06-14T22:28:26.579 INFO:journalctl@ceph.mon.a.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:28:26.579 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:28:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout: self.do_handshake() 2024-06-14T22:28:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:28:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:28:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:28:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:26 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:26 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:28:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:26 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2015970762' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-14T22:28:30.576 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-14T22:28:30.577 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 8, 2024-06-14T22:28:30.577 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-14T22:28:30.577 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-06-14T22:28:30.577 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-14T22:28:30.577 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-14T22:28:30.577 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for the mgr to restart... 2024-06-14T22:28:30.577 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr epoch 8... 2024-06-14T22:28:31.650 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-14T22:28:31.651 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 8, 2024-06-14T22:28:31.651 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-14T22:28:31.651 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-14T22:28:31.651 INFO:teuthology.orchestra.run.smithi086.stdout:mgr epoch 8 is available 2024-06-14T22:28:31.651 INFO:teuthology.orchestra.run.smithi086.stdout:Setting orchestrator backend to cephadm... 2024-06-14T22:28:32.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:32 smithi086 ceph-mon[29889]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-14T22:28:32.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:32 smithi086 ceph-mon[29889]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-14T22:28:33.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:33 smithi086 ceph-mon[29889]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:28:33.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:33 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:33.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:33 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:28:33.953 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-14T22:28:33.953 INFO:teuthology.orchestra.run.smithi086.stdout:Generating ssh key... 2024-06-14T22:28:34.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-mon[29889]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:28:35.079 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: Generating public/private rsa key pair. 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: Your identification has been saved in /tmp/tmp5gi20fxq/key 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: Your public key has been saved in /tmp/tmp5gi20fxq/key.pub 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: The key fingerprint is: 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: SHA256:hwxez9Jo9PeilnoC6OijQxYuZmEv7lNO1KjYJkfFQsk ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: The key's randomart image is: 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: +---[RSA 3072]----+ 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: | o.o | 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: | E o | 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: | oo . o | 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: | +.o o = B | 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: |+o* .. S * . | 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: |+O++. .. o . . | 2024-06-14T22:28:35.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: |B+=o . .. . | 2024-06-14T22:28:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: |.oo.. . +. . | 2024-06-14T22:28:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: |o=o. .=. | 2024-06-14T22:28:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: +----[SHA256]-----+ 2024-06-14T22:28:35.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:35 smithi086 ceph-mon[29889]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:28:35.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:35 smithi086 ceph-mon[29889]: Generating ssh key... 2024-06-14T22:28:35.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:35 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:35.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:35 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:37.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:36 smithi086 ceph-mon[29889]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:28:37.184 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLrkJcfdOkD59/WZVB9VBcfpeqG81dkfHrIGPhmjo55mzDY8ReH5aEI1xUu1RXz+zxuuGiIwcdQ14pJbVOCSxwdIF00ZdOJwwa+zMeuUmfhMie0TOIu2fHrpjOGEFjgCLx6nOg/2O/FH7xFU0o4WJi68thk324KZxJYlPxKHWdFdzJHj1brGwEkWli84KweMTHWjEqs35F1rAyQcwjA/RxRFmHJLvseQrui8n5YL26zD+tWzYKFIpBWLG3b/CSlqWXFRr8ImVLs77kTJvkz2cX0o8FUkB5YYNT/dOl6Y5/srDf5+ClzfX7zKkCvDHzIJY6uu/J74Ev4yJMrikdyMTLladjy23LjI8/+ZkHR+Iq4YlfbBcJZIM1qsy4e6HcmSehpSEc67aTmJatXuf6REjJKGenFneMd7kf2QcYTEGYInn6ZoJp5tooKETgPiav8MW0+tXM2rMwM1KBopW4+mCe5mPjwq0fqRCzoaDpGYWX+/whupH5gLLcldVhtx9Px3s= ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:28:37.184 INFO:teuthology.orchestra.run.smithi086.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-14T22:28:37.185 INFO:teuthology.orchestra.run.smithi086.stdout:Adding key to root@localhost authorized_keys... 2024-06-14T22:28:37.185 INFO:teuthology.orchestra.run.smithi086.stdout:Adding host smithi086... 2024-06-14T22:28:38.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:38 smithi086 ceph-mon[29889]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "addr": "172.21.15.86", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:28:39.160 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:39 smithi086 ceph-mon[29889]: Deploying cephadm binary to smithi086 2024-06-14T22:28:41.233 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Added host 'smithi086' with addr '172.21.15.86' 2024-06-14T22:28:41.233 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying unmanaged mon service... 2024-06-14T22:28:41.903 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:41 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:41.903 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:41 smithi086 ceph-mon[29889]: Added host smithi086 2024-06-14T22:28:41.903 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:41 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:28:42.717 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-14T22:28:42.717 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying unmanaged mgr service... 2024-06-14T22:28:43.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:42 smithi086 ceph-mon[29889]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:28:43.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:42 smithi086 ceph-mon[29889]: Saving service mon spec with placement count:5 2024-06-14T22:28:43.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:42 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:43.950 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-14T22:28:44.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:44 smithi086 ceph-mon[29889]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:28:44.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:44 smithi086 ceph-mon[29889]: Saving service mgr spec with placement count:2 2024-06-14T22:28:44.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:44 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:44.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:44 smithi086 ceph-mon[29889]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:28:46.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:46 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:46.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:46 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:46.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:46 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1553390278' entity='client.admin' 2024-06-14T22:28:46.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:46 smithi086 ceph-mon[29889]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:28:47.326 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling the dashboard module... 2024-06-14T22:28:47.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:47 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1816053579' entity='client.admin' 2024-06-14T22:28:48.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:48 smithi086 ceph-mon[29889]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:28:48.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:48 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1369586201' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-14T22:28:48.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:48 smithi086 ceph-mon[29889]: from='mgr.14118 172.21.15.86:0/3103996048' entity='mgr.y' 2024-06-14T22:28:48.993 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ignoring --setuser ceph since I am not root 2024-06-14T22:28:48.993 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ignoring --setgroup ceph since I am not root 2024-06-14T22:28:48.993 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:48.867+0000 7f3e5a2a8ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:28:48.993 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:48.997+0000 7f3e5a2a8ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:28:49.303 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:49.065+0000 7f3e5a2a8ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:28:49.303 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:49.308+0000 7f3e5a2a8ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:28:49.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:49.373+0000 7f3e5a2a8ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:28:49.860 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1369586201' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-mon[29889]: mgrmap e9: y(active, since 25s) 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:49.683+0000 7f3e5a2a8ec0 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:49.683+0000 7f3e5a2a8ec0 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from .module import Module 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:49.683+0000 7f3e5a2a8ec0 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:28:49.861 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:49.683+0000 7f3e5a2a8ec0 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:28:50.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:50 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:50.313+0000 7f3e5a2a8ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:28:50.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:50 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:50.429+0000 7f3e5a2a8ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:28:50.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:50 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:50.486+0000 7f3e5a2a8ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:28:50.718 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-14T22:28:50.718 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-14T22:28:50.718 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-14T22:28:50.718 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-06-14T22:28:50.718 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-14T22:28:50.718 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-14T22:28:50.718 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for the mgr to restart... 2024-06-14T22:28:50.718 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr epoch 9... 2024-06-14T22:28:50.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:50 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2822925245' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-14T22:28:50.941 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:50 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:50.683+0000 7f3e5a2a8ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:28:50.941 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:50 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:50.747+0000 7f3e5a2a8ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:28:50.941 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:50 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:50.815+0000 7f3e5a2a8ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:28:50.941 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:50 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:50.946+0000 7f3e5a2a8ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:28:51.666 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:51 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:51.377+0000 7f3e5a2a8ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:28:51.666 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:51 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:51.671+0000 7f3e5a2a8ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:28:52.079 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:51 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:51.737+0000 7f3e5a2a8ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:28:52.079 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:51 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:51.866+0000 7f3e5a2a8ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:28:52.425 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:52 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:52.122+0000 7f3e5a2a8ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:28:52.425 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:52 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:52.182+0000 7f3e5a2a8ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:28:52.426 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:52 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:52.367+0000 7f3e5a2a8ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:28:52.829 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:52 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:52.430+0000 7f3e5a2a8ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:28:52.829 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:52 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:52.514+0000 7f3e5a2a8ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:28:52.829 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:52 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:52.572+0000 7f3e5a2a8ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:28:53.384 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:53.098+0000 7f3e5a2a8ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:28:53.384 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:53.236+0000 7f3e5a2a8ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:28:53.384 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:53.236+0000 7f3e5a2a8ec0 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:28:53.384 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: Active manager daemon y restarted 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: Activating manager daemon y 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: osdmap e3: 0 total, 0 up, 0 in 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: mgrmap e10: y(active, starting, since 0.005766s) 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: Manager daemon y is now available 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:28:53.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:28:55.192 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:54.851+0000 7f3e0aee3640 -1 log_channel(cephadm) log [ERR] : [14/Jun/2024:22:28:54] ENGINE Error in HTTPServer.serve 2024-06-14T22:28:55.192 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: Traceback (most recent call last): 2024-06-14T22:28:55.192 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:28:55.192 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self._connections.run(self.expiration_interval) 2024-06-14T22:28:55.192 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:28:55.192 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self._run(expiration_interval) 2024-06-14T22:28:55.192 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:28:55.192 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:28:55.192 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: s = self.context.wrap_socket( 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: return self.sslsocket_class._create( 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self.do_handshake() 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self._sslobj.do_handshake() 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:28:55.193 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:28:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:28:55.194 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-06-14T22:28:55.194 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-14T22:28:55.194 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-14T22:28:55.194 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-06-14T22:28:55.194 INFO:teuthology.orchestra.run.smithi086.stdout:mgr epoch 9 is available 2024-06-14T22:28:55.194 INFO:teuthology.orchestra.run.smithi086.stdout:Generating a dashboard self-signed certificate... 2024-06-14T22:28:55.450 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:55 smithi086 ceph-mon[29889]: mgrmap e11: y(active, since 1.00518s) 2024-06-14T22:28:55.450 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:55 smithi086 ceph-mon[29889]: [14/Jun/2024:22:28:54] ENGINE Bus STARTING 2024-06-14T22:28:55.451 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:55 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:28:55.451 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:55 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:28:56.417 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-14T22:28:56.417 INFO:teuthology.orchestra.run.smithi086.stdout:Creating initial admin user... 2024-06-14T22:28:56.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:56 smithi086 ceph-mon[29889]: [14/Jun/2024:22:28:54] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:28:56.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:56 smithi086 ceph-mon[29889]: [14/Jun/2024:22:28:54] ENGINE Bus STARTED 2024-06-14T22:28:56.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:56 smithi086 ceph-mon[29889]: [14/Jun/2024:22:28:54] ENGINE Error in HTTPServer.serve 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:28:56.830 INFO:journalctl@ceph.mon.a.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:28:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:28:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout: self.do_handshake() 2024-06-14T22:28:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:28:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:28:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:28:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:28:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:56 smithi086 ceph-mon[29889]: mgrmap e12: y(active, since 2s) 2024-06-14T22:28:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:28:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:28:57.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:57 smithi086 ceph-mon[29889]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:28:57.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:57 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:28:58.388 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Fq7npYSI1/YsCC5815R1eeHeygWPBoO.inL30KMTpMOZABSDkYKu.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1718404137, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-14T22:28:58.388 INFO:teuthology.orchestra.run.smithi086.stdout:Fetching dashboard port number... 2024-06-14T22:28:58.643 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:58 smithi086 ceph-mon[29889]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:28:59.585 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 8443 2024-06-14T22:28:59.595 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-14T22:28:59.595 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-06-14T22:28:59.602 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-14T22:28:59.603 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-06-14T22:28:59.603 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-06-14T22:28:59.603 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-14T22:28:59.604 INFO:teuthology.orchestra.run.smithi086.stdout:Ceph Dashboard is now available at: 2024-06-14T22:28:59.604 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:28:59.604 INFO:teuthology.orchestra.run.smithi086.stdout: URL: https://smithi086.front.sepia.ceph.com:8443/ 2024-06-14T22:28:59.604 INFO:teuthology.orchestra.run.smithi086.stdout: User: admin 2024-06-14T22:28:59.604 INFO:teuthology.orchestra.run.smithi086.stdout: Password: wvjmqmjtig 2024-06-14T22:28:59.605 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:28:59.605 INFO:teuthology.orchestra.run.smithi086.stdout:Saving cluster configuration to /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config directory 2024-06-14T22:28:59.605 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling autotune for osd_memory_target 2024-06-14T22:28:59.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:28:59 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3963846357' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-14T22:29:00.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:00 smithi086 ceph-mon[29889]: mgrmap e13: y(active, since 6s) 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:Or, if you are only running a single cluster on this host: 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout: ceph telemetry on 2024-06-14T22:29:02.107 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:29:02.108 INFO:teuthology.orchestra.run.smithi086.stdout:For more information see: 2024-06-14T22:29:02.108 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:29:02.108 INFO:teuthology.orchestra.run.smithi086.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-14T22:29:02.108 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:29:02.108 INFO:teuthology.orchestra.run.smithi086.stdout:Bootstrap complete. 2024-06-14T22:29:02.142 INFO:tasks.cephadm:Fetching config... 2024-06-14T22:29:02.142 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:29:02.142 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-14T22:29:02.158 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-14T22:29:02.158 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:29:02.158 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-14T22:29:02.213 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-14T22:29:02.213 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:29:02.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/keyring of=/dev/stdout 2024-06-14T22:29:02.277 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-14T22:29:02.278 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:29:02.278 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-14T22:29:02.333 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-14T22:29:02.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLrkJcfdOkD59/WZVB9VBcfpeqG81dkfHrIGPhmjo55mzDY8ReH5aEI1xUu1RXz+zxuuGiIwcdQ14pJbVOCSxwdIF00ZdOJwwa+zMeuUmfhMie0TOIu2fHrpjOGEFjgCLx6nOg/2O/FH7xFU0o4WJi68thk324KZxJYlPxKHWdFdzJHj1brGwEkWli84KweMTHWjEqs35F1rAyQcwjA/RxRFmHJLvseQrui8n5YL26zD+tWzYKFIpBWLG3b/CSlqWXFRr8ImVLs77kTJvkz2cX0o8FUkB5YYNT/dOl6Y5/srDf5+ClzfX7zKkCvDHzIJY6uu/J74Ev4yJMrikdyMTLladjy23LjI8/+ZkHR+Iq4YlfbBcJZIM1qsy4e6HcmSehpSEc67aTmJatXuf6REjJKGenFneMd7kf2QcYTEGYInn6ZoJp5tooKETgPiav8MW0+tXM2rMwM1KBopW4+mCe5mPjwq0fqRCzoaDpGYWX+/whupH5gLLcldVhtx9Px3s= ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-14T22:29:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLrkJcfdOkD59/WZVB9VBcfpeqG81dkfHrIGPhmjo55mzDY8ReH5aEI1xUu1RXz+zxuuGiIwcdQ14pJbVOCSxwdIF00ZdOJwwa+zMeuUmfhMie0TOIu2fHrpjOGEFjgCLx6nOg/2O/FH7xFU0o4WJi68thk324KZxJYlPxKHWdFdzJHj1brGwEkWli84KweMTHWjEqs35F1rAyQcwjA/RxRFmHJLvseQrui8n5YL26zD+tWzYKFIpBWLG3b/CSlqWXFRr8ImVLs77kTJvkz2cX0o8FUkB5YYNT/dOl6Y5/srDf5+ClzfX7zKkCvDHzIJY6uu/J74Ev4yJMrikdyMTLladjy23LjI8/+ZkHR+Iq4YlfbBcJZIM1qsy4e6HcmSehpSEc67aTmJatXuf6REjJKGenFneMd7kf2QcYTEGYInn6ZoJp5tooKETgPiav8MW0+tXM2rMwM1KBopW4+mCe5mPjwq0fqRCzoaDpGYWX+/whupH5gLLcldVhtx9Px3s= ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:29:02.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLrkJcfdOkD59/WZVB9VBcfpeqG81dkfHrIGPhmjo55mzDY8ReH5aEI1xUu1RXz+zxuuGiIwcdQ14pJbVOCSxwdIF00ZdOJwwa+zMeuUmfhMie0TOIu2fHrpjOGEFjgCLx6nOg/2O/FH7xFU0o4WJi68thk324KZxJYlPxKHWdFdzJHj1brGwEkWli84KweMTHWjEqs35F1rAyQcwjA/RxRFmHJLvseQrui8n5YL26zD+tWzYKFIpBWLG3b/CSlqWXFRr8ImVLs77kTJvkz2cX0o8FUkB5YYNT/dOl6Y5/srDf5+ClzfX7zKkCvDHzIJY6uu/J74Ev4yJMrikdyMTLladjy23LjI8/+ZkHR+Iq4YlfbBcJZIM1qsy4e6HcmSehpSEc67aTmJatXuf6REjJKGenFneMd7kf2QcYTEGYInn6ZoJp5tooKETgPiav8MW0+tXM2rMwM1KBopW4+mCe5mPjwq0fqRCzoaDpGYWX+/whupH5gLLcldVhtx9Px3s= ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-14T22:29:02.457 INFO:teuthology.orchestra.run.smithi137.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLrkJcfdOkD59/WZVB9VBcfpeqG81dkfHrIGPhmjo55mzDY8ReH5aEI1xUu1RXz+zxuuGiIwcdQ14pJbVOCSxwdIF00ZdOJwwa+zMeuUmfhMie0TOIu2fHrpjOGEFjgCLx6nOg/2O/FH7xFU0o4WJi68thk324KZxJYlPxKHWdFdzJHj1brGwEkWli84KweMTHWjEqs35F1rAyQcwjA/RxRFmHJLvseQrui8n5YL26zD+tWzYKFIpBWLG3b/CSlqWXFRr8ImVLs77kTJvkz2cX0o8FUkB5YYNT/dOl6Y5/srDf5+ClzfX7zKkCvDHzIJY6uu/J74Ev4yJMrikdyMTLladjy23LjI8/+ZkHR+Iq4YlfbBcJZIM1qsy4e6HcmSehpSEc67aTmJatXuf6REjJKGenFneMd7kf2QcYTEGYInn6ZoJp5tooKETgPiav8MW0+tXM2rMwM1KBopW4+mCe5mPjwq0fqRCzoaDpGYWX+/whupH5gLLcldVhtx9Px3s= ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:29:02.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-14T22:29:02.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:02 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/40627213' entity='client.admin' 2024-06-14T22:29:02.813 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:29:05.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:05.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:05.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:29:05.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:05.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:05.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:05.648 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-14T22:29:05.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-14T22:29:05.893 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:29:06.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:06 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/169520897' entity='client.admin' 2024-06-14T22:29:06.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:06 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:06.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:06 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:08.415 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi137 2024-06-14T22:29:08.415 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:29:08.415 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.conf 2024-06-14T22:29:08.438 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:29:08.438 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:29:08.493 INFO:tasks.cephadm:Adding host smithi137 to orchestrator... 2024-06-14T22:29:08.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch host add smithi137 2024-06-14T22:29:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:29:08.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:08 smithi086 ceph-mon[29889]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:08.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:08.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:08.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:08.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:08.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:08 smithi086 ceph-mon[29889]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:29:08.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:08.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:08.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:09.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:09 smithi086 ceph-mon[29889]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:09.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:09 smithi086 ceph-mon[29889]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:29:09.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:09 smithi086 ceph-mon[29889]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:29:11.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:10 smithi086 ceph-mon[29889]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:12.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:11 smithi086 ceph-mon[29889]: Deploying cephadm binary to smithi137 2024-06-14T22:29:13.261 INFO:teuthology.orchestra.run.smithi086.stdout:Added host 'smithi137' with addr '172.21.15.137' 2024-06-14T22:29:13.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch host ls --format=json 2024-06-14T22:29:14.142 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:29:14.538 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:14 smithi086 ceph-mon[29889]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:14.538 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:14 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:14.538 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:14 smithi086 ceph-mon[29889]: Added host smithi137 2024-06-14T22:29:14.538 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:14 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:14.538 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:14 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:14.539 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:14 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:15.809 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:29:15.809 INFO:teuthology.orchestra.run.smithi086.stdout:[{"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}, {"addr": "172.21.15.137", "hostname": "smithi137", "labels": [], "status": ""}] 2024-06-14T22:29:15.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:15 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:16.380 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-14T22:29:16.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd crush tunables default 2024-06-14T22:29:16.625 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:29:16.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:16 smithi086 ceph-mon[29889]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:17.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:17 smithi086 ceph-mon[29889]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:29:17.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:17.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:17.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:17.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:17.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi137", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:29:17.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:17.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:19.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:18 smithi086 ceph-mon[29889]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:19.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:18 smithi086 ceph-mon[29889]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:29:19.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:18 smithi086 ceph-mon[29889]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:19.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:18 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:19.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:18 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:19.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:18 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:19.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:18 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4251212281' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-14T22:29:19.272 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2024-06-14T22:29:19.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:19 smithi086 ceph-mon[29889]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:29:19.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:19 smithi086 ceph-mon[29889]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:29:19.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:19 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4251212281' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-14T22:29:19.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:19 smithi086 ceph-mon[29889]: osdmap e4: 0 total, 0 up, 0 in 2024-06-14T22:29:19.898 INFO:tasks.cephadm:Adding mon.a on smithi086 2024-06-14T22:29:19.898 INFO:tasks.cephadm:Adding mon.c on smithi086 2024-06-14T22:29:19.899 INFO:tasks.cephadm:Adding mon.b on smithi137 2024-06-14T22:29:19.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch apply mon '3;smithi086:172.21.15.86=a;smithi086:[v2:172.21.15.86:3301,v1:172.21.15.86:6790]=c;smithi137:172.21.15.137=b' 2024-06-14T22:29:20.148 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:21.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:20 smithi086 ceph-mon[29889]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:21.656 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled mon update... 2024-06-14T22:29:22.291 DEBUG:teuthology.orchestra.run.smithi086:mon.c> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.c.service 2024-06-14T22:29:22.294 DEBUG:teuthology.orchestra.run.smithi137:mon.b> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.b.service 2024-06-14T22:29:22.296 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-14T22:29:22.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-14T22:29:22.931 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:23.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:23.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi086:172.21.15.86=a;smithi086:[v2:172.21.15.86:3301,v1:172.21.15.86:6790]=c;smithi137:172.21.15.137=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:23.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: Saving service mon spec with placement smithi086:172.21.15.86=a;smithi086:[v2:172.21.15.86:3301,v1:172.21.15.86:6790]=c;smithi137:172.21.15.137=b;count:3 2024-06-14T22:29:23.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:23.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:23.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:23.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:23.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:23.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:23.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:23.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:22 smithi086 ceph-mon[29889]: Deploying daemon mon.b on smithi137 2024-06-14T22:29:25.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:24 smithi086 ceph-mon[29889]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:26.064 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:29:26.064 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","modified":"2024-06-14T22:28:00.129790Z","created":"2024-06-14T22:28:00.129790Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-14T22:29:26.064 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-06-14T22:29:26.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:26 smithi137 ceph-mon[32951]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-14T22:29:26.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:26 smithi137 bash[32885]: bd4aed27097c3e761c223fe064ebccaee77a0e06b62d1f222bbbcadb7da718cc 2024-06-14T22:29:26.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:26 smithi137 systemd[1]: Started Ceph mon.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:29:27.735 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-14T22:29:27.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-14T22:29:27.992 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:29:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:29:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: mon.a calling monitor election 2024-06-14T22:29:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: mon.b calling monitor election 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: monmap e2: 2 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: fsmap 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: osdmap e4: 0 total, 0 up, 0 in 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: mgrmap e13: y(active, since 37s) 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: overall HEALTH_OK 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:31.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: mon.a calling monitor election 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: mon.b calling monitor election 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-06-14T22:29:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: monmap e2: 2 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:29:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: fsmap 2024-06-14T22:29:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: osdmap e4: 0 total, 0 up, 0 in 2024-06-14T22:29:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: mgrmap e13: y(active, since 37s) 2024-06-14T22:29:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: overall HEALTH_OK 2024-06-14T22:29:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:32.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:32 smithi137 ceph-mon[32951]: Deploying daemon mon.c on smithi086 2024-06-14T22:29:32.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:32 smithi137 ceph-mon[32951]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:32.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:32 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:32.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:32 smithi086 ceph-mon[29889]: Deploying daemon mon.c on smithi086 2024-06-14T22:29:32.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:32 smithi086 ceph-mon[29889]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:32.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:32 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:32.720 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:29:32.720 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":2,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","modified":"2024-06-14T22:29:26.147293Z","created":"2024-06-14T22:28:00.129790Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-14T22:29:32.720 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 2 2024-06-14T22:29:33.335 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:33 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/4056684411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T22:29:33.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:33 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/4056684411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T22:29:34.337 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-14T22:29:34.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-14T22:29:34.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[29889]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:34.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:34 smithi137 ceph-mon[32951]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:34.589 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:29:34.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(???) e0 preinit fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:29:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).mds e1 new map 2024-06-14T22:29:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).mds e1 print_map 2024-06-14T22:29:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout: e1 2024-06-14T22:29:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-14T22:29:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-14T22:29:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout: legacy client fscid: -1 2024-06-14T22:29:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:29:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout: No filesystems configured 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:29:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4251212281' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4251212281' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: osdmap e4: 0 total, 0 up, 0 in 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi086:172.21.15.86=a;smithi086:[v2:172.21.15.86:3301,v1:172.21.15.86:6790]=c;smithi137:172.21.15.137=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:35.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: Saving service mon spec with placement smithi086:172.21.15.86=a;smithi086:[v2:172.21.15.86:3301,v1:172.21.15.86:6790]=c;smithi137:172.21.15.137=b;count:3 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: Deploying daemon mon.b on smithi137 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.a calling monitor election 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.b calling monitor election 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: monmap e2: 2 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: fsmap 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: osdmap e4: 0 total, 0 up, 0 in 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mgrmap e13: y(active, since 37s) 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: overall HEALTH_OK 2024-06-14T22:29:35.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: Deploying daemon mon.c on smithi086 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/4056684411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 ceph-mon[38904]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 bash[38865]: e02d9b886f41e1da45f684fb73c25a7bd7086eb9bf94b02aac4aa937f900af12 2024-06-14T22:29:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:34 smithi086 systemd[1]: Started Ceph mon.c for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:29:39.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:29:39.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: mon.a calling monitor election 2024-06-14T22:29:39.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:39.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:39.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: mon.b calling monitor election 2024-06-14T22:29:39.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:39.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:39.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: mon.c calling monitor election 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: monmap e3: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: fsmap 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: osdmap e4: 0 total, 0 up, 0 in 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: mgrmap e13: y(active, since 46s) 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: overall HEALTH_OK 2024-06-14T22:29:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:39.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:39.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:39.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:39 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:40.002 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-06-14T22:29:40.002 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":3,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","modified":"2024-06-14T22:29:34.628631Z","created":"2024-06-14T22:28:00.129790Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-06-14T22:29:40.002 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 3 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: mon.a calling monitor election 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: mon.b calling monitor election 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: mon.c calling monitor election 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: monmap e3: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: fsmap 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: osdmap e4: 0 total, 0 up, 0 in 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: mgrmap e13: y(active, since 46s) 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: overall HEALTH_OK 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: mon.a calling monitor election 2024-06-14T22:29:40.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: mon.b calling monitor election 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: mon.c calling monitor election 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: monmap e3: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: fsmap 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: osdmap e4: 0 total, 0 up, 0 in 2024-06-14T22:29:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: mgrmap e13: y(active, since 46s) 2024-06-14T22:29:40.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: overall HEALTH_OK 2024-06-14T22:29:40.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:40.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:40.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:40.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:39 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:40.540 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-14T22:29:40.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph config generate-minimal-conf 2024-06-14T22:29:40.798 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:29:40.834 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:40 smithi137 ceph-mon[32951]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:40.834 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:40 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:40.834 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:40 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/1983861346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T22:29:40.834 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:40 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:40 smithi086 ceph-mon[29889]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:40.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:40 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:40.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:40 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/1983861346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T22:29:40.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:40 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:40.950 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:40 smithi086 ceph-mon[38904]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:40.950 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:40 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:40.951 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:40 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/1983861346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T22:29:40.951 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:40 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:29:41.982 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.982 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.982 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.982 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.982 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:41.982 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: Reconfiguring mon.a (unknown last config time)... 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:41.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[38904]: Reconfiguring daemon mon.a on smithi086 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:41.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: Reconfiguring mon.a (unknown last config time)... 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:41.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:41 smithi086 ceph-mon[29889]: Reconfiguring daemon mon.a on smithi086 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:42.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: Reconfiguring mon.a (unknown last config time)... 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:42.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:41 smithi137 ceph-mon[32951]: Reconfiguring daemon mon.a on smithi086 2024-06-14T22:29:43.008 INFO:teuthology.orchestra.run.smithi086.stdout:# minimal ceph.conf for 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:29:43.008 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2024-06-14T22:29:43.008 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:29:43.008 INFO:teuthology.orchestra.run.smithi086.stdout: mon_host = [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-06-14T22:29:43.450 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:43 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3574686318' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:43.450 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:43 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3574686318' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:43.487 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:43 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3574686318' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:43.711 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-14T22:29:43.712 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:29:43.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-06-14T22:29:43.739 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:29:43.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:29:43.803 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:29:43.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.conf 2024-06-14T22:29:43.833 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:29:43.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:29:43.900 INFO:tasks.cephadm:Adding mgr.y on smithi086 2024-06-14T22:29:43.900 INFO:tasks.cephadm:Adding mgr.x on smithi137 2024-06-14T22:29:43.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch apply mgr '2;smithi086=y;smithi137=x' 2024-06-14T22:29:44.184 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:29:44.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[29889]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:44.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:44.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:44.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:44.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:44.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:29:44.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:44.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[38904]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:44.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:44.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:44.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:44.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:44.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:29:44.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:44.622 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:44 smithi137 ceph-mon[32951]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:44.622 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:44.622 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:44.622 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:44.622 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:44.622 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:29:44.622 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:45.545 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:45 smithi137 ceph-mon[32951]: Reconfiguring mon.c (monmap changed)... 2024-06-14T22:29:45.546 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:45 smithi137 ceph-mon[32951]: Reconfiguring daemon mon.c on smithi086 2024-06-14T22:29:45.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:45 smithi086 ceph-mon[38904]: Reconfiguring mon.c (monmap changed)... 2024-06-14T22:29:45.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:45 smithi086 ceph-mon[38904]: Reconfiguring daemon mon.c on smithi086 2024-06-14T22:29:45.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:45 smithi086 ceph-mon[29889]: Reconfiguring mon.c (monmap changed)... 2024-06-14T22:29:45.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:45 smithi086 ceph-mon[29889]: Reconfiguring daemon mon.c on smithi086 2024-06-14T22:29:45.821 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled mgr update... 2024-06-14T22:29:46.415 DEBUG:teuthology.orchestra.run.smithi137:mgr.x> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.x.service 2024-06-14T22:29:46.417 INFO:tasks.cephadm:Deploying OSDs... 2024-06-14T22:29:46.417 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:29:46.417 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-06-14T22:29:46.436 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-14T22:29:46.436 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-06-14T22:29:46.492 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-14T22:29:46.492 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-14T22:29:46.492 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-06-14T22:29:46.492 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-14T22:29:46.492 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-06-14T22:29:46.493 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-06-14 22:29:34.042403392 +0000 2024-06-14T22:29:46.493 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-06-14 22:24:28.269209740 +0000 2024-06-14T22:29:46.493 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-06-14 22:24:28.269209740 +0000 2024-06-14T22:29:46.493 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-06-14 22:24:28.269209740 +0000 2024-06-14T22:29:46.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-14T22:29:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-06-14T22:29:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-06-14T22:29:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000288325 s, 1.8 MB/s 2024-06-14T22:29:46.560 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-14T22:29:46.618 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-06-14T22:29:46.674 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-14T22:29:46.674 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-14T22:29:46.674 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 692 Links: 1 2024-06-14T22:29:46.674 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-14T22:29:46.674 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-06-14T22:29:46.674 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-06-14 22:29:34.043403371 +0000 2024-06-14T22:29:46.674 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-06-14 22:24:28.597202843 +0000 2024-06-14T22:29:46.675 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-06-14 22:24:28.597202843 +0000 2024-06-14T22:29:46.675 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-06-14 22:24:28.597202843 +0000 2024-06-14T22:29:46.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-14T22:29:46.681 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:46 smithi137 ceph-mon[32951]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:46.681 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:46 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:46.681 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:46 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:46.681 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:46 smithi137 ceph-mon[32951]: Reconfiguring mon.b (monmap changed)... 2024-06-14T22:29:46.681 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:46 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:46.681 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:46 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:29:46.681 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:46 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:46.681 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:46 smithi137 ceph-mon[32951]: Reconfiguring daemon mon.b on smithi137 2024-06-14T22:29:46.681 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:46 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:46.738 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-06-14T22:29:46.738 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-06-14T22:29:46.738 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000204931 s, 2.5 MB/s 2024-06-14T22:29:46.739 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-14T22:29:46.796 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-06-14T22:29:46.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[38904]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:46.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:46.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:46.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[38904]: Reconfiguring mon.b (monmap changed)... 2024-06-14T22:29:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:29:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[38904]: Reconfiguring daemon mon.b on smithi137 2024-06-14T22:29:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[29889]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[29889]: Reconfiguring mon.b (monmap changed)... 2024-06-14T22:29:46.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:29:46.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:29:46.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:46.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[29889]: Reconfiguring daemon mon.b on smithi137 2024-06-14T22:29:46.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:46 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:46.845 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-14T22:29:46.846 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-14T22:29:46.846 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 710 Links: 1 2024-06-14T22:29:46.846 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-14T22:29:46.846 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-06-14T22:29:46.846 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-06-14 22:29:34.043403371 +0000 2024-06-14T22:29:46.846 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-06-14 22:24:28.925195947 +0000 2024-06-14T22:29:46.846 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-06-14 22:24:28.925195947 +0000 2024-06-14T22:29:46.846 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-06-14 22:24:28.925195947 +0000 2024-06-14T22:29:46.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-14T22:29:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-06-14T22:29:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-06-14T22:29:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000217492 s, 2.4 MB/s 2024-06-14T22:29:46.911 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-14T22:29:46.967 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-06-14T22:29:47.024 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-14T22:29:47.024 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-14T22:29:47.024 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 729 Links: 1 2024-06-14T22:29:47.024 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-14T22:29:47.024 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-06-14T22:29:47.024 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-06-14 22:29:34.044403351 +0000 2024-06-14T22:29:47.024 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-06-14 22:24:29.262188861 +0000 2024-06-14T22:29:47.024 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-06-14 22:24:29.262188861 +0000 2024-06-14T22:29:47.024 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-06-14 22:24:29.262188861 +0000 2024-06-14T22:29:47.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-14T22:29:47.087 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-06-14T22:29:47.087 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-06-14T22:29:47.087 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000214544 s, 2.4 MB/s 2024-06-14T22:29:47.088 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-14T22:29:47.145 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:29:47.145 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-06-14T22:29:47.163 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-14T22:29:47.163 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-06-14T22:29:47.219 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-14T22:29:47.219 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-14T22:29:47.219 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-14T22:29:47.219 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-14T22:29:47.219 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-06-14T22:29:47.219 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-06-14 22:29:25.499309161 +0000 2024-06-14T22:29:47.219 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-06-14 22:24:26.016242631 +0000 2024-06-14T22:29:47.219 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-06-14 22:24:26.016242631 +0000 2024-06-14T22:29:47.219 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-06-14 22:24:26.016242631 +0000 2024-06-14T22:29:47.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-14T22:29:47.284 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-06-14T22:29:47.284 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-06-14T22:29:47.284 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000204258 s, 2.5 MB/s 2024-06-14T22:29:47.285 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-14T22:29:47.341 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-06-14T22:29:47.397 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-14T22:29:47.397 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-14T22:29:47.397 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-14T22:29:47.397 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-14T22:29:47.397 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-06-14T22:29:47.397 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-06-14 22:29:25.499309161 +0000 2024-06-14T22:29:47.397 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-06-14 22:24:26.350236827 +0000 2024-06-14T22:29:47.397 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-06-14 22:24:26.350236827 +0000 2024-06-14T22:29:47.397 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-06-14 22:24:26.350236827 +0000 2024-06-14T22:29:47.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-14T22:29:47.460 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-06-14T22:29:47.460 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-06-14T22:29:47.460 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000198088 s, 2.6 MB/s 2024-06-14T22:29:47.461 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-14T22:29:47.517 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-06-14T22:29:47.573 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-14T22:29:47.573 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-14T22:29:47.573 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-06-14T22:29:47.573 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-14T22:29:47.573 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-06-14T22:29:47.573 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-06-14 22:29:25.500309144 +0000 2024-06-14T22:29:47.573 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-06-14 22:24:26.678231128 +0000 2024-06-14T22:29:47.573 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-06-14 22:24:26.678231128 +0000 2024-06-14T22:29:47.573 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-06-14 22:24:26.678231128 +0000 2024-06-14T22:29:47.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-14T22:29:47.636 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-06-14T22:29:47.636 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-06-14T22:29:47.636 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000252347 s, 2.0 MB/s 2024-06-14T22:29:47.637 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-14T22:29:47.693 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-06-14T22:29:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-14T22:29:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-14T22:29:47.749 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-14T22:29:47.749 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-14T22:29:47.750 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-06-14T22:29:47.750 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-06-14 22:29:25.500309144 +0000 2024-06-14T22:29:47.750 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-06-14 22:24:27.016225256 +0000 2024-06-14T22:29:47.750 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-06-14 22:24:27.016225256 +0000 2024-06-14T22:29:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-06-14 22:24:27.016225256 +0000 2024-06-14T22:29:47.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-14T22:29:47.811 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:47 smithi137 ceph-mon[32951]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi086=y;smithi137=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:47.811 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:47 smithi137 ceph-mon[32951]: Saving service mgr spec with placement smithi086=y;smithi137=x;count:2 2024-06-14T22:29:47.813 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-06-14T22:29:47.813 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-06-14T22:29:47.814 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.00018654 s, 2.7 MB/s 2024-06-14T22:29:47.815 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-14T22:29:47.871 INFO:tasks.cephadm:Deploying osd.0 on smithi086 with /dev/vg_nvme/lv_4... 2024-06-14T22:29:47.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-14T22:29:47.894 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:47 smithi086 ceph-mon[29889]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi086=y;smithi137=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:47.894 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:47 smithi086 ceph-mon[29889]: Saving service mgr spec with placement smithi086=y;smithi137=x;count:2 2024-06-14T22:29:47.895 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:47 smithi086 ceph-mon[38904]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi086=y;smithi137=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:47.895 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:47 smithi086 ceph-mon[38904]: Saving service mgr spec with placement smithi086=y;smithi137=x;count:2 2024-06-14T22:29:48.115 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:29:48.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:48.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:48.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:48.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[29889]: Deploying daemon mgr.x on smithi137 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:48.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:48.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:48.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:48.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:48.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:29:48.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-14T22:29:48.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:29:48.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:48.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:48 smithi086 ceph-mon[38904]: Deploying daemon mgr.x on smithi137 2024-06-14T22:29:48.893 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:48.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:48 smithi137 ceph-mon[32951]: Deploying daemon mgr.x on smithi137 2024-06-14T22:29:49.587 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:49 smithi137 podman[34988]: 2024-06-14 22:29:49.500838764 +0000 UTC m=+0.117507171 container create 2817a158009e4f7062c5cc70d279647804d87e455bfd590ef50434c97327c2d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-06-14T22:29:49.903 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-14T22:29:49.955 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:49 smithi137 podman[34988]: 2024-06-14 22:29:49.702974022 +0000 UTC m=+0.319625176 container init 2817a158009e4f7062c5cc70d279647804d87e455bfd590ef50434c97327c2d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b) 2024-06-14T22:29:49.955 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:49 smithi137 podman[34988]: 2024-06-14 22:29:49.708834581 +0000 UTC m=+0.325485730 container start 2817a158009e4f7062c5cc70d279647804d87e455bfd590ef50434c97327c2d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:29:49.956 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:49 smithi137 ceph-mgr[35026]: -- 172.21.15.137:0/1762190725 <== mon.1 v2:172.21.15.137:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e0514509c0 con 0x55e05145c800 2024-06-14T22:29:49.956 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:49 smithi137 bash[34988]: 2817a158009e4f7062c5cc70d279647804d87e455bfd590ef50434c97327c2d6 2024-06-14T22:29:49.956 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:49 smithi137 systemd[1]: Started Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:29:49.956 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:49.826+0000 7f1015c4fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:29:49.956 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:49.960+0000 7f1015c4fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:29:50.010 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-14T22:29:50.026 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-06-14T22:29:50.026 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-06-14T22:29:50.026 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130626 s, 803 MB/s 2024-06-14T22:29:50.026 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-06-14T22:29:50.238 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:50.030+0000 7f1015c4fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:29:50.575 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:50.303+0000 7f1015c4fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:29:50.575 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:50.367+0000 7f1015c4fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:29:50.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-06-14T22:29:50.873 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:29:50.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-mon[32951]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:50.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:50.988 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:50.688+0000 7f1015c4fec0 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:29:50.988 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:50.688+0000 7f1015c4fec0 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:29:50.988 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:29:50.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: from .module import Module 2024-06-14T22:29:50.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:29:50.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:29:50.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:29:50.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:50.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:50.689+0000 7f1015c4fec0 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:29:50.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:50.689+0000 7f1015c4fec0 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:29:50.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[38904]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[29889]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:50.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:50 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:51.672 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:51.321+0000 7f1015c4fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:29:51.673 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:51.433+0000 7f1015c4fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:29:51.673 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:51.488+0000 7f1015c4fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:29:51.673 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:51.677+0000 7f1015c4fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:29:51.923 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:51.737+0000 7f1015c4fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:29:51.923 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:51.803+0000 7f1015c4fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:29:51.923 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:51.927+0000 7f1015c4fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:29:52.151 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:52.151 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: Reconfiguring mgr.y (unknown last config time)... 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: Reconfiguring daemon mgr.y on smithi086 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[38904]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:52.152 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:52.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:52.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:52.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:52.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: Reconfiguring mgr.y (unknown last config time)... 2024-06-14T22:29:52.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:29:52.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:29:52.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:52.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: Reconfiguring daemon mgr.y on smithi086 2024-06-14T22:29:52.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:52 smithi086 ceph-mon[29889]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: Reconfiguring mgr.y (unknown last config time)... 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: Reconfiguring daemon mgr.y on smithi086 2024-06-14T22:29:52.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-mon[32951]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:52.632 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:52.345+0000 7f1015c4fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:29:52.632 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:52.636+0000 7f1015c4fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:29:52.988 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:52.702+0000 7f1015c4fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:29:52.988 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:52.830+0000 7f1015c4fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:29:53.324 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-mon[32951]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:53.324 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:29:53.324 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:29:53.324 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:53.324 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:53.085+0000 7f1015c4fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:29:53.325 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:53.145+0000 7f1015c4fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:29:53.325 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:53.329+0000 7f1015c4fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:29:53.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:53 smithi086 ceph-mon[38904]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:53.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:53 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:29:53.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:53 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:29:53.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:53 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:53.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:53 smithi086 ceph-mon[29889]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:29:53.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:29:53.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:29:53.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:53.738 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:53.392+0000 7f1015c4fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:29:53.738 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:53.473+0000 7f1015c4fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:29:53.738 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:53.531+0000 7f1015c4fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:29:54.488 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:54.064+0000 7f1015c4fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:29:54.488 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:54.201+0000 7f1015c4fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:29:54.488 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:29:54.201+0000 7f1015c4fec0 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: Standby manager daemon x started 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:29:54.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:29:54.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:54 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:29:54.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: Standby manager daemon x started 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:29:54.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: Standby manager daemon x started 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:29:54.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:54 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/474734879' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:29:55.422 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:55 smithi086 ceph-mon[38904]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:29:55.422 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:55 smithi086 ceph-mon[38904]: mgrmap e14: y(active, since 61s), standbys: x 2024-06-14T22:29:55.422 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:55 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:29:55.423 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:55 smithi086 ceph-mon[29889]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:29:55.423 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:55 smithi086 ceph-mon[29889]: mgrmap e14: y(active, since 61s), standbys: x 2024-06-14T22:29:55.423 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:55 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:29:55.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:55 smithi137 ceph-mon[32951]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:29:55.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:55 smithi137 ceph-mon[32951]: mgrmap e14: y(active, since 61s), standbys: x 2024-06-14T22:29:55.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:55 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[29889]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1309422362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db19a6c7-4c6f-4c95-aa2f-3ddc1e121270"}]: dispatch 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1309422362' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db19a6c7-4c6f-4c95-aa2f-3ddc1e121270"}]': finished 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[29889]: osdmap e5: 1 total, 0 up, 1 in 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3757332834' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[38904]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1309422362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db19a6c7-4c6f-4c95-aa2f-3ddc1e121270"}]: dispatch 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1309422362' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db19a6c7-4c6f-4c95-aa2f-3ddc1e121270"}]': finished 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[38904]: osdmap e5: 1 total, 0 up, 1 in 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:29:56.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:56 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3757332834' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:29:56.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:56 smithi137 ceph-mon[32951]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:56.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:56 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1309422362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db19a6c7-4c6f-4c95-aa2f-3ddc1e121270"}]: dispatch 2024-06-14T22:29:56.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:56 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1309422362' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db19a6c7-4c6f-4c95-aa2f-3ddc1e121270"}]': finished 2024-06-14T22:29:56.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:56 smithi137 ceph-mon[32951]: osdmap e5: 1 total, 0 up, 1 in 2024-06-14T22:29:56.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:56 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:29:56.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:56 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3757332834' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:29:58.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:29:58 smithi086 ceph-mon[29889]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:58.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:29:58 smithi086 ceph-mon[38904]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:29:58.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:29:58 smithi137 ceph-mon[32951]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:00.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:00 smithi086 ceph-mon[29889]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:00.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:00 smithi086 ceph-mon[29889]: overall HEALTH_OK 2024-06-14T22:30:00.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:00 smithi086 ceph-mon[38904]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:00.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:00 smithi086 ceph-mon[38904]: overall HEALTH_OK 2024-06-14T22:30:00.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:00 smithi137 ceph-mon[32951]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:00.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:00 smithi137 ceph-mon[32951]: overall HEALTH_OK 2024-06-14T22:30:02.540 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:02 smithi137 ceph-mon[32951]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:02.541 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:02 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-14T22:30:02.541 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:02 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:02.543 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:02 smithi086 ceph-mon[29889]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:02.543 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:02 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-14T22:30:02.543 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:02 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:02.543 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:02 smithi086 ceph-mon[38904]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:02.543 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:02 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-14T22:30:02.544 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:02 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:03.392 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:03 smithi086 ceph-mon[29889]: Deploying daemon osd.0 on smithi086 2024-06-14T22:30:03.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:03 smithi086 ceph-mon[38904]: Deploying daemon osd.0 on smithi086 2024-06-14T22:30:03.722 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:03 smithi137 ceph-mon[32951]: Deploying daemon osd.0 on smithi086 2024-06-14T22:30:04.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:04 smithi086 ceph-mon[29889]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:04.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:04 smithi086 ceph-mon[38904]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:04.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:04 smithi137 ceph-mon[32951]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:06.490 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:06 smithi086 ceph-mon[29889]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:06.490 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:06 smithi086 ceph-mon[38904]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:06.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:06 smithi137 ceph-mon[32951]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:07.574 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:07 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:07.574 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:07 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:07.574 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:07 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:07.574 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:07 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:07.575 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:07 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:07.575 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:07 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:07.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:07 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:07.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:07 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:07.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:07 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:08.549 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:08 smithi086 ceph-mon[29889]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:08.549 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:08 smithi086 ceph-mon[38904]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:08.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:08 smithi137 ceph-mon[32951]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:10.175 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 0 on host 'smithi086' 2024-06-14T22:30:10.525 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[29889]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:10.525 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:10.525 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:10.525 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[29889]: from='osd.0 [v2:172.21.15.86:6802/2092557056,v1:172.21.15.86:6803/2092557056]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-14T22:30:10.525 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[29889]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-14T22:30:10.525 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[38904]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:10.525 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:10.525 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:10.526 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[38904]: from='osd.0 [v2:172.21.15.86:6802/2092557056,v1:172.21.15.86:6803/2092557056]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-14T22:30:10.526 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:10 smithi086 ceph-mon[38904]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-14T22:30:10.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:10 smithi137 ceph-mon[32951]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:10.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:10 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:10.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:10 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:10.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:10 smithi137 ceph-mon[32951]: from='osd.0 [v2:172.21.15.86:6802/2092557056,v1:172.21.15.86:6803/2092557056]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-14T22:30:10.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:10 smithi137 ceph-mon[32951]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-14T22:30:10.838 DEBUG:teuthology.orchestra.run.smithi086:osd.0> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.0.service 2024-06-14T22:30:10.840 INFO:tasks.cephadm:Deploying osd.1 on smithi086 with /dev/vg_nvme/lv_3... 2024-06-14T22:30:10.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-14T22:30:11.142 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[38904]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[38904]: osdmap e6: 1 total, 0 up, 1 in 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[38904]: from='osd.0 [v2:172.21.15.86:6802/2092557056,v1:172.21.15.86:6803/2092557056]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[38904]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[29889]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-14T22:30:11.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[29889]: osdmap e6: 1 total, 0 up, 1 in 2024-06-14T22:30:11.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:11.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[29889]: from='osd.0 [v2:172.21.15.86:6802/2092557056,v1:172.21.15.86:6803/2092557056]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:11.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:11 smithi086 ceph-mon[29889]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:11.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:11 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:11.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:11 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:11.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:11 smithi137 ceph-mon[32951]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-14T22:30:11.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:11 smithi137 ceph-mon[32951]: osdmap e6: 1 total, 0 up, 1 in 2024-06-14T22:30:11.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:11 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:11.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:11 smithi137 ceph-mon[32951]: from='osd.0 [v2:172.21.15.86:6802/2092557056,v1:172.21.15.86:6803/2092557056]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:11.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:11 smithi137 ceph-mon[32951]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:12.329 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[43191]: 2024-06-14T22:30:12.189+0000 7f3cc8dd6640 -1 osd.0 0 waiting for initial osdmap 2024-06-14T22:30:12.330 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[43191]: 2024-06-14T22:30:12.191+0000 7f3cc5c11640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:30:12.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:12 smithi137 ceph-mon[32951]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:12.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:12 smithi137 ceph-mon[32951]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:30:12.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:12 smithi137 ceph-mon[32951]: osdmap e7: 1 total, 0 up, 1 in 2024-06-14T22:30:12.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:12.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:12.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[29889]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:12.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[29889]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:30:12.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[29889]: osdmap e7: 1 total, 0 up, 1 in 2024-06-14T22:30:12.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:12.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:12.739 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[38904]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-14T22:30:12.739 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[38904]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:30:12.739 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[38904]: osdmap e7: 1 total, 0 up, 1 in 2024-06-14T22:30:12.739 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:12.740 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:14.189 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[38904]: purged_snaps scrub starts 2024-06-14T22:30:14.245 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-14T22:30:14.348 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-14T22:30:14.363 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-06-14T22:30:14.363 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-06-14T22:30:14.363 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122221 s, 858 MB/s 2024-06-14T22:30:14.363 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-06-14T22:30:14.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:14 smithi137 ceph-mon[32951]: purged_snaps scrub starts 2024-06-14T22:30:14.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:14 smithi137 ceph-mon[32951]: purged_snaps scrub ok 2024-06-14T22:30:14.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:14 smithi137 ceph-mon[32951]: osd.0 [v2:172.21.15.86:6802/2092557056,v1:172.21.15.86:6803/2092557056] boot 2024-06-14T22:30:14.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:14 smithi137 ceph-mon[32951]: osdmap e8: 1 total, 1 up, 1 in 2024-06-14T22:30:14.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:14 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:14.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:14 smithi137 ceph-mon[32951]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:14.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[38904]: purged_snaps scrub ok 2024-06-14T22:30:14.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[38904]: osd.0 [v2:172.21.15.86:6802/2092557056,v1:172.21.15.86:6803/2092557056] boot 2024-06-14T22:30:14.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[38904]: osdmap e8: 1 total, 1 up, 1 in 2024-06-14T22:30:14.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:14.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[38904]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[29889]: purged_snaps scrub starts 2024-06-14T22:30:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[29889]: purged_snaps scrub ok 2024-06-14T22:30:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[29889]: osd.0 [v2:172.21.15.86:6802/2092557056,v1:172.21.15.86:6803/2092557056] boot 2024-06-14T22:30:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[29889]: osdmap e8: 1 total, 1 up, 1 in 2024-06-14T22:30:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:30:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:14 smithi086 ceph-mon[29889]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:15.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-06-14T22:30:15.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:15 smithi086 ceph-mon[29889]: osdmap e9: 1 total, 1 up, 1 in 2024-06-14T22:30:15.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:15 smithi086 ceph-mon[38904]: osdmap e9: 1 total, 1 up, 1 in 2024-06-14T22:30:15.438 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:30:15.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:15 smithi137 ceph-mon[32951]: osdmap e9: 1 total, 1 up, 1 in 2024-06-14T22:30:16.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:16 smithi137 ceph-mon[32951]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:16.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:16 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:16.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:16 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:16.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:16 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:30:16.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:16 smithi086 ceph-mon[29889]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:16.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:16 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:16.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:16 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:16.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:16 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:30:16.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:16 smithi086 ceph-mon[38904]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:16.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:16 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:16.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:16 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:16.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:16 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: Detected new or changed devices on smithi086 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi086 to 16064M 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:17.432 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: Detected new or changed devices on smithi086 2024-06-14T22:30:17.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi086 to 16064M 2024-06-14T22:30:17.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:17.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:17.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:17.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:17.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:17.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:17.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:17.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:17 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:17.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: Detected new or changed devices on smithi086 2024-06-14T22:30:17.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi086 to 16064M 2024-06-14T22:30:17.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:17.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:17.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:17.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:17.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:17.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:17.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:17.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:17 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:18.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[38904]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:18.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[38904]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:30:18.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:30:18.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:30:18.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:18.379 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[29889]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:18.379 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[29889]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:30:18.379 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:30:18.379 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:30:18.379 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:18 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:18.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:18 smithi137 ceph-mon[32951]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:18.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:18 smithi137 ceph-mon[32951]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:30:18.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:18 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:30:18.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:18 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:30:18.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:18 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:20.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[38904]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:20.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1846986505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f021e24a-e7b4-46f0-94ed-4db138b60d64"}]: dispatch 2024-06-14T22:30:20.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f021e24a-e7b4-46f0-94ed-4db138b60d64"}]: dispatch 2024-06-14T22:30:20.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f021e24a-e7b4-46f0-94ed-4db138b60d64"}]': finished 2024-06-14T22:30:20.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[38904]: osdmap e10: 2 total, 1 up, 2 in 2024-06-14T22:30:20.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:20.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[29889]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:20.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1846986505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f021e24a-e7b4-46f0-94ed-4db138b60d64"}]: dispatch 2024-06-14T22:30:20.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f021e24a-e7b4-46f0-94ed-4db138b60d64"}]: dispatch 2024-06-14T22:30:20.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f021e24a-e7b4-46f0-94ed-4db138b60d64"}]': finished 2024-06-14T22:30:20.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[29889]: osdmap e10: 2 total, 1 up, 2 in 2024-06-14T22:30:20.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:20 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:20.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:20 smithi137 ceph-mon[32951]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:20.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:20 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1846986505' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f021e24a-e7b4-46f0-94ed-4db138b60d64"}]: dispatch 2024-06-14T22:30:20.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:20 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f021e24a-e7b4-46f0-94ed-4db138b60d64"}]: dispatch 2024-06-14T22:30:20.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:20 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f021e24a-e7b4-46f0-94ed-4db138b60d64"}]': finished 2024-06-14T22:30:20.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:20 smithi137 ceph-mon[32951]: osdmap e10: 2 total, 1 up, 2 in 2024-06-14T22:30:20.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:20 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:21.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:21 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3598855751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:30:21.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:21 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3598855751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:30:21.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:21 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3598855751' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:30:22.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:22 smithi086 ceph-mon[29889]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:22.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:22 smithi086 ceph-mon[38904]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:22.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:22 smithi137 ceph-mon[32951]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:24.460 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:24 smithi086 ceph-mon[29889]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:24.461 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:24 smithi086 ceph-mon[38904]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:24.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:24 smithi137 ceph-mon[32951]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:26.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:26 smithi086 ceph-mon[29889]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:26.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:26 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-14T22:30:26.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:26 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:26.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:26 smithi086 ceph-mon[38904]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:26.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:26 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-14T22:30:26.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:26 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:26.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:26 smithi137 ceph-mon[32951]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:26.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:26 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-14T22:30:26.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:26 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:27.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:27 smithi086 ceph-mon[38904]: Deploying daemon osd.1 on smithi086 2024-06-14T22:30:27.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:27 smithi086 ceph-mon[29889]: Deploying daemon osd.1 on smithi086 2024-06-14T22:30:27.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:27 smithi137 ceph-mon[32951]: Deploying daemon osd.1 on smithi086 2024-06-14T22:30:28.466 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:28 smithi086 ceph-mon[29889]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:28.466 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:28 smithi086 ceph-mon[38904]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:28.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:28 smithi137 ceph-mon[32951]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:30.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:30 smithi086 ceph-mon[38904]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:30.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:30 smithi086 ceph-mon[29889]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:30.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:30 smithi137 ceph-mon[32951]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:31.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:31.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:31 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:31.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:31 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:31.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:31 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:31.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:31.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:31.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:32.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:32 smithi086 ceph-mon[29889]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:32.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:32 smithi086 ceph-mon[38904]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:32.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:32 smithi137 ceph-mon[32951]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:34.375 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:34 smithi086 ceph-mon[29889]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:34.376 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:34 smithi086 ceph-mon[29889]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-14T22:30:34.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:34 smithi086 ceph-mon[38904]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:34.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:34 smithi086 ceph-mon[38904]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-14T22:30:34.407 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 1 on host 'smithi086' 2024-06-14T22:30:34.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:34 smithi137 ceph-mon[32951]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:34.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:34 smithi137 ceph-mon[32951]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-14T22:30:34.990 DEBUG:teuthology.orchestra.run.smithi086:osd.1> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.1.service 2024-06-14T22:30:34.992 INFO:tasks.cephadm:Deploying osd.2 on smithi086 with /dev/vg_nvme/lv_2... 2024-06-14T22:30:34.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-14T22:30:35.293 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:30:35.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[29889]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-14T22:30:35.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[29889]: osdmap e11: 2 total, 1 up, 2 in 2024-06-14T22:30:35.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:35.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[29889]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:35.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[38904]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-14T22:30:35.569 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[38904]: osdmap e11: 2 total, 1 up, 2 in 2024-06-14T22:30:35.569 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:35.569 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[38904]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:35.569 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.569 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.569 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.569 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.569 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[49514]: 2024-06-14T22:30:35.344+0000 7fd6b3887640 -1 osd.1 0 waiting for initial osdmap 2024-06-14T22:30:35.569 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:30:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[49514]: 2024-06-14T22:30:35.351+0000 7fd6ae6be640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:30:35.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:35 smithi137 ceph-mon[32951]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-14T22:30:35.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:35 smithi137 ceph-mon[32951]: osdmap e11: 2 total, 1 up, 2 in 2024-06-14T22:30:35.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:35 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:35.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:35 smithi137 ceph-mon[32951]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:35.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:35 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:35 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:35 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:35.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:35 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:36.468 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[29889]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[29889]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[29889]: osdmap e12: 2 total, 1 up, 2 in 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[38904]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[38904]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[38904]: osdmap e12: 2 total, 1 up, 2 in 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:36.469 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:36.470 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:36 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:36.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:36 smithi137 ceph-mon[32951]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:36.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:36 smithi137 ceph-mon[32951]: from='osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:30:36.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:36 smithi137 ceph-mon[32951]: osdmap e12: 2 total, 1 up, 2 in 2024-06-14T22:30:36.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:36 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:36.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:36 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:36.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:36 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:37.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[38904]: purged_snaps scrub starts 2024-06-14T22:30:37.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[38904]: purged_snaps scrub ok 2024-06-14T22:30:37.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[38904]: osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430] boot 2024-06-14T22:30:37.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[38904]: osdmap e13: 2 total, 2 up, 2 in 2024-06-14T22:30:37.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:37.702 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[29889]: purged_snaps scrub starts 2024-06-14T22:30:37.702 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[29889]: purged_snaps scrub ok 2024-06-14T22:30:37.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[29889]: osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430] boot 2024-06-14T22:30:37.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[29889]: osdmap e13: 2 total, 2 up, 2 in 2024-06-14T22:30:37.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:37 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:37.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:37 smithi137 ceph-mon[32951]: purged_snaps scrub starts 2024-06-14T22:30:37.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:37 smithi137 ceph-mon[32951]: purged_snaps scrub ok 2024-06-14T22:30:37.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:37 smithi137 ceph-mon[32951]: osd.1 [v2:172.21.15.86:6810/796510430,v1:172.21.15.86:6811/796510430] boot 2024-06-14T22:30:37.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:37 smithi137 ceph-mon[32951]: osdmap e13: 2 total, 2 up, 2 in 2024-06-14T22:30:37.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:37 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:30:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-14T22:30:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-14T22:30:37.882 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-06-14T22:30:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-06-14T22:30:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140759 s, 745 MB/s 2024-06-14T22:30:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-06-14T22:30:38.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:38 smithi086 ceph-mon[29889]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:38.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:38 smithi086 ceph-mon[29889]: osdmap e14: 2 total, 2 up, 2 in 2024-06-14T22:30:38.548 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:38 smithi086 ceph-mon[38904]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:38.548 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:38 smithi086 ceph-mon[38904]: osdmap e14: 2 total, 2 up, 2 in 2024-06-14T22:30:38.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-06-14T22:30:38.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:38 smithi137 ceph-mon[32951]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-14T22:30:38.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:38 smithi137 ceph-mon[32951]: osdmap e14: 2 total, 2 up, 2 in 2024-06-14T22:30:39.062 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:30:40.392 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:40 smithi086 ceph-mon[29889]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:40.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:40 smithi086 ceph-mon[38904]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:40.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:40 smithi137 ceph-mon[32951]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:41.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: Detected new or changed devices on smithi086 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi086 to 8032M 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:41 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: Detected new or changed devices on smithi086 2024-06-14T22:30:41.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi086 to 8032M 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: Detected new or changed devices on smithi086 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi086 to 8032M 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:30:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:41 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:42.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:42 smithi137 ceph-mon[32951]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:42.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:42 smithi137 ceph-mon[32951]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:30:42.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:42 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:30:42.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:42 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:30:42.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:42 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:42.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[29889]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[29889]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:30:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:30:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:30:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:42.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[38904]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:42.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[38904]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:30:42.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:30:42.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:30:42.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:42 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:44.545 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[38904]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:44.545 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/860610687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65dbd353-d1a7-4f6d-8112-b77fbc51f9ce"}]: dispatch 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65dbd353-d1a7-4f6d-8112-b77fbc51f9ce"}]: dispatch 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65dbd353-d1a7-4f6d-8112-b77fbc51f9ce"}]': finished 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[38904]: osdmap e15: 3 total, 2 up, 3 in 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[29889]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/860610687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65dbd353-d1a7-4f6d-8112-b77fbc51f9ce"}]: dispatch 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65dbd353-d1a7-4f6d-8112-b77fbc51f9ce"}]: dispatch 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65dbd353-d1a7-4f6d-8112-b77fbc51f9ce"}]': finished 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[29889]: osdmap e15: 3 total, 2 up, 3 in 2024-06-14T22:30:44.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:30:44.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:44 smithi137 ceph-mon[32951]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:44.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:44 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/860610687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65dbd353-d1a7-4f6d-8112-b77fbc51f9ce"}]: dispatch 2024-06-14T22:30:44.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:44 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65dbd353-d1a7-4f6d-8112-b77fbc51f9ce"}]: dispatch 2024-06-14T22:30:44.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:44 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65dbd353-d1a7-4f6d-8112-b77fbc51f9ce"}]': finished 2024-06-14T22:30:44.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:44 smithi137 ceph-mon[32951]: osdmap e15: 3 total, 2 up, 3 in 2024-06-14T22:30:44.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:30:45.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:45 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3686208887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:30:45.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:45 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3686208887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:30:45.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:45 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3686208887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:30:46.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:46 smithi086 ceph-mon[29889]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:46.721 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:46 smithi086 ceph-mon[38904]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:46.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:46 smithi137 ceph-mon[32951]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:48.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:48 smithi137 ceph-mon[32951]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:48.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:48 smithi086 ceph-mon[29889]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:48.808 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:48 smithi086 ceph-mon[38904]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:50.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:50 smithi137 ceph-mon[32951]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:50.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:50 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-14T22:30:50.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:50 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:50.766 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:50 smithi086 ceph-mon[38904]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:50.767 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:50 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-14T22:30:50.767 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:50 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:50.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:50 smithi086 ceph-mon[29889]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:50.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:50 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-14T22:30:50.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:50 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:30:51.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:51 smithi137 ceph-mon[32951]: Deploying daemon osd.2 on smithi086 2024-06-14T22:30:51.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:51 smithi086 ceph-mon[29889]: Deploying daemon osd.2 on smithi086 2024-06-14T22:30:51.767 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:51 smithi086 ceph-mon[38904]: Deploying daemon osd.2 on smithi086 2024-06-14T22:30:52.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:52 smithi086 ceph-mon[29889]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:52.645 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:52 smithi086 ceph-mon[38904]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:52.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:52 smithi137 ceph-mon[32951]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:54.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:54 smithi086 ceph-mon[29889]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:54.545 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:54 smithi086 ceph-mon[38904]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:54.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:54 smithi137 ceph-mon[32951]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:55.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:55 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:55.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:55 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:55.640 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:55 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:55.640 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:55 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:55.640 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:55 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:55.640 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:55 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:55.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:55 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:30:55.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:55 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:55.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:55 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:56.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:56 smithi086 ceph-mon[29889]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:56.646 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:56 smithi086 ceph-mon[38904]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:56.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:56 smithi137 ceph-mon[32951]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:58.421 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 2 on host 'smithi086' 2024-06-14T22:30:58.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:58 smithi086 ceph-mon[38904]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:58.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:58 smithi086 ceph-mon[38904]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T22:30:58.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:58 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:58.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:58 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:58.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:58 smithi086 ceph-mon[29889]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:58.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:58 smithi086 ceph-mon[29889]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T22:30:58.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:58 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:58.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:58 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:58.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:58 smithi137 ceph-mon[32951]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:30:58.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:58 smithi137 ceph-mon[32951]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T22:30:58.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:58 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:58.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:58 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:59.019 DEBUG:teuthology.orchestra.run.smithi086:osd.2> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.2.service 2024-06-14T22:30:59.022 INFO:tasks.cephadm:Deploying osd.3 on smithi086 with /dev/vg_nvme/lv_1... 2024-06-14T22:30:59.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-14T22:30:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:30:59.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:59 smithi137 ceph-mon[32951]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-14T22:30:59.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:59 smithi137 ceph-mon[32951]: osdmap e16: 3 total, 2 up, 3 in 2024-06-14T22:30:59.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:59 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:30:59.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:59 smithi137 ceph-mon[32951]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:59.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:59 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:59.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:59 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:59.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:59 smithi137 ceph-mon[32951]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:30:59.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:59 smithi137 ceph-mon[32951]: osdmap e17: 3 total, 2 up, 3 in 2024-06-14T22:30:59.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:30:59 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:30:59.764 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[55497]: 2024-06-14T22:30:59.462+0000 7f5995b51640 -1 osd.2 0 waiting for initial osdmap 2024-06-14T22:30:59.764 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[55497]: 2024-06-14T22:30:59.466+0000 7f5990988640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:30:59.764 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[29889]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-14T22:30:59.764 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[29889]: osdmap e16: 3 total, 2 up, 3 in 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[29889]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[29889]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[29889]: osdmap e17: 3 total, 2 up, 3 in 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[38904]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[38904]: osdmap e16: 3 total, 2 up, 3 in 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:30:59.765 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[38904]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:30:59.766 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:59.766 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:30:59.766 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[38904]: from='osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:30:59.766 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[38904]: osdmap e17: 3 total, 2 up, 3 in 2024-06-14T22:30:59.766 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:30:59 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:00.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:00 smithi137 ceph-mon[32951]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:31:00.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:00 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:00.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:00 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:00.744 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:00 smithi086 ceph-mon[29889]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:31:00.744 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:00 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:00.744 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:00 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:00.744 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:00 smithi086 ceph-mon[38904]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:31:00.744 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:00 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:00.745 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:00 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:01.660 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[29889]: purged_snaps scrub starts 2024-06-14T22:31:01.660 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[29889]: purged_snaps scrub ok 2024-06-14T22:31:01.660 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[29889]: osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667] boot 2024-06-14T22:31:01.660 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[29889]: osdmap e18: 3 total, 3 up, 3 in 2024-06-14T22:31:01.660 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:01.660 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[38904]: purged_snaps scrub starts 2024-06-14T22:31:01.660 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[38904]: purged_snaps scrub ok 2024-06-14T22:31:01.660 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[38904]: osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667] boot 2024-06-14T22:31:01.661 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[38904]: osdmap e18: 3 total, 3 up, 3 in 2024-06-14T22:31:01.661 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:01.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:01 smithi137 ceph-mon[32951]: purged_snaps scrub starts 2024-06-14T22:31:01.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:01 smithi137 ceph-mon[32951]: purged_snaps scrub ok 2024-06-14T22:31:01.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:01 smithi137 ceph-mon[32951]: osd.2 [v2:172.21.15.86:6818/3531892667,v1:172.21.15.86:6819/3531892667] boot 2024-06-14T22:31:01.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:01 smithi137 ceph-mon[32951]: osdmap e18: 3 total, 3 up, 3 in 2024-06-14T22:31:01.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:31:02.802 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-14T22:31:02.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[29889]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:31:02.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-14T22:31:02.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-14T22:31:02.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[29889]: osdmap e19: 3 total, 3 up, 3 in 2024-06-14T22:31:02.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:31:02.803 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[38904]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:31:02.803 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-14T22:31:02.803 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-14T22:31:02.804 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[38904]: osdmap e19: 3 total, 3 up, 3 in 2024-06-14T22:31:02.804 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:02 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:31:02.889 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:02 smithi137 ceph-mon[32951]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-14T22:31:02.889 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:02 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-14T22:31:02.889 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:02 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-14T22:31:02.889 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:02 smithi137 ceph-mon[32951]: osdmap e19: 3 total, 3 up, 3 in 2024-06-14T22:31:02.890 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:02 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:31:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-14T22:31:02.920 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-06-14T22:31:02.920 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-06-14T22:31:02.920 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116172 s, 903 MB/s 2024-06-14T22:31:02.920 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-06-14T22:31:03.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:03 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:31:03.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:03 smithi086 ceph-mon[29889]: osdmap e20: 3 total, 3 up, 3 in 2024-06-14T22:31:03.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:03 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:31:03.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:03 smithi086 ceph-mon[38904]: osdmap e20: 3 total, 3 up, 3 in 2024-06-14T22:31:03.830 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:31:03 smithi086 sudo[59494]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-14T22:31:03.830 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:31:03 smithi086 sudo[59494]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:03.830 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:31:03 smithi086 sudo[59494]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:03.831 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:31:03 smithi086 sudo[59494]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:03.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_1 2024-06-14T22:31:03.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:03 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:31:03.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:03 smithi137 ceph-mon[32951]: osdmap e20: 3 total, 3 up, 3 in 2024-06-14T22:31:04.152 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:31:03 smithi086 sudo[59497]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-14T22:31:04.152 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:31:03 smithi086 sudo[59497]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:04.152 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:31:03 smithi086 sudo[59497]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:04.152 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:31:03 smithi086 sudo[59497]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:04.414 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59538]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-14T22:31:04.414 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59538]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:04.414 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59538]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:04.414 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59538]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:04.514 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:31:04.676 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59543]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-14T22:31:04.676 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59543]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:04.676 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59543]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:04.676 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59543]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: osdmap e21: 3 total, 3 up, 3 in 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:04.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: osdmap e21: 3 total, 3 up, 3 in 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.678 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:04.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:04 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.930 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59650]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-14T22:31:04.930 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59650]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:04.930 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59650]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:04.930 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59650]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:04.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:04.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: osdmap e21: 3 total, 3 up, 3 in 2024-06-14T22:31:04.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:04 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:05.188 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59681]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-14T22:31:05.188 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59681]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:05.188 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59681]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:05.188 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:31:04 smithi086 sudo[59681]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:05.447 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 sudo[59694]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-14T22:31:05.447 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 sudo[59694]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:05.447 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 sudo[59694]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:05.447 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 sudo[59694]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:05.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 sudo[59703]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-14T22:31:05.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 sudo[59703]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:05.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 sudo[59703]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:05.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 sudo[59703]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:05.947 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 sudo[35680]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-14T22:31:05.947 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 sudo[35680]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:05.947 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 sudo[35680]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:05.947 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 sudo[35680]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:06.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: Detected new or changed devices on smithi086 2024-06-14T22:31:06.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi086 to 5354M 2024-06-14T22:31:06.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-14T22:31:06.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-14T22:31:06.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: Detected new or changed devices on smithi086 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi086 to 5354M 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:31:06.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:31:06.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:31:06.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:31:06.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:05 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:31:06.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 sudo[35683]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-14T22:31:06.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 sudo[35683]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:06.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 sudo[35683]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:06.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 sudo[35683]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:06.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: Detected new or changed devices on smithi086 2024-06-14T22:31:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi086 to 5354M 2024-06-14T22:31:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-14T22:31:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-14T22:31:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:31:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:31:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:31:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:31:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:31:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:05 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:31:06.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 sudo[59775]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-14T22:31:06.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 sudo[59775]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:06.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 sudo[59775]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:06.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 sudo[59775]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:06.761 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 sudo[59845]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-14T22:31:06.761 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 sudo[59845]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-14T22:31:06.761 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 sudo[59845]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-14T22:31:06.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 sudo[59845]: pam_unix(sudo:session): session closed for user root 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:31:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[29889]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:31:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:31:07.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:31:07.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:31:07.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:31:07.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:07.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:06 smithi086 ceph-mon[38904]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-14T22:31:07.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:07.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-14T22:31:07.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:06 smithi137 ceph-mon[32951]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-14T22:31:08.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:07 smithi086 ceph-mon[29889]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:08.031 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:07 smithi086 ceph-mon[38904]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:08.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:07 smithi137 ceph-mon[32951]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:09.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:08 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/733507799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e141877-8275-4b3a-bb90-277e99a725d8"}]: dispatch 2024-06-14T22:31:09.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:08 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e141877-8275-4b3a-bb90-277e99a725d8"}]: dispatch 2024-06-14T22:31:09.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:08 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e141877-8275-4b3a-bb90-277e99a725d8"}]': finished 2024-06-14T22:31:09.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:08 smithi137 ceph-mon[32951]: osdmap e22: 4 total, 3 up, 4 in 2024-06-14T22:31:09.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:09.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/733507799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e141877-8275-4b3a-bb90-277e99a725d8"}]: dispatch 2024-06-14T22:31:09.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e141877-8275-4b3a-bb90-277e99a725d8"}]: dispatch 2024-06-14T22:31:09.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e141877-8275-4b3a-bb90-277e99a725d8"}]': finished 2024-06-14T22:31:09.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[38904]: osdmap e22: 4 total, 3 up, 4 in 2024-06-14T22:31:09.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/733507799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e141877-8275-4b3a-bb90-277e99a725d8"}]: dispatch 2024-06-14T22:31:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e141877-8275-4b3a-bb90-277e99a725d8"}]: dispatch 2024-06-14T22:31:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e141877-8275-4b3a-bb90-277e99a725d8"}]': finished 2024-06-14T22:31:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[29889]: osdmap e22: 4 total, 3 up, 4 in 2024-06-14T22:31:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:10.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:09 smithi137 ceph-mon[32951]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:10.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:09 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3484250654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:31:10.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:09 smithi086 ceph-mon[29889]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:10.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:09 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3484250654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:31:10.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:09 smithi086 ceph-mon[38904]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:10.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:09 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3484250654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:31:12.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:12 smithi086 ceph-mon[29889]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:12.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:12 smithi086 ceph-mon[38904]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:12.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:12 smithi137 ceph-mon[32951]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:14.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:14 smithi086 ceph-mon[29889]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:14.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:14 smithi086 ceph-mon[38904]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:14.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:14 smithi137 ceph-mon[32951]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:15.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:15 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-14T22:31:15.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:15 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:15.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:15 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-14T22:31:15.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:15 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:15.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:15 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-14T22:31:15.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:15 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:16.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:16 smithi086 ceph-mon[29889]: Deploying daemon osd.3 on smithi086 2024-06-14T22:31:16.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:16 smithi086 ceph-mon[29889]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:16.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:16 smithi086 ceph-mon[38904]: Deploying daemon osd.3 on smithi086 2024-06-14T22:31:16.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:16 smithi086 ceph-mon[38904]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:16.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:16 smithi137 ceph-mon[32951]: Deploying daemon osd.3 on smithi086 2024-06-14T22:31:16.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:16 smithi137 ceph-mon[32951]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:18.458 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:18 smithi086 ceph-mon[29889]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:18.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:18 smithi086 ceph-mon[38904]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:18.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:18 smithi137 ceph-mon[32951]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:20.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:20 smithi086 ceph-mon[38904]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:20.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:20 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:20.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:20 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:20.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:20 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:20.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:20 smithi086 ceph-mon[29889]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:20.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:20 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:20.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:20 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:20.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:20 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:20.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:20 smithi137 ceph-mon[32951]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:20.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:20 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:20.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:20 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:20.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:20 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:22.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:22 smithi086 ceph-mon[29889]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:22.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:22 smithi086 ceph-mon[38904]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:22.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:22 smithi137 ceph-mon[32951]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:23.420 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 3 on host 'smithi086' 2024-06-14T22:31:23.720 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:23 smithi086 ceph-mon[29889]: from='osd.3 [v2:172.21.15.86:6826/287823501,v1:172.21.15.86:6827/287823501]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:31:23.720 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:23 smithi086 ceph-mon[29889]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:31:23.721 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:23 smithi086 ceph-mon[38904]: from='osd.3 [v2:172.21.15.86:6826/287823501,v1:172.21.15.86:6827/287823501]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:31:23.721 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:23 smithi086 ceph-mon[38904]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:31:23.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:23 smithi137 ceph-mon[32951]: from='osd.3 [v2:172.21.15.86:6826/287823501,v1:172.21.15.86:6827/287823501]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:31:23.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:23 smithi137 ceph-mon[32951]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:31:23.985 DEBUG:teuthology.orchestra.run.smithi086:osd.3> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.3.service 2024-06-14T22:31:23.987 INFO:tasks.cephadm:Deploying osd.4 on smithi137 with /dev/vg_nvme/lv_4... 2024-06-14T22:31:23.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-14T22:31:24.244 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:31:24.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: osdmap e23: 4 total, 3 up, 4 in 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: from='osd.3 [v2:172.21.15.86:6826/287823501,v1:172.21.15.86:6827/287823501]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-14T22:31:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: osdmap e23: 4 total, 3 up, 4 in 2024-06-14T22:31:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: from='osd.3 [v2:172.21.15.86:6826/287823501,v1:172.21.15.86:6827/287823501]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:31:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:31:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.581 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[61825]: 2024-06-14T22:31:24.349+0000 7f49dbefb640 -1 osd.3 0 waiting for initial osdmap 2024-06-14T22:31:24.582 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:31:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[61825]: 2024-06-14T22:31:24.353+0000 7f49d6d32640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:31:24.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:24.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-14T22:31:24.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: osdmap e23: 4 total, 3 up, 4 in 2024-06-14T22:31:24.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:24.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: from='osd.3 [v2:172.21.15.86:6826/287823501,v1:172.21.15.86:6827/287823501]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:31:24.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:31:24.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:24.640 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:24 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:25.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:25 smithi086 ceph-mon[38904]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:31:25.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:25 smithi086 ceph-mon[38904]: osdmap e24: 4 total, 3 up, 4 in 2024-06-14T22:31:25.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:25 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:25.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:25 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:25.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:25 smithi086 ceph-mon[29889]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:31:25.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:25 smithi086 ceph-mon[29889]: osdmap e24: 4 total, 3 up, 4 in 2024-06-14T22:31:25.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:25 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:25.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:25 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:25.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:25 smithi137 ceph-mon[32951]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-06-14T22:31:25.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:25 smithi137 ceph-mon[32951]: osdmap e24: 4 total, 3 up, 4 in 2024-06-14T22:31:25.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:25 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:25.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:25 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:25.973 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-14T22:31:26.073 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-14T22:31:26.087 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-06-14T22:31:26.087 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-06-14T22:31:26.087 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114864 s, 913 MB/s 2024-06-14T22:31:26.087 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-06-14T22:31:26.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[38904]: purged_snaps scrub starts 2024-06-14T22:31:26.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[38904]: purged_snaps scrub ok 2024-06-14T22:31:26.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[38904]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:26.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[38904]: osd.3 [v2:172.21.15.86:6826/287823501,v1:172.21.15.86:6827/287823501] boot 2024-06-14T22:31:26.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[38904]: osdmap e25: 4 total, 4 up, 4 in 2024-06-14T22:31:26.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[29889]: purged_snaps scrub starts 2024-06-14T22:31:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[29889]: purged_snaps scrub ok 2024-06-14T22:31:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[29889]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[29889]: osd.3 [v2:172.21.15.86:6826/287823501,v1:172.21.15.86:6827/287823501] boot 2024-06-14T22:31:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[29889]: osdmap e25: 4 total, 4 up, 4 in 2024-06-14T22:31:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:26 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:26.722 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:26 smithi137 ceph-mon[32951]: purged_snaps scrub starts 2024-06-14T22:31:26.722 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:26 smithi137 ceph-mon[32951]: purged_snaps scrub ok 2024-06-14T22:31:26.723 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:26 smithi137 ceph-mon[32951]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-14T22:31:26.723 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:26 smithi137 ceph-mon[32951]: osd.3 [v2:172.21.15.86:6826/287823501,v1:172.21.15.86:6827/287823501] boot 2024-06-14T22:31:26.723 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:26 smithi137 ceph-mon[32951]: osdmap e25: 4 total, 4 up, 4 in 2024-06-14T22:31:26.723 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:26 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:31:26.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi137:vg_nvme/lv_4 2024-06-14T22:31:26.993 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:31:27.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:27 smithi086 ceph-mon[38904]: osdmap e26: 4 total, 4 up, 4 in 2024-06-14T22:31:27.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:27 smithi086 ceph-mon[29889]: osdmap e26: 4 total, 4 up, 4 in 2024-06-14T22:31:27.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:27 smithi137 ceph-mon[32951]: osdmap e26: 4 total, 4 up, 4 in 2024-06-14T22:31:28.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:28.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:28.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:28.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:28.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:28.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:28.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:28 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:28.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:28.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:28.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:28.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:28.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:28 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:29.737 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:29 smithi137 ceph-mon[32951]: Detected new or changed devices on smithi086 2024-06-14T22:31:29.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:29 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi086 to 4016M 2024-06-14T22:31:29.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:29 smithi137 ceph-mon[32951]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:31:29.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:29 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:31:29.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:29 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:31:29.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:29 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:29.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[29889]: Detected new or changed devices on smithi086 2024-06-14T22:31:29.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi086 to 4016M 2024-06-14T22:31:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[29889]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:31:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:31:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:31:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[38904]: Detected new or changed devices on smithi086 2024-06-14T22:31:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi086 to 4016M 2024-06-14T22:31:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[38904]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:31:29.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:31:29.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:31:29.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:29 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:30.680 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:30 smithi137 ceph-mon[32951]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:30.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:30 smithi086 ceph-mon[29889]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:30.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:30 smithi086 ceph-mon[38904]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:31 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/1776176382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd061-11f3-4d2c-9395-65c3ae5489f5"}]: dispatch 2024-06-14T22:31:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:31 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd061-11f3-4d2c-9395-65c3ae5489f5"}]: dispatch 2024-06-14T22:31:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:31 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f0bd061-11f3-4d2c-9395-65c3ae5489f5"}]': finished 2024-06-14T22:31:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:31 smithi137 ceph-mon[32951]: osdmap e27: 5 total, 4 up, 5 in 2024-06-14T22:31:31.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:31 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:31.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/1776176382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd061-11f3-4d2c-9395-65c3ae5489f5"}]: dispatch 2024-06-14T22:31:31.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd061-11f3-4d2c-9395-65c3ae5489f5"}]: dispatch 2024-06-14T22:31:31.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f0bd061-11f3-4d2c-9395-65c3ae5489f5"}]': finished 2024-06-14T22:31:31.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[29889]: osdmap e27: 5 total, 4 up, 5 in 2024-06-14T22:31:31.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:31.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/1776176382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd061-11f3-4d2c-9395-65c3ae5489f5"}]: dispatch 2024-06-14T22:31:31.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd061-11f3-4d2c-9395-65c3ae5489f5"}]: dispatch 2024-06-14T22:31:31.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f0bd061-11f3-4d2c-9395-65c3ae5489f5"}]': finished 2024-06-14T22:31:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[38904]: osdmap e27: 5 total, 4 up, 5 in 2024-06-14T22:31:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:31 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:32.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:32 smithi137 ceph-mon[32951]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:32.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:32 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/3457388830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:31:32.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:32 smithi086 ceph-mon[29889]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:32.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:32 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/3457388830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:31:32.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:32 smithi086 ceph-mon[38904]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:32.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:32 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/3457388830' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:31:34.718 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:34 smithi137 ceph-mon[32951]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:34.729 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:34 smithi086 ceph-mon[29889]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:34.729 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:34 smithi086 ceph-mon[38904]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:36.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:36 smithi086 ceph-mon[29889]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:36.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:36 smithi086 ceph-mon[38904]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:36.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:36 smithi137 ceph-mon[32951]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:37.636 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:37 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-14T22:31:37.636 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:37 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:37.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:37 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-14T22:31:37.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:37 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:37.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:37 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-14T22:31:37.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:37 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:38.574 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:38 smithi137 ceph-mon[32951]: Deploying daemon osd.4 on smithi137 2024-06-14T22:31:38.575 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:38 smithi137 ceph-mon[32951]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:38.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:38 smithi086 ceph-mon[29889]: Deploying daemon osd.4 on smithi137 2024-06-14T22:31:38.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:38 smithi086 ceph-mon[29889]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:38.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:38 smithi086 ceph-mon[38904]: Deploying daemon osd.4 on smithi137 2024-06-14T22:31:38.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:38 smithi086 ceph-mon[38904]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:40.642 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:40 smithi137 ceph-mon[32951]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:40.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:40 smithi086 ceph-mon[29889]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:40.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:40 smithi086 ceph-mon[38904]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:42.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:42 smithi137 ceph-mon[32951]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:42.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:42 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:42.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:42 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:42.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:42 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:42 smithi086 ceph-mon[29889]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:42 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:42 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:42 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:42.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:42 smithi086 ceph-mon[38904]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:42.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:42 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:42.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:42 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:42 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:44.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:44 smithi137 ceph-mon[32951]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:44.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:44.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:44 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:44.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[29889]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:44.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:44.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:44.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:44.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:44.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:44.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[38904]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:44.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:44.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:44.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:44.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:44.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:44 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:45.073 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 4 on host 'smithi137' 2024-06-14T22:31:45.649 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:45 smithi137 ceph-mon[32951]: from='osd.4 [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-14T22:31:45.649 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:45 smithi137 ceph-mon[32951]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-14T22:31:45.649 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:45 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:45.649 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:45 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:45.650 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:45 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:45.677 DEBUG:teuthology.orchestra.run.smithi137:osd.4> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.4.service 2024-06-14T22:31:45.679 INFO:tasks.cephadm:Deploying osd.5 on smithi137 with /dev/vg_nvme/lv_3... 2024-06-14T22:31:45.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-14T22:31:45.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[29889]: from='osd.4 [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-14T22:31:45.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[29889]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-14T22:31:45.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:45.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:45.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:45.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[38904]: from='osd.4 [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-14T22:31:45.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[38904]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-14T22:31:45.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:31:45.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:45.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:45 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:46.352 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:31:46.715 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:46 smithi137 ceph-mon[32951]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:46.715 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:46 smithi137 ceph-mon[32951]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-14T22:31:46.715 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:46 smithi137 ceph-mon[32951]: osdmap e28: 5 total, 4 up, 5 in 2024-06-14T22:31:46.715 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:46 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:46.716 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:46 smithi137 ceph-mon[32951]: from='osd.4 [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:31:46.716 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:46 smithi137 ceph-mon[32951]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:31:46.716 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 22:31:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T22:31:46.418+0000 7fedda142640 -1 osd.4 0 waiting for initial osdmap 2024-06-14T22:31:46.716 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 22:31:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T22:31:46.421+0000 7fedd3776640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:31:46.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[29889]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:46.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[29889]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[29889]: osdmap e28: 5 total, 4 up, 5 in 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[29889]: from='osd.4 [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[29889]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[38904]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[38904]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[38904]: osdmap e28: 5 total, 4 up, 5 in 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[38904]: from='osd.4 [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:31:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:46 smithi086 ceph-mon[38904]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:31:47.517 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:47 smithi137 ceph-mon[32951]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:31:47.517 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:47 smithi137 ceph-mon[32951]: osdmap e29: 5 total, 4 up, 5 in 2024-06-14T22:31:47.517 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:47 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:47.517 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:47 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:47.517 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:47 smithi137 ceph-mon[32951]: osd.4 [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531] boot 2024-06-14T22:31:47.517 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:47 smithi137 ceph-mon[32951]: osdmap e30: 5 total, 5 up, 5 in 2024-06-14T22:31:47.517 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:47 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[29889]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[29889]: osdmap e29: 5 total, 4 up, 5 in 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[29889]: osd.4 [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531] boot 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[29889]: osdmap e30: 5 total, 5 up, 5 in 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[38904]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[38904]: osdmap e29: 5 total, 4 up, 5 in 2024-06-14T22:31:47.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:47.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:47.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[38904]: osd.4 [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531] boot 2024-06-14T22:31:47.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[38904]: osdmap e30: 5 total, 5 up, 5 in 2024-06-14T22:31:47.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:47 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:31:48.893 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:48 smithi137 ceph-mon[32951]: purged_snaps scrub starts 2024-06-14T22:31:48.893 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:48 smithi137 ceph-mon[32951]: purged_snaps scrub ok 2024-06-14T22:31:48.893 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:48 smithi137 ceph-mon[32951]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:48.893 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:48 smithi137 ceph-mon[32951]: osdmap e31: 5 total, 5 up, 5 in 2024-06-14T22:31:49.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:48 smithi086 ceph-mon[29889]: purged_snaps scrub starts 2024-06-14T22:31:49.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:48 smithi086 ceph-mon[29889]: purged_snaps scrub ok 2024-06-14T22:31:49.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:48 smithi086 ceph-mon[29889]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:49.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:48 smithi086 ceph-mon[29889]: osdmap e31: 5 total, 5 up, 5 in 2024-06-14T22:31:49.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:48 smithi086 ceph-mon[38904]: purged_snaps scrub starts 2024-06-14T22:31:49.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:48 smithi086 ceph-mon[38904]: purged_snaps scrub ok 2024-06-14T22:31:49.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:48 smithi086 ceph-mon[38904]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-14T22:31:49.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:48 smithi086 ceph-mon[38904]: osdmap e31: 5 total, 5 up, 5 in 2024-06-14T22:31:49.552 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-14T22:31:49.640 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-14T22:31:49.654 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-06-14T22:31:49.654 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-06-14T22:31:49.654 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113965 s, 920 MB/s 2024-06-14T22:31:49.654 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-06-14T22:31:49.965 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:49 smithi137 ceph-mon[32951]: osdmap e32: 5 total, 5 up, 5 in 2024-06-14T22:31:49.965 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:49 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:49.965 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:49 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:49.965 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:49 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:49.965 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:49 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:49.966 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:49 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:49.966 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:49 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:49.966 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:49 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:50.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[29889]: osdmap e32: 5 total, 5 up, 5 in 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[38904]: osdmap e32: 5 total, 5 up, 5 in 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:31:50.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:50.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:50.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:31:50.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:49 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:31:50.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi137:vg_nvme/lv_3 2024-06-14T22:31:50.487 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:31:50.858 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:50 smithi137 ceph-mon[32951]: Detected new or changed devices on smithi137 2024-06-14T22:31:50.858 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:50 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi137 to 17088M 2024-06-14T22:31:50.858 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:50 smithi137 ceph-mon[32951]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:31:50.859 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:50 smithi137 ceph-mon[32951]: osdmap e33: 5 total, 5 up, 5 in 2024-06-14T22:31:50.859 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:50 smithi137 ceph-mon[32951]: osdmap e34: 5 total, 5 up, 5 in 2024-06-14T22:31:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[29889]: Detected new or changed devices on smithi137 2024-06-14T22:31:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi137 to 17088M 2024-06-14T22:31:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[29889]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:31:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[29889]: osdmap e33: 5 total, 5 up, 5 in 2024-06-14T22:31:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[29889]: osdmap e34: 5 total, 5 up, 5 in 2024-06-14T22:31:51.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[38904]: Detected new or changed devices on smithi137 2024-06-14T22:31:51.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi137 to 17088M 2024-06-14T22:31:51.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[38904]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:31:51.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[38904]: osdmap e33: 5 total, 5 up, 5 in 2024-06-14T22:31:51.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:50 smithi086 ceph-mon[38904]: osdmap e34: 5 total, 5 up, 5 in 2024-06-14T22:31:52.817 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:52 smithi137 ceph-mon[32951]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:31:52.818 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:31:52.818 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:31:52.818 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:52 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:53.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:52 smithi086 ceph-mon[29889]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:31:53.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:31:53.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:31:53.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:52 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:53.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:52 smithi086 ceph-mon[38904]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:31:53.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:31:53.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:31:53.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:52 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:31:53.933 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:53 smithi137 ceph-mon[32951]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:31:53.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:53 smithi086 ceph-mon[29889]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:31:53.981 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:53 smithi086 ceph-mon[38904]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:31:54.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:54 smithi137 ceph-mon[32951]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-06-14T22:31:54.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:54 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/3202694008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c27edac8-f0e3-4b4c-a1e6-9e57165aea80"}]: dispatch 2024-06-14T22:31:54.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:54 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c27edac8-f0e3-4b4c-a1e6-9e57165aea80"}]: dispatch 2024-06-14T22:31:54.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:54 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c27edac8-f0e3-4b4c-a1e6-9e57165aea80"}]': finished 2024-06-14T22:31:54.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:54 smithi137 ceph-mon[32951]: osdmap e35: 6 total, 5 up, 6 in 2024-06-14T22:31:54.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:54 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:31:55.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[29889]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/3202694008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c27edac8-f0e3-4b4c-a1e6-9e57165aea80"}]: dispatch 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c27edac8-f0e3-4b4c-a1e6-9e57165aea80"}]: dispatch 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c27edac8-f0e3-4b4c-a1e6-9e57165aea80"}]': finished 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[29889]: osdmap e35: 6 total, 5 up, 6 in 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[38904]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/3202694008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c27edac8-f0e3-4b4c-a1e6-9e57165aea80"}]: dispatch 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c27edac8-f0e3-4b4c-a1e6-9e57165aea80"}]: dispatch 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c27edac8-f0e3-4b4c-a1e6-9e57165aea80"}]': finished 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[38904]: osdmap e35: 6 total, 5 up, 6 in 2024-06-14T22:31:55.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:54 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:31:55.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:55 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/893534246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:31:56.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:55 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/893534246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:31:56.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:55 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/893534246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:31:56.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:56 smithi137 ceph-mon[32951]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T22:31:57.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:56 smithi086 ceph-mon[29889]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T22:31:57.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:56 smithi086 ceph-mon[38904]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T22:31:58.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:31:58 smithi137 ceph-mon[32951]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-06-14T22:31:59.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:31:58 smithi086 ceph-mon[29889]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-06-14T22:31:59.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:31:58 smithi086 ceph-mon[38904]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-06-14T22:32:00.970 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:00 smithi137 ceph-mon[32951]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-06-14T22:32:00.970 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:00 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-14T22:32:00.970 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:00 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:01.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:00 smithi086 ceph-mon[29889]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-06-14T22:32:01.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:00 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-14T22:32:01.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:00 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:01.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:00 smithi086 ceph-mon[38904]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-06-14T22:32:01.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:00 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-14T22:32:01.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:00 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:01.816 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:01 smithi137 ceph-mon[32951]: Deploying daemon osd.5 on smithi137 2024-06-14T22:32:02.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:01 smithi086 ceph-mon[29889]: Deploying daemon osd.5 on smithi137 2024-06-14T22:32:02.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:01 smithi086 ceph-mon[38904]: Deploying daemon osd.5 on smithi137 2024-06-14T22:32:02.945 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:02 smithi137 ceph-mon[32951]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T22:32:03.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:02 smithi086 ceph-mon[29889]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T22:32:03.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:02 smithi086 ceph-mon[38904]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T22:32:04.869 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:04 smithi137 ceph-mon[32951]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:05.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:04 smithi086 ceph-mon[29889]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:05.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:04 smithi086 ceph-mon[38904]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:05.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:05 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:05.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:05 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:05.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:05 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:06.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:05 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:06.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:05 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:06.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:05 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:06.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:05 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:06.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:05 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:06.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:05 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:06.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:06 smithi137 ceph-mon[32951]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:06 smithi086 ceph-mon[29889]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:07.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:06 smithi086 ceph-mon[38904]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:08.712 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:08 smithi137 ceph-mon[32951]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:08.713 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:08.713 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:08.713 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:08.713 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:08.713 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:08.713 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:08 smithi137 ceph-mon[32951]: from='osd.5 [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T22:32:08.713 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:08 smithi137 ceph-mon[32951]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T22:32:08.748 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 5 on host 'smithi137' 2024-06-14T22:32:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[29889]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[29889]: from='osd.5 [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T22:32:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[29889]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T22:32:09.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[38904]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:09.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:09.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:09.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:09.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:09.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:09.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[38904]: from='osd.5 [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T22:32:09.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:08 smithi086 ceph-mon[38904]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T22:32:09.372 DEBUG:teuthology.orchestra.run.smithi137:osd.5> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.5.service 2024-06-14T22:32:09.375 INFO:tasks.cephadm:Deploying osd.6 on smithi137 with /dev/vg_nvme/lv_2... 2024-06-14T22:32:09.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-14T22:32:09.936 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:09.936 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:09.936 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:09.936 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-mon[32951]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-14T22:32:09.936 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-mon[32951]: osdmap e36: 6 total, 5 up, 6 in 2024-06-14T22:32:09.936 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:09.936 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-mon[32951]: from='osd.5 [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:09.937 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-mon[32951]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:10.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:10.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:10.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:10.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[29889]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-14T22:32:10.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[29889]: osdmap e36: 6 total, 5 up, 6 in 2024-06-14T22:32:10.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:10.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[29889]: from='osd.5 [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:10.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[29889]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:10.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:10.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:10.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:10.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[38904]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-14T22:32:10.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[38904]: osdmap e36: 6 total, 5 up, 6 in 2024-06-14T22:32:10.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:10.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[38904]: from='osd.5 [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:10.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:09 smithi086 ceph-mon[38904]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:10.154 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:32:10.238 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[44686]: 2024-06-14T22:32:09.974+0000 7f5c435ec640 -1 osd.5 0 waiting for initial osdmap 2024-06-14T22:32:10.239 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 22:32:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[44686]: 2024-06-14T22:32:09.981+0000 7f5c3ec24640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:32:10.821 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:10 smithi137 ceph-mon[32951]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:10.821 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:10 smithi137 ceph-mon[32951]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:32:10.821 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:10 smithi137 ceph-mon[32951]: osdmap e37: 6 total, 5 up, 6 in 2024-06-14T22:32:10.821 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:10 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:10.821 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:10 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:11.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[29889]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:11.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[29889]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:32:11.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[29889]: osdmap e37: 6 total, 5 up, 6 in 2024-06-14T22:32:11.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:11.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:11.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[38904]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:11.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[38904]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:32:11.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[38904]: osdmap e37: 6 total, 5 up, 6 in 2024-06-14T22:32:11.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:11.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:10 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:11.974 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:11 smithi137 ceph-mon[32951]: purged_snaps scrub starts 2024-06-14T22:32:11.974 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:11 smithi137 ceph-mon[32951]: purged_snaps scrub ok 2024-06-14T22:32:11.974 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:11 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:11.974 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:11 smithi137 ceph-mon[32951]: osd.5 [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324] boot 2024-06-14T22:32:11.974 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:11 smithi137 ceph-mon[32951]: osdmap e38: 6 total, 6 up, 6 in 2024-06-14T22:32:11.974 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:11 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:12.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[38904]: purged_snaps scrub starts 2024-06-14T22:32:12.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[38904]: purged_snaps scrub ok 2024-06-14T22:32:12.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:12.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[38904]: osd.5 [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324] boot 2024-06-14T22:32:12.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[38904]: osdmap e38: 6 total, 6 up, 6 in 2024-06-14T22:32:12.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:12.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[29889]: purged_snaps scrub starts 2024-06-14T22:32:12.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[29889]: purged_snaps scrub ok 2024-06-14T22:32:12.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:12.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[29889]: osd.5 [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324] boot 2024-06-14T22:32:12.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[29889]: osdmap e38: 6 total, 6 up, 6 in 2024-06-14T22:32:12.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:11 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:32:12.929 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:12.930 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: osdmap e39: 6 total, 6 up, 6 in 2024-06-14T22:32:12.930 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:12.930 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:12.930 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:12.930 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:12.930 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:12.930 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:12.930 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:12.930 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:12 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:13.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:13.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: osdmap e39: 6 total, 6 up, 6 in 2024-06-14T22:32:13.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:13.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:13.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:13.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: osdmap e39: 6 total, 6 up, 6 in 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:13.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:13.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:12 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:13.141 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-14T22:32:13.228 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-14T22:32:13.242 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-06-14T22:32:13.243 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-06-14T22:32:13.243 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129992 s, 807 MB/s 2024-06-14T22:32:13.243 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-06-14T22:32:13.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:13 smithi137 ceph-mon[32951]: Detected new or changed devices on smithi137 2024-06-14T22:32:13.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:13 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi137 to 8544M 2024-06-14T22:32:13.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:13 smithi137 ceph-mon[32951]: osdmap e40: 6 total, 6 up, 6 in 2024-06-14T22:32:13.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi137:vg_nvme/lv_2 2024-06-14T22:32:14.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:13 smithi086 ceph-mon[29889]: Detected new or changed devices on smithi137 2024-06-14T22:32:14.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:13 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi137 to 8544M 2024-06-14T22:32:14.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:13 smithi086 ceph-mon[29889]: osdmap e40: 6 total, 6 up, 6 in 2024-06-14T22:32:14.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:13 smithi086 ceph-mon[38904]: Detected new or changed devices on smithi137 2024-06-14T22:32:14.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:13 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi137 to 8544M 2024-06-14T22:32:14.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:13 smithi086 ceph-mon[38904]: osdmap e40: 6 total, 6 up, 6 in 2024-06-14T22:32:14.119 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:32:14.830 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:14 smithi137 ceph-mon[32951]: pgmap v130: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-14T22:32:14.830 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:14 smithi137 ceph-mon[32951]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-06-14T22:32:14.830 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:14 smithi137 ceph-mon[32951]: osdmap e41: 6 total, 6 up, 6 in 2024-06-14T22:32:15.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:14 smithi086 ceph-mon[29889]: pgmap v130: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-14T22:32:15.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:14 smithi086 ceph-mon[29889]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-06-14T22:32:15.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:14 smithi086 ceph-mon[29889]: osdmap e41: 6 total, 6 up, 6 in 2024-06-14T22:32:15.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:14 smithi086 ceph-mon[38904]: pgmap v130: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-14T22:32:15.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:14 smithi086 ceph-mon[38904]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-06-14T22:32:15.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:14 smithi086 ceph-mon[38904]: osdmap e41: 6 total, 6 up, 6 in 2024-06-14T22:32:16.296 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:15 smithi137 ceph-mon[32951]: osdmap e42: 6 total, 6 up, 6 in 2024-06-14T22:32:16.296 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:15 smithi137 ceph-mon[32951]: pgmap v133: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-14T22:32:16.296 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:15 smithi137 ceph-mon[32951]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:32:16.296 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:15 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:32:16.296 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:15 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:32:16.296 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:15 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:16.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[29889]: osdmap e42: 6 total, 6 up, 6 in 2024-06-14T22:32:16.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[29889]: pgmap v133: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-14T22:32:16.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[29889]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:32:16.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:32:16.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:32:16.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:16.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[38904]: osdmap e42: 6 total, 6 up, 6 in 2024-06-14T22:32:16.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[38904]: pgmap v133: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-06-14T22:32:16.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[38904]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:32:16.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:32:16.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:32:16.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:15 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:17.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:17 smithi137 ceph-mon[32951]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-06-14T22:32:17.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:17 smithi137 ceph-mon[32951]: Cluster is now healthy 2024-06-14T22:32:17.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:17 smithi086 ceph-mon[38904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-06-14T22:32:17.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:17 smithi086 ceph-mon[38904]: Cluster is now healthy 2024-06-14T22:32:17.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:17 smithi086 ceph-mon[29889]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-06-14T22:32:17.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:17 smithi086 ceph-mon[29889]: Cluster is now healthy 2024-06-14T22:32:18.671 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:18 smithi137 ceph-mon[32951]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:18.671 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:18 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/3048178734' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44e654f2-c297-49dd-94c0-9d0ef90bffe2"}]: dispatch 2024-06-14T22:32:18.671 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:18 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44e654f2-c297-49dd-94c0-9d0ef90bffe2"}]: dispatch 2024-06-14T22:32:18.671 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:18 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44e654f2-c297-49dd-94c0-9d0ef90bffe2"}]': finished 2024-06-14T22:32:18.671 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:18 smithi137 ceph-mon[32951]: osdmap e43: 7 total, 6 up, 7 in 2024-06-14T22:32:18.671 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:18 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[29889]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/3048178734' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44e654f2-c297-49dd-94c0-9d0ef90bffe2"}]: dispatch 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44e654f2-c297-49dd-94c0-9d0ef90bffe2"}]: dispatch 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44e654f2-c297-49dd-94c0-9d0ef90bffe2"}]': finished 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[29889]: osdmap e43: 7 total, 6 up, 7 in 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[38904]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/3048178734' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44e654f2-c297-49dd-94c0-9d0ef90bffe2"}]: dispatch 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44e654f2-c297-49dd-94c0-9d0ef90bffe2"}]: dispatch 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44e654f2-c297-49dd-94c0-9d0ef90bffe2"}]': finished 2024-06-14T22:32:18.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[38904]: osdmap e43: 7 total, 6 up, 7 in 2024-06-14T22:32:18.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:18 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:19.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:19 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/4214428381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:32:19.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:19 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/4214428381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:32:19.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:19 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/4214428381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:32:20.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:20 smithi137 ceph-mon[32951]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T22:32:20.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:20 smithi086 ceph-mon[29889]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T22:32:20.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:20 smithi086 ceph-mon[38904]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T22:32:22.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:22 smithi137 ceph-mon[32951]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-06-14T22:32:22.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:22 smithi086 ceph-mon[29889]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-06-14T22:32:22.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:22 smithi086 ceph-mon[38904]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-06-14T22:32:24.642 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:24 smithi137 ceph-mon[32951]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-06-14T22:32:24.643 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:24 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-14T22:32:24.643 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:24 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:24.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:24 smithi086 ceph-mon[29889]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-06-14T22:32:24.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:24 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-14T22:32:24.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:24 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:24.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:24 smithi086 ceph-mon[38904]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-06-14T22:32:24.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:24 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-14T22:32:24.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:24 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:25.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:25 smithi137 ceph-mon[32951]: Deploying daemon osd.6 on smithi137 2024-06-14T22:32:25.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:25 smithi086 ceph-mon[29889]: Deploying daemon osd.6 on smithi137 2024-06-14T22:32:25.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:25 smithi086 ceph-mon[38904]: Deploying daemon osd.6 on smithi137 2024-06-14T22:32:26.669 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:26 smithi137 ceph-mon[32951]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T22:32:26.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:26 smithi086 ceph-mon[29889]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T22:32:26.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:26 smithi086 ceph-mon[38904]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T22:32:28.706 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:28 smithi137 ceph-mon[32951]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:28.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:28 smithi086 ceph-mon[29889]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:28.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:28 smithi086 ceph-mon[38904]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:29.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:29 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:29.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:29 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:29.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:29 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:29.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:29 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:29.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:29 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:29.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:29 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:29.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:29 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:29.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:29 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:29.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:29 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:30.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:30 smithi137 ceph-mon[32951]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:30.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:30 smithi086 ceph-mon[29889]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:30.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:30 smithi086 ceph-mon[38904]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:32.490 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 6 on host 'smithi137' 2024-06-14T22:32:32.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='osd.6 [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:32.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:32 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.059 DEBUG:teuthology.orchestra.run.smithi137:osd.6> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.6.service 2024-06-14T22:32:33.062 INFO:tasks.cephadm:Deploying osd.7 on smithi137 with /dev/vg_nvme/lv_1... 2024-06-14T22:32:33.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='osd.6 [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='osd.6 [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:32 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:33.783 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:32:34.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:34 smithi137 ceph-mon[32951]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-14T22:32:34.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:34 smithi137 ceph-mon[32951]: osdmap e44: 7 total, 6 up, 7 in 2024-06-14T22:32:34.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:34 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:34.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:34 smithi137 ceph-mon[32951]: from='osd.6 [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:34.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:34 smithi137 ceph-mon[32951]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:34.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:34 smithi137 ceph-mon[32951]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:34.489 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 22:32:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[50586]: 2024-06-14T22:32:34.202+0000 7fcb4acfc640 -1 osd.6 0 waiting for initial osdmap 2024-06-14T22:32:34.489 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 22:32:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[50586]: 2024-06-14T22:32:34.208+0000 7fcb43b2f640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:32:34.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[38904]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-14T22:32:34.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[38904]: osdmap e44: 7 total, 6 up, 7 in 2024-06-14T22:32:34.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:34.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[38904]: from='osd.6 [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:34.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[38904]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:34.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[38904]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:34.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[29889]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-14T22:32:34.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[29889]: osdmap e44: 7 total, 6 up, 7 in 2024-06-14T22:32:34.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:34.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[29889]: from='osd.6 [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:34.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[29889]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:34.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:34 smithi086 ceph-mon[29889]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-14T22:32:35.357 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:35 smithi137 ceph-mon[32951]: purged_snaps scrub starts 2024-06-14T22:32:35.357 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:35 smithi137 ceph-mon[32951]: purged_snaps scrub ok 2024-06-14T22:32:35.357 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:35 smithi137 ceph-mon[32951]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:32:35.357 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:35 smithi137 ceph-mon[32951]: osdmap e45: 7 total, 6 up, 7 in 2024-06-14T22:32:35.357 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:35 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:35.357 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:35 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:35.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[29889]: purged_snaps scrub starts 2024-06-14T22:32:35.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[29889]: purged_snaps scrub ok 2024-06-14T22:32:35.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[29889]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:32:35.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[29889]: osdmap e45: 7 total, 6 up, 7 in 2024-06-14T22:32:35.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:35.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:35.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[38904]: purged_snaps scrub starts 2024-06-14T22:32:35.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[38904]: purged_snaps scrub ok 2024-06-14T22:32:35.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[38904]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:32:35.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[38904]: osdmap e45: 7 total, 6 up, 7 in 2024-06-14T22:32:35.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:35.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:35 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:36.265 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-14T22:32:36.354 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-14T22:32:36.369 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-06-14T22:32:36.369 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-06-14T22:32:36.369 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012828 s, 817 MB/s 2024-06-14T22:32:36.369 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-06-14T22:32:36.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:36 smithi137 ceph-mon[32951]: osd.6 [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362] boot 2024-06-14T22:32:36.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:36 smithi137 ceph-mon[32951]: osdmap e46: 7 total, 7 up, 7 in 2024-06-14T22:32:36.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:36 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:36.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:36 smithi137 ceph-mon[32951]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:36.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:36 smithi137 ceph-mon[32951]: osdmap e47: 7 total, 7 up, 7 in 2024-06-14T22:32:36.541 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[29889]: osd.6 [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362] boot 2024-06-14T22:32:36.541 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[29889]: osdmap e46: 7 total, 7 up, 7 in 2024-06-14T22:32:36.541 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:36.541 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[29889]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:36.541 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[29889]: osdmap e47: 7 total, 7 up, 7 in 2024-06-14T22:32:36.541 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[38904]: osd.6 [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362] boot 2024-06-14T22:32:36.541 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[38904]: osdmap e46: 7 total, 7 up, 7 in 2024-06-14T22:32:36.541 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:32:36.542 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[38904]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:36.542 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:36 smithi086 ceph-mon[38904]: osdmap e47: 7 total, 7 up, 7 in 2024-06-14T22:32:37.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi137:vg_nvme/lv_1 2024-06-14T22:32:38.062 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:32:38.480 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: Detected new or changed devices on smithi137 2024-06-14T22:32:38.480 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: osdmap e48: 7 total, 7 up, 7 in 2024-06-14T22:32:38.480 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.480 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.480 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:38.481 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:38.481 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:38.481 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi137 to 5696M 2024-06-14T22:32:38.481 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.481 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:38.481 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:38.481 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.481 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:38.481 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:38 smithi137 ceph-mon[32951]: osdmap e49: 7 total, 7 up, 7 in 2024-06-14T22:32:38.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: Detected new or changed devices on smithi137 2024-06-14T22:32:38.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: osdmap e48: 7 total, 7 up, 7 in 2024-06-14T22:32:38.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi137 to 5696M 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[29889]: osdmap e49: 7 total, 7 up, 7 in 2024-06-14T22:32:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: Detected new or changed devices on smithi137 2024-06-14T22:32:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: osdmap e48: 7 total, 7 up, 7 in 2024-06-14T22:32:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:32:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi137 to 5696M 2024-06-14T22:32:38.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:38.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:38.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:38.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:38.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:38 smithi086 ceph-mon[38904]: osdmap e49: 7 total, 7 up, 7 in 2024-06-14T22:32:39.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:39 smithi137 ceph-mon[32951]: osdmap e50: 7 total, 7 up, 7 in 2024-06-14T22:32:40.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:39 smithi086 ceph-mon[29889]: osdmap e50: 7 total, 7 up, 7 in 2024-06-14T22:32:40.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:39 smithi086 ceph-mon[38904]: osdmap e50: 7 total, 7 up, 7 in 2024-06-14T22:32:40.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:40 smithi137 ceph-mon[32951]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:40.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:40 smithi137 ceph-mon[32951]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:32:40.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:40 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:32:40.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:40 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:32:40.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:40 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:41.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[29889]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:41.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[29889]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:32:41.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:32:41.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:32:41.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:41.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[38904]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:41.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[38904]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:32:41.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-14T22:32:41.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-14T22:32:41.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:40 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:42.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:42 smithi137 ceph-mon[32951]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-06-14T22:32:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:42 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/2536916699' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f867732b-d642-4650-a57e-b01280b47e6e"}]: dispatch 2024-06-14T22:32:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:42 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f867732b-d642-4650-a57e-b01280b47e6e"}]: dispatch 2024-06-14T22:32:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:42 smithi137 ceph-mon[32951]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f867732b-d642-4650-a57e-b01280b47e6e"}]': finished 2024-06-14T22:32:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:42 smithi137 ceph-mon[32951]: osdmap e51: 8 total, 7 up, 8 in 2024-06-14T22:32:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:42 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:42 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/128666960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:32:43.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[29889]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-06-14T22:32:43.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/2536916699' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f867732b-d642-4650-a57e-b01280b47e6e"}]: dispatch 2024-06-14T22:32:43.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f867732b-d642-4650-a57e-b01280b47e6e"}]: dispatch 2024-06-14T22:32:43.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[29889]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f867732b-d642-4650-a57e-b01280b47e6e"}]': finished 2024-06-14T22:32:43.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[29889]: osdmap e51: 8 total, 7 up, 8 in 2024-06-14T22:32:43.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:43.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/128666960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:32:43.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[38904]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-06-14T22:32:43.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/2536916699' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f867732b-d642-4650-a57e-b01280b47e6e"}]: dispatch 2024-06-14T22:32:43.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f867732b-d642-4650-a57e-b01280b47e6e"}]: dispatch 2024-06-14T22:32:43.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[38904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f867732b-d642-4650-a57e-b01280b47e6e"}]': finished 2024-06-14T22:32:43.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[38904]: osdmap e51: 8 total, 7 up, 8 in 2024-06-14T22:32:43.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:43.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:42 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/128666960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-14T22:32:44.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:44 smithi137 ceph-mon[32951]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T22:32:45.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:44 smithi086 ceph-mon[29889]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T22:32:45.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:44 smithi086 ceph-mon[38904]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T22:32:46.940 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:46 smithi137 ceph-mon[32951]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-06-14T22:32:47.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:46 smithi086 ceph-mon[29889]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-06-14T22:32:47.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:46 smithi086 ceph-mon[38904]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-06-14T22:32:48.916 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:48 smithi137 ceph-mon[32951]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-06-14T22:32:48.916 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-14T22:32:48.917 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:48 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:49.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:48 smithi086 ceph-mon[29889]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-06-14T22:32:49.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-14T22:32:49.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:48 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:49.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:48 smithi086 ceph-mon[38904]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-06-14T22:32:49.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-14T22:32:49.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:48 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:49.947 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:49 smithi137 ceph-mon[32951]: Deploying daemon osd.7 on smithi137 2024-06-14T22:32:50.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:49 smithi086 ceph-mon[29889]: Deploying daemon osd.7 on smithi137 2024-06-14T22:32:50.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:49 smithi086 ceph-mon[38904]: Deploying daemon osd.7 on smithi137 2024-06-14T22:32:50.834 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:50 smithi137 ceph-mon[32951]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T22:32:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:50 smithi086 ceph-mon[29889]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T22:32:51.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:50 smithi086 ceph-mon[38904]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T22:32:52.692 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:52 smithi137 ceph-mon[32951]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:53.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:52 smithi086 ceph-mon[29889]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:53.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:52 smithi086 ceph-mon[38904]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:53.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:53 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:53.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:53 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:53.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:53 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:54.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:54.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:54.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:53 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:54.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:53 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:54.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:53 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:54.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:53 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:54.843 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:54 smithi137 ceph-mon[32951]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:54.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:54 smithi086 ceph-mon[29889]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:54.981 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:54 smithi086 ceph-mon[38904]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:56.540 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 7 on host 'smithi137' 2024-06-14T22:32:56.840 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='osd.7 [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:56.841 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:56 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='osd.7 [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-14T22:32:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-14T22:32:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='osd.7 [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:56 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:32:57.116 DEBUG:teuthology.orchestra.run.smithi137:osd.7> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.7.service 2024-06-14T22:32:57.118 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-06-14T22:32:57.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd stat -f json 2024-06-14T22:32:57.379 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:32:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:57 smithi137 ceph-mon[32951]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-14T22:32:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:57 smithi137 ceph-mon[32951]: osdmap e52: 8 total, 7 up, 8 in 2024-06-14T22:32:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:57 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:57 smithi137 ceph-mon[32951]: from='osd.7 [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:57 smithi137 ceph-mon[32951]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:57 smithi137 ceph-mon[32951]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:57.989 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 22:32:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T22:32:57.890+0000 7fec39532640 -1 osd.7 0 waiting for initial osdmap 2024-06-14T22:32:57.989 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 22:32:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T22:32:57.896+0000 7fec33b68640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:32:58.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[29889]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-14T22:32:58.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[29889]: osdmap e52: 8 total, 7 up, 8 in 2024-06-14T22:32:58.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:58.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[29889]: from='osd.7 [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:58.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[29889]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:58.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[29889]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[38904]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-14T22:32:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[38904]: osdmap e52: 8 total, 7 up, 8 in 2024-06-14T22:32:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[38904]: from='osd.7 [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[38904]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T22:32:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:57 smithi086 ceph-mon[38904]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:32:58.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:58 smithi137 ceph-mon[32951]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:32:58.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:58 smithi137 ceph-mon[32951]: osdmap e53: 8 total, 7 up, 8 in 2024-06-14T22:32:58.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:58 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:58.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:58 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:59.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:58 smithi086 ceph-mon[29889]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:32:59.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:58 smithi086 ceph-mon[29889]: osdmap e53: 8 total, 7 up, 8 in 2024-06-14T22:32:59.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:58 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:59.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:58 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:59.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:58 smithi086 ceph-mon[38904]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-06-14T22:32:59.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:58 smithi086 ceph-mon[38904]: osdmap e53: 8 total, 7 up, 8 in 2024-06-14T22:32:59.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:58 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:59.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:58 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:32:59.169 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:32:59.850 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1718404378,"num_in_osds":8,"osd_in_since":1718404362,"num_remapped_pgs":1} 2024-06-14T22:32:59.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-14T22:33:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[38904]: purged_snaps scrub starts 2024-06-14T22:33:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[38904]: purged_snaps scrub ok 2024-06-14T22:33:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[38904]: osd.7 [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786] boot 2024-06-14T22:33:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[38904]: osdmap e54: 8 total, 8 up, 8 in 2024-06-14T22:33:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:33:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1504038683' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-14T22:33:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[38904]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:33:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[38904]: osdmap e55: 8 total, 8 up, 8 in 2024-06-14T22:33:00.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[29889]: purged_snaps scrub starts 2024-06-14T22:33:00.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[29889]: purged_snaps scrub ok 2024-06-14T22:33:00.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[29889]: osd.7 [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786] boot 2024-06-14T22:33:00.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[29889]: osdmap e54: 8 total, 8 up, 8 in 2024-06-14T22:33:00.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:33:00.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1504038683' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-14T22:33:00.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[29889]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:33:00.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:32:59 smithi086 ceph-mon[29889]: osdmap e55: 8 total, 8 up, 8 in 2024-06-14T22:33:00.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:59 smithi137 ceph-mon[32951]: purged_snaps scrub starts 2024-06-14T22:33:00.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:59 smithi137 ceph-mon[32951]: purged_snaps scrub ok 2024-06-14T22:33:00.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:59 smithi137 ceph-mon[32951]: osd.7 [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786] boot 2024-06-14T22:33:00.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:59 smithi137 ceph-mon[32951]: osdmap e54: 8 total, 8 up, 8 in 2024-06-14T22:33:00.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:59 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:33:00.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:59 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1504038683' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-14T22:33:00.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:59 smithi137 ceph-mon[32951]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-06-14T22:33:00.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:32:59 smithi137 ceph-mon[32951]: osdmap e55: 8 total, 8 up, 8 in 2024-06-14T22:33:00.104 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:01.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: Detected new or changed devices on smithi137 2024-06-14T22:33:01.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi137 to 4272M 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[29889]: osdmap e56: 8 total, 8 up, 8 in 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: Detected new or changed devices on smithi137 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi137 to 4272M 2024-06-14T22:33:01.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:33:01.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:33:01.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:01 smithi086 ceph-mon[38904]: osdmap e56: 8 total, 8 up, 8 in 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: Detected new or changed devices on smithi137 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi137 to 4272M 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:33:01.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:33:01.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:01.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:01 smithi137 ceph-mon[32951]: osdmap e56: 8 total, 8 up, 8 in 2024-06-14T22:33:02.050 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:33:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":57,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T22:33:01.888419+0000","last_up_change":"2024-06-14T22:32:58.884084+0000","last_in_change":"2024-06-14T22:32:42.063674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6803","nonce":2092557056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6805","nonce":2092557056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6809","nonce":2092557056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6807","nonce":2092557056}]},"public_addr":"172.21.15.86:6803/2092557056","cluster_addr":"172.21.15.86:6805/2092557056","heartbeat_back_addr":"172.21.15.86:6809/2092557056","heartbeat_front_addr":"172.21.15.86:6807/2092557056","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6811","nonce":796510430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6813","nonce":796510430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6817","nonce":796510430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6815","nonce":796510430}]},"public_addr":"172.21.15.86:6811/796510430","cluster_addr":"172.21.15.86:6813/796510430","heartbeat_back_addr":"172.21.15.86:6817/796510430","heartbeat_front_addr":"172.21.15.86:6815/796510430","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6819","nonce":3531892667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6821","nonce":3531892667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6825","nonce":3531892667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6823","nonce":3531892667}]},"public_addr":"172.21.15.86:6819/3531892667","cluster_addr":"172.21.15.86:6821/3531892667","heartbeat_back_addr":"172.21.15.86:6825/3531892667","heartbeat_front_addr":"172.21.15.86:6823/3531892667","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6827","nonce":287823501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6829","nonce":287823501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6833","nonce":287823501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6831","nonce":287823501}]},"public_addr":"172.21.15.86:6827/287823501","cluster_addr":"172.21.15.86:6829/287823501","heartbeat_back_addr":"172.21.15.86:6833/287823501","heartbeat_front_addr":"172.21.15.86:6831/287823501","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6809","nonce":2122959324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6811","nonce":2122959324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6815","nonce":2122959324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6813","nonce":2122959324}]},"public_addr":"172.21.15.137:6809/2122959324","cluster_addr":"172.21.15.137:6811/2122959324","heartbeat_back_addr":"172.21.15.137:6815/2122959324","heartbeat_front_addr":"172.21.15.137:6813/2122959324","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6817","nonce":2398798362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6819","nonce":2398798362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6823","nonce":2398798362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6821","nonce":2398798362}]},"public_addr":"172.21.15.137:6817/2398798362","cluster_addr":"172.21.15.137:6819/2398798362","heartbeat_back_addr":"172.21.15.137:6823/2398798362","heartbeat_front_addr":"172.21.15.137:6821/2398798362","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T22:33:02.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:02 smithi086 ceph-mon[29889]: pgmap v170: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:02.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:02 smithi086 ceph-mon[29889]: osdmap e57: 8 total, 8 up, 8 in 2024-06-14T22:33:02.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:02 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1941617830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:02.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:02 smithi086 ceph-mon[38904]: pgmap v170: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:02.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:02 smithi086 ceph-mon[38904]: osdmap e57: 8 total, 8 up, 8 in 2024-06-14T22:33:02.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:02 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1941617830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:02.390 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:02 smithi137 ceph-mon[32951]: pgmap v170: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:02.390 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:02 smithi137 ceph-mon[32951]: osdmap e57: 8 total, 8 up, 8 in 2024-06-14T22:33:02.390 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:02 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1941617830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:02.696 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-14T22:31:01.536068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-14T22:33:02.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-14T22:33:02.946 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:04.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:03 smithi086 ceph-mon[29889]: osdmap e58: 8 total, 8 up, 8 in 2024-06-14T22:33:04.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:03 smithi086 ceph-mon[29889]: pgmap v173: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:04.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:03 smithi086 ceph-mon[38904]: osdmap e58: 8 total, 8 up, 8 in 2024-06-14T22:33:04.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:03 smithi086 ceph-mon[38904]: pgmap v173: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:04.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:03 smithi137 ceph-mon[32951]: osdmap e58: 8 total, 8 up, 8 in 2024-06-14T22:33:04.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:03 smithi137 ceph-mon[32951]: pgmap v173: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:pg_num: 1 2024-06-14T22:33:05.150 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:04 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2754173290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-14T22:33:05.150 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:04 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2754173290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-14T22:33:05.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:04 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2754173290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-14T22:33:05.542 INFO:tasks.cephadm:Adding prometheus.a on smithi137 2024-06-14T22:33:05.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch apply prometheus '1;smithi137=a' 2024-06-14T22:33:05.807 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:33:06.160 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:05 smithi137 ceph-mon[32951]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-06-14T22:33:06.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:05 smithi086 ceph-mon[29889]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-06-14T22:33:06.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:05 smithi086 ceph-mon[38904]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-06-14T22:33:07.464 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled prometheus update... 2024-06-14T22:33:08.021 DEBUG:teuthology.orchestra.run.smithi137:prometheus.a> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@prometheus.a.service 2024-06-14T22:33:08.023 INFO:tasks.cephadm:Adding node-exporter.a on smithi086 2024-06-14T22:33:08.023 INFO:tasks.cephadm:Adding node-exporter.b on smithi137 2024-06-14T22:33:08.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch apply node-exporter '2;smithi086=a;smithi137=b' 2024-06-14T22:33:08.331 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:33:08.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:08 smithi137 ceph-mon[32951]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-06-14T22:33:08.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:08 smithi137 ceph-mon[32951]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:08.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:08 smithi137 ceph-mon[32951]: Saving service prometheus spec with placement smithi137=a;count:1 2024-06-14T22:33:08.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:08.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:33:08.711 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:33:08.711 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:33:08.711 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:08 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:08.711 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:08 smithi137 ceph-mon[32951]: Deploying daemon prometheus.a on smithi137 2024-06-14T22:33:08.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[29889]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-06-14T22:33:08.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[29889]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[29889]: Saving service prometheus spec with placement smithi137=a;count:1 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[29889]: Deploying daemon prometheus.a on smithi137 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[38904]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[38904]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[38904]: Saving service prometheus spec with placement smithi137=a;count:1 2024-06-14T22:33:08.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:08.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:33:08.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:33:08.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:33:08.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:08.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:08 smithi086 ceph-mon[38904]: Deploying daemon prometheus.a on smithi137 2024-06-14T22:33:10.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:10 smithi137 ceph-mon[32951]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-14T22:33:10.797 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled node-exporter update... 2024-06-14T22:33:10.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:10 smithi086 ceph-mon[29889]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-14T22:33:10.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:10 smithi086 ceph-mon[38904]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-14T22:33:11.769 DEBUG:teuthology.orchestra.run.smithi086:node-exporter.a> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.a.service 2024-06-14T22:33:11.772 DEBUG:teuthology.orchestra.run.smithi137:node-exporter.b> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.b.service 2024-06-14T22:33:11.775 INFO:tasks.cephadm:Adding alertmanager.a on smithi086 2024-06-14T22:33:11.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch apply alertmanager '1;smithi086=a' 2024-06-14T22:33:11.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:11 smithi137 ceph-mon[32951]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi086=a;smithi137=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:11.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:11 smithi137 ceph-mon[32951]: Saving service node-exporter spec with placement smithi086=a;smithi137=b;count:2 2024-06-14T22:33:11.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:11 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:12.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:11 smithi086 ceph-mon[29889]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi086=a;smithi137=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:12.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:11 smithi086 ceph-mon[29889]: Saving service node-exporter spec with placement smithi086=a;smithi137=b;count:2 2024-06-14T22:33:12.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:11 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:12.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:11 smithi086 ceph-mon[38904]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi086=a;smithi137=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:12.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:11 smithi086 ceph-mon[38904]: Saving service node-exporter spec with placement smithi086=a;smithi137=b;count:2 2024-06-14T22:33:12.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:11 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:12.086 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:33:12.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:12 smithi137 ceph-mon[32951]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-14T22:33:13.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:12 smithi086 ceph-mon[29889]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-14T22:33:13.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:12 smithi086 ceph-mon[38904]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-14T22:33:14.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:14 smithi137 ceph-mon[32951]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-06-14T22:33:15.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:14 smithi086 ceph-mon[29889]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-06-14T22:33:15.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:14 smithi086 ceph-mon[38904]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-06-14T22:33:15.919 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled alertmanager update... 2024-06-14T22:33:16.820 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:16 smithi137 ceph-mon[32951]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-14T22:33:16.820 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:16 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:16.847 DEBUG:teuthology.orchestra.run.smithi086:alertmanager.a> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@alertmanager.a.service 2024-06-14T22:33:16.850 INFO:tasks.cephadm:Adding grafana.a on smithi137 2024-06-14T22:33:16.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph orch apply grafana '1;smithi137=a' 2024-06-14T22:33:17.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:16 smithi086 ceph-mon[29889]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-14T22:33:17.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:16 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:17.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:16 smithi086 ceph-mon[38904]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-06-14T22:33:17.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:16 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:17.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:17 smithi137 ceph-mon[32951]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:17.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:17 smithi137 ceph-mon[32951]: Saving service alertmanager spec with placement smithi086=a;count:1 2024-06-14T22:33:18.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:17 smithi086 ceph-mon[29889]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:18.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:17 smithi086 ceph-mon[29889]: Saving service alertmanager spec with placement smithi086=a;count:1 2024-06-14T22:33:18.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:17 smithi086 ceph-mon[38904]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:18.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:17 smithi086 ceph-mon[38904]: Saving service alertmanager spec with placement smithi086=a;count:1 2024-06-14T22:33:18.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:18 smithi137 ceph-mon[32951]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:19.041 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:33:19.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:18 smithi086 ceph-mon[29889]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:19.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:18 smithi086 ceph-mon[38904]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:20.910 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled grafana update... 2024-06-14T22:33:21.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:20 smithi086 ceph-mon[29889]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:21.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:20 smithi086 ceph-mon[38904]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:21.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:20 smithi137 ceph-mon[32951]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:21.642 DEBUG:teuthology.orchestra.run.smithi137:grafana.a> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@grafana.a.service 2024-06-14T22:33:21.644 INFO:tasks.cephadm:Setting up client nodes... 2024-06-14T22:33:21.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-14T22:33:21.865 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:33:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:33:21.840Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-06-14T22:33:21.902 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:22.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:21 smithi086 ceph-mon[29889]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:22.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:21 smithi086 ceph-mon[29889]: Saving service grafana spec with placement smithi137=a;count:1 2024-06-14T22:33:22.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:21 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:22.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:21 smithi086 ceph-mon[29889]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:22.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:21 smithi086 ceph-mon[38904]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:22.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:21 smithi086 ceph-mon[38904]: Saving service grafana spec with placement smithi137=a;count:1 2024-06-14T22:33:22.027 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:21 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:22.027 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:21 smithi086 ceph-mon[38904]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:22.238 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:33:21 smithi137 bash[62101]: 020e411403a82ed9842212d86e7f860d980c83ad2d17fa17b114ac201bde2923 2024-06-14T22:33:22.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:33:21 smithi137 systemd[1]: Started Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:33:22.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:21 smithi137 ceph-mon[32951]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:33:22.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:21 smithi137 ceph-mon[32951]: Saving service grafana spec with placement smithi137=a;count:1 2024-06-14T22:33:22.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:21 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:22.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:21 smithi137 ceph-mon[32951]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:23.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:23.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:23.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:23.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-14T22:33:23.079 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ignoring --setuser ceph since I am not root 2024-06-14T22:33:23.079 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ignoring --setgroup ceph since I am not root 2024-06-14T22:33:23.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-mgr[30165]: -- 172.21.15.86:0/1551239451 <== mon.0 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563ba129e9c0 con 0x563ba12aa800 2024-06-14T22:33:23.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:23.082+0000 7fec319f2ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:33:23.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:23.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:23.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:23.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:22 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-14T22:33:23.210 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:22 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:23.211 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:22 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:23.211 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:22 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' 2024-06-14T22:33:23.211 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:22 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-14T22:33:23.211 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ignoring --setuser ceph since I am not root 2024-06-14T22:33:23.211 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ignoring --setgroup ceph since I am not root 2024-06-14T22:33:23.211 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:22 smithi137 ceph-mgr[35026]: -- 172.21.15.137:0/799350555 <== mon.2 v2:172.21.15.86:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55825ba769c0 con 0x55825ba82c00 2024-06-14T22:33:23.211 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:23.089+0000 7ff3fd9b3ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:33:23.211 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:23.215+0000 7ff3fd9b3ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:33:23.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:23.211+0000 7fec319f2ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:33:23.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:23.282+0000 7fec319f2ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:33:23.488 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:23.281+0000 7ff3fd9b3ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:33:23.592 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:23.537+0000 7fec319f2ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:33:23.592 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:23.596+0000 7fec319f2ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:33:23.872 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:23.523+0000 7ff3fd9b3ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:33:23.872 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:23.581+0000 7ff3fd9b3ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:33:23.968 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2024-06-14T22:33:23.968 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAzxWxmrPuvORAA0r6CC0K8u5YvN/MnQ/mQjQ== 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-mon[32951]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-mon[32951]: mgrmap e16: y(active, since 4m), standbys: x 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:23.876+0000 7ff3fd9b3ec0 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:23.876+0000 7ff3fd9b3ec0 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: from .module import Module 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:23.877+0000 7ff3fd9b3ec0 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:33:24.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:23.877+0000 7ff3fd9b3ec0 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:33:24.279 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-mon[29889]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-14T22:33:24.279 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-mon[29889]: mgrmap e16: y(active, since 4m), standbys: x 2024-06-14T22:33:24.279 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-mon[38904]: from='mgr.14150 172.21.15.86:0/2809433469' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-14T22:33:24.279 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-mon[38904]: mgrmap e16: y(active, since 4m), standbys: x 2024-06-14T22:33:24.279 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:23.928+0000 7fec319f2ec0 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:33:24.279 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:23.928+0000 7fec319f2ec0 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from .module import Module 2024-06-14T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:23.929+0000 7fec319f2ec0 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:23.929+0000 7fec319f2ec0 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:33:24.656 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-06-14T22:33:24.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-14T22:33:24.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-14T22:33:24.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-14T22:33:24.738 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:24.484+0000 7ff3fd9b3ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:33:24.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:24.596+0000 7ff3fd9b3ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:33:24.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:24.651+0000 7ff3fd9b3ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:33:24.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:24.560+0000 7fec319f2ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:33:24.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:24.677+0000 7fec319f2ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:33:24.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:24.736+0000 7fec319f2ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:33:24.956 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:33:25.100 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:24 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2618981922' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:25.100 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:24 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:25.100 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:24 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-14T22:33:25.100 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:24.852+0000 7ff3fd9b3ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:33:25.100 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:24.919+0000 7ff3fd9b3ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:33:25.100 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:24.995+0000 7ff3fd9b3ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:33:25.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2618981922' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:25.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:25.181 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-14T22:33:25.181 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2618981922' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:25.181 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:25.181 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-14T22:33:25.181 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:24.930+0000 7fec319f2ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:33:25.181 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:24.993+0000 7fec319f2ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:33:25.181 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:25.059+0000 7fec319f2ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:33:25.181 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:25.185+0000 7fec319f2ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:33:25.361 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:25 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:25.125+0000 7ff3fd9b3ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:33:25.738 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:25 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:25.556+0000 7ff3fd9b3ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:33:25.898 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:25.601+0000 7fec319f2ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:33:25.898 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:25.901+0000 7fec319f2ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:33:26.218 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:25 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:25.855+0000 7ff3fd9b3ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:33:26.218 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:25 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:25.921+0000 7ff3fd9b3ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:33:26.218 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:26 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:26.054+0000 7ff3fd9b3ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:33:26.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:25.970+0000 7fec319f2ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:33:26.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:26 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:26.106+0000 7fec319f2ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:33:26.488 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:26 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:26.323+0000 7ff3fd9b3ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:33:26.488 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:26 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:26.384+0000 7ff3fd9b3ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:33:26.625 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:26 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:26.376+0000 7fec319f2ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:33:26.625 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:26 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:26.438+0000 7fec319f2ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:33:26.791 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:26 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:26.581+0000 7ff3fd9b3ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:33:26.791 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:26 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:26.646+0000 7ff3fd9b3ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:33:26.792 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:26 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:26.735+0000 7ff3fd9b3ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:33:26.792 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:26 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:26.795+0000 7ff3fd9b3ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:33:26.917 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:26 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:26.629+0000 7fec319f2ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:33:26.917 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:26 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:26.694+0000 7fec319f2ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:33:26.917 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:26 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:26.780+0000 7fec319f2ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:33:26.917 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:26 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:26.840+0000 7fec319f2ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:33:26.963 INFO:teuthology.orchestra.run.smithi137.stdout:[client.1] 2024-06-14T22:33:26.963 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQA2xWxm69N/ORAA8W4vKXrnF4gERd/c792eaA== 2024-06-14T22:33:27.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-mon[32951]: from='client.? 172.21.15.137:0/3733625442' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:27.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:27.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-14T22:33:27.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-mon[38904]: from='client.? 172.21.15.137:0/3733625442' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:27.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:27.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-14T22:33:27.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-mon[29889]: from='client.? 172.21.15.137:0/3733625442' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:27.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-14T22:33:27.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-14T22:33:27.539 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:27.345+0000 7ff3fd9b3ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:33:27.539 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:27.485+0000 7ff3fd9b3ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:33:27.539 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:27.485+0000 7ff3fd9b3ec0 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:33:27.539 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: [14/Jun/2024:22:33:27] ENGINE Bus STARTING 2024-06-14T22:33:27.539 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: CherryPy Checker: 2024-06-14T22:33:27.539 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: The Application mounted at '' has an empty config. 2024-06-14T22:33:27.539 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: 2024-06-14T22:33:27.563 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-06-14T22:33:27.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-14T22:33:27.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-14T22:33:27.600 INFO:tasks.cephadm:Waiting for OSDs to come up 2024-06-14T22:33:27.600 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-14T22:33:27.713 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:27.376+0000 7fec319f2ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:33:27.713 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:27.516+0000 7fec319f2ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:33:27.713 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:27.516+0000 7fec319f2ec0 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:33:27.713 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:33:27] ENGINE Bus STARTING 2024-06-14T22:33:27.713 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: CherryPy Checker: 2024-06-14T22:33:27.713 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: The Application mounted at '' has an empty config. 2024-06-14T22:33:27.713 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:27.821 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: [14/Jun/2024:22:33:27] ENGINE Serving on http://:::9283 2024-06-14T22:33:27.821 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: [14/Jun/2024:22:33:27] ENGINE Bus STARTED 2024-06-14T22:33:27.944 INFO:teuthology.misc.health.smithi086.stdout: 2024-06-14T22:33:27.944 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":59,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T22:33:27.518450+0000","last_up_change":"2024-06-14T22:32:58.884084+0000","last_in_change":"2024-06-14T22:32:42.063674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6803","nonce":2092557056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6805","nonce":2092557056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6809","nonce":2092557056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6807","nonce":2092557056}]},"public_addr":"172.21.15.86:6803/2092557056","cluster_addr":"172.21.15.86:6805/2092557056","heartbeat_back_addr":"172.21.15.86:6809/2092557056","heartbeat_front_addr":"172.21.15.86:6807/2092557056","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6811","nonce":796510430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6813","nonce":796510430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6817","nonce":796510430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6815","nonce":796510430}]},"public_addr":"172.21.15.86:6811/796510430","cluster_addr":"172.21.15.86:6813/796510430","heartbeat_back_addr":"172.21.15.86:6817/796510430","heartbeat_front_addr":"172.21.15.86:6815/796510430","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6819","nonce":3531892667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6821","nonce":3531892667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6825","nonce":3531892667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6823","nonce":3531892667}]},"public_addr":"172.21.15.86:6819/3531892667","cluster_addr":"172.21.15.86:6821/3531892667","heartbeat_back_addr":"172.21.15.86:6825/3531892667","heartbeat_front_addr":"172.21.15.86:6823/3531892667","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6827","nonce":287823501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6829","nonce":287823501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6833","nonce":287823501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6831","nonce":287823501}]},"public_addr":"172.21.15.86:6827/287823501","cluster_addr":"172.21.15.86:6829/287823501","heartbeat_back_addr":"172.21.15.86:6833/287823501","heartbeat_front_addr":"172.21.15.86:6831/287823501","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6809","nonce":2122959324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6811","nonce":2122959324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6815","nonce":2122959324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6813","nonce":2122959324}]},"public_addr":"172.21.15.137:6809/2122959324","cluster_addr":"172.21.15.137:6811/2122959324","heartbeat_back_addr":"172.21.15.137:6815/2122959324","heartbeat_front_addr":"172.21.15.137:6813/2122959324","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6817","nonce":2398798362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6819","nonce":2398798362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6823","nonce":2398798362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6821","nonce":2398798362}]},"public_addr":"172.21.15.137:6817/2398798362","cluster_addr":"172.21.15.137:6819/2398798362","heartbeat_back_addr":"172.21.15.137:6823/2398798362","heartbeat_front_addr":"172.21.15.137:6821/2398798362","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3941467574":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/2741142752":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/1759721543":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3724463773":"2024-06-15T22:33:27.518416+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T22:33:27.957 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-06-14T22:33:27.957 INFO:tasks.cephadm:Creating RBD pool 2024-06-14T22:33:27.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- sudo ceph --cluster ceph osd pool create rbd 8 2024-06-14T22:33:27.982 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:33:27] ENGINE Serving on http://:::9283 2024-06-14T22:33:27.982 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:33:27] ENGINE Bus STARTED 2024-06-14T22:33:28.239 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:28.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:33:28.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: Standby manager daemon x restarted 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: Standby manager daemon x started 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: Active manager daemon y restarted 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: Activating manager daemon y 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: osdmap e59: 8 total, 8 up, 8 in 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: mgrmap e17: y(active, starting, since 0.00855752s), standbys: x 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:33:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:33:28.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: Manager daemon y is now available 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:33:28.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:28 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1911260217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:28.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:33:28.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: Standby manager daemon x restarted 2024-06-14T22:33:28.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: Standby manager daemon x started 2024-06-14T22:33:28.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:33:28.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: Active manager daemon y restarted 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: Activating manager daemon y 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: osdmap e59: 8 total, 8 up, 8 in 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: mgrmap e17: y(active, starting, since 0.00855752s), standbys: x 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:33:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: Manager daemon y is now available 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:33:28.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:33:28.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:33:28.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1911260217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:28.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:33:28.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: Standby manager daemon x restarted 2024-06-14T22:33:28.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: Standby manager daemon x started 2024-06-14T22:33:28.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:33:28.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:33:28.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:33:28.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/2938437038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: Active manager daemon y restarted 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: Activating manager daemon y 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: osdmap e59: 8 total, 8 up, 8 in 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: mgrmap e17: y(active, starting, since 0.00855752s), standbys: x 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:33:28.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: Manager daemon y is now available 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:33:28.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:33:28.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:28 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1911260217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:29.534 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:29.177+0000 7febd4852640 -1 log_channel(cephadm) log [ERR] : [14/Jun/2024:22:33:29] ENGINE Error in HTTPServer.serve 2024-06-14T22:33:29.534 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: Traceback (most recent call last): 2024-06-14T22:33:29.534 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:33:29.534 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self._connections.run(self.expiration_interval) 2024-06-14T22:33:29.534 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:33:29.534 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self._run(expiration_interval) 2024-06-14T22:33:29.534 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:33:29.534 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:33:29.534 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: s = self.context.wrap_socket( 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: return self.sslsocket_class._create( 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self.do_handshake() 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: self._sslobj.do_handshake() 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:33:29.535 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:33:29.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:29 smithi137 ceph-mon[32951]: mgrmap e18: y(active, since 1.017s), standbys: x 2024-06-14T22:33:29.738 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:29 smithi137 ceph-mon[32951]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:29.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:29 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:29.789 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-mon[29889]: mgrmap e18: y(active, since 1.017s), standbys: x 2024-06-14T22:33:29.789 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-mon[29889]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:29.789 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:29.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-mon[38904]: mgrmap e18: y(active, since 1.017s), standbys: x 2024-06-14T22:33:29.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-mon[38904]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:29.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:29 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:30.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:33:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:33:30.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: [14/Jun/2024:22:33:29] ENGINE Bus STARTING 2024-06-14T22:33:30.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: [14/Jun/2024:22:33:29] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:33:30.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: [14/Jun/2024:22:33:29] ENGINE Bus STARTED 2024-06-14T22:33:30.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: [14/Jun/2024:22:33:29] ENGINE Error in HTTPServer.serve 2024-06-14T22:33:30.989 INFO:journalctl@ceph.mon.b.smithi137.stdout: Traceback (most recent call last): 2024-06-14T22:33:30.989 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:33:30.989 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._run(expiration_interval) 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: s = self.context.wrap_socket( 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: return self.sslsocket_class._create( 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: self.do_handshake() 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._sslobj.do_handshake() 2024-06-14T22:33:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: mgrmap e19: y(active, since 2s), standbys: x 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: [14/Jun/2024:22:33:29] ENGINE Bus STARTING 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: [14/Jun/2024:22:33:29] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: [14/Jun/2024:22:33:29] ENGINE Bus STARTED 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: [14/Jun/2024:22:33:29] ENGINE Error in HTTPServer.serve 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:33:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: self.do_handshake() 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:31.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: mgrmap e19: y(active, since 2s), standbys: x 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: [14/Jun/2024:22:33:29] ENGINE Bus STARTING 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: [14/Jun/2024:22:33:29] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: [14/Jun/2024:22:33:29] ENGINE Bus STARTED 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: [14/Jun/2024:22:33:29] ENGINE Error in HTTPServer.serve 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:33:31.334 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: self.do_handshake() 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: mgrmap e19: y(active, since 2s), standbys: x 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:33:31.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'rbd' created 2024-06-14T22:33:32.224 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi137 to 4016M 2024-06-14T22:33:32.224 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.224 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.224 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4232215213' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-06-14T22:33:32.224 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.224 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.224 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:33:32.224 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:33:32.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi137 to 4016M 2024-06-14T22:33:32.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4232215213' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-06-14T22:33:32.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:33:32.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:31 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:33:32.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:31 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi137 to 4016M 2024-06-14T22:33:32.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:31 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:31 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:31 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4232215213' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-06-14T22:33:32.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:31 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:31 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:32.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:31 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:33:32.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:31 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:33:32.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-06-14T22:33:33.008 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:33.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:33:33.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:33:33.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:33.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4232215213' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: osdmap e60: 8 total, 8 up, 8 in 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[29889]: mgrmap e20: y(active, since 4s), standbys: x 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4232215213' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: osdmap e60: 8 total, 8 up, 8 in 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:33:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:32 smithi086 ceph-mon[38904]: mgrmap e20: y(active, since 4s), standbys: x 2024-06-14T22:33:33.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:33:33.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:33:33.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:33.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:33:33.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:33:33.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4232215213' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-06-14T22:33:33.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: osdmap e60: 8 total, 8 up, 8 in 2024-06-14T22:33:33.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:33:33.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:33:33.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:33.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:32 smithi137 ceph-mon[32951]: mgrmap e20: y(active, since 4s), standbys: x 2024-06-14T22:33:34.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:33 smithi137 ceph-mon[32951]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:33:34.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:33 smithi137 ceph-mon[32951]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:33:34.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:33 smithi137 ceph-mon[32951]: Deploying daemon alertmanager.a on smithi086 2024-06-14T22:33:34.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:33 smithi137 ceph-mon[32951]: osdmap e61: 8 total, 8 up, 8 in 2024-06-14T22:33:34.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:33 smithi086 ceph-mon[29889]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:33:34.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:33 smithi086 ceph-mon[29889]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:33:34.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:33 smithi086 ceph-mon[29889]: Deploying daemon alertmanager.a on smithi086 2024-06-14T22:33:34.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:33 smithi086 ceph-mon[29889]: osdmap e61: 8 total, 8 up, 8 in 2024-06-14T22:33:34.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:33 smithi086 ceph-mon[38904]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:33:34.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:33 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:33:34.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:33 smithi086 ceph-mon[38904]: Deploying daemon alertmanager.a on smithi086 2024-06-14T22:33:34.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:33 smithi086 ceph-mon[38904]: osdmap e61: 8 total, 8 up, 8 in 2024-06-14T22:33:35.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:34 smithi086 ceph-mon[29889]: pgmap v8: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:35.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:34 smithi086 ceph-mon[29889]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:33:35.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:34 smithi086 ceph-mon[29889]: osdmap e62: 8 total, 8 up, 8 in 2024-06-14T22:33:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:34 smithi086 ceph-mon[38904]: pgmap v8: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:34 smithi086 ceph-mon[38904]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:33:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:34 smithi086 ceph-mon[38904]: osdmap e62: 8 total, 8 up, 8 in 2024-06-14T22:33:35.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:34 smithi137 ceph-mon[32951]: pgmap v8: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:35.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:34 smithi137 ceph-mon[32951]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:33:35.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:34 smithi137 ceph-mon[32951]: osdmap e62: 8 total, 8 up, 8 in 2024-06-14T22:33:36.891 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rbd' on pool 'rbd' 2024-06-14T22:33:37.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:36 smithi137 ceph-mon[32951]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 19 op/s 2024-06-14T22:33:37.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:36 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/920538471' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:33:37.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:36 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:33:37.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:36 smithi086 ceph-mon[29889]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 19 op/s 2024-06-14T22:33:37.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:36 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/920538471' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:33:37.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:36 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:33:37.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:36 smithi086 ceph-mon[38904]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 19 op/s 2024-06-14T22:33:37.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:36 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/920538471' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:33:37.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:36 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:33:37.857 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-14T22:33:37.857 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-14T22:33:37.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph mgr dump --format=json 2024-06-14T22:33:38.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:37 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:33:38.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:37 smithi086 ceph-mon[29889]: osdmap e63: 8 total, 8 up, 8 in 2024-06-14T22:33:38.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:37 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:33:38.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:37 smithi086 ceph-mon[38904]: osdmap e63: 8 total, 8 up, 8 in 2024-06-14T22:33:38.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:37 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:33:38.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:37 smithi137 ceph-mon[32951]: osdmap e63: 8 total, 8 up, 8 in 2024-06-14T22:33:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:39.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:38 smithi086 ceph-mon[29889]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-06-14T22:33:39.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:38 smithi086 ceph-mon[29889]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:33:39.122 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:38 smithi086 ceph-mon[29889]: Cluster is now healthy 2024-06-14T22:33:39.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:38 smithi086 ceph-mon[38904]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-06-14T22:33:39.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:38 smithi086 ceph-mon[38904]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:33:39.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:38 smithi086 ceph-mon[38904]: Cluster is now healthy 2024-06-14T22:33:39.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:38 smithi137 ceph-mon[32951]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-06-14T22:33:39.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:38 smithi137 ceph-mon[32951]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:33:39.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:38 smithi137 ceph-mon[32951]: Cluster is now healthy 2024-06-14T22:33:40.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:40 smithi137 ceph-mon[32951]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-14T22:33:40.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:33:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:33:41.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:40 smithi086 ceph-mon[29889]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-14T22:33:41.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:40 smithi086 ceph-mon[38904]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-14T22:33:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:33:43.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:42 smithi137 ceph-mon[32951]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 14 op/s 2024-06-14T22:33:43.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:42 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:33:43.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:43.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:42 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3746193624' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-14T22:33:43.310 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:42 smithi086 ceph-mon[29889]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 14 op/s 2024-06-14T22:33:43.310 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:42 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:33:43.310 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:43.310 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:42 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3746193624' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-14T22:33:43.311 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:42 smithi086 ceph-mon[38904]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 14 op/s 2024-06-14T22:33:43.311 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:42 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:33:43.311 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:43.311 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:42 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3746193624' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-14T22:33:43.579 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:33:43] ENGINE Bus STOPPING 2024-06-14T22:33:43.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:33:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T22:33:43.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:33:43] ENGINE Bus STOPPED 2024-06-14T22:33:43.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:33:43] ENGINE Bus STARTING 2024-06-14T22:33:43.947 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":20,"flags":0,"active_gid":14424,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":2254669699},{"type":"v1","addr":"172.21.15.86:6801","nonce":2254669699}]},"active_addr":"172.21.15.86:6801/2254669699","active_change":"2024-06-14T22:33:27.518569+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24319,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.86:8443/","prometheus":"http://172.21.15.86:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":418538598}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3107951276}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3167284010}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":1167860215}]}]} 2024-06-14T22:33:43.950 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-14T22:33:43.950 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-14T22:33:43.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-14T22:33:44.201 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:44.323 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:33:43] ENGINE Serving on http://:::9283 2024-06-14T22:33:44.323 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:33:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:33:43] ENGINE Bus STARTED 2024-06-14T22:33:44.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[38904]: Deploying daemon grafana.a on smithi137 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:44.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-14T22:33:44.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-14T22:33:44.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:44 smithi086 ceph-mon[29889]: Deploying daemon grafana.a on smithi137 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:44.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:44 smithi137 ceph-mon[32951]: Deploying daemon grafana.a on smithi137 2024-06-14T22:33:46.079 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:33:46.080 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":63,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T22:33:36.889267+0000","last_up_change":"2024-06-14T22:32:58.884084+0000","last_in_change":"2024-06-14T22:32:42.063674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-14T22:33:31.126149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6803","nonce":2092557056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6805","nonce":2092557056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6809","nonce":2092557056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6807","nonce":2092557056}]},"public_addr":"172.21.15.86:6803/2092557056","cluster_addr":"172.21.15.86:6805/2092557056","heartbeat_back_addr":"172.21.15.86:6809/2092557056","heartbeat_front_addr":"172.21.15.86:6807/2092557056","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6811","nonce":796510430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6813","nonce":796510430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6817","nonce":796510430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6815","nonce":796510430}]},"public_addr":"172.21.15.86:6811/796510430","cluster_addr":"172.21.15.86:6813/796510430","heartbeat_back_addr":"172.21.15.86:6817/796510430","heartbeat_front_addr":"172.21.15.86:6815/796510430","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6819","nonce":3531892667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6821","nonce":3531892667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6825","nonce":3531892667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6823","nonce":3531892667}]},"public_addr":"172.21.15.86:6819/3531892667","cluster_addr":"172.21.15.86:6821/3531892667","heartbeat_back_addr":"172.21.15.86:6825/3531892667","heartbeat_front_addr":"172.21.15.86:6823/3531892667","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6827","nonce":287823501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6829","nonce":287823501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6833","nonce":287823501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6831","nonce":287823501}]},"public_addr":"172.21.15.86:6827/287823501","cluster_addr":"172.21.15.86:6829/287823501","heartbeat_back_addr":"172.21.15.86:6833/287823501","heartbeat_front_addr":"172.21.15.86:6831/287823501","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6809","nonce":2122959324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6811","nonce":2122959324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6815","nonce":2122959324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6813","nonce":2122959324}]},"public_addr":"172.21.15.137:6809/2122959324","cluster_addr":"172.21.15.137:6811/2122959324","heartbeat_back_addr":"172.21.15.137:6815/2122959324","heartbeat_front_addr":"172.21.15.137:6813/2122959324","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6817","nonce":2398798362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6819","nonce":2398798362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6823","nonce":2398798362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6821","nonce":2398798362}]},"public_addr":"172.21.15.137:6817/2398798362","cluster_addr":"172.21.15.137:6819/2398798362","heartbeat_back_addr":"172.21.15.137:6823/2398798362","heartbeat_front_addr":"172.21.15.137:6821/2398798362","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3941467574":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/2741142752":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/1759721543":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3724463773":"2024-06-15T22:33:27.518416+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T22:33:46.794 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-14T22:33:46.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-14T22:33:46.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:46 smithi137 ceph-mon[32951]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:46.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:46 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2785011457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:47.042 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:47.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:46 smithi086 ceph-mon[29889]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:47.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:46 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2785011457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:47.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:46 smithi086 ceph-mon[38904]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:47.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:46 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2785011457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:48.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:48 smithi086 ceph-mon[38904]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:48.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:48 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:48.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:48 smithi086 ceph-mon[29889]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:48.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:48 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:48.972 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:33:48.972 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":63,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T22:33:36.889267+0000","last_up_change":"2024-06-14T22:32:58.884084+0000","last_in_change":"2024-06-14T22:32:42.063674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-14T22:33:31.126149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6803","nonce":2092557056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6805","nonce":2092557056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6809","nonce":2092557056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6807","nonce":2092557056}]},"public_addr":"172.21.15.86:6803/2092557056","cluster_addr":"172.21.15.86:6805/2092557056","heartbeat_back_addr":"172.21.15.86:6809/2092557056","heartbeat_front_addr":"172.21.15.86:6807/2092557056","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6811","nonce":796510430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6813","nonce":796510430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6817","nonce":796510430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6815","nonce":796510430}]},"public_addr":"172.21.15.86:6811/796510430","cluster_addr":"172.21.15.86:6813/796510430","heartbeat_back_addr":"172.21.15.86:6817/796510430","heartbeat_front_addr":"172.21.15.86:6815/796510430","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6819","nonce":3531892667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6821","nonce":3531892667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6825","nonce":3531892667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6823","nonce":3531892667}]},"public_addr":"172.21.15.86:6819/3531892667","cluster_addr":"172.21.15.86:6821/3531892667","heartbeat_back_addr":"172.21.15.86:6825/3531892667","heartbeat_front_addr":"172.21.15.86:6823/3531892667","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6827","nonce":287823501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6829","nonce":287823501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6833","nonce":287823501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6831","nonce":287823501}]},"public_addr":"172.21.15.86:6827/287823501","cluster_addr":"172.21.15.86:6829/287823501","heartbeat_back_addr":"172.21.15.86:6833/287823501","heartbeat_front_addr":"172.21.15.86:6831/287823501","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6809","nonce":2122959324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6811","nonce":2122959324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6815","nonce":2122959324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6813","nonce":2122959324}]},"public_addr":"172.21.15.137:6809/2122959324","cluster_addr":"172.21.15.137:6811/2122959324","heartbeat_back_addr":"172.21.15.137:6815/2122959324","heartbeat_front_addr":"172.21.15.137:6813/2122959324","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6817","nonce":2398798362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6819","nonce":2398798362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6823","nonce":2398798362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6821","nonce":2398798362}]},"public_addr":"172.21.15.137:6817/2398798362","cluster_addr":"172.21.15.137:6819/2398798362","heartbeat_back_addr":"172.21.15.137:6823/2398798362","heartbeat_front_addr":"172.21.15.137:6821/2398798362","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3941467574":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/2741142752":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/1759721543":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3724463773":"2024-06-15T22:33:27.518416+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T22:33:48.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:48 smithi137 ceph-mon[32951]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:48.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:48 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:49.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-14T22:33:49.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-14T22:33:49.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-14T22:33:49.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-14T22:33:49.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-14T22:33:49.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-14T22:33:49.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-14T22:33:49.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-14T22:33:49.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:49 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2964339089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:49.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:49 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2964339089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:49.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:49 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2964339089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:33:50.113 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:50.113 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:50.114 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:50.116 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:50.120 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:50.121 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:50.123 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:50.123 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:33:50.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:50 smithi086 ceph-mon[29889]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:50.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:50 smithi086 ceph-mon[38904]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:50.988 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:33:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:33:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:33:50.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:50 smithi137 ceph-mon[32951]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:52.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:52 smithi137 ceph-mon[32951]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:53.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:52 smithi086 ceph-mon[29889]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:53.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:52 smithi086 ceph-mon[38904]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:53.525 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:33:53 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[69600]: ts=2024-06-14T22:33:53.338Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003499091s 2024-06-14T22:33:54.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:54 smithi086 ceph-mon[29889]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:54.714 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:54 smithi086 ceph-mon[38904]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:54.937 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:54 smithi137 ceph-mon[32951]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:56.400 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 podman[64232]: 2024-06-14 22:33:56.298104865 +0000 UTC m=+0.123962233 container create a38b34e5bca6d227bf29d808fc97fc787503cedb7063d9d9999f35eb9a4fc078 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, name=ubi8, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.tags=base rhel8, io.buildah.version=1.32.0, distribution-scope=public, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-06-14T22:33:56.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-mon[32951]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:56.748 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 podman[64232]: 2024-06-14 22:33:56.501066368 +0000 UTC m=+0.326923708 container init a38b34e5bca6d227bf29d808fc97fc787503cedb7063d9d9999f35eb9a4fc078 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, vendor=Red Hat, Inc., description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.buildah.version=1.32.0, name=ubi8, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-type=git, version=8.5, maintainer=Paul Cuzner ) 2024-06-14T22:33:56.748 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 podman[64232]: 2024-06-14 22:33:56.506308734 +0000 UTC m=+0.332166059 container start a38b34e5bca6d227bf29d808fc97fc787503cedb7063d9d9999f35eb9a4fc078 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-type=git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.buildah.version=1.32.0, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-06-14T22:33:56.748 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 bash[64232]: a38b34e5bca6d227bf29d808fc97fc787503cedb7063d9d9999f35eb9a4fc078 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 systemd[1]: Started Ceph grafana.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659416189Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659592345Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659612812Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659624564Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659634397Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659643957Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659652603Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659660935Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659670605Z level=info msg="Path Home" path=/usr/share/grafana 2024-06-14T22:33:56.749 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659680095Z level=info msg="Path Data" path=/var/lib/grafana 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659689113Z level=info msg="Path Logs" path=/var/log/grafana 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659698263Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659708582Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=settings t=2024-06-14T22:33:56.659719339Z level=info msg="App mode production" 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=sqlstore t=2024-06-14T22:33:56.65978339Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=sqlstore t=2024-06-14T22:33:56.659807564Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.660956329Z level=info msg="Starting DB migrations" 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.661050782Z level=info msg="Executing migration" id="create migration_log table" 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.661568396Z level=info msg="Executing migration" id="create user table" 2024-06-14T22:33:56.750 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.662634413Z level=info msg="Executing migration" id="add unique index user.login" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.663584012Z level=info msg="Executing migration" id="add unique index user.email" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.664030095Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.664623306Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.664992469Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.66568882Z level=info msg="Executing migration" id="create user table v2" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.666085329Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.666488271Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.666900677Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.667279601Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.667611879Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.66799851Z level=info msg="Executing migration" id="Update user table charset" 2024-06-14T22:33:56.751 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.668305152Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.668697676Z level=info msg="Executing migration" id="Add missing user data" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.669069866Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.669484519Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.669859803Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.670290042Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.671435916Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.67192361Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.672407008Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.672841661Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.673294653Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-06-14T22:33:56.752 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.673739457Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-06-14T22:33:56.753 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.674086931Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-06-14T22:33:56.753 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.674481293Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-06-14T22:33:56.753 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.674860347Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-06-14T22:33:56.753 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.675265163Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.675666355Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.67633792Z level=info msg="Executing migration" id="create temp_user v2" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.676765161Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.677182279Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.677593869Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.677976893Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.678382172Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.678708972Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.679080591Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-06-14T22:33:56.754 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.679437535Z level=info msg="Executing migration" id="create star table" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.679791921Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.680209516Z level=info msg="Executing migration" id="create org table v1" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.680606714Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.681346228Z level=info msg="Executing migration" id="create org_user table v1" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.681749379Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.682176803Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.682595669Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.683045031Z level=info msg="Executing migration" id="Update org table charset" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.683857056Z level=info msg="Executing migration" id="Update org_user table charset" 2024-06-14T22:33:56.755 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.684164371Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.686956368Z level=info msg="Executing migration" id="create dashboard table" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.687344593Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.68970693Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.692362586Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.69276079Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.695736256Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.698361416Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.701106559Z level=info msg="Executing migration" id="create dashboard v2" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.701521975Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.704373977Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-06-14T22:33:56.756 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.707174608Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.709766629Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.710203157Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.713001821Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.715836908Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.718477181Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.719071541Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.721945462Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.724604093Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.727149595Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.727763425Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-06-14T22:33:56.757 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.728665254Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.729534094Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.730259754Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.730769223Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.731623427Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.732466377Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.733349562Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.733797362Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.734771183Z level=info msg="Executing migration" id="Update dashboard title length" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.735458709Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.736277213Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-06-14T22:33:56.758 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.736691617Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.737612344Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.738474549Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.739334257Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.740224399Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.741047278Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.741897013Z level=info msg="Executing migration" id="Add check_sum column" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.742780582Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.743688894Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.744018271Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.744882115Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.745711855Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-06-14T22:33:56.759 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.746639876Z level=info msg="Executing migration" id="create data_source table" 2024-06-14T22:33:56.760 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.747084653Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-06-14T22:33:56.760 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.748041827Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-06-14T22:33:56.760 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.748755548Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-06-14T22:33:56.760 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.749748449Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-06-14T22:33:56.760 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.750253611Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-06-14T22:33:56.760 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.752853293Z level=info msg="Executing migration" id="create data_source table v2" 2024-06-14T22:33:56.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:56 smithi086 ceph-mon[29889]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:56.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:56 smithi086 ceph-mon[38904]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:56.999 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.75573787Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-06-14T22:33:56.999 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.75856395Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-06-14T22:33:56.999 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.759013838Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-06-14T22:33:56.999 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.761727995Z level=info msg="Executing migration" id="Add column with_credentials" 2024-06-14T22:33:56.999 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.764593699Z level=info msg="Executing migration" id="Add secure json data column" 2024-06-14T22:33:56.999 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.767287765Z level=info msg="Executing migration" id="Update data_source table charset" 2024-06-14T22:33:56.999 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.767636585Z level=info msg="Executing migration" id="Update initial version to 1" 2024-06-14T22:33:56.999 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.770559866Z level=info msg="Executing migration" id="Add read_only data column" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.771176732Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.77329315Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.776113936Z level=info msg="Executing migration" id="Add uid column" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.776671115Z level=info msg="Executing migration" id="Update uid value" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.779318658Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.779801057Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.782217942Z level=info msg="Executing migration" id="create api_key table" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.785148419Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.78562608Z level=info msg="Executing migration" id="add index api_key.key" 2024-06-14T22:33:57.000 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.788566023Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-06-14T22:33:57.001 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.791071436Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-06-14T22:33:57.002 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.793789296Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-06-14T22:33:57.002 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.796290835Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-06-14T22:33:57.002 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.798847751Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-06-14T22:33:57.002 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.801612938Z level=info msg="Executing migration" id="create api_key table v2" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.804119552Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.806893134Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.809719041Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.812179281Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.814858345Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.817503536Z level=info msg="Executing migration" id="Update api_key table charset" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.820169207Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.822799485Z level=info msg="Executing migration" id="Add service account foreign key" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.825272218Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.827967715Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-06-14T22:33:57.003 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.828616042Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.829510927Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.830325585Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.831209992Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.832017449Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.83277832Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.833701615Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.834610835Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.835574766Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.836328773Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.83713402Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.837960344Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-06-14T22:33:57.004 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.838807749Z level=info msg="Executing migration" id="create quota table v1" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.839656257Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.840645589Z level=info msg="Executing migration" id="Update quota table charset" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.841441606Z level=info msg="Executing migration" id="create plugin_setting table" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.842162966Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.843182768Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.843882771Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.844941663Z level=info msg="Executing migration" id="create session table" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.845707652Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.846507516Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.846838956Z level=info msg="Executing migration" id="create playlist table v2" 2024-06-14T22:33:57.005 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.847393916Z level=info msg="Executing migration" id="create playlist item table v2" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.848396281Z level=info msg="Executing migration" id="Update playlist table charset" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.849127846Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.850027463Z level=info msg="Executing migration" id="drop preferences table v2" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.850966683Z level=info msg="Executing migration" id="drop preferences table v3" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.851753822Z level=info msg="Executing migration" id="create preferences table v3" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.852695795Z level=info msg="Executing migration" id="Update preferences table charset" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.853464543Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.854458123Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.855257954Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-06-14T22:33:57.006 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.856049937Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.856959348Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.859725569Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.862660351Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.865503968Z level=info msg="Executing migration" id="create alert table v1" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.868409804Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.871141499Z level=info msg="Executing migration" id="add index alert state" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.873864298Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.876834455Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.879783661Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.882496734Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.885911196Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-06-14T22:33:57.007 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.888587823Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.891383325Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.89433675Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.897151439Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.899848406Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.902849937Z level=info msg="Executing migration" id="Add column is_default" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.903685968Z level=info msg="Executing migration" id="Add column frequency" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.906422439Z level=info msg="Executing migration" id="Add column send_reminder" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.90886189Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.911534778Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.91203219Z level=info msg="Executing migration" id="Update alert table charset" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.914779686Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-06-14T22:33:57.008 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.917429151Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.92008687Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.922720397Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.925344415Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.928066126Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.930886955Z level=info msg="Executing migration" id="Add for to alert table" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.931660413Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.934243614Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.93684899Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.939610322Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.940137115Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.943081026Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-06-14T22:33:57.009 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.945849999Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.948396965Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.948816865Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.9513257Z level=info msg="Executing migration" id="create annotation table v5" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.95410404Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.956682998Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.957152557Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.959736078Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.962400369Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.964909008Z level=info msg="Executing migration" id="Update annotation table charset" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.965223657Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.967836262Z level=info msg="Executing migration" id="Drop category_id index" 2024-06-14T22:33:57.010 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.968266675Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.970485909Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.973151708Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.975737219Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.978302308Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.980900278Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.981658029Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.982198712Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.983079751Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.983794053Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.984717278Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.985469992Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-06-14T22:33:57.011 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.986300858Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.987134154Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.98811992Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.988533887Z level=info msg="Executing migration" id="Add epoch_end column" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.989364529Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.990257629Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.991013428Z level=info msg="Executing migration" id="Move region to single row" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.991399165Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.992238943Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.992665534Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.993094874Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.993927747Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-06-14T22:33:57.012 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.994337658Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.995250923Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.995678978Z level=info msg="Executing migration" id="create test_data table" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.996081002Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.996466141Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.996883999Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.997726079Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.998620352Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.999167851Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.999474242Z level=info msg="Executing migration" id="create team table" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:56.999890111Z level=info msg="Executing migration" id="add index team.org_id" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.000758926Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-06-14T22:33:57.013 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.001560215Z level=info msg="Executing migration" id="create team member table" 2024-06-14T22:33:57.014 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.002399653Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-06-14T22:33:57.014 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.002863309Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-06-14T22:33:57.014 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.003712048Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-06-14T22:33:57.014 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.004560776Z level=info msg="Executing migration" id="Add column email to team table" 2024-06-14T22:33:57.014 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.005401189Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-06-14T22:33:57.014 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.006137947Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-06-14T22:33:57.250 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.008858357Z level=info msg="Executing migration" id="create dashboard acl table" 2024-06-14T22:33:57.250 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.011565462Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.014374452Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.014801331Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.015238641Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.018030454Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.020643695Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.023565545Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.026051512Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.029013661Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.031648038Z level=info msg="Executing migration" id="create tag table" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.034528923Z level=info msg="Executing migration" id="add index tag.key_value" 2024-06-14T22:33:57.251 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.037112663Z level=info msg="Executing migration" id="create login attempt table" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.039994643Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.042602794Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.045311959Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.048149681Z level=info msg="Executing migration" id="create login_attempt v2" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.050978428Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.053569791Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.056057266Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.058746941Z level=info msg="Executing migration" id="create user auth table" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.06150788Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.063964243Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.066621691Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-06-14T22:33:57.252 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.069021829Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.071616135Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.074273774Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.076883652Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.077354025Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.078116983Z level=info msg="Executing migration" id="create server_lock table" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.078516134Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.078966906Z level=info msg="Executing migration" id="create user auth token table" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.079409405Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.079851899Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.080288122Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.080723549Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-06-14T22:33:57.253 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.081543551Z level=info msg="Executing migration" id="create cache_data table" 2024-06-14T22:33:57.254 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.082004562Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-06-14T22:33:57.255 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.082463425Z level=info msg="Executing migration" id="create short_url table v1" 2024-06-14T22:33:57.255 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.082881017Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-06-14T22:33:57.255 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.085541821Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-06-14T22:33:57.255 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.085857407Z level=info msg="Executing migration" id="delete alert_definition table" 2024-06-14T22:33:57.255 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.086190615Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-06-14T22:33:57.255 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.086623034Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-06-14T22:33:57.255 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.089254975Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.089695304Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.08996183Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.090362795Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.090745649Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.091181942Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.09160429Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.092416486Z level=info msg="Executing migration" id="drop alert_definition table" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.092818056Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.093127436Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.09353543Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.09394589Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-06-14T22:33:57.256 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.094356094Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.094611466Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.09498811Z level=info msg="Executing migration" id="create alert_instance table" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.095390717Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.095778084Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.096213866Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.096984743Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.097399963Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.097776552Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.101234925Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.104588639Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.105043984Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-06-14T22:33:57.257 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.105460586Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.106269184Z level=info msg="Executing migration" id="create alert_rule table" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.106665488Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.107084137Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.107489513Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.107880179Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.108174851Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.108964639Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.109762123Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.110577111Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.110977075Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.113839832Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-06-14T22:33:57.258 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.116365702Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.117214209Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.117626709Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.120279551Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.122844076Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.125398883Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.12621741Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.127167447Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.127872838Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.12886153Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.129829732Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-06-14T22:33:57.259 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.130668468Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.131559846Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.132430504Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.133302742Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.134302876Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.135006533Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.135895684Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.136778896Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.137676158Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.138481615Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.139428493Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.140226835Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-06-14T22:33:57.260 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.141207635Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.142100268Z level=info msg="Executing migration" id="create provenance_type table" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.143021032Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.144021973Z level=info msg="Executing migration" id="create alert_image table" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.14488744Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.145629909Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.146432659Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.146890842Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.147846133Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.148079875Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.148559511Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.149477283Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-06-14T22:33:57.261 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.150115752Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-06-14T22:33:57.262 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.150229535Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-06-14T22:33:57.262 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.150341627Z level=info msg="alerts found to migrate" alerts=0 2024-06-14T22:33:57.262 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.150741146Z level=info msg="Executing migration" id="create library_element table v1" 2024-06-14T22:33:57.262 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.151712577Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-06-14T22:33:57.262 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.152466503Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-06-14T22:33:57.262 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.153487245Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-06-14T22:33:57.262 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.154180791Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-06-14T22:33:57.262 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.155077861Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.157843438Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.160797488Z level=info msg="Executing migration" id="create data_keys table" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.16370171Z level=info msg="Executing migration" id="create secrets table" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.166544629Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.170813565Z level=info msg="Executing migration" id="add name column into data_keys" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.173418134Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.176312057Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.180564297Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.184736925Z level=info msg="Executing migration" id="create kv_store table v1" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.187517958Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.190331393Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.193385882Z level=info msg="Executing migration" id="create permission table" 2024-06-14T22:33:57.263 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.196180161Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.199090244Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.20204613Z level=info msg="Executing migration" id="create role table" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.204782346Z level=info msg="Executing migration" id="add column display_name" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.207629398Z level=info msg="Executing migration" id="add column group_name" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.210299983Z level=info msg="Executing migration" id="add index role.org_id" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.213019989Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.215911363Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.21836825Z level=info msg="Executing migration" id="create team role table" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.221030055Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.223795407Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.226254858Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-06-14T22:33:57.264 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.228959938Z level=info msg="Executing migration" id="create user role table" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.231445179Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.234308279Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.236944519Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.239665396Z level=info msg="Executing migration" id="create builtin role table" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.242102487Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.244805044Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.24750286Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.250195118Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.252559922Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.255041493Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.255848852Z level=info msg="Executing migration" id="add unique index role.uid" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.256740171Z level=info msg="Executing migration" id="create seed assignment table" 2024-06-14T22:33:57.265 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.257569282Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-06-14T22:33:57.266 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.258425285Z level=info msg="Executing migration" id="add column hidden to role table" 2024-06-14T22:33:57.647 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.259418886Z level=info msg="Executing migration" id="create query_history table v1" 2024-06-14T22:33:57.647 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.260197875Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-06-14T22:33:57.647 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.261066408Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-06-14T22:33:57.647 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.261951858Z level=info msg="Executing migration" id="teams permissions migration" 2024-06-14T22:33:57.647 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.262787863Z level=info msg="Executing migration" id="dashboard permissions" 2024-06-14T22:33:57.647 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.26364273Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-06-14T22:33:57.647 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.264495722Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.265400719Z level=info msg="Executing migration" id="alerting notification permissions" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.266219478Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.267022382Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.26800509Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.268993574Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.269834657Z level=info msg="Executing migration" id="create correlation table v1" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.270759457Z level=info msg="Executing migration" id="add index correlations.uid" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.271711783Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.27249419Z level=info msg="Executing migration" id="add correlation config column" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.273529232Z level=info msg="Executing migration" id="create entity_events table" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.274425136Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-06-14T22:33:57.648 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.275392624Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-06-14T22:33:57.649 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.275601919Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-06-14T22:33:57.649 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.27612659Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-06-14T22:33:57.649 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.276221379Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-06-14T22:33:57.649 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.277088976Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-06-14T22:33:57.649 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.277871554Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.278853951Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.279706528Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.280475188Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.281315493Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.282170052Z level=info msg="Executing migration" id="Drop public config table" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.283149593Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.283884941Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.284847106Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.285640684Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.288573476Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.291669837Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-06-14T22:33:57.650 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.29468005Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.297455379Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.300299671Z level=info msg="Executing migration" id="add share column" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.30307818Z level=info msg="Executing migration" id="create default alerting folders" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.306099447Z level=info msg="Executing migration" id="create file table" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.308891102Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.31199714Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.314750743Z level=info msg="Executing migration" id="create file_meta table" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.317531601Z level=info msg="Executing migration" id="file table idx: path key" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.320043641Z level=info msg="Executing migration" id="set path collation in file table" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.323073573Z level=info msg="Executing migration" id="managed permissions migration" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.326025565Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-06-14T22:33:57.651 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.328752978Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.331627207Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.334166224Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.337042681Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.339562678Z level=info msg="Executing migration" id="update group index for alert rules" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.342310367Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.344977125Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.347567033Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.348703768Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.349737069Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.350588431Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.360679744Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-06-14T22:33:57.652 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.3614571Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.362479991Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.365879477Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.366822801Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.367574525Z level=info msg="Executing migration" id="create folder table" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.368476415Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.369183139Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.36964237Z level=info msg="Executing migration" id="Update folder title length" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.372403478Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=migrator t=2024-06-14T22:33:57.372987422Z level=info msg="migrations completed" performed=485 skipped=0 duration=711.964491ms 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=sqlstore t=2024-06-14T22:33:57.373378146Z level=info msg="Created default organization" 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=plugin.loader t=2024-06-14T22:33:57.430265889Z level=info msg="Plugin registered" pluginID=input 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=plugin.loader t=2024-06-14T22:33:57.457410005Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-06-14T22:33:57.653 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=plugin.loader t=2024-06-14T22:33:57.457430576Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-06-14T22:33:57.654 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=secrets t=2024-06-14T22:33:57.457461269Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-06-14T22:33:57.654 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=query_data t=2024-06-14T22:33:57.459082907Z level=info msg="Query Service initialization" 2024-06-14T22:33:57.654 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=live.push_http t=2024-06-14T22:33:57.46134823Z level=info msg="Live Push Gateway initialization" 2024-06-14T22:33:57.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[29889]: Deploying daemon node-exporter.a on smithi086 2024-06-14T22:33:57.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:33:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[38904]: Deploying daemon node-exporter.a on smithi086 2024-06-14T22:33:57.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:33:57.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:57 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-mon[32951]: Deploying daemon node-exporter.a on smithi086 2024-06-14T22:33:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:33:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:57 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:33:58.489 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=infra.usagestats.collector t=2024-06-14T22:33:58.1676931Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-06-14T22:33:58.489 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=provisioning.datasources t=2024-06-14T22:33:58.168042726Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-06-14T22:33:58.489 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=provisioning.datasources t=2024-06-14T22:33:58.199459375Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-06-14T22:33:58.489 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=provisioning.datasources t=2024-06-14T22:33:58.231079866Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-06-14T22:33:58.489 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=provisioning.alerting t=2024-06-14T22:33:58.252694689Z level=info msg="starting to provision alerting" 2024-06-14T22:33:58.489 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=provisioning.alerting t=2024-06-14T22:33:58.252713865Z level=info msg="finished to provision alerting" 2024-06-14T22:33:58.489 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=ngalert.state.manager t=2024-06-14T22:33:58.252811553Z level=info msg="Warming state cache for startup" 2024-06-14T22:33:58.489 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=grafanaStorageLogger t=2024-06-14T22:33:58.252865733Z level=info msg="storage starting" 2024-06-14T22:33:58.489 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=http.server t=2024-06-14T22:33:58.255295223Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-06-14T22:33:58.490 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=sqlstore.transactions t=2024-06-14T22:33:58.263159549Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-06-14T22:33:58.490 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=ngalert.state.manager t=2024-06-14T22:33:58.263286808Z level=info msg="State cache has been initialized" states=0 duration=10.47358ms 2024-06-14T22:33:58.490 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=ticker t=2024-06-14T22:33:58.263362464Z level=info msg=starting first_tick=2024-06-14T22:34:00Z 2024-06-14T22:33:58.490 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=ngalert.multiorg.alertmanager t=2024-06-14T22:33:58.263348581Z level=info msg="starting MultiOrg Alertmanager" 2024-06-14T22:33:58.490 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=sqlstore.transactions t=2024-06-14T22:33:58.297233468Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-06-14T22:33:58.490 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=sqlstore.transactions t=2024-06-14T22:33:58.337805685Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-06-14T22:33:58.490 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[64247]: logger=sqlstore.transactions t=2024-06-14T22:33:58.349133162Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-06-14T22:33:58.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:33:58 smithi086 ceph-mon[29889]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:58.938 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:33:58 smithi086 ceph-mon[38904]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:58.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:33:58 smithi137 ceph-mon[32951]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:33:59.797 INFO:teuthology.orchestra.run.smithi086.stdout:163208757272 2024-06-14T22:33:59.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-14T22:34:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:34359738414 2024-06-14T22:34:00.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-14T22:34:00.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:00 smithi137 ceph-mon[32951]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:00.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:34:00 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:34:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:34:01.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:00 smithi086 ceph-mon[29889]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:01.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:00 smithi086 ceph-mon[38904]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:02.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:02 smithi086 ceph-mon[29889]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:02.928 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:02 smithi086 ceph-mon[38904]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:02.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:02 smithi137 ceph-mon[32951]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:03.328 INFO:teuthology.orchestra.run.smithi086.stdout:231928233998 2024-06-14T22:34:03.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-14T22:34:04.629 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:04 smithi086 bash[71738]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-06-14T22:34:04.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:04 smithi086 ceph-mon[29889]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:04.967 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:04 smithi086 ceph-mon[38904]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:04.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:04 smithi137 ceph-mon[32951]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:77309411368 2024-06-14T22:34:04.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-14T22:34:05.330 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:04 smithi086 bash[71738]: Getting image source signatures 2024-06-14T22:34:05.330 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:04 smithi086 bash[71738]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-06-14T22:34:05.330 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:04 smithi086 bash[71738]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-06-14T22:34:05.330 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:04 smithi086 bash[71738]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-06-14T22:34:05.825 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:05.860 INFO:teuthology.orchestra.run.smithi086.stdout:107374182434 2024-06-14T22:34:05.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-14T22:34:06.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:06 smithi086 ceph-mon[29889]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:06.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:06 smithi086 ceph-mon[38904]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:06.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:06 smithi137 ceph-mon[32951]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:07.216 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:07 smithi086 bash[71738]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-06-14T22:34:07.567 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:07 smithi086 bash[71738]: Writing manifest to image destination 2024-06-14T22:34:08.080 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:07 smithi086 podman[71738]: 2024-06-14 22:34:07.802315571 +0000 UTC m=+3.926808062 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-06-14T22:34:08.525 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:08 smithi086 podman[71738]: 2024-06-14 22:34:08.429781909 +0000 UTC m=+4.554274389 container create d83ddcfcdc25409dfba8b2471316d7907ab387c84b6cdbc17ae8059a0e314b52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:08.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:08 smithi137 ceph-mon[32951]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:09.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:08 smithi086 ceph-mon[38904]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:08 smithi086 ceph-mon[29889]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:09.447 INFO:teuthology.orchestra.run.smithi086.stdout:197568495636 2024-06-14T22:34:09.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-14T22:34:09.792 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 podman[71738]: 2024-06-14 22:34:09.587178012 +0000 UTC m=+5.711670516 container init d83ddcfcdc25409dfba8b2471316d7907ab387c84b6cdbc17ae8059a0e314b52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:09.792 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 podman[71738]: 2024-06-14 22:34:09.593168717 +0000 UTC m=+5.717661202 container start d83ddcfcdc25409dfba8b2471316d7907ab387c84b6cdbc17ae8059a0e314b52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:09.792 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.597Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-06-14T22:34:09.792 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.597Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-06-14T22:34:09.792 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.597Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-06-14T22:34:09.792 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.597Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-14T22:34:09.792 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-06-14T22:34:09.792 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-06-14T22:34:09.792 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-06-14T22:34:09.793 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=arp 2024-06-14T22:34:09.793 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=bcache 2024-06-14T22:34:09.793 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=bonding 2024-06-14T22:34:09.793 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=btrfs 2024-06-14T22:34:09.793 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=conntrack 2024-06-14T22:34:09.793 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=cpu 2024-06-14T22:34:09.793 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=diskstats 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=dmi 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=edac 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=entropy 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=filefd 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=filesystem 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=hwmon 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=infiniband 2024-06-14T22:34:09.794 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=ipvs 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=loadavg 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=mdadm 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=meminfo 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=netclass 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=netdev 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=netstat 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=nfs 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=nfsd 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=nvme 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=os 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=pressure 2024-06-14T22:34:09.795 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=rapl 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=schedstat 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=selinux 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=sockstat 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=softnet 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=stat 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=tapestats 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=textfile 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=time 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=uname 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=vmstat 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=xfs 2024-06-14T22:34:09.796 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.598Z caller=node_exporter.go:117 level=info collector=zfs 2024-06-14T22:34:09.797 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.599Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-06-14T22:34:09.797 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[72198]: ts=2024-06-14T22:34:09.599Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-06-14T22:34:09.797 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 bash[71738]: d83ddcfcdc25409dfba8b2471316d7907ab387c84b6cdbc17ae8059a0e314b52 2024-06-14T22:34:09.797 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:34:09 smithi086 systemd[1]: Started Ceph node-exporter.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:34:10.734 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:10 smithi137 bash[64528]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-06-14T22:34:10.988 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:34:10 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:34:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:34:10.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:10 smithi137 ceph-mon[32951]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:10.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:10 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:10.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:10 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:10.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:10 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:10.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:10 smithi137 ceph-mon[32951]: Deploying daemon node-exporter.b on smithi137 2024-06-14T22:34:10.989 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:10 smithi137 bash[64528]: Getting image source signatures 2024-06-14T22:34:10.990 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:10 smithi137 bash[64528]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-06-14T22:34:10.990 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:10 smithi137 bash[64528]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-06-14T22:34:10.990 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:10 smithi137 bash[64528]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-06-14T22:34:11.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[29889]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:11.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:11.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:11.051 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:11.051 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[29889]: Deploying daemon node-exporter.b on smithi137 2024-06-14T22:34:11.051 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[38904]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:11.051 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:11.051 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:11.051 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:11.051 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:10 smithi086 ceph-mon[38904]: Deploying daemon node-exporter.b on smithi137 2024-06-14T22:34:11.371 INFO:teuthology.orchestra.run.smithi086.stdout:128849018910 2024-06-14T22:34:11.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-14T22:34:11.956 INFO:teuthology.orchestra.run.smithi086.stdout:55834574893 2024-06-14T22:34:11.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-14T22:34:12.738 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:12 smithi137 bash[64528]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-06-14T22:34:12.739 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:12 smithi137 bash[64528]: Writing manifest to image destination 2024-06-14T22:34:12.894 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:12 smithi086 ceph-mon[29889]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:12.894 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:12 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:12.895 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:12 smithi086 ceph-mon[38904]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:12.895 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:12 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:12.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:12 smithi137 ceph-mon[32951]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:12.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:12 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:12.989 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:12 smithi137 podman[64528]: 2024-06-14 22:34:12.718599381 +0000 UTC m=+2.176236882 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-06-14T22:34:12.989 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:12 smithi137 podman[64528]: 2024-06-14 22:34:12.825325641 +0000 UTC m=+2.282963143 container create 6082a0b8781a5b874d805cfaeb6d84b6331b30f22b2994833b8fa5ec9e4c9f5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T22:34:13.068 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:13.159 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:13.159 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:13.352 INFO:teuthology.orchestra.run.smithi086.stdout:163208757275 2024-06-14T22:34:13.489 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 podman[64528]: 2024-06-14 22:34:13.052346153 +0000 UTC m=+2.509983654 container init 6082a0b8781a5b874d805cfaeb6d84b6331b30f22b2994833b8fa5ec9e4c9f5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T22:34:13.489 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 podman[64528]: 2024-06-14 22:34:13.058462421 +0000 UTC m=+2.516099923 container start 6082a0b8781a5b874d805cfaeb6d84b6331b30f22b2994833b8fa5ec9e4c9f5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T22:34:13.489 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.062Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-06-14T22:34:13.489 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.062Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=node_exporter.go:117 level=info collector=arp 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=node_exporter.go:117 level=info collector=bcache 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=node_exporter.go:117 level=info collector=bonding 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=node_exporter.go:117 level=info collector=btrfs 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.063Z caller=node_exporter.go:117 level=info collector=conntrack 2024-06-14T22:34:13.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=cpu 2024-06-14T22:34:13.491 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-06-14T22:34:13.491 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=diskstats 2024-06-14T22:34:13.491 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=dmi 2024-06-14T22:34:13.491 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=edac 2024-06-14T22:34:13.491 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=entropy 2024-06-14T22:34:13.491 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=filefd 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=filesystem 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=hwmon 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=infiniband 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=ipvs 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=loadavg 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=mdadm 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=meminfo 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=netclass 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=netdev 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=netstat 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=nfs 2024-06-14T22:34:13.492 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=nfsd 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=nvme 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=os 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=pressure 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=rapl 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=schedstat 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=selinux 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=sockstat 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=softnet 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=stat 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=tapestats 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=textfile 2024-06-14T22:34:13.493 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=time 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=uname 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=vmstat 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=xfs 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=node_exporter.go:117 level=info collector=zfs 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[64579]: ts=2024-06-14T22:34:13.064Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 bash[64528]: 6082a0b8781a5b874d805cfaeb6d84b6331b30f22b2994833b8fa5ec9e4c9f5b 2024-06-14T22:34:13.494 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:34:13 smithi137 systemd[1]: Started Ceph node-exporter.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:34:14.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/576113767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[29889]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/576113767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T22:34:14.463 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:14 smithi086 ceph-mon[38904]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:14.488 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:14 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:14 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:14 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:14 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:14 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:14.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:14 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:34:14.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:14 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/576113767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T22:34:14.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:14 smithi137 ceph-mon[32951]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:14.819 INFO:teuthology.orchestra.run.smithi086.stdout:34359738417 2024-06-14T22:34:15.392 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:15 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2214713611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T22:34:15.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:15 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2214713611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T22:34:15.476 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:15 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2214713611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T22:34:16.369 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:16 smithi137 ceph-mon[32951]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:16.459 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:16 smithi086 ceph-mon[29889]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:16.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:16 smithi086 ceph-mon[38904]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:16.739 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757272 got 163208757275 for osd.5 2024-06-14T22:34:16.740 DEBUG:teuthology.parallel:result is None 2024-06-14T22:34:16.744 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:17.435 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:17.615 INFO:teuthology.orchestra.run.smithi086.stdout:77309411370 2024-06-14T22:34:17.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:17 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:17 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:17 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:17 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1716336204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T22:34:17.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:17 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:17.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:17 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:17.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:17 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:17.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:17 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1716336204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T22:34:18.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:17 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:18.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:17 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:18.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:17 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:18.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:17 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1716336204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T22:34:18.521 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411370 for osd.2 2024-06-14T22:34:18.521 DEBUG:teuthology.parallel:result is None 2024-06-14T22:34:19.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:18 smithi086 ceph-mon[29889]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:19.075 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:18 smithi086 ceph-mon[38904]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:19.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:18 smithi137 ceph-mon[32951]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:20.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:20 smithi137 ceph-mon[32951]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:20.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:34:20 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:34:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:34:21.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:20 smithi086 ceph-mon[38904]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:21.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:20 smithi086 ceph-mon[29889]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:21.711 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738417 for osd.0 2024-06-14T22:34:21.712 DEBUG:teuthology.parallel:result is None 2024-06-14T22:34:22.354 INFO:teuthology.orchestra.run.smithi086.stdout:55834574896 2024-06-14T22:34:22.823 INFO:teuthology.orchestra.run.smithi086.stdout:231928234002 2024-06-14T22:34:22.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:22 smithi086 ceph-mon[38904]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:22.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:22 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3222107043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T22:34:22.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:22 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4287187735' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T22:34:22.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:22 smithi086 ceph-mon[29889]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:22.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:22 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3222107043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T22:34:22.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:22 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4287187735' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T22:34:23.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:22 smithi137 ceph-mon[32951]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:23.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:22 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3222107043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T22:34:23.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:22 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4287187735' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T22:34:23.562 INFO:teuthology.orchestra.run.smithi086.stdout:107374182437 2024-06-14T22:34:23.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:23 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4159048844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T22:34:23.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:23 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4159048844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T22:34:23.882 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233998 got 231928234002 for osd.7 2024-06-14T22:34:23.882 DEBUG:teuthology.parallel:result is None 2024-06-14T22:34:24.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:23 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4159048844' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T22:34:25.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:24 smithi086 ceph-mon[29889]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:25.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:24 smithi086 ceph-mon[38904]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:25.223 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574896 for osd.1 2024-06-14T22:34:25.223 DEBUG:teuthology.parallel:result is None 2024-06-14T22:34:25.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:24 smithi137 ceph-mon[32951]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:26.194 INFO:teuthology.orchestra.run.smithi086.stdout:128849018914 2024-06-14T22:34:26.779 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018914 for osd.4 2024-06-14T22:34:26.780 DEBUG:teuthology.parallel:result is None 2024-06-14T22:34:27.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:26 smithi086 ceph-mon[29889]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:27.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:26 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3927462112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T22:34:27.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:26 smithi086 ceph-mon[38904]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:27.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:26 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3927462112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T22:34:27.207 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182437 for osd.3 2024-06-14T22:34:27.207 DEBUG:teuthology.parallel:result is None 2024-06-14T22:34:27.238 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:26 smithi137 ceph-mon[32951]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:27.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:26 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3927462112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T22:34:27.856 INFO:teuthology.orchestra.run.smithi086.stdout:197568495639 2024-06-14T22:34:28.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:27 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:28.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:27 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:28.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:27 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:28.480 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495639 for osd.6 2024-06-14T22:34:28.480 DEBUG:teuthology.parallel:result is None 2024-06-14T22:34:28.480 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-14T22:34:28.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:34:29.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:28 smithi086 ceph-mon[38904]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:29.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:28 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2725087821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T22:34:29.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:28 smithi086 ceph-mon[29889]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:29.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:28 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2725087821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T22:34:29.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:28 smithi137 ceph-mon[32951]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:29.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:28 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2725087821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T22:34:29.391 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:30.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:30 smithi086 ceph-mon[29889]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:30.970 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:30 smithi086 ceph-mon[38904]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:30.988 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:30 smithi137 ceph-mon[32951]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:30.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:34:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:34:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:34:31.569 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:34:31.570 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:34:32.380 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-06-14T22:34:31.540274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385504,"kb_used_data":6656,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575712,"statfs":{"total":767960285184,"available":765517529088,"internally_reserved":0,"allocated":6815744,"data_stored":3875977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002209"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:34.405729+0000","last_change":"2024-06-14T22:33:32.889977+0000","last_active":"2024-06-14T22:33:34.405729+0000","last_peered":"2024-06-14T22:33:34.405729+0000","last_clean":"2024-06-14T22:33:34.405729+0000","last_became_active":"2024-06-14T22:33:32.889626+0000","last_became_peered":"2024-06-14T22:33:32.889626+0000","last_unstale":"2024-06-14T22:33:34.405729+0000","last_undegraded":"2024-06-14T22:33:34.405729+0000","last_fullsized":"2024-06-14T22:33:34.405729+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:17:06.530667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887659+0000","last_change":"2024-06-14T22:33:33.356640+0000","last_active":"2024-06-14T22:33:33.887659+0000","last_peered":"2024-06-14T22:33:33.887659+0000","last_clean":"2024-06-14T22:33:33.887659+0000","last_became_active":"2024-06-14T22:33:33.356350+0000","last_became_peered":"2024-06-14T22:33:33.356350+0000","last_unstale":"2024-06-14T22:33:33.887659+0000","last_undegraded":"2024-06-14T22:33:33.887659+0000","last_fullsized":"2024-06-14T22:33:33.887659+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:11.719635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887863+0000","last_change":"2024-06-14T22:33:33.356876+0000","last_active":"2024-06-14T22:33:33.887863+0000","last_peered":"2024-06-14T22:33:33.887863+0000","last_clean":"2024-06-14T22:33:33.887863+0000","last_became_active":"2024-06-14T22:33:33.356363+0000","last_became_peered":"2024-06-14T22:33:33.356363+0000","last_unstale":"2024-06-14T22:33:33.887863+0000","last_undegraded":"2024-06-14T22:33:33.887863+0000","last_fullsized":"2024-06-14T22:33:33.887863+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:44.687332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887751+0000","last_change":"2024-06-14T22:33:32.889192+0000","last_active":"2024-06-14T22:33:33.887751+0000","last_peered":"2024-06-14T22:33:33.887751+0000","last_clean":"2024-06-14T22:33:33.887751+0000","last_became_active":"2024-06-14T22:33:32.888868+0000","last_became_peered":"2024-06-14T22:33:32.888868+0000","last_unstale":"2024-06-14T22:33:33.887751+0000","last_undegraded":"2024-06-14T22:33:33.887751+0000","last_fullsized":"2024-06-14T22:33:33.887751+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:44:45.963989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:34.405695+0000","last_change":"2024-06-14T22:33:32.889827+0000","last_active":"2024-06-14T22:33:34.405695+0000","last_peered":"2024-06-14T22:33:34.405695+0000","last_clean":"2024-06-14T22:33:34.405695+0000","last_became_active":"2024-06-14T22:33:32.888965+0000","last_became_peered":"2024-06-14T22:33:32.888965+0000","last_unstale":"2024-06-14T22:33:34.405695+0000","last_undegraded":"2024-06-14T22:33:34.405695+0000","last_fullsized":"2024-06-14T22:33:34.405695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:58:20.273165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:34.405528+0000","last_change":"2024-06-14T22:33:33.332830+0000","last_active":"2024-06-14T22:33:34.405528+0000","last_peered":"2024-06-14T22:33:34.405528+0000","last_clean":"2024-06-14T22:33:34.405528+0000","last_became_active":"2024-06-14T22:33:33.332537+0000","last_became_peered":"2024-06-14T22:33:33.332537+0000","last_unstale":"2024-06-14T22:33:34.405528+0000","last_undegraded":"2024-06-14T22:33:34.405528+0000","last_fullsized":"2024-06-14T22:33:34.405528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:04:42.394657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887863+0000","last_change":"2024-06-14T22:33:32.890674+0000","last_active":"2024-06-14T22:33:33.887863+0000","last_peered":"2024-06-14T22:33:33.887863+0000","last_clean":"2024-06-14T22:33:33.887863+0000","last_became_active":"2024-06-14T22:33:32.890268+0000","last_became_peered":"2024-06-14T22:33:32.890268+0000","last_unstale":"2024-06-14T22:33:33.887863+0000","last_undegraded":"2024-06-14T22:33:33.887863+0000","last_fullsized":"2024-06-14T22:33:33.887863+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:19:58.775230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:34.405695+0000","last_change":"2024-06-14T22:33:32.889322+0000","last_active":"2024-06-14T22:33:34.405695+0000","last_peered":"2024-06-14T22:33:34.405695+0000","last_clean":"2024-06-14T22:33:34.405695+0000","last_became_active":"2024-06-14T22:33:32.888906+0000","last_became_peered":"2024-06-14T22:33:32.888906+0000","last_unstale":"2024-06-14T22:33:34.405695+0000","last_undegraded":"2024-06-14T22:33:34.405695+0000","last_fullsized":"2024-06-14T22:33:34.405695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:44:26.407894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":270,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887856+0000","last_change":"2024-06-14T22:33:02.896315+0000","last_active":"2024-06-14T22:33:33.887856+0000","last_peered":"2024-06-14T22:33:33.887856+0000","last_clean":"2024-06-14T22:33:33.887856+0000","last_became_active":"2024-06-14T22:33:02.895971+0000","last_became_peered":"2024-06-14T22:33:02.895971+0000","last_unstale":"2024-06-14T22:33:33.887856+0000","last_undegraded":"2024-06-14T22:33:33.887856+0000","last_fullsized":"2024-06-14T22:33:33.887856+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:59:26.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1224704,"data_stored":853591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.60599999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.64000000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.505},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.58299999999999996}]},{"osd":5,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.66000000000000003}]},{"osd":6,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.628}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298548,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446604,"statfs":{"total":95995035648,"available":95689322496,"internally_reserved":0,"allocated":1220608,"data_stored":853268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.50900000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.498}]},{"osd":3,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.36699999999999999}]},{"osd":5,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.45700000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:33:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.441},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.40500000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297968,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447184,"statfs":{"total":95995035648,"available":95689916416,"internally_reserved":0,"allocated":626688,"data_stored":262900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.61699999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.64100000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.68000000000000005}]},{"osd":1,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.45100000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.66000000000000003,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.504,"5min":0.52300000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66000000000000003,"5min":0.748,"15min":0.748},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.38900000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.67100000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71799999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.627}]},{"osd":3,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.42499999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.67800000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.40600000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65700000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.65800000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39600000000000002,"15min":0.39200000000000002},"min":{"1min":0.32200000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53700000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.40500000000000003,"15min":0.39700000000000002},"min":{"1min":0.35499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.38600000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.55600000000000005}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297968,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447184,"statfs":{"total":95995035648,"available":95689916416,"internally_reserved":0,"allocated":626688,"data_stored":262958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52700000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.60599999999999998,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.498}]},{"osd":2,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52300000000000002}]},{"osd":3,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.621,"5min":0.748,"15min":0.748},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70199999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.51000000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.39100000000000001,"15min":0.38800000000000001},"min":{"1min":0.28999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.316},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.29699999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.36599999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:33:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.40699999999999997}]},{"osd":7,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.42599999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297972,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447180,"statfs":{"total":95995035648,"available":95689912320,"internally_reserved":0,"allocated":630784,"data_stored":263223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48399999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.312},{"interface":"front","average":{"1min":0.38,"5min":0.39200000000000002,"15min":0.39400000000000002},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.52300000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.216}]},{"osd":1,"last update":"Fri Jun 14 22:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39900000000000002,"15min":0.39700000000000002},"min":{"1min":0.311,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.25},{"interface":"front","average":{"1min":0.375,"5min":0.39900000000000002,"15min":0.40300000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53900000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.26200000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29799999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.28100000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.42099999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.39500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.38}]},{"osd":5,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.38200000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.44400000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.501},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.50900000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297972,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447180,"statfs":{"total":95995035648,"available":95689912320,"internally_reserved":0,"allocated":630784,"data_stored":263223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.378,"15min":0.36799999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38,"15min":0.38100000000000001},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34200000000000003}]},{"osd":1,"last update":"Fri Jun 14 22:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.378,"15min":0.376},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40999999999999998,"15min":0.40699999999999997},"min":{"1min":0.26400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60699999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.31}]},{"osd":3,"last update":"Fri Jun 14 22:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45300000000000001,"15min":0.45400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.37}]},{"osd":4,"last update":"Fri Jun 14 22:33:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54200000000000004,"15min":0.55100000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71699999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Fri Jun 14 22:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71499999999999997,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56799999999999995}]},{"osd":6,"last update":"Fri Jun 14 22:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.745}]},{"osd":7,"last update":"Fri Jun 14 22:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.60499999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297972,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447180,"statfs":{"total":95995035648,"available":95689912320,"internally_reserved":0,"allocated":630784,"data_stored":263223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:33:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.32300000000000001,"15min":0.29299999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.318,"15min":0.28699999999999998},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.27800000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:34:10 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.375,"15min":0.371},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46600000000000003,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.253},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.36099999999999999,"15min":0.35799999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.36599999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.44500000000000001,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.47899999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.442,"15min":0.45800000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.50800000000000001,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.34899999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.65900000000000003,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Jun 14 22:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56899999999999995,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.59299999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.67300000000000004,"5min":0.876,"15min":0.876},"last":0.60699999999999998}]},{"osd":6,"last update":"Fri Jun 14 22:33:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.58099999999999996}]},{"osd":7,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.67800000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1224704,"data_stored":853591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.312,"15min":0.28599999999999998},"min":{"1min":0.252,"5min":0.156,"15min":0.156},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.33200000000000002,"15min":0.308},"min":{"1min":0.27800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47899999999999998,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.39700000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36699999999999999,"15min":0.35899999999999999},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.48799999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36299999999999999,"15min":0.35099999999999998},"min":{"1min":0.26200000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.497,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.32300000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.432,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.40600000000000003,"15min":0.40400000000000003},"min":{"1min":0.28799999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.34100000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.38100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.62,"5min":0.625,"15min":0.625},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.54300000000000004}]},{"osd":5,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.68200000000000005}]},{"osd":6,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Jun 14 22:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.66800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:34:32.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:34:32.668 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:32.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:32 smithi137 ceph-mon[32951]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:32.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:32 smithi137 ceph-mon[32951]: from='client.24506 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:34:32.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:32 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:32.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:32 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:32.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:32 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:32.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:32 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:34:32.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:32 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:33.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[38904]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:33.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[38904]: from='client.24506 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:34:33.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:33.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:33.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:33.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:34:33.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[29889]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[29889]: from='client.24506 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:34:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:34:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:32 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:33.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:33 smithi137 ceph-mon[32951]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-14T22:34:33.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:33 smithi137 ceph-mon[32951]: Reconfiguring daemon alertmanager.a on smithi086 2024-06-14T22:34:34.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:33 smithi086 ceph-mon[29889]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-14T22:34:34.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:33 smithi086 ceph-mon[29889]: Reconfiguring daemon alertmanager.a on smithi086 2024-06-14T22:34:34.069 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:33 smithi086 ceph-mon[38904]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-14T22:34:34.070 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:33 smithi086 ceph-mon[38904]: Reconfiguring daemon alertmanager.a on smithi086 2024-06-14T22:34:34.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:34 smithi137 ceph-mon[32951]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:35.063 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:34 smithi086 ceph-mon[29889]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:35.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:34 smithi086 ceph-mon[38904]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:35.586 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:34:35.587 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:34:36.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:36 smithi137 ceph-mon[32951]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:36.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:36 smithi137 ceph-mon[32951]: from='client.14607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:34:37.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:36 smithi086 ceph-mon[29889]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:37.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:36 smithi086 ceph-mon[29889]: from='client.14607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:34:37.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:36 smithi086 ceph-mon[38904]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:37.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:36 smithi086 ceph-mon[38904]: from='client.14607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:34:37.436 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-06-14T22:34:35.540779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385504,"kb_used_data":6656,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575712,"statfs":{"total":767960285184,"available":765517529088,"internally_reserved":0,"allocated":6815744,"data_stored":3875977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001928"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:34.405729+0000","last_change":"2024-06-14T22:33:32.889977+0000","last_active":"2024-06-14T22:33:34.405729+0000","last_peered":"2024-06-14T22:33:34.405729+0000","last_clean":"2024-06-14T22:33:34.405729+0000","last_became_active":"2024-06-14T22:33:32.889626+0000","last_became_peered":"2024-06-14T22:33:32.889626+0000","last_unstale":"2024-06-14T22:33:34.405729+0000","last_undegraded":"2024-06-14T22:33:34.405729+0000","last_fullsized":"2024-06-14T22:33:34.405729+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:17:06.530667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887659+0000","last_change":"2024-06-14T22:33:33.356640+0000","last_active":"2024-06-14T22:33:33.887659+0000","last_peered":"2024-06-14T22:33:33.887659+0000","last_clean":"2024-06-14T22:33:33.887659+0000","last_became_active":"2024-06-14T22:33:33.356350+0000","last_became_peered":"2024-06-14T22:33:33.356350+0000","last_unstale":"2024-06-14T22:33:33.887659+0000","last_undegraded":"2024-06-14T22:33:33.887659+0000","last_fullsized":"2024-06-14T22:33:33.887659+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:11.719635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887863+0000","last_change":"2024-06-14T22:33:33.356876+0000","last_active":"2024-06-14T22:33:33.887863+0000","last_peered":"2024-06-14T22:33:33.887863+0000","last_clean":"2024-06-14T22:33:33.887863+0000","last_became_active":"2024-06-14T22:33:33.356363+0000","last_became_peered":"2024-06-14T22:33:33.356363+0000","last_unstale":"2024-06-14T22:33:33.887863+0000","last_undegraded":"2024-06-14T22:33:33.887863+0000","last_fullsized":"2024-06-14T22:33:33.887863+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:44.687332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887751+0000","last_change":"2024-06-14T22:33:32.889192+0000","last_active":"2024-06-14T22:33:33.887751+0000","last_peered":"2024-06-14T22:33:33.887751+0000","last_clean":"2024-06-14T22:33:33.887751+0000","last_became_active":"2024-06-14T22:33:32.888868+0000","last_became_peered":"2024-06-14T22:33:32.888868+0000","last_unstale":"2024-06-14T22:33:33.887751+0000","last_undegraded":"2024-06-14T22:33:33.887751+0000","last_fullsized":"2024-06-14T22:33:33.887751+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:44:45.963989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:34.405695+0000","last_change":"2024-06-14T22:33:32.889827+0000","last_active":"2024-06-14T22:33:34.405695+0000","last_peered":"2024-06-14T22:33:34.405695+0000","last_clean":"2024-06-14T22:33:34.405695+0000","last_became_active":"2024-06-14T22:33:32.888965+0000","last_became_peered":"2024-06-14T22:33:32.888965+0000","last_unstale":"2024-06-14T22:33:34.405695+0000","last_undegraded":"2024-06-14T22:33:34.405695+0000","last_fullsized":"2024-06-14T22:33:34.405695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:58:20.273165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:34.405528+0000","last_change":"2024-06-14T22:33:33.332830+0000","last_active":"2024-06-14T22:33:34.405528+0000","last_peered":"2024-06-14T22:33:34.405528+0000","last_clean":"2024-06-14T22:33:34.405528+0000","last_became_active":"2024-06-14T22:33:33.332537+0000","last_became_peered":"2024-06-14T22:33:33.332537+0000","last_unstale":"2024-06-14T22:33:34.405528+0000","last_undegraded":"2024-06-14T22:33:34.405528+0000","last_fullsized":"2024-06-14T22:33:34.405528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:04:42.394657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887863+0000","last_change":"2024-06-14T22:33:32.890674+0000","last_active":"2024-06-14T22:33:33.887863+0000","last_peered":"2024-06-14T22:33:33.887863+0000","last_clean":"2024-06-14T22:33:33.887863+0000","last_became_active":"2024-06-14T22:33:32.890268+0000","last_became_peered":"2024-06-14T22:33:32.890268+0000","last_unstale":"2024-06-14T22:33:33.887863+0000","last_undegraded":"2024-06-14T22:33:33.887863+0000","last_fullsized":"2024-06-14T22:33:33.887863+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:19:58.775230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:34.405695+0000","last_change":"2024-06-14T22:33:32.889322+0000","last_active":"2024-06-14T22:33:34.405695+0000","last_peered":"2024-06-14T22:33:34.405695+0000","last_clean":"2024-06-14T22:33:34.405695+0000","last_became_active":"2024-06-14T22:33:32.888906+0000","last_became_peered":"2024-06-14T22:33:32.888906+0000","last_unstale":"2024-06-14T22:33:34.405695+0000","last_undegraded":"2024-06-14T22:33:34.405695+0000","last_fullsized":"2024-06-14T22:33:34.405695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:44:26.407894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":270,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-06-14T22:33:33.887856+0000","last_change":"2024-06-14T22:33:02.896315+0000","last_active":"2024-06-14T22:33:33.887856+0000","last_peered":"2024-06-14T22:33:33.887856+0000","last_clean":"2024-06-14T22:33:33.887856+0000","last_became_active":"2024-06-14T22:33:02.895971+0000","last_became_peered":"2024-06-14T22:33:02.895971+0000","last_unstale":"2024-06-14T22:33:33.887856+0000","last_undegraded":"2024-06-14T22:33:33.887856+0000","last_fullsized":"2024-06-14T22:33:33.887856+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:59:26.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1224704,"data_stored":853591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.505}]},{"osd":1,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.44}]},{"osd":2,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.52600000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.315},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.33000000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.379}]},{"osd":6,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.39000000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298548,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446604,"statfs":{"total":95995035648,"available":95689322496,"internally_reserved":0,"allocated":1220608,"data_stored":853268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.35799999999999998}]},{"osd":2,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.502},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.41799999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.53300000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.438}]},{"osd":5,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Jun 14 22:33:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56499999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297968,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447184,"statfs":{"total":95995035648,"available":95689916416,"internally_reserved":0,"allocated":626688,"data_stored":262900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.61699999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.495},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52900000000000003,"15min":0.53000000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.64100000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.60999999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.45100000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.66000000000000003,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.52300000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66000000000000003,"5min":0.748,"15min":0.748},"last":0.56200000000000006}]},{"osd":2,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.38900000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.67100000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71799999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.42499999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.67800000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.63},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.40600000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65700000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69999999999999996}]},{"osd":4,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39600000000000002,"15min":0.39200000000000002},"min":{"1min":0.32200000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53700000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.40500000000000003,"15min":0.39700000000000002},"min":{"1min":0.35499999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.67100000000000004}]},{"osd":6,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.76800000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.68700000000000006}]}]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297968,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447184,"statfs":{"total":95995035648,"available":95689916416,"internally_reserved":0,"allocated":626688,"data_stored":262958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52700000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.51000000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.39400000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.60599999999999998,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.627},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.621,"5min":0.748,"15min":0.748},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70199999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Jun 14 22:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.39100000000000001,"15min":0.38800000000000001},"min":{"1min":0.28999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.29699999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.38400000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:33:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.439}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297972,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447180,"statfs":{"total":95995035648,"available":95689912320,"internally_reserved":0,"allocated":630784,"data_stored":263223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48399999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.312},{"interface":"front","average":{"1min":0.38,"5min":0.39200000000000002,"15min":0.39400000000000002},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.52300000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.216}]},{"osd":1,"last update":"Fri Jun 14 22:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.39900000000000002,"15min":0.39700000000000002},"min":{"1min":0.311,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.25},{"interface":"front","average":{"1min":0.375,"5min":0.39900000000000002,"15min":0.40300000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.53900000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.26200000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29799999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56499999999999995,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40200000000000002},"min":{"1min":0.307,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.56200000000000006,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.28100000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.42099999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.39500000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.38}]},{"osd":5,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.38200000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.44400000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.47099999999999997}]},{"osd":7,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.501},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.50900000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297972,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447180,"statfs":{"total":95995035648,"available":95689912320,"internally_reserved":0,"allocated":630784,"data_stored":263223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.378,"15min":0.36799999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.379},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38,"15min":0.38100000000000001},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59299999999999997,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.48599999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.378,"15min":0.376},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40999999999999998,"15min":0.40699999999999997},"min":{"1min":0.26400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60699999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.47099999999999997}]},{"osd":3,"last update":"Fri Jun 14 22:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.45300000000000001,"15min":0.45400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.44},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.51000000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:33:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54200000000000004,"15min":0.55100000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71699999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59099999999999997,"15min":0.59599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Jun 14 22:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71499999999999997,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.69499999999999995}]},{"osd":6,"last update":"Fri Jun 14 22:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.751},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.67800000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297972,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447180,"statfs":{"total":95995035648,"available":95689912320,"internally_reserved":0,"allocated":630784,"data_stored":263223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:33:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.32300000000000001,"15min":0.29299999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.54000000000000004,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.318,"15min":0.28699999999999998},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.27100000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:34:10 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.375,"15min":0.371},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46600000000000003,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.318},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.36099999999999999,"15min":0.35799999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.252}]},{"osd":3,"last update":"Fri Jun 14 22:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.44500000000000001,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.47899999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.442,"15min":0.45800000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.50800000000000001,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.35799999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.65900000000000003,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.45500000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56899999999999995,"15min":0.57999999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.59299999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.67300000000000004,"5min":0.876,"15min":0.876},"last":0.51400000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:33:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.505},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.46899999999999997}]},{"osd":7,"last update":"Fri Jun 14 22:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1224704,"data_stored":853591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:33:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.312,"15min":0.28599999999999998},"min":{"1min":0.252,"5min":0.156,"15min":0.156},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.33200000000000002,"15min":0.308},"min":{"1min":0.27800000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47899999999999998,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36699999999999999,"15min":0.35899999999999999},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.48799999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36299999999999999,"15min":0.35099999999999998},"min":{"1min":0.26200000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.497,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.36899999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.41599999999999998,"15min":0.42399999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.434,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.39300000000000002,"15min":0.39900000000000002},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42299999999999999,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.39300000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.38100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.62,"5min":0.625,"15min":0.625},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.502,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.73299999999999998}]},{"osd":5,"last update":"Fri Jun 14 22:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.66700000000000004}]},{"osd":6,"last update":"Fri Jun 14 22:33:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.77100000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.73999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:34:37.438 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-14T22:34:37.438 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-14T22:34:37.438 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-14T22:34:37.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-14T22:34:38.142 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:38.266 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:38 smithi086 systemd[1]: Stopping Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:34:38.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:38 smithi086 ceph-mon[29889]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:38.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:38 smithi086 ceph-mon[38904]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:38.831 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:38 smithi086 podman[74932]: 2024-06-14 22:34:38.642398936 +0000 UTC m=+0.528507451 container died bfd0c963e8679a6e7b999897b35f29b8b1646facfad4a5b51344faf6357e8320 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=cool_perlman, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-14T22:34:38.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:38 smithi137 ceph-mon[32951]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:39.292 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:38 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[69600]: ts=2024-06-14T22:34:38.949Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-06-14T22:34:39.292 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:38 smithi086 podman[75026]: 2024-06-14 22:34:38.96986828 +0000 UTC m=+0.122317281 container stop 86d4352ad0d0e82994a1d1bbaa2c48d57d3d620ac4deb0f138730a8d46baccaa (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:39.292 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:38 smithi086 podman[75026]: 2024-06-14 22:34:38.999055128 +0000 UTC m=+0.151504148 container died 86d4352ad0d0e82994a1d1bbaa2c48d57d3d620ac4deb0f138730a8d46baccaa (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:39.580 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:39 smithi086 podman[75026]: 2024-06-14 22:34:39.395770138 +0000 UTC m=+0.548219141 container cleanup 86d4352ad0d0e82994a1d1bbaa2c48d57d3d620ac4deb0f138730a8d46baccaa (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:39.580 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:39 smithi086 bash[75026]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a 2024-06-14T22:34:40.080 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:39 smithi086 podman[75045]: 2024-06-14 22:34:39.605139298 +0000 UTC m=+0.629407895 container remove 86d4352ad0d0e82994a1d1bbaa2c48d57d3d620ac4deb0f138730a8d46baccaa (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:40.080 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:39 smithi086 podman[75045]: 2024-06-14 22:34:39.680685375 +0000 UTC m=+0.704953927 volume remove 3fede1e92427da4538f3f3d2696da84e983c7cc26e48cc46d51430cf02fb8da0 2024-06-14T22:34:40.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:40 smithi086 ceph-mon[29889]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:40.811 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:40 smithi086 ceph-mon[38904]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:40.811 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:40 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-06-14T22:34:40.811 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:40 smithi086 systemd[1]: Stopped Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:34:40.898 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:34:40.898 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-14T22:34:40.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:40 smithi137 ceph-mon[32951]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:40.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:34:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:34:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:34:41.080 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:40 smithi086 systemd[1]: Starting Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:34:41.504 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 podman[75214]: 2024-06-14 22:34:41.116689126 +0000 UTC m=+0.198323305 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-06-14T22:34:41.505 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 podman[75214]: 2024-06-14 22:34:41.221697812 +0000 UTC m=+0.303331992 volume create 43058b324957ee1cb23aeb5b4c53846bba8e9eaa4cfe93137444878a09197c7a 2024-06-14T22:34:41.505 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 podman[75214]: 2024-06-14 22:34:41.313824844 +0000 UTC m=+0.395459069 container create 3c3fa36d4b10054f84174258c50f477108018943b6b5eefde88677273874da19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:41.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:41 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4108957196' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-14T22:34:42.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4108957196' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-14T22:34:42.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4108957196' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-14T22:34:42.081 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 podman[75214]: 2024-06-14 22:34:41.650196948 +0000 UTC m=+0.731831133 container init 3c3fa36d4b10054f84174258c50f477108018943b6b5eefde88677273874da19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:42.081 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 podman[75214]: 2024-06-14 22:34:41.655764541 +0000 UTC m=+0.737398724 container start 3c3fa36d4b10054f84174258c50f477108018943b6b5eefde88677273874da19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:42.081 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:41.668Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-06-14T22:34:42.081 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:41.668Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-06-14T22:34:42.081 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:41.669Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.86 port=9094 2024-06-14T22:34:42.081 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:41.670Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-06-14T22:34:42.081 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:41.692Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-14T22:34:42.081 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:41.693Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-14T22:34:42.082 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:41.695Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-06-14T22:34:42.082 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:41.695Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-06-14T22:34:42.082 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 bash[75214]: 3c3fa36d4b10054f84174258c50f477108018943b6b5eefde88677273874da19 2024-06-14T22:34:42.082 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:41 smithi086 systemd[1]: Started Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:34:42.257 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-14T22:34:42.257 INFO:tasks.cephadm:Setup complete, yielding 2024-06-14T22:34:42.258 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-14T22:34:42.267 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi086.front.sepia.ceph.com 2024-06-14T22:34:42.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-06-14T22:34:42.521 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[38904]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[38904]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[38904]: Reconfiguring daemon prometheus.a on smithi137 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[29889]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:42.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[29889]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-14T22:34:42.978 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[29889]: Reconfiguring daemon prometheus.a on smithi137 2024-06-14T22:34:42.978 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:42 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:42 smithi137 ceph-mon[32951]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:42 smithi137 ceph-mon[32951]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-14T22:34:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:42 smithi137 ceph-mon[32951]: Reconfiguring daemon prometheus.a on smithi137 2024-06-14T22:34:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:42 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:44.025 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:43.671Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000983567s 2024-06-14T22:34:45.037 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:44 smithi137 ceph-mon[32951]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:45.037 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:44 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-06-14T22:34:45.037 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:44 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-06-14T22:34:45.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:44 smithi086 ceph-mon[29889]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:45.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:44 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-06-14T22:34:45.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:44 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-06-14T22:34:45.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:44 smithi086 ceph-mon[38904]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:45.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:44 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-06-14T22:34:45.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:44 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-06-14T22:34:45.300 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 systemd[1]: Stopping Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:34:45.688 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.304Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-14T22:34:45.688 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.304Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-06-14T22:34:45.688 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.304Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.304Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.304Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.304Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.304Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.304Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.304Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.305Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.305Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[62129]: ts=2024-06-14T22:34:45.305Z caller=main.go:1147 level=info msg="See you next time!" 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 podman[65464]: 2024-06-14 22:34:45.314752715 +0000 UTC m=+0.123053071 container died 020e411403a82ed9842212d86e7f860d980c83ad2d17fa17b114ac201bde2923 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 podman[65464]: 2024-06-14 22:34:45.693015132 +0000 UTC m=+0.501315533 container cleanup 020e411403a82ed9842212d86e7f860d980c83ad2d17fa17b114ac201bde2923 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:45.689 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 bash[65464]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a 2024-06-14T22:34:45.940 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-mon[32951]: from='client.24524 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:34:45.940 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-06-14T22:34:45.940 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-mon[32951]: osdmap e64: 8 total, 8 up, 8 in 2024-06-14T22:34:45.940 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:45.940 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:45 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:46.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[38904]: from='client.24524 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:34:46.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-06-14T22:34:46.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[38904]: osdmap e64: 8 total, 8 up, 8 in 2024-06-14T22:34:46.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:46.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:46.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[29889]: from='client.24524 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:34:46.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-06-14T22:34:46.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[29889]: osdmap e64: 8 total, 8 up, 8 in 2024-06-14T22:34:46.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:46.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:46.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:45 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[29866]: 2024-06-14T22:34:45.807+0000 7f96021cd640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-14T22:34:46.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:45 smithi137 podman[65483]: 2024-06-14 22:34:45.944780303 +0000 UTC m=+0.632406020 container remove 020e411403a82ed9842212d86e7f860d980c83ad2d17fa17b114ac201bde2923 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:46.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-06-14T22:34:46.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 systemd[1]: Stopped Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:34:46.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@prometheus.a.service: Consumed 1.138s CPU time. 2024-06-14T22:34:46.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 systemd[1]: Starting Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:34:46.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-06-14T22:34:46.576 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 podman[65580]: 2024-06-14 22:34:46.191181658 +0000 UTC m=+0.017950746 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-06-14T22:34:46.576 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 podman[65580]: 2024-06-14 22:34:46.329232098 +0000 UTC m=+0.156001214 container create 4b95a815ba8cdfbd6e30b11b8b8f964dec63a58f773430ec39be740fa1f3052f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:46.796 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:46] ENGINE Bus STOPPING 2024-06-14T22:34:46.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: pgmap v47: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:46.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:34:46.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-06-14T22:34:46.832 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: pgmap v47: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:34:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-06-14T22:34:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: osdmap e65: 8 total, 8 up, 8 in 2024-06-14T22:34:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: osdmap e66: 8 total, 8 up, 8 in 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: fsmap foo:0 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: Saving service mds.foo spec with placement count:2 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-06-14T22:34:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:46.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:34:46.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-06-14T22:34:46.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:46.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:34:46.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 podman[65580]: 2024-06-14 22:34:46.581120555 +0000 UTC m=+0.407889641 container init 4b95a815ba8cdfbd6e30b11b8b8f964dec63a58f773430ec39be740fa1f3052f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:46.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 podman[65580]: 2024-06-14 22:34:46.58682101 +0000 UTC m=+0.413590087 container start 4b95a815ba8cdfbd6e30b11b8b8f964dec63a58f773430ec39be740fa1f3052f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:34:46.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.610Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-06-14T22:34:46.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.610Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-06-14T22:34:46.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.610Z caller=main.go:570 level=info host_details="(Linux 5.14.0-457.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024 x86_64 smithi137 (none))" 2024-06-14T22:34:46.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.610Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.610Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.612Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.612Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.613Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.613Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.614Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.614Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.567µs 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.614Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.615Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.615Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.616Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.756µs wal_replay_duration=1.022706ms wbl_replay_duration=137ns total_replay_duration=1.058118ms 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.617Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.617Z caller=main.go:1029 level=info msg="TSDB started" 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.617Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-14T22:34:46.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.631Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=13.780391ms db_storage=722ns remote_storage=1.094µs web_handler=349ns query_engine=530ns scrape=398.422µs scrape_sd=30.517µs notify=31.879µs notify_sd=6.029µs rules=13.092514ms tracing=7.06µs 2024-06-14T22:34:46.993 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.631Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-06-14T22:34:46.993 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:34:46.631Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-06-14T22:34:46.993 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 bash[65580]: 4b95a815ba8cdfbd6e30b11b8b8f964dec63a58f773430ec39be740fa1f3052f 2024-06-14T22:34:46.993 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:34:46 smithi137 systemd[1]: Started Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:34:47.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T22:34:47.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:46] ENGINE Bus STOPPED 2024-06-14T22:34:47.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:46] ENGINE Bus STARTING 2024-06-14T22:34:47.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:47] ENGINE Serving on http://:::9283 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:47] ENGINE Bus STARTED 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:47] ENGINE Bus STOPPING 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: pgmap v47: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: osdmap e65: 8 total, 8 up, 8 in 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-14T22:34:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: osdmap e66: 8 total, 8 up, 8 in 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: fsmap foo:0 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: Saving service mds.foo spec with placement count:2 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-06-14T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: osdmap e65: 8 total, 8 up, 8 in 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: osdmap e66: 8 total, 8 up, 8 in 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-06-14T22:34:47.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: fsmap foo:0 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: Saving service mds.foo spec with placement count:2 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:47.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:46 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:34:47.629 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T22:34:47.629 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:47] ENGINE Bus STOPPED 2024-06-14T22:34:47.629 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:47] ENGINE Bus STARTING 2024-06-14T22:34:47.629 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:47] ENGINE Serving on http://:::9283 2024-06-14T22:34:47.629 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:47] ENGINE Bus STARTED 2024-06-14T22:34:47.995 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:47] ENGINE Bus STOPPING 2024-06-14T22:34:47.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:34:47.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-14T22:34:47.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:34:47.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[38904]: osdmap e67: 8 total, 8 up, 8 in 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-06-14T22:34:47.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:47 smithi086 ceph-mon[29889]: osdmap e67: 8 total, 8 up, 8 in 2024-06-14T22:34:48.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:47 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:34:48.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:47 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi086.front.sepia.ceph.com:9093"}]: dispatch 2024-06-14T22:34:48.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:47 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:34:48.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:47 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-06-14T22:34:48.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:47 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:34:48.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:47 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-06-14T22:34:48.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:47 smithi137 ceph-mon[32951]: osdmap e67: 8 total, 8 up, 8 in 2024-06-14T22:34:48.267 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T22:34:48.267 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:48] ENGINE Bus STOPPED 2024-06-14T22:34:48.267 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:48] ENGINE Bus STARTING 2024-06-14T22:34:48.267 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:48] ENGINE Serving on http://:::9283 2024-06-14T22:34:48.267 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:34:48] ENGINE Bus STARTED 2024-06-14T22:34:48.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-mon[29889]: pgmap v51: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:48.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-mon[29889]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:34:48.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-mon[29889]: osdmap e68: 8 total, 8 up, 8 in 2024-06-14T22:34:48.869 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-mon[38904]: pgmap v51: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:48.869 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-mon[38904]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:34:48.869 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:48 smithi086 ceph-mon[38904]: osdmap e68: 8 total, 8 up, 8 in 2024-06-14T22:34:49.166 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:48 smithi137 ceph-mon[32951]: pgmap v51: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:49.166 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:48 smithi137 ceph-mon[32951]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:34:49.166 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:48 smithi137 ceph-mon[32951]: osdmap e68: 8 total, 8 up, 8 in 2024-06-14T22:34:49.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-06-14T22:34:50.233 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:50 smithi086 ceph-mon[29889]: pgmap v53: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:50 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:50 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:51.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:50 smithi086 ceph-mon[38904]: pgmap v53: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:51.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:50 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:51.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:50 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:51.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:50 smithi137 ceph-mon[32951]: pgmap v53: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:51.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:50 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:51.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:50 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:51.940 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:34:51 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:34:51.673Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002689684s 2024-06-14T22:34:53.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:52 smithi086 ceph-mon[29889]: pgmap v54: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:53.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:52 smithi086 ceph-mon[38904]: pgmap v54: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:53.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:52 smithi137 ceph-mon[32951]: pgmap v54: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:53.561 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-06-14T22:34:53.570 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-14T22:34:53.570 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-14T22:34:53.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph mgr dump --format=json 2024-06-14T22:34:54.157 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:34:55.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:55.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:55.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:55.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-14T22:34:55.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:54 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:34:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-14T22:34:55.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:54 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:55.904 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:55 smithi086 ceph-mon[29889]: Deploying daemon mds.foo.smithi086.nhdjdb on smithi086 2024-06-14T22:34:55.904 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:55 smithi086 ceph-mon[38904]: Deploying daemon mds.foo.smithi086.nhdjdb on smithi086 2024-06-14T22:34:56.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:55 smithi137 ceph-mon[32951]: Deploying daemon mds.foo.smithi086.nhdjdb on smithi086 2024-06-14T22:34:56.799 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:34:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:56 smithi086 ceph-mon[29889]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:56 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/139270759' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-14T22:34:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:56 smithi086 ceph-mon[38904]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:56 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/139270759' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-14T22:34:57.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:56 smithi137 ceph-mon[32951]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:57.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:56 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/139270759' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-14T22:34:57.814 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":20,"flags":0,"active_gid":14424,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":2254669699},{"type":"v1","addr":"172.21.15.86:6801","nonce":2254669699}]},"active_addr":"172.21.15.86:6801/2254669699","active_change":"2024-06-14T22:33:27.518569+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24319,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.86:8443/","prometheus":"http://172.21.15.86:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":418538598}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3107951276}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3167284010}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":1167860215}]}]} 2024-06-14T22:34:57.819 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-14T22:34:57.819 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-14T22:34:57.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-14T22:34:58.382 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-14T22:34:58.789 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-14T22:34:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:58 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:58.847 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:34:58.848 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.848 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:34:58.848 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.848 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.848 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:34:58.848 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:58.848 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:34:58.848 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-14T22:34:58.848 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:58 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:34:59.826 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:34:59] "GET /metrics HTTP/1.1" 200 208114 "" "Prometheus/2.43.0" 2024-06-14T22:35:00.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: Deploying daemon mds.foo.smithi137.yjevpg on smithi137 2024-06-14T22:35:00.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: mds.? [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] up:boot 2024-06-14T22:35:00.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: daemon mds.foo.smithi086.nhdjdb assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-06-14T22:35:00.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-14T22:35:00.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: Cluster is now healthy 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: fsmap foo:0 1 up:standby 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:creating} 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[29889]: daemon mds.foo.smithi086.nhdjdb is now active in filesystem foo as rank 0 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: Deploying daemon mds.foo.smithi137.yjevpg on smithi137 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: mds.? [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] up:boot 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: daemon mds.foo.smithi086.nhdjdb assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: Cluster is now healthy 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: fsmap foo:0 1 up:standby 2024-06-14T22:35:00.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:35:00.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:creating} 2024-06-14T22:35:00.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:34:59 smithi086 ceph-mon[38904]: daemon mds.foo.smithi086.nhdjdb is now active in filesystem foo as rank 0 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: Deploying daemon mds.foo.smithi137.yjevpg on smithi137 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: mds.? [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] up:boot 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: daemon mds.foo.smithi086.nhdjdb assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: Cluster is now healthy 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: fsmap foo:0 1 up:standby 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:creating} 2024-06-14T22:35:00.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:34:59 smithi137 ceph-mon[32951]: daemon mds.foo.smithi086.nhdjdb is now active in filesystem foo as rank 0 2024-06-14T22:35:00.336 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:35:00.336 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":68,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T22:34:47.813980+0000","last_up_change":"2024-06-14T22:32:58.884084+0000","last_in_change":"2024-06-14T22:32:42.063674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-14T22:33:31.126149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-06-14T22:34:44.677867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-06-14T22:34:45.034624+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6803","nonce":2092557056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6805","nonce":2092557056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6809","nonce":2092557056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6807","nonce":2092557056}]},"public_addr":"172.21.15.86:6803/2092557056","cluster_addr":"172.21.15.86:6805/2092557056","heartbeat_back_addr":"172.21.15.86:6809/2092557056","heartbeat_front_addr":"172.21.15.86:6807/2092557056","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6811","nonce":796510430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6813","nonce":796510430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6817","nonce":796510430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6815","nonce":796510430}]},"public_addr":"172.21.15.86:6811/796510430","cluster_addr":"172.21.15.86:6813/796510430","heartbeat_back_addr":"172.21.15.86:6817/796510430","heartbeat_front_addr":"172.21.15.86:6815/796510430","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6819","nonce":3531892667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6821","nonce":3531892667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6825","nonce":3531892667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6823","nonce":3531892667}]},"public_addr":"172.21.15.86:6819/3531892667","cluster_addr":"172.21.15.86:6821/3531892667","heartbeat_back_addr":"172.21.15.86:6825/3531892667","heartbeat_front_addr":"172.21.15.86:6823/3531892667","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6827","nonce":287823501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6829","nonce":287823501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6833","nonce":287823501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6831","nonce":287823501}]},"public_addr":"172.21.15.86:6827/287823501","cluster_addr":"172.21.15.86:6829/287823501","heartbeat_back_addr":"172.21.15.86:6833/287823501","heartbeat_front_addr":"172.21.15.86:6831/287823501","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6809","nonce":2122959324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6811","nonce":2122959324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6815","nonce":2122959324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6813","nonce":2122959324}]},"public_addr":"172.21.15.137:6809/2122959324","cluster_addr":"172.21.15.137:6811/2122959324","heartbeat_back_addr":"172.21.15.137:6815/2122959324","heartbeat_front_addr":"172.21.15.137:6813/2122959324","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6817","nonce":2398798362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6819","nonce":2398798362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6823","nonce":2398798362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6821","nonce":2398798362}]},"public_addr":"172.21.15.137:6817/2398798362","cluster_addr":"172.21.15.137:6819/2398798362","heartbeat_back_addr":"172.21.15.137:6823/2398798362","heartbeat_front_addr":"172.21.15.137:6821/2398798362","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3941467574":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/2741142752":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/1759721543":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3724463773":"2024-06-15T22:33:27.518416+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T22:35:00.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:35:00.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-mon[32951]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-mon[32951]: mds.? [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] up:active 2024-06-14T22:35:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-mon[32951]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 2024-06-14T22:35:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/492424370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:35:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:00 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.042 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[38904]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:01.042 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[38904]: mds.? [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] up:active 2024-06-14T22:35:01.042 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[38904]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 2024-06-14T22:35:01.042 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/492424370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:35:01.042 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.042 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[29889]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[29889]: mds.? [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] up:active 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[29889]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/492424370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:00 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:01.067 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-14T22:35:01.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-14T22:35:02.085 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:02.208 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:35:02.208 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[29889]: mds.? [v2:172.21.15.137:6832/1266342281,v1:172.21.15.137:6833/1266342281] up:boot 2024-06-14T22:35:02.208 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[29889]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:02.208 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[29889]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:02.208 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:35:02.208 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:35:02.209 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[38904]: mds.? [v2:172.21.15.137:6832/1266342281,v1:172.21.15.137:6833/1266342281] up:boot 2024-06-14T22:35:02.209 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[38904]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:02.209 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[38904]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:02.209 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:01 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:35:02.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:01 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:35:02.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:01 smithi137 ceph-mon[32951]: mds.? [v2:172.21.15.137:6832/1266342281,v1:172.21.15.137:6833/1266342281] up:boot 2024-06-14T22:35:02.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:01 smithi137 ceph-mon[32951]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:02.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:01 smithi137 ceph-mon[32951]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:02.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:01 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:35:03.134 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:02 smithi137 ceph-mon[32951]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:03.134 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:02 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:03.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:02 smithi086 ceph-mon[29889]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:03.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:02 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:03.302 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:02 smithi086 ceph-mon[38904]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:03.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:02 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:04.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:35:04.053 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":68,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T22:34:47.813980+0000","last_up_change":"2024-06-14T22:32:58.884084+0000","last_in_change":"2024-06-14T22:32:42.063674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-14T22:33:31.126149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-06-14T22:34:44.677867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-06-14T22:34:45.034624+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6803","nonce":2092557056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6805","nonce":2092557056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6809","nonce":2092557056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2092557056},{"type":"v1","addr":"172.21.15.86:6807","nonce":2092557056}]},"public_addr":"172.21.15.86:6803/2092557056","cluster_addr":"172.21.15.86:6805/2092557056","heartbeat_back_addr":"172.21.15.86:6809/2092557056","heartbeat_front_addr":"172.21.15.86:6807/2092557056","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6811","nonce":796510430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6813","nonce":796510430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6817","nonce":796510430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":796510430},{"type":"v1","addr":"172.21.15.86:6815","nonce":796510430}]},"public_addr":"172.21.15.86:6811/796510430","cluster_addr":"172.21.15.86:6813/796510430","heartbeat_back_addr":"172.21.15.86:6817/796510430","heartbeat_front_addr":"172.21.15.86:6815/796510430","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6819","nonce":3531892667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6821","nonce":3531892667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6825","nonce":3531892667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3531892667},{"type":"v1","addr":"172.21.15.86:6823","nonce":3531892667}]},"public_addr":"172.21.15.86:6819/3531892667","cluster_addr":"172.21.15.86:6821/3531892667","heartbeat_back_addr":"172.21.15.86:6825/3531892667","heartbeat_front_addr":"172.21.15.86:6823/3531892667","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6827","nonce":287823501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6829","nonce":287823501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6833","nonce":287823501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":287823501},{"type":"v1","addr":"172.21.15.86:6831","nonce":287823501}]},"public_addr":"172.21.15.86:6827/287823501","cluster_addr":"172.21.15.86:6829/287823501","heartbeat_back_addr":"172.21.15.86:6833/287823501","heartbeat_front_addr":"172.21.15.86:6831/287823501","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6809","nonce":2122959324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6811","nonce":2122959324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6815","nonce":2122959324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6813","nonce":2122959324}]},"public_addr":"172.21.15.137:6809/2122959324","cluster_addr":"172.21.15.137:6811/2122959324","heartbeat_back_addr":"172.21.15.137:6815/2122959324","heartbeat_front_addr":"172.21.15.137:6813/2122959324","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6817","nonce":2398798362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6819","nonce":2398798362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6823","nonce":2398798362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6821","nonce":2398798362}]},"public_addr":"172.21.15.137:6817/2398798362","cluster_addr":"172.21.15.137:6819/2398798362","heartbeat_back_addr":"172.21.15.137:6823/2398798362","heartbeat_front_addr":"172.21.15.137:6821/2398798362","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3941467574":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/2741142752":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/1759721543":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3724463773":"2024-06-15T22:33:27.518416+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T22:35:04.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[29889]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:04.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[29889]: mds.? [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] up:active 2024-06-14T22:35:04.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[29889]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:04.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2837713413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:35:04.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:04.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:04.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[38904]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:04.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[38904]: mds.? [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] up:active 2024-06-14T22:35:04.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[38904]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:04.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2837713413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:35:04.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:04.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:04 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:04.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-14T22:35:04.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-14T22:35:04.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-14T22:35:04.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-14T22:35:04.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-14T22:35:04.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-14T22:35:04.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-14T22:35:04.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-14T22:35:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:04 smithi137 ceph-mon[32951]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:04 smithi137 ceph-mon[32951]: mds.? [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] up:active 2024-06-14T22:35:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:04 smithi137 ceph-mon[32951]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:04 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2837713413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:35:04.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:04 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:04.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:04 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:06.049 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:06.049 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:06.890 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:06 smithi086 ceph-mon[29889]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:06.890 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:06 smithi086 ceph-mon[29889]: mds.? [v2:172.21.15.137:6832/1266342281,v1:172.21.15.137:6833/1266342281] up:standby 2024-06-14T22:35:06.890 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:06 smithi086 ceph-mon[29889]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:06.890 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:06 smithi086 ceph-mon[38904]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:06.890 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:06 smithi086 ceph-mon[38904]: mds.? [v2:172.21.15.137:6832/1266342281,v1:172.21.15.137:6833/1266342281] up:standby 2024-06-14T22:35:06.890 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:06 smithi086 ceph-mon[38904]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:06.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:06 smithi137 ceph-mon[32951]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:06.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:06 smithi137 ceph-mon[32951]: mds.? [v2:172.21.15.137:6832/1266342281,v1:172.21.15.137:6833/1266342281] up:standby 2024-06-14T22:35:06.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:06 smithi137 ceph-mon[32951]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:35:08.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:08 smithi137 ceph-mon[32951]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:09.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:08 smithi086 ceph-mon[29889]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:09.050 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:08 smithi086 ceph-mon[38904]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:09.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:09] "GET /metrics HTTP/1.1" 200 208114 "" "Prometheus/2.43.0" 2024-06-14T22:35:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:10.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[29889]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:10.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:10.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:10.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[38904]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:10.822 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:10 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:35:10.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:35:10 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:35:10.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:10 smithi137 ceph-mon[32951]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:10.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:10 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:10.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:10 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:10.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:10 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:35:10.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:10 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:35:10.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:10 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:10.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:10 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:35:11.631 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:11.632 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:12.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:12 smithi086 ceph-mon[29889]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:12.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:12 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:12.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:12 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:12 smithi086 ceph-mon[38904]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:12 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:12 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:12.983 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:12 smithi137 ceph-mon[32951]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:12.983 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:12 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:12.983 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:12 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:13.808 INFO:teuthology.orchestra.run.smithi086.stdout:107374182448 2024-06-14T22:35:13.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-14T22:35:14.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:14 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:14.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:14 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:14.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:14 smithi086 ceph-mon[29889]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:14.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:14 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:14.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:14 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:14.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:14 smithi086 ceph-mon[38904]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:14.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:14 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:14.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:14 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:14.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:14 smithi137 ceph-mon[32951]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-06-14T22:35:15.669 INFO:teuthology.orchestra.run.smithi086.stdout:197568495649 2024-06-14T22:35:15.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-14T22:35:16.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:16 smithi137 ceph-mon[32951]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-06-14T22:35:16.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:16 smithi086 ceph-mon[38904]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-06-14T22:35:16.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:16 smithi086 ceph-mon[29889]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-06-14T22:35:17.339 INFO:teuthology.orchestra.run.smithi086.stdout:55834574908 2024-06-14T22:35:17.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-14T22:35:18.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:18 smithi086 ceph-mon[29889]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:18.935 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:18 smithi086 ceph-mon[38904]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:18.960 INFO:teuthology.orchestra.run.smithi086.stdout:128849018925 2024-06-14T22:35:18.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-14T22:35:18.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:18 smithi137 ceph-mon[32951]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:19.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:19] "GET /metrics HTTP/1.1" 200 239146 "" "Prometheus/2.43.0" 2024-06-14T22:35:20.725 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:20 smithi086 ceph-mon[38904]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:20.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:20 smithi086 ceph-mon[29889]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:20.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:20 smithi137 ceph-mon[32951]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:20.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:35:20 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:35:22.128 INFO:teuthology.orchestra.run.smithi086.stdout:163208757290 2024-06-14T22:35:22.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-14T22:35:22.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:22 smithi086 ceph-mon[29889]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:22.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:22 smithi086 ceph-mon[38904]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:22.916 INFO:teuthology.orchestra.run.smithi086.stdout:77309411384 2024-06-14T22:35:22.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-14T22:35:22.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:22 smithi137 ceph-mon[32951]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:23.308 INFO:teuthology.orchestra.run.smithi086.stdout:34359738432 2024-06-14T22:35:23.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-14T22:35:24.147 INFO:teuthology.orchestra.run.smithi086.stdout:231928234015 2024-06-14T22:35:24.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-14T22:35:24.502 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:24.805 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:24 smithi086 ceph-mon[29889]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:24.805 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:24 smithi086 ceph-mon[38904]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:24.939 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:24.941 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:24.944 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:24.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:24 smithi137 ceph-mon[32951]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:26.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:26 smithi137 ceph-mon[32951]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:27.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:26 smithi086 ceph-mon[29889]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:27.058 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:26 smithi086 ceph-mon[38904]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:27.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:27 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-06-14T22:35:27.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:27 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-14T22:35:27.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:27 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-06-14T22:35:27.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:27 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-14T22:35:27.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:27 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-14T22:35:27.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:27 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-14T22:35:27.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:27 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:28.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-14T22:35:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-14T22:35:28.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-14T22:35:28.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:27 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:28.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[38904]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:28.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-06-14T22:35:28.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-06-14T22:35:28.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-06-14T22:35:28.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[38904]: osdmap e69: 8 total, 8 up, 8 in 2024-06-14T22:35:28.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[29889]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:28.802 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-06-14T22:35:28.802 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-06-14T22:35:28.802 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-06-14T22:35:28.802 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:28 smithi086 ceph-mon[29889]: osdmap e69: 8 total, 8 up, 8 in 2024-06-14T22:35:28.835 INFO:teuthology.orchestra.run.smithi086.stdout:197568495652 2024-06-14T22:35:28.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:28 smithi137 ceph-mon[32951]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:28.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:28 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-06-14T22:35:28.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:28 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-06-14T22:35:28.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:28 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-06-14T22:35:28.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:28 smithi137 ceph-mon[32951]: osdmap e69: 8 total, 8 up, 8 in 2024-06-14T22:35:29.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:29 smithi086 ceph-mon[29889]: osdmap e70: 8 total, 8 up, 8 in 2024-06-14T22:35:29.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:29 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1061784994' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T22:35:29.724 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:29] "GET /metrics HTTP/1.1" 200 239126 "" "Prometheus/2.43.0" 2024-06-14T22:35:29.724 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:29 smithi086 ceph-mon[38904]: osdmap e70: 8 total, 8 up, 8 in 2024-06-14T22:35:29.724 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:29 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1061784994' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T22:35:29.869 INFO:teuthology.orchestra.run.smithi086.stdout:107374182452 2024-06-14T22:35:29.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:29 smithi137 ceph-mon[32951]: osdmap e70: 8 total, 8 up, 8 in 2024-06-14T22:35:29.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:29 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1061784994' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T22:35:30.690 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:30 smithi086 ceph-mon[29889]: pgmap v75: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:30.690 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:30 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1449401967' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T22:35:30.690 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:30 smithi086 ceph-mon[38904]: pgmap v75: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:30.691 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:30 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1449401967' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T22:35:30.952 INFO:teuthology.orchestra.run.smithi086.stdout:55834574911 2024-06-14T22:35:30.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:30 smithi137 ceph-mon[32951]: pgmap v75: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:30.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:30 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1449401967' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T22:35:30.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:35:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:35:31.147 INFO:teuthology.orchestra.run.smithi086.stdout:128849018928 2024-06-14T22:35:31.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:31 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/371096193' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T22:35:31.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:31 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1770791131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T22:35:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:31 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/371096193' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T22:35:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:31 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1770791131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T22:35:31.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182448 got 107374182452 for osd.3 2024-06-14T22:35:31.972 DEBUG:teuthology.parallel:result is None 2024-06-14T22:35:31.977 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:31.979 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:31.979 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:31.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:31 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/371096193' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T22:35:31.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:31 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1770791131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T22:35:32.459 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495649 got 197568495652 for osd.6 2024-06-14T22:35:32.459 DEBUG:teuthology.parallel:result is None 2024-06-14T22:35:32.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:32 smithi086 ceph-mon[29889]: pgmap v76: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:32.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:32 smithi086 ceph-mon[38904]: pgmap v76: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:32.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:32 smithi137 ceph-mon[32951]: pgmap v76: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:33.318 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574911 for osd.1 2024-06-14T22:35:33.319 DEBUG:teuthology.parallel:result is None 2024-06-14T22:35:33.775 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018925 got 128849018928 for osd.4 2024-06-14T22:35:33.775 DEBUG:teuthology.parallel:result is None 2024-06-14T22:35:34.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:34 smithi086 ceph-mon[38904]: pgmap v77: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:34.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:34 smithi086 ceph-mon[29889]: pgmap v77: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:34.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:34 smithi137 ceph-mon[32951]: pgmap v77: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:35.631 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:36.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:36 smithi137 ceph-mon[32951]: pgmap v78: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:37.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:36 smithi086 ceph-mon[29889]: pgmap v78: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:37.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:36 smithi086 ceph-mon[38904]: pgmap v78: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:38.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:38 smithi086 ceph-mon[29889]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:38.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:38 smithi086 ceph-mon[38904]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:38.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:38 smithi137 ceph-mon[32951]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:39.260 INFO:teuthology.orchestra.run.smithi086.stdout:34359738435 2024-06-14T22:35:39.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:39 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/595976363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T22:35:39.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:39 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/595976363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T22:35:39.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:39] "GET /metrics HTTP/1.1" 200 239126 "" "Prometheus/2.43.0" 2024-06-14T22:35:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:39 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/595976363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T22:35:40.110 INFO:teuthology.orchestra.run.smithi086.stdout:77309411388 2024-06-14T22:35:40.765 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:40 smithi086 ceph-mon[38904]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:40.765 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:40 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3451841690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T22:35:40.766 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:40 smithi086 ceph-mon[29889]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:40.766 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:40 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3451841690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T22:35:40.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:40 smithi137 ceph-mon[32951]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:40.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:40 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3451841690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T22:35:40.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:35:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:35:41.086 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738432 got 34359738435 for osd.0 2024-06-14T22:35:41.086 DEBUG:teuthology.parallel:result is None 2024-06-14T22:35:41.995 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411384 got 77309411388 for osd.2 2024-06-14T22:35:41.995 DEBUG:teuthology.parallel:result is None 2024-06-14T22:35:42.340 INFO:teuthology.orchestra.run.smithi086.stdout:163208757294 2024-06-14T22:35:42.554 INFO:teuthology.orchestra.run.smithi086.stdout:231928234019 2024-06-14T22:35:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[29889]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:42.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3152667019' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2558369580' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[38904]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3152667019' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2558369580' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T22:35:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:42 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:42 smithi137 ceph-mon[32951]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:42 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3152667019' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T22:35:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:42 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2558369580' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T22:35:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:42 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:43.132 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757290 got 163208757294 for osd.5 2024-06-14T22:35:43.133 DEBUG:teuthology.parallel:result is None 2024-06-14T22:35:43.697 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234015 got 231928234019 for osd.7 2024-06-14T22:35:43.697 DEBUG:teuthology.parallel:result is None 2024-06-14T22:35:43.697 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-14T22:35:43.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:35:43.950 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:43.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:43 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:35:43.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:43 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:35:43.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:43 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:44.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:43 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:35:44.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:43 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:35:44.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:43 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:44.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:43 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:35:44.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:43 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:35:44.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:43 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:44.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:44 smithi137 ceph-mon[32951]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:45.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:44 smithi086 ceph-mon[29889]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:45.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:44 smithi086 ceph-mon[38904]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:35:45.811 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:35:46.561 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-06-14T22:35:45.555547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592706,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":224,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167,"ondisk_log_size":167,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386284,"kb_used_data":7424,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747574932,"statfs":{"total":767960285184,"available":765516730368,"internally_reserved":0,"allocated":7602176,"data_stored":4286407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002570"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632890+0000","last_change":"2024-06-14T22:34:46.493127+0000","last_active":"2024-06-14T22:35:27.632890+0000","last_peered":"2024-06-14T22:35:27.632890+0000","last_clean":"2024-06-14T22:35:27.632890+0000","last_became_active":"2024-06-14T22:34:46.492766+0000","last_became_peered":"2024-06-14T22:34:46.492766+0000","last_unstale":"2024-06-14T22:35:27.632890+0000","last_undegraded":"2024-06-14T22:35:27.632890+0000","last_fullsized":"2024-06-14T22:35:27.632890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:31:41.431477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370073+0000","last_change":"2024-06-14T22:34:45.814323+0000","last_active":"2024-06-14T22:35:28.370073+0000","last_peered":"2024-06-14T22:35:28.370073+0000","last_clean":"2024-06-14T22:35:28.370073+0000","last_became_active":"2024-06-14T22:34:45.814041+0000","last_became_peered":"2024-06-14T22:34:45.814041+0000","last_unstale":"2024-06-14T22:35:28.370073+0000","last_undegraded":"2024-06-14T22:35:28.370073+0000","last_fullsized":"2024-06-14T22:35:28.370073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:07:38.056038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627797+0000","last_change":"2024-06-14T22:34:46.494622+0000","last_active":"2024-06-14T22:35:27.627797+0000","last_peered":"2024-06-14T22:35:27.627797+0000","last_clean":"2024-06-14T22:35:27.627797+0000","last_became_active":"2024-06-14T22:34:46.494331+0000","last_became_peered":"2024-06-14T22:34:46.494331+0000","last_unstale":"2024-06-14T22:35:27.627797+0000","last_undegraded":"2024-06-14T22:35:27.627797+0000","last_fullsized":"2024-06-14T22:35:27.627797+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:23:44.353759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630479+0000","last_change":"2024-06-14T22:34:45.817622+0000","last_active":"2024-06-14T22:35:27.630479+0000","last_peered":"2024-06-14T22:35:27.630479+0000","last_clean":"2024-06-14T22:35:27.630479+0000","last_became_active":"2024-06-14T22:34:45.817350+0000","last_became_peered":"2024-06-14T22:34:45.817350+0000","last_unstale":"2024-06-14T22:35:27.630479+0000","last_undegraded":"2024-06-14T22:35:27.630479+0000","last_fullsized":"2024-06-14T22:35:27.630479+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:45:12.669650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631250+0000","last_change":"2024-06-14T22:34:47.047477+0000","last_active":"2024-06-14T22:35:27.631250+0000","last_peered":"2024-06-14T22:35:27.631250+0000","last_clean":"2024-06-14T22:35:27.631250+0000","last_became_active":"2024-06-14T22:34:47.047206+0000","last_became_peered":"2024-06-14T22:34:47.047206+0000","last_unstale":"2024-06-14T22:35:27.631250+0000","last_undegraded":"2024-06-14T22:35:27.631250+0000","last_fullsized":"2024-06-14T22:35:27.631250+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:54:08.872928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369787+0000","last_change":"2024-06-14T22:34:45.813267+0000","last_active":"2024-06-14T22:35:28.369787+0000","last_peered":"2024-06-14T22:35:28.369787+0000","last_clean":"2024-06-14T22:35:28.369787+0000","last_became_active":"2024-06-14T22:34:45.813003+0000","last_became_peered":"2024-06-14T22:34:45.813003+0000","last_unstale":"2024-06-14T22:35:28.369787+0000","last_undegraded":"2024-06-14T22:35:28.369787+0000","last_fullsized":"2024-06-14T22:35:28.369787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:46.446483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.633616+0000","last_change":"2024-06-14T22:34:46.494754+0000","last_active":"2024-06-14T22:35:27.633616+0000","last_peered":"2024-06-14T22:35:27.633616+0000","last_clean":"2024-06-14T22:35:27.633616+0000","last_became_active":"2024-06-14T22:34:46.494412+0000","last_became_peered":"2024-06-14T22:34:46.494412+0000","last_unstale":"2024-06-14T22:35:27.633616+0000","last_undegraded":"2024-06-14T22:35:27.633616+0000","last_fullsized":"2024-06-14T22:35:27.633616+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:51.607597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369599+0000","last_change":"2024-06-14T22:34:45.811556+0000","last_active":"2024-06-14T22:35:28.369599+0000","last_peered":"2024-06-14T22:35:28.369599+0000","last_clean":"2024-06-14T22:35:28.369599+0000","last_became_active":"2024-06-14T22:34:45.811307+0000","last_became_peered":"2024-06-14T22:34:45.811307+0000","last_unstale":"2024-06-14T22:35:28.369599+0000","last_undegraded":"2024-06-14T22:35:28.369599+0000","last_fullsized":"2024-06-14T22:35:28.369599+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:36:07.685165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370239+0000","last_change":"2024-06-14T22:34:45.822087+0000","last_active":"2024-06-14T22:35:28.370239+0000","last_peered":"2024-06-14T22:35:28.370239+0000","last_clean":"2024-06-14T22:35:28.370239+0000","last_became_active":"2024-06-14T22:34:45.819146+0000","last_became_peered":"2024-06-14T22:34:45.819146+0000","last_unstale":"2024-06-14T22:35:28.370239+0000","last_undegraded":"2024-06-14T22:35:28.370239+0000","last_fullsized":"2024-06-14T22:35:28.370239+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:50:31.815369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627526+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:35:27.627526+0000","last_peered":"2024-06-14T22:35:27.627526+0000","last_clean":"2024-06-14T22:35:27.627526+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:35:27.627526+0000","last_undegraded":"2024-06-14T22:35:27.627526+0000","last_fullsized":"2024-06-14T22:35:27.627526+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370220+0000","last_change":"2024-06-14T22:34:45.821848+0000","last_active":"2024-06-14T22:35:28.370220+0000","last_peered":"2024-06-14T22:35:28.370220+0000","last_clean":"2024-06-14T22:35:28.370220+0000","last_became_active":"2024-06-14T22:34:45.821567+0000","last_became_peered":"2024-06-14T22:34:45.821567+0000","last_unstale":"2024-06-14T22:35:28.370220+0000","last_undegraded":"2024-06-14T22:35:28.370220+0000","last_fullsized":"2024-06-14T22:35:28.370220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:21.984416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631256+0000","last_change":"2024-06-14T22:34:45.836854+0000","last_active":"2024-06-14T22:35:27.631256+0000","last_peered":"2024-06-14T22:35:27.631256+0000","last_clean":"2024-06-14T22:35:27.631256+0000","last_became_active":"2024-06-14T22:34:45.836595+0000","last_became_peered":"2024-06-14T22:34:45.836595+0000","last_unstale":"2024-06-14T22:35:27.631256+0000","last_undegraded":"2024-06-14T22:35:27.631256+0000","last_fullsized":"2024-06-14T22:35:27.631256+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:58:46.720927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369985+0000","last_change":"2024-06-14T22:34:45.810186+0000","last_active":"2024-06-14T22:35:28.369985+0000","last_peered":"2024-06-14T22:35:28.369985+0000","last_clean":"2024-06-14T22:35:28.369985+0000","last_became_active":"2024-06-14T22:34:45.809869+0000","last_became_peered":"2024-06-14T22:34:45.809869+0000","last_unstale":"2024-06-14T22:35:28.369985+0000","last_undegraded":"2024-06-14T22:35:28.369985+0000","last_fullsized":"2024-06-14T22:35:28.369985+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:03:45.652442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369765+0000","last_change":"2024-06-14T22:34:47.048088+0000","last_active":"2024-06-14T22:35:28.369765+0000","last_peered":"2024-06-14T22:35:28.369765+0000","last_clean":"2024-06-14T22:35:28.369765+0000","last_became_active":"2024-06-14T22:34:47.047800+0000","last_became_peered":"2024-06-14T22:34:47.047800+0000","last_unstale":"2024-06-14T22:35:28.369765+0000","last_undegraded":"2024-06-14T22:35:28.369765+0000","last_fullsized":"2024-06-14T22:35:28.369765+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:58:07.949059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627736+0000","last_change":"2024-06-14T22:34:45.815407+0000","last_active":"2024-06-14T22:35:27.627736+0000","last_peered":"2024-06-14T22:35:27.627736+0000","last_clean":"2024-06-14T22:35:27.627736+0000","last_became_active":"2024-06-14T22:34:45.815128+0000","last_became_peered":"2024-06-14T22:34:45.815128+0000","last_unstale":"2024-06-14T22:35:27.627736+0000","last_undegraded":"2024-06-14T22:35:27.627736+0000","last_fullsized":"2024-06-14T22:35:27.627736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:23:30.777594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630679+0000","last_change":"2024-06-14T22:34:46.941258+0000","last_active":"2024-06-14T22:35:27.630679+0000","last_peered":"2024-06-14T22:35:27.630679+0000","last_clean":"2024-06-14T22:35:27.630679+0000","last_became_active":"2024-06-14T22:34:46.940911+0000","last_became_peered":"2024-06-14T22:34:46.940911+0000","last_unstale":"2024-06-14T22:35:27.630679+0000","last_undegraded":"2024-06-14T22:35:27.630679+0000","last_fullsized":"2024-06-14T22:35:27.630679+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:48:02.568050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632701+0000","last_change":"2024-06-14T22:34:45.828009+0000","last_active":"2024-06-14T22:35:27.632701+0000","last_peered":"2024-06-14T22:35:27.632701+0000","last_clean":"2024-06-14T22:35:27.632701+0000","last_became_active":"2024-06-14T22:34:45.827333+0000","last_became_peered":"2024-06-14T22:34:45.827333+0000","last_unstale":"2024-06-14T22:35:27.632701+0000","last_undegraded":"2024-06-14T22:35:27.632701+0000","last_fullsized":"2024-06-14T22:35:27.632701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:53.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631589+0000","last_change":"2024-06-14T22:34:46.495386+0000","last_active":"2024-06-14T22:35:27.631589+0000","last_peered":"2024-06-14T22:35:27.631589+0000","last_clean":"2024-06-14T22:35:27.631589+0000","last_became_active":"2024-06-14T22:34:46.495044+0000","last_became_peered":"2024-06-14T22:34:46.495044+0000","last_unstale":"2024-06-14T22:35:27.631589+0000","last_undegraded":"2024-06-14T22:35:27.631589+0000","last_fullsized":"2024-06-14T22:35:27.631589+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:14:25.297282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369891+0000","last_change":"2024-06-14T22:34:45.820658+0000","last_active":"2024-06-14T22:35:28.369891+0000","last_peered":"2024-06-14T22:35:28.369891+0000","last_clean":"2024-06-14T22:35:28.369891+0000","last_became_active":"2024-06-14T22:34:45.820398+0000","last_became_peered":"2024-06-14T22:34:45.820398+0000","last_unstale":"2024-06-14T22:35:28.369891+0000","last_undegraded":"2024-06-14T22:35:28.369891+0000","last_fullsized":"2024-06-14T22:35:28.369891+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:07:26.958484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-06-14T22:35:29.087859+0000","last_change":"2024-06-14T22:35:29.087859+0000","last_active":"2024-06-14T22:35:29.087859+0000","last_peered":"2024-06-14T22:35:29.087859+0000","last_clean":"2024-06-14T22:35:29.087859+0000","last_became_active":"2024-06-14T22:35:29.087470+0000","last_became_peered":"2024-06-14T22:35:29.087470+0000","last_unstale":"2024-06-14T22:35:29.087859+0000","last_undegraded":"2024-06-14T22:35:29.087859+0000","last_fullsized":"2024-06-14T22:35:29.087859+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:49:37.524268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631267+0000","last_change":"2024-06-14T22:34:45.811353+0000","last_active":"2024-06-14T22:35:27.631267+0000","last_peered":"2024-06-14T22:35:27.631267+0000","last_clean":"2024-06-14T22:35:27.631267+0000","last_became_active":"2024-06-14T22:34:45.811101+0000","last_became_peered":"2024-06-14T22:34:45.811101+0000","last_unstale":"2024-06-14T22:35:27.631267+0000","last_undegraded":"2024-06-14T22:35:27.631267+0000","last_fullsized":"2024-06-14T22:35:27.631267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:38.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631780+0000","last_change":"2024-06-14T22:34:46.494233+0000","last_active":"2024-06-14T22:35:27.631780+0000","last_peered":"2024-06-14T22:35:27.631780+0000","last_clean":"2024-06-14T22:35:27.631780+0000","last_became_active":"2024-06-14T22:34:46.493933+0000","last_became_peered":"2024-06-14T22:34:46.493933+0000","last_unstale":"2024-06-14T22:35:27.631780+0000","last_undegraded":"2024-06-14T22:35:27.631780+0000","last_fullsized":"2024-06-14T22:35:27.631780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:06:52.440503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632254+0000","last_change":"2024-06-14T22:34:45.820681+0000","last_active":"2024-06-14T22:35:27.632254+0000","last_peered":"2024-06-14T22:35:27.632254+0000","last_clean":"2024-06-14T22:35:27.632254+0000","last_became_active":"2024-06-14T22:34:45.820408+0000","last_became_peered":"2024-06-14T22:34:45.820408+0000","last_unstale":"2024-06-14T22:35:27.632254+0000","last_undegraded":"2024-06-14T22:35:27.632254+0000","last_fullsized":"2024-06-14T22:35:27.632254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:39:39.570801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630241+0000","last_change":"2024-06-14T22:34:46.941193+0000","last_active":"2024-06-14T22:35:27.630241+0000","last_peered":"2024-06-14T22:35:27.630241+0000","last_clean":"2024-06-14T22:35:27.630241+0000","last_became_active":"2024-06-14T22:34:46.940866+0000","last_became_peered":"2024-06-14T22:34:46.940866+0000","last_unstale":"2024-06-14T22:35:27.630241+0000","last_undegraded":"2024-06-14T22:35:27.630241+0000","last_fullsized":"2024-06-14T22:35:27.630241+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:09:58.642955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":277,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369696+0000","last_change":"2024-06-14T22:33:02.896315+0000","last_active":"2024-06-14T22:35:28.369696+0000","last_peered":"2024-06-14T22:35:28.369696+0000","last_clean":"2024-06-14T22:35:28.369696+0000","last_became_active":"2024-06-14T22:33:02.895971+0000","last_became_peered":"2024-06-14T22:33:02.895971+0000","last_unstale":"2024-06-14T22:35:28.369696+0000","last_undegraded":"2024-06-14T22:35:28.369696+0000","last_fullsized":"2024-06-14T22:35:28.369696+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:59:26.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627642+0000","last_change":"2024-06-14T22:33:32.889322+0000","last_active":"2024-06-14T22:35:27.627642+0000","last_peered":"2024-06-14T22:35:27.627642+0000","last_clean":"2024-06-14T22:35:27.627642+0000","last_became_active":"2024-06-14T22:33:32.888906+0000","last_became_peered":"2024-06-14T22:33:32.888906+0000","last_unstale":"2024-06-14T22:35:27.627642+0000","last_undegraded":"2024-06-14T22:35:27.627642+0000","last_fullsized":"2024-06-14T22:35:27.627642+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:44:26.407894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632553+0000","last_change":"2024-06-14T22:34:45.812679+0000","last_active":"2024-06-14T22:35:27.632553+0000","last_peered":"2024-06-14T22:35:27.632553+0000","last_clean":"2024-06-14T22:35:27.632553+0000","last_became_active":"2024-06-14T22:34:45.812413+0000","last_became_peered":"2024-06-14T22:34:45.812413+0000","last_unstale":"2024-06-14T22:35:27.632553+0000","last_undegraded":"2024-06-14T22:35:27.632553+0000","last_fullsized":"2024-06-14T22:35:27.632553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:17:20.857814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370118+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:35:28.370118+0000","last_peered":"2024-06-14T22:35:28.370118+0000","last_clean":"2024-06-14T22:35:28.370118+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:35:28.370118+0000","last_undegraded":"2024-06-14T22:35:28.370118+0000","last_fullsized":"2024-06-14T22:35:28.370118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369599+0000","last_change":"2024-06-14T22:33:32.890674+0000","last_active":"2024-06-14T22:35:28.369599+0000","last_peered":"2024-06-14T22:35:28.369599+0000","last_clean":"2024-06-14T22:35:28.369599+0000","last_became_active":"2024-06-14T22:33:32.890268+0000","last_became_peered":"2024-06-14T22:33:32.890268+0000","last_unstale":"2024-06-14T22:35:28.369599+0000","last_undegraded":"2024-06-14T22:35:28.369599+0000","last_fullsized":"2024-06-14T22:35:28.369599+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:19:58.775230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630480+0000","last_change":"2024-06-14T22:34:45.813620+0000","last_active":"2024-06-14T22:35:27.630480+0000","last_peered":"2024-06-14T22:35:27.630480+0000","last_clean":"2024-06-14T22:35:27.630480+0000","last_became_active":"2024-06-14T22:34:45.813320+0000","last_became_peered":"2024-06-14T22:34:45.813320+0000","last_unstale":"2024-06-14T22:35:27.630480+0000","last_undegraded":"2024-06-14T22:35:27.630480+0000","last_fullsized":"2024-06-14T22:35:27.630480+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:05:50.010677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-06-14T22:35:29.088026+0000","last_change":"2024-06-14T22:35:29.088026+0000","last_active":"2024-06-14T22:35:29.088026+0000","last_peered":"2024-06-14T22:35:29.088026+0000","last_clean":"2024-06-14T22:35:29.088026+0000","last_became_active":"2024-06-14T22:35:29.087671+0000","last_became_peered":"2024-06-14T22:35:29.087671+0000","last_unstale":"2024-06-14T22:35:29.088026+0000","last_undegraded":"2024-06-14T22:35:29.088026+0000","last_fullsized":"2024-06-14T22:35:29.088026+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:38:46.679989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632647+0000","last_change":"2024-06-14T22:33:33.332830+0000","last_active":"2024-06-14T22:35:27.632647+0000","last_peered":"2024-06-14T22:35:27.632647+0000","last_clean":"2024-06-14T22:35:27.632647+0000","last_became_active":"2024-06-14T22:33:33.332537+0000","last_became_peered":"2024-06-14T22:33:33.332537+0000","last_unstale":"2024-06-14T22:35:27.632647+0000","last_undegraded":"2024-06-14T22:35:27.632647+0000","last_fullsized":"2024-06-14T22:35:27.632647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:04:42.394657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629361+0000","last_change":"2024-06-14T22:34:45.825225+0000","last_active":"2024-06-14T22:35:27.629361+0000","last_peered":"2024-06-14T22:35:27.629361+0000","last_clean":"2024-06-14T22:35:27.629361+0000","last_became_active":"2024-06-14T22:34:45.824340+0000","last_became_peered":"2024-06-14T22:34:45.824340+0000","last_unstale":"2024-06-14T22:35:27.629361+0000","last_undegraded":"2024-06-14T22:35:27.629361+0000","last_fullsized":"2024-06-14T22:35:27.629361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:16.925338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630278+0000","last_change":"2024-06-14T22:34:46.825958+0000","last_active":"2024-06-14T22:35:27.630278+0000","last_peered":"2024-06-14T22:35:27.630278+0000","last_clean":"2024-06-14T22:35:27.630278+0000","last_became_active":"2024-06-14T22:34:46.825690+0000","last_became_peered":"2024-06-14T22:34:46.825690+0000","last_unstale":"2024-06-14T22:35:27.630278+0000","last_undegraded":"2024-06-14T22:35:27.630278+0000","last_fullsized":"2024-06-14T22:35:27.630278+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:05:12.707288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627588+0000","last_change":"2024-06-14T22:33:32.889827+0000","last_active":"2024-06-14T22:35:27.627588+0000","last_peered":"2024-06-14T22:35:27.627588+0000","last_clean":"2024-06-14T22:35:27.627588+0000","last_became_active":"2024-06-14T22:33:32.888965+0000","last_became_peered":"2024-06-14T22:33:32.888965+0000","last_unstale":"2024-06-14T22:35:27.627588+0000","last_undegraded":"2024-06-14T22:35:27.627588+0000","last_fullsized":"2024-06-14T22:35:27.627588+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:58:20.273165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631650+0000","last_change":"2024-06-14T22:34:45.818906+0000","last_active":"2024-06-14T22:35:27.631650+0000","last_peered":"2024-06-14T22:35:27.631650+0000","last_clean":"2024-06-14T22:35:27.631650+0000","last_became_active":"2024-06-14T22:34:45.813746+0000","last_became_peered":"2024-06-14T22:34:45.813746+0000","last_unstale":"2024-06-14T22:35:27.631650+0000","last_undegraded":"2024-06-14T22:35:27.631650+0000","last_fullsized":"2024-06-14T22:35:27.631650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:54:48.659260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629559+0000","last_change":"2024-06-14T22:34:46.826241+0000","last_active":"2024-06-14T22:35:27.629559+0000","last_peered":"2024-06-14T22:35:27.629559+0000","last_clean":"2024-06-14T22:35:27.629559+0000","last_became_active":"2024-06-14T22:34:46.825315+0000","last_became_peered":"2024-06-14T22:34:46.825315+0000","last_unstale":"2024-06-14T22:35:27.629559+0000","last_undegraded":"2024-06-14T22:35:27.629559+0000","last_fullsized":"2024-06-14T22:35:27.629559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:16:54.588027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627894+0000","last_change":"2024-06-14T22:34:45.814788+0000","last_active":"2024-06-14T22:35:27.627894+0000","last_peered":"2024-06-14T22:35:27.627894+0000","last_clean":"2024-06-14T22:35:27.627894+0000","last_became_active":"2024-06-14T22:34:45.814514+0000","last_became_peered":"2024-06-14T22:34:45.814514+0000","last_unstale":"2024-06-14T22:35:27.627894+0000","last_undegraded":"2024-06-14T22:35:27.627894+0000","last_fullsized":"2024-06-14T22:35:27.627894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:41:42.814577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629709+0000","last_change":"2024-06-14T22:33:32.889192+0000","last_active":"2024-06-14T22:35:27.629709+0000","last_peered":"2024-06-14T22:35:27.629709+0000","last_clean":"2024-06-14T22:35:27.629709+0000","last_became_active":"2024-06-14T22:33:32.888868+0000","last_became_peered":"2024-06-14T22:33:32.888868+0000","last_unstale":"2024-06-14T22:35:27.629709+0000","last_undegraded":"2024-06-14T22:35:27.629709+0000","last_fullsized":"2024-06-14T22:35:27.629709+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:44:45.963989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629404+0000","last_change":"2024-06-14T22:34:46.941210+0000","last_active":"2024-06-14T22:35:27.629404+0000","last_peered":"2024-06-14T22:35:27.629404+0000","last_clean":"2024-06-14T22:35:27.629404+0000","last_became_active":"2024-06-14T22:34:46.940895+0000","last_became_peered":"2024-06-14T22:34:46.940895+0000","last_unstale":"2024-06-14T22:35:27.629404+0000","last_undegraded":"2024-06-14T22:35:27.629404+0000","last_fullsized":"2024-06-14T22:35:27.629404+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:20:37.459295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629648+0000","last_change":"2024-06-14T22:34:45.826249+0000","last_active":"2024-06-14T22:35:27.629648+0000","last_peered":"2024-06-14T22:35:27.629648+0000","last_clean":"2024-06-14T22:35:27.629648+0000","last_became_active":"2024-06-14T22:34:45.825993+0000","last_became_peered":"2024-06-14T22:34:45.825993+0000","last_unstale":"2024-06-14T22:35:27.629648+0000","last_undegraded":"2024-06-14T22:35:27.629648+0000","last_fullsized":"2024-06-14T22:35:27.629648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:49:14.625774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370049+0000","last_change":"2024-06-14T22:33:33.356876+0000","last_active":"2024-06-14T22:35:28.370049+0000","last_peered":"2024-06-14T22:35:28.370049+0000","last_clean":"2024-06-14T22:35:28.370049+0000","last_became_active":"2024-06-14T22:33:33.356363+0000","last_became_peered":"2024-06-14T22:33:33.356363+0000","last_unstale":"2024-06-14T22:35:28.370049+0000","last_undegraded":"2024-06-14T22:35:28.370049+0000","last_fullsized":"2024-06-14T22:35:28.370049+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:44.687332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630691+0000","last_change":"2024-06-14T22:34:47.047386+0000","last_active":"2024-06-14T22:35:27.630691+0000","last_peered":"2024-06-14T22:35:27.630691+0000","last_clean":"2024-06-14T22:35:27.630691+0000","last_became_active":"2024-06-14T22:34:47.047037+0000","last_became_peered":"2024-06-14T22:34:47.047037+0000","last_unstale":"2024-06-14T22:35:27.630691+0000","last_undegraded":"2024-06-14T22:35:27.630691+0000","last_fullsized":"2024-06-14T22:35:27.630691+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:59:01.638170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.633685+0000","last_change":"2024-06-14T22:34:45.824114+0000","last_active":"2024-06-14T22:35:27.633685+0000","last_peered":"2024-06-14T22:35:27.633685+0000","last_clean":"2024-06-14T22:35:27.633685+0000","last_became_active":"2024-06-14T22:34:45.823632+0000","last_became_peered":"2024-06-14T22:34:45.823632+0000","last_unstale":"2024-06-14T22:35:27.633685+0000","last_undegraded":"2024-06-14T22:35:27.633685+0000","last_fullsized":"2024-06-14T22:35:27.633685+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:45.353525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629829+0000","last_change":"2024-06-14T22:33:33.356640+0000","last_active":"2024-06-14T22:35:27.629829+0000","last_peered":"2024-06-14T22:35:27.629829+0000","last_clean":"2024-06-14T22:35:27.629829+0000","last_became_active":"2024-06-14T22:33:33.356350+0000","last_became_peered":"2024-06-14T22:33:33.356350+0000","last_unstale":"2024-06-14T22:35:27.629829+0000","last_undegraded":"2024-06-14T22:35:27.629829+0000","last_fullsized":"2024-06-14T22:35:27.629829+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:11.719635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632162+0000","last_change":"2024-06-14T22:34:46.494728+0000","last_active":"2024-06-14T22:35:27.632162+0000","last_peered":"2024-06-14T22:35:27.632162+0000","last_clean":"2024-06-14T22:35:27.632162+0000","last_became_active":"2024-06-14T22:34:46.494329+0000","last_became_peered":"2024-06-14T22:34:46.494329+0000","last_unstale":"2024-06-14T22:35:27.632162+0000","last_undegraded":"2024-06-14T22:35:27.632162+0000","last_fullsized":"2024-06-14T22:35:27.632162+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:29:10.758774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631334+0000","last_change":"2024-06-14T22:34:45.822132+0000","last_active":"2024-06-14T22:35:27.631334+0000","last_peered":"2024-06-14T22:35:27.631334+0000","last_clean":"2024-06-14T22:35:27.631334+0000","last_became_active":"2024-06-14T22:34:45.821772+0000","last_became_peered":"2024-06-14T22:34:45.821772+0000","last_unstale":"2024-06-14T22:35:27.631334+0000","last_undegraded":"2024-06-14T22:35:27.631334+0000","last_fullsized":"2024-06-14T22:35:27.631334+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:09:15.977711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370324+0000","last_change":"2024-06-14T22:33:32.889977+0000","last_active":"2024-06-14T22:35:28.370324+0000","last_peered":"2024-06-14T22:35:28.370324+0000","last_clean":"2024-06-14T22:35:28.370324+0000","last_became_active":"2024-06-14T22:33:32.889626+0000","last_became_peered":"2024-06-14T22:33:32.889626+0000","last_unstale":"2024-06-14T22:35:28.370324+0000","last_undegraded":"2024-06-14T22:35:28.370324+0000","last_fullsized":"2024-06-14T22:35:28.370324+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:17:06.530667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631141+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:35:27.631141+0000","last_peered":"2024-06-14T22:35:27.631141+0000","last_clean":"2024-06-14T22:35:27.631141+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:35:27.631141+0000","last_undegraded":"2024-06-14T22:35:27.631141+0000","last_fullsized":"2024-06-14T22:35:27.631141+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631723+0000","last_change":"2024-06-14T22:34:46.493764+0000","last_active":"2024-06-14T22:35:27.631723+0000","last_peered":"2024-06-14T22:35:27.631723+0000","last_clean":"2024-06-14T22:35:27.631723+0000","last_became_active":"2024-06-14T22:34:46.493470+0000","last_became_peered":"2024-06-14T22:34:46.493470+0000","last_unstale":"2024-06-14T22:35:27.631723+0000","last_undegraded":"2024-06-14T22:35:27.631723+0000","last_fullsized":"2024-06-14T22:35:27.631723+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:56:25.184221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631589+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:35:27.631589+0000","last_peered":"2024-06-14T22:35:27.631589+0000","last_clean":"2024-06-14T22:35:27.631589+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:35:27.631589+0000","last_undegraded":"2024-06-14T22:35:27.631589+0000","last_fullsized":"2024-06-14T22:35:27.631589+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629413+0000","last_change":"2024-06-14T22:34:46.825597+0000","last_active":"2024-06-14T22:35:27.629413+0000","last_peered":"2024-06-14T22:35:27.629413+0000","last_clean":"2024-06-14T22:35:27.629413+0000","last_became_active":"2024-06-14T22:34:46.825326+0000","last_became_peered":"2024-06-14T22:34:46.825326+0000","last_unstale":"2024-06-14T22:35:27.629413+0000","last_undegraded":"2024-06-14T22:35:27.629413+0000","last_fullsized":"2024-06-14T22:35:27.629413+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:58:52.948909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369808+0000","last_change":"2024-06-14T22:34:45.814302+0000","last_active":"2024-06-14T22:35:28.369808+0000","last_peered":"2024-06-14T22:35:28.369808+0000","last_clean":"2024-06-14T22:35:28.369808+0000","last_became_active":"2024-06-14T22:34:45.814024+0000","last_became_peered":"2024-06-14T22:34:45.814024+0000","last_unstale":"2024-06-14T22:35:28.369808+0000","last_undegraded":"2024-06-14T22:35:28.369808+0000","last_fullsized":"2024-06-14T22:35:28.369808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:01:48.646538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627713+0000","last_change":"2024-06-14T22:34:45.821858+0000","last_active":"2024-06-14T22:35:27.627713+0000","last_peered":"2024-06-14T22:35:27.627713+0000","last_clean":"2024-06-14T22:35:27.627713+0000","last_became_active":"2024-06-14T22:34:45.821478+0000","last_became_peered":"2024-06-14T22:34:45.821478+0000","last_unstale":"2024-06-14T22:35:27.627713+0000","last_undegraded":"2024-06-14T22:35:27.627713+0000","last_fullsized":"2024-06-14T22:35:27.627713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:02:38.862686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632453+0000","last_change":"2024-06-14T22:34:46.495167+0000","last_active":"2024-06-14T22:35:27.632453+0000","last_peered":"2024-06-14T22:35:27.632453+0000","last_clean":"2024-06-14T22:35:27.632453+0000","last_became_active":"2024-06-14T22:34:46.494841+0000","last_became_peered":"2024-06-14T22:34:46.494841+0000","last_unstale":"2024-06-14T22:35:27.632453+0000","last_undegraded":"2024-06-14T22:35:27.632453+0000","last_fullsized":"2024-06-14T22:35:27.632453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:29:14.557081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630948+0000","last_change":"2024-06-14T22:34:45.816388+0000","last_active":"2024-06-14T22:35:27.630948+0000","last_peered":"2024-06-14T22:35:27.630948+0000","last_clean":"2024-06-14T22:35:27.630948+0000","last_became_active":"2024-06-14T22:34:45.816110+0000","last_became_peered":"2024-06-14T22:34:45.816110+0000","last_unstale":"2024-06-14T22:35:27.630948+0000","last_undegraded":"2024-06-14T22:35:27.630948+0000","last_fullsized":"2024-06-14T22:35:27.630948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:10:34.839829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632114+0000","last_change":"2024-06-14T22:34:46.494774+0000","last_active":"2024-06-14T22:35:27.632114+0000","last_peered":"2024-06-14T22:35:27.632114+0000","last_clean":"2024-06-14T22:35:27.632114+0000","last_became_active":"2024-06-14T22:34:46.494419+0000","last_became_peered":"2024-06-14T22:34:46.494419+0000","last_unstale":"2024-06-14T22:35:27.632114+0000","last_undegraded":"2024-06-14T22:35:27.632114+0000","last_fullsized":"2024-06-14T22:35:27.632114+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:52:53.094366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370119+0000","last_change":"2024-06-14T22:34:46.827532+0000","last_active":"2024-06-14T22:35:28.370119+0000","last_peered":"2024-06-14T22:35:28.370119+0000","last_clean":"2024-06-14T22:35:28.370119+0000","last_became_active":"2024-06-14T22:34:46.827236+0000","last_became_peered":"2024-06-14T22:34:46.827236+0000","last_unstale":"2024-06-14T22:35:28.370119+0000","last_undegraded":"2024-06-14T22:35:28.370119+0000","last_fullsized":"2024-06-14T22:35:28.370119+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:35:44.648613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632225+0000","last_change":"2024-06-14T22:34:45.822922+0000","last_active":"2024-06-14T22:35:27.632225+0000","last_peered":"2024-06-14T22:35:27.632225+0000","last_clean":"2024-06-14T22:35:27.632225+0000","last_became_active":"2024-06-14T22:34:45.822512+0000","last_became_peered":"2024-06-14T22:34:45.822512+0000","last_unstale":"2024-06-14T22:35:27.632225+0000","last_undegraded":"2024-06-14T22:35:27.632225+0000","last_fullsized":"2024-06-14T22:35:27.632225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:30.970649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630776+0000","last_change":"2024-06-14T22:34:47.047452+0000","last_active":"2024-06-14T22:35:27.630776+0000","last_peered":"2024-06-14T22:35:27.630776+0000","last_clean":"2024-06-14T22:35:27.630776+0000","last_became_active":"2024-06-14T22:34:47.047212+0000","last_became_peered":"2024-06-14T22:34:47.047212+0000","last_unstale":"2024-06-14T22:35:27.630776+0000","last_undegraded":"2024-06-14T22:35:27.630776+0000","last_fullsized":"2024-06-14T22:35:27.630776+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:59.291709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629413+0000","last_change":"2024-06-14T22:34:45.829172+0000","last_active":"2024-06-14T22:35:27.629413+0000","last_peered":"2024-06-14T22:35:27.629413+0000","last_clean":"2024-06-14T22:35:27.629413+0000","last_became_active":"2024-06-14T22:34:45.828859+0000","last_became_peered":"2024-06-14T22:34:45.828859+0000","last_unstale":"2024-06-14T22:35:27.629413+0000","last_undegraded":"2024-06-14T22:35:27.629413+0000","last_fullsized":"2024-06-14T22:35:27.629413+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:29:46.335986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-06-14T22:35:29.086555+0000","last_change":"2024-06-14T22:35:29.086555+0000","last_active":"2024-06-14T22:35:29.086555+0000","last_peered":"2024-06-14T22:35:29.086555+0000","last_clean":"2024-06-14T22:35:29.086555+0000","last_became_active":"2024-06-14T22:35:29.086297+0000","last_became_peered":"2024-06-14T22:35:29.086297+0000","last_unstale":"2024-06-14T22:35:29.086555+0000","last_undegraded":"2024-06-14T22:35:29.086555+0000","last_fullsized":"2024-06-14T22:35:29.086555+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:33:26.603821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631441+0000","last_change":"2024-06-14T22:34:45.822913+0000","last_active":"2024-06-14T22:35:27.631441+0000","last_peered":"2024-06-14T22:35:27.631441+0000","last_clean":"2024-06-14T22:35:27.631441+0000","last_became_active":"2024-06-14T22:34:45.822330+0000","last_became_peered":"2024-06-14T22:34:45.822330+0000","last_unstale":"2024-06-14T22:35:27.631441+0000","last_undegraded":"2024-06-14T22:35:27.631441+0000","last_fullsized":"2024-06-14T22:35:27.631441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:17:22.638197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632605+0000","last_change":"2024-06-14T22:34:45.836816+0000","last_active":"2024-06-14T22:35:27.632605+0000","last_peered":"2024-06-14T22:35:27.632605+0000","last_clean":"2024-06-14T22:35:27.632605+0000","last_became_active":"2024-06-14T22:34:45.836535+0000","last_became_peered":"2024-06-14T22:34:45.836535+0000","last_unstale":"2024-06-14T22:35:27.632605+0000","last_undegraded":"2024-06-14T22:35:27.632605+0000","last_fullsized":"2024-06-14T22:35:27.632605+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:53:49.296682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630820+0000","last_change":"2024-06-14T22:34:45.811823+0000","last_active":"2024-06-14T22:35:27.630820+0000","last_peered":"2024-06-14T22:35:27.630820+0000","last_clean":"2024-06-14T22:35:27.630820+0000","last_became_active":"2024-06-14T22:34:45.811440+0000","last_became_peered":"2024-06-14T22:34:45.811440+0000","last_unstale":"2024-06-14T22:35:27.630820+0000","last_undegraded":"2024-06-14T22:35:27.630820+0000","last_fullsized":"2024-06-14T22:35:27.630820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:03.265328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631557+0000","last_change":"2024-06-14T22:34:45.836936+0000","last_active":"2024-06-14T22:35:27.631557+0000","last_peered":"2024-06-14T22:35:27.631557+0000","last_clean":"2024-06-14T22:35:27.631557+0000","last_became_active":"2024-06-14T22:34:45.836687+0000","last_became_peered":"2024-06-14T22:34:45.836687+0000","last_unstale":"2024-06-14T22:35:27.631557+0000","last_undegraded":"2024-06-14T22:35:27.631557+0000","last_fullsized":"2024-06-14T22:35:27.631557+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:44:49.187441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627662+0000","last_change":"2024-06-14T22:34:45.816712+0000","last_active":"2024-06-14T22:35:27.627662+0000","last_peered":"2024-06-14T22:35:27.627662+0000","last_clean":"2024-06-14T22:35:27.627662+0000","last_became_active":"2024-06-14T22:34:45.816438+0000","last_became_peered":"2024-06-14T22:34:45.816438+0000","last_unstale":"2024-06-14T22:35:27.627662+0000","last_undegraded":"2024-06-14T22:35:27.627662+0000","last_fullsized":"2024-06-14T22:35:27.627662+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:02:31.452920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631363+0000","last_change":"2024-06-14T22:34:46.247253+0000","last_active":"2024-06-14T22:35:27.631363+0000","last_peered":"2024-06-14T22:35:27.631363+0000","last_clean":"2024-06-14T22:35:27.631363+0000","last_became_active":"2024-06-14T22:34:46.247020+0000","last_became_peered":"2024-06-14T22:34:46.247020+0000","last_unstale":"2024-06-14T22:35:27.631363+0000","last_undegraded":"2024-06-14T22:35:27.631363+0000","last_fullsized":"2024-06-14T22:35:27.631363+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:59:33.350586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627837+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:35:27.627837+0000","last_peered":"2024-06-14T22:35:27.627837+0000","last_clean":"2024-06-14T22:35:27.627837+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:35:27.627837+0000","last_undegraded":"2024-06-14T22:35:27.627837+0000","last_fullsized":"2024-06-14T22:35:27.627837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.633132+0000","last_change":"2024-06-14T22:34:46.246908+0000","last_active":"2024-06-14T22:35:27.633132+0000","last_peered":"2024-06-14T22:35:27.633132+0000","last_clean":"2024-06-14T22:35:27.633132+0000","last_became_active":"2024-06-14T22:34:46.246554+0000","last_became_peered":"2024-06-14T22:34:46.246554+0000","last_unstale":"2024-06-14T22:35:27.633132+0000","last_undegraded":"2024-06-14T22:35:27.633132+0000","last_fullsized":"2024-06-14T22:35:27.633132+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:49.875407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.633568+0000","last_change":"2024-06-14T22:34:45.814299+0000","last_active":"2024-06-14T22:35:27.633568+0000","last_peered":"2024-06-14T22:35:27.633568+0000","last_clean":"2024-06-14T22:35:27.633568+0000","last_became_active":"2024-06-14T22:34:45.814017+0000","last_became_peered":"2024-06-14T22:34:45.814017+0000","last_unstale":"2024-06-14T22:35:27.633568+0000","last_undegraded":"2024-06-14T22:35:27.633568+0000","last_fullsized":"2024-06-14T22:35:27.633568+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:49:50.153404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631064+0000","last_change":"2024-06-14T22:34:46.494835+0000","last_active":"2024-06-14T22:35:27.631064+0000","last_peered":"2024-06-14T22:35:27.631064+0000","last_clean":"2024-06-14T22:35:27.631064+0000","last_became_active":"2024-06-14T22:34:46.494588+0000","last_became_peered":"2024-06-14T22:34:46.494588+0000","last_unstale":"2024-06-14T22:35:27.631064+0000","last_undegraded":"2024-06-14T22:35:27.631064+0000","last_fullsized":"2024-06-14T22:35:27.631064+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:51:28.347865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630922+0000","last_change":"2024-06-14T22:34:45.810611+0000","last_active":"2024-06-14T22:35:27.630922+0000","last_peered":"2024-06-14T22:35:27.630922+0000","last_clean":"2024-06-14T22:35:27.630922+0000","last_became_active":"2024-06-14T22:34:45.810339+0000","last_became_peered":"2024-06-14T22:34:45.810339+0000","last_unstale":"2024-06-14T22:35:27.630922+0000","last_undegraded":"2024-06-14T22:35:27.630922+0000","last_fullsized":"2024-06-14T22:35:27.630922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:10:25.525863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234020,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298668,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446484,"statfs":{"total":95995035648,"available":95689199616,"internally_reserved":0,"allocated":1339392,"data_stored":905786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.47799999999999998,"15min":0.46200000000000002},"min":{"1min":0.41499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.505,"15min":0.496},"min":{"1min":0.40799999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.58699999999999997}]},{"osd":1,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.50600000000000001,"15min":0.49399999999999999},"min":{"1min":0.42799999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.49299999999999999,"15min":0.48299999999999998},"min":{"1min":0.41999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.48199999999999998,"15min":0.47099999999999997},"min":{"1min":0.44500000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.49399999999999999,"15min":0.47899999999999998},"min":{"1min":0.43099999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.505},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.40799999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55500000000000005}]},{"osd":4,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.442,"15min":0.437},"min":{"1min":0.29399999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68600000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41099999999999998}]},{"osd":5,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72299999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51900000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.746,"5min":0.753,"15min":0.753},"last":0.438},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72399999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.56899999999999995}]}]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298652,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446500,"statfs":{"total":95995035648,"available":95689216000,"internally_reserved":0,"allocated":1323008,"data_stored":905317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.502},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66400000000000003,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.35999999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65000000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35999999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.62,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.498},{"interface":"front","average":{"1min":0.5,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.63600000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.376}]},{"osd":2,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.499,"15min":0.495},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70299999999999996,"5min":0.752,"15min":0.752},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.40999999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51200000000000001,"15min":0.52400000000000002},"min":{"1min":0.38,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68600000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.438},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52000000000000002,"15min":0.52500000000000002},"min":{"1min":0.40400000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.42599999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.41099999999999998,"15min":0.40699999999999997},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.61599999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.29699999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.629,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.309}]},{"osd":5,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.44400000000000001,"15min":0.42999999999999999},"min":{"1min":0.34300000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.72199999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.47399999999999998,"15min":0.46400000000000002},"min":{"1min":0.32800000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70499999999999996,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.53800000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298064,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447088,"statfs":{"total":95995035648,"available":95689818112,"internally_reserved":0,"allocated":720896,"data_stored":313967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66900000000000004,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.41199999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68700000000000006,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.38200000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69899999999999995,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67500000000000004,"5min":0.748,"15min":0.748},"last":0.44800000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.67600000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.621},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68300000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.53300000000000003}]},{"osd":3,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.53800000000000003},"min":{"1min":0.38700000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.497},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69999999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.47499999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.30399999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.42499999999999999,"15min":0.40300000000000002},"min":{"1min":0.312,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64900000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46800000000000003,"15min":0.45500000000000002},"min":{"1min":0.32000000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.47499999999999998,"15min":0.46000000000000002},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71399999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Jun 14 22:35:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.52000000000000002}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298080,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447072,"statfs":{"total":95995035648,"available":95689801728,"internally_reserved":0,"allocated":737280,"data_stored":315083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67600000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.502},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.61599999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.48099999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.66100000000000003,"5min":0.755,"15min":0.755},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66800000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72299999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67400000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Jun 14 22:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54700000000000004,"15min":0.55900000000000005},"min":{"1min":0.376,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69399999999999995,"5min":0.748,"15min":0.748},"last":0.432},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65100000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.44900000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39700000000000002,"15min":0.39000000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.63,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36399999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42499999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Jun 14 22:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.64100000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.48399999999999999,"15min":0.47799999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.57899999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298064,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447088,"statfs":{"total":95995035648,"available":95689818112,"internally_reserved":0,"allocated":720896,"data_stored":313740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.372,"15min":0.38300000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.45100000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.316},{"interface":"front","average":{"1min":0.318,"5min":0.36699999999999999,"15min":0.38500000000000001},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.49399999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.27000000000000002}]},{"osd":1,"last update":"Fri Jun 14 22:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.45400000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38200000000000001,"15min":0.39800000000000002},"min":{"1min":0.26700000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.46899999999999997,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.21299999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.27100000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51800000000000002,"5min":1.276,"15min":1.276},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39500000000000002,"15min":0.39900000000000002},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53900000000000003,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.38,"5min":0.371,"15min":0.371},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.40500000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.39200000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68899999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.379}]},{"osd":6,"last update":"Fri Jun 14 22:34:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72799999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.39400000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.59199999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.60699999999999998,"15min":0.621},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76000000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.38700000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298064,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447088,"statfs":{"total":95995035648,"available":95689818112,"internally_reserved":0,"allocated":724992,"data_stored":314254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.375,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55700000000000005,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.374,"15min":0.379},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.48599999999999999,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.255}]},{"osd":1,"last update":"Fri Jun 14 22:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.371,"15min":0.373},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45400000000000001,"5min":0.629,"15min":0.629},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.48799999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.30499999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.43099999999999999,"15min":0.44700000000000001},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.51400000000000001,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.41099999999999998,"15min":0.442},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.498,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.317}]},{"osd":4,"last update":"Fri Jun 14 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.41099999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72599999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.36499999999999999}]},{"osd":5,"last update":"Fri Jun 14 22:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69499999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38200000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.54800000000000004},"min":{"1min":0.39600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.745,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.40300000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60599999999999998,"15min":0.61299999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68200000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.433}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298052,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447100,"statfs":{"total":95995035648,"available":95689830400,"internally_reserved":0,"allocated":716800,"data_stored":313650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.33100000000000002,"15min":0.29499999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51400000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.22},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.32800000000000001,"15min":0.28999999999999998},"min":{"1min":0.22600000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45000000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.315}]},{"osd":2,"last update":"Fri Jun 14 22:35:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36699999999999999,"15min":0.36799999999999999},"min":{"1min":0.23699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.53900000000000003,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.25},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36299999999999999,"15min":0.35899999999999999},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.26300000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.43099999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56200000000000006,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.42499999999999999,"15min":0.45200000000000001},"min":{"1min":0.27000000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.29099999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.66100000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70799999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.55500000000000005}]},{"osd":5,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55900000000000005,"15min":0.57699999999999996},"min":{"1min":0.39200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78300000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.34399999999999997}]},{"osd":6,"last update":"Fri Jun 14 22:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.58499999999999996,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.66000000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72599999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.372}]},{"osd":7,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96299999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.69199999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.437}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298640,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446512,"statfs":{"total":95995035648,"available":95689228288,"internally_reserved":0,"allocated":1318912,"data_stored":904610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.33200000000000002,"15min":0.29299999999999998},"min":{"1min":0.19400000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.34499999999999997,"15min":0.312},"min":{"1min":0.216,"5min":0.183,"15min":0.183},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.28100000000000003}]},{"osd":2,"last update":"Fri Jun 14 22:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36699999999999999,"15min":0.35899999999999999},"min":{"1min":0.25600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.313},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36299999999999999,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.20399999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:35:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40799999999999997,"15min":0.42199999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.56100000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.38700000000000001,"15min":0.39700000000000002},"min":{"1min":0.27300000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56399999999999995,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.24099999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.42599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53700000000000003}]},{"osd":6,"last update":"Fri Jun 14 22:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.40999999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.56200000000000006}]},{"osd":7,"last update":"Fri Jun 14 22:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.439,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:35:46.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:35:46.820 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:46 smithi086 ceph-mon[38904]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:46.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:46 smithi086 ceph-mon[38904]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:35:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:46 smithi086 ceph-mon[29889]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:46.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:46 smithi086 ceph-mon[29889]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:35:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:46 smithi137 ceph-mon[32951]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:46 smithi137 ceph-mon[32951]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:35:48.672 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:35:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:35:48.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:48 smithi086 ceph-mon[29889]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:48.966 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:48 smithi086 ceph-mon[38904]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:49.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:48 smithi137 ceph-mon[32951]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:49.382 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-06-14T22:35:47.555903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592706,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":224,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167,"ondisk_log_size":167,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386284,"kb_used_data":7424,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747574932,"statfs":{"total":767960285184,"available":765516730368,"internally_reserved":0,"allocated":7602176,"data_stored":4286407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002522"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632890+0000","last_change":"2024-06-14T22:34:46.493127+0000","last_active":"2024-06-14T22:35:27.632890+0000","last_peered":"2024-06-14T22:35:27.632890+0000","last_clean":"2024-06-14T22:35:27.632890+0000","last_became_active":"2024-06-14T22:34:46.492766+0000","last_became_peered":"2024-06-14T22:34:46.492766+0000","last_unstale":"2024-06-14T22:35:27.632890+0000","last_undegraded":"2024-06-14T22:35:27.632890+0000","last_fullsized":"2024-06-14T22:35:27.632890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:31:41.431477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370073+0000","last_change":"2024-06-14T22:34:45.814323+0000","last_active":"2024-06-14T22:35:28.370073+0000","last_peered":"2024-06-14T22:35:28.370073+0000","last_clean":"2024-06-14T22:35:28.370073+0000","last_became_active":"2024-06-14T22:34:45.814041+0000","last_became_peered":"2024-06-14T22:34:45.814041+0000","last_unstale":"2024-06-14T22:35:28.370073+0000","last_undegraded":"2024-06-14T22:35:28.370073+0000","last_fullsized":"2024-06-14T22:35:28.370073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:07:38.056038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627797+0000","last_change":"2024-06-14T22:34:46.494622+0000","last_active":"2024-06-14T22:35:27.627797+0000","last_peered":"2024-06-14T22:35:27.627797+0000","last_clean":"2024-06-14T22:35:27.627797+0000","last_became_active":"2024-06-14T22:34:46.494331+0000","last_became_peered":"2024-06-14T22:34:46.494331+0000","last_unstale":"2024-06-14T22:35:27.627797+0000","last_undegraded":"2024-06-14T22:35:27.627797+0000","last_fullsized":"2024-06-14T22:35:27.627797+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:23:44.353759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630479+0000","last_change":"2024-06-14T22:34:45.817622+0000","last_active":"2024-06-14T22:35:27.630479+0000","last_peered":"2024-06-14T22:35:27.630479+0000","last_clean":"2024-06-14T22:35:27.630479+0000","last_became_active":"2024-06-14T22:34:45.817350+0000","last_became_peered":"2024-06-14T22:34:45.817350+0000","last_unstale":"2024-06-14T22:35:27.630479+0000","last_undegraded":"2024-06-14T22:35:27.630479+0000","last_fullsized":"2024-06-14T22:35:27.630479+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:45:12.669650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631250+0000","last_change":"2024-06-14T22:34:47.047477+0000","last_active":"2024-06-14T22:35:27.631250+0000","last_peered":"2024-06-14T22:35:27.631250+0000","last_clean":"2024-06-14T22:35:27.631250+0000","last_became_active":"2024-06-14T22:34:47.047206+0000","last_became_peered":"2024-06-14T22:34:47.047206+0000","last_unstale":"2024-06-14T22:35:27.631250+0000","last_undegraded":"2024-06-14T22:35:27.631250+0000","last_fullsized":"2024-06-14T22:35:27.631250+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:54:08.872928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369787+0000","last_change":"2024-06-14T22:34:45.813267+0000","last_active":"2024-06-14T22:35:28.369787+0000","last_peered":"2024-06-14T22:35:28.369787+0000","last_clean":"2024-06-14T22:35:28.369787+0000","last_became_active":"2024-06-14T22:34:45.813003+0000","last_became_peered":"2024-06-14T22:34:45.813003+0000","last_unstale":"2024-06-14T22:35:28.369787+0000","last_undegraded":"2024-06-14T22:35:28.369787+0000","last_fullsized":"2024-06-14T22:35:28.369787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:46.446483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.633616+0000","last_change":"2024-06-14T22:34:46.494754+0000","last_active":"2024-06-14T22:35:27.633616+0000","last_peered":"2024-06-14T22:35:27.633616+0000","last_clean":"2024-06-14T22:35:27.633616+0000","last_became_active":"2024-06-14T22:34:46.494412+0000","last_became_peered":"2024-06-14T22:34:46.494412+0000","last_unstale":"2024-06-14T22:35:27.633616+0000","last_undegraded":"2024-06-14T22:35:27.633616+0000","last_fullsized":"2024-06-14T22:35:27.633616+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:51.607597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369599+0000","last_change":"2024-06-14T22:34:45.811556+0000","last_active":"2024-06-14T22:35:28.369599+0000","last_peered":"2024-06-14T22:35:28.369599+0000","last_clean":"2024-06-14T22:35:28.369599+0000","last_became_active":"2024-06-14T22:34:45.811307+0000","last_became_peered":"2024-06-14T22:34:45.811307+0000","last_unstale":"2024-06-14T22:35:28.369599+0000","last_undegraded":"2024-06-14T22:35:28.369599+0000","last_fullsized":"2024-06-14T22:35:28.369599+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:36:07.685165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370239+0000","last_change":"2024-06-14T22:34:45.822087+0000","last_active":"2024-06-14T22:35:28.370239+0000","last_peered":"2024-06-14T22:35:28.370239+0000","last_clean":"2024-06-14T22:35:28.370239+0000","last_became_active":"2024-06-14T22:34:45.819146+0000","last_became_peered":"2024-06-14T22:34:45.819146+0000","last_unstale":"2024-06-14T22:35:28.370239+0000","last_undegraded":"2024-06-14T22:35:28.370239+0000","last_fullsized":"2024-06-14T22:35:28.370239+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:50:31.815369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627526+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:35:27.627526+0000","last_peered":"2024-06-14T22:35:27.627526+0000","last_clean":"2024-06-14T22:35:27.627526+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:35:27.627526+0000","last_undegraded":"2024-06-14T22:35:27.627526+0000","last_fullsized":"2024-06-14T22:35:27.627526+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370220+0000","last_change":"2024-06-14T22:34:45.821848+0000","last_active":"2024-06-14T22:35:28.370220+0000","last_peered":"2024-06-14T22:35:28.370220+0000","last_clean":"2024-06-14T22:35:28.370220+0000","last_became_active":"2024-06-14T22:34:45.821567+0000","last_became_peered":"2024-06-14T22:34:45.821567+0000","last_unstale":"2024-06-14T22:35:28.370220+0000","last_undegraded":"2024-06-14T22:35:28.370220+0000","last_fullsized":"2024-06-14T22:35:28.370220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:21.984416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631256+0000","last_change":"2024-06-14T22:34:45.836854+0000","last_active":"2024-06-14T22:35:27.631256+0000","last_peered":"2024-06-14T22:35:27.631256+0000","last_clean":"2024-06-14T22:35:27.631256+0000","last_became_active":"2024-06-14T22:34:45.836595+0000","last_became_peered":"2024-06-14T22:34:45.836595+0000","last_unstale":"2024-06-14T22:35:27.631256+0000","last_undegraded":"2024-06-14T22:35:27.631256+0000","last_fullsized":"2024-06-14T22:35:27.631256+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:58:46.720927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369985+0000","last_change":"2024-06-14T22:34:45.810186+0000","last_active":"2024-06-14T22:35:28.369985+0000","last_peered":"2024-06-14T22:35:28.369985+0000","last_clean":"2024-06-14T22:35:28.369985+0000","last_became_active":"2024-06-14T22:34:45.809869+0000","last_became_peered":"2024-06-14T22:34:45.809869+0000","last_unstale":"2024-06-14T22:35:28.369985+0000","last_undegraded":"2024-06-14T22:35:28.369985+0000","last_fullsized":"2024-06-14T22:35:28.369985+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:03:45.652442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369765+0000","last_change":"2024-06-14T22:34:47.048088+0000","last_active":"2024-06-14T22:35:28.369765+0000","last_peered":"2024-06-14T22:35:28.369765+0000","last_clean":"2024-06-14T22:35:28.369765+0000","last_became_active":"2024-06-14T22:34:47.047800+0000","last_became_peered":"2024-06-14T22:34:47.047800+0000","last_unstale":"2024-06-14T22:35:28.369765+0000","last_undegraded":"2024-06-14T22:35:28.369765+0000","last_fullsized":"2024-06-14T22:35:28.369765+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:58:07.949059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627736+0000","last_change":"2024-06-14T22:34:45.815407+0000","last_active":"2024-06-14T22:35:27.627736+0000","last_peered":"2024-06-14T22:35:27.627736+0000","last_clean":"2024-06-14T22:35:27.627736+0000","last_became_active":"2024-06-14T22:34:45.815128+0000","last_became_peered":"2024-06-14T22:34:45.815128+0000","last_unstale":"2024-06-14T22:35:27.627736+0000","last_undegraded":"2024-06-14T22:35:27.627736+0000","last_fullsized":"2024-06-14T22:35:27.627736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:23:30.777594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630679+0000","last_change":"2024-06-14T22:34:46.941258+0000","last_active":"2024-06-14T22:35:27.630679+0000","last_peered":"2024-06-14T22:35:27.630679+0000","last_clean":"2024-06-14T22:35:27.630679+0000","last_became_active":"2024-06-14T22:34:46.940911+0000","last_became_peered":"2024-06-14T22:34:46.940911+0000","last_unstale":"2024-06-14T22:35:27.630679+0000","last_undegraded":"2024-06-14T22:35:27.630679+0000","last_fullsized":"2024-06-14T22:35:27.630679+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:48:02.568050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632701+0000","last_change":"2024-06-14T22:34:45.828009+0000","last_active":"2024-06-14T22:35:27.632701+0000","last_peered":"2024-06-14T22:35:27.632701+0000","last_clean":"2024-06-14T22:35:27.632701+0000","last_became_active":"2024-06-14T22:34:45.827333+0000","last_became_peered":"2024-06-14T22:34:45.827333+0000","last_unstale":"2024-06-14T22:35:27.632701+0000","last_undegraded":"2024-06-14T22:35:27.632701+0000","last_fullsized":"2024-06-14T22:35:27.632701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:53.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631589+0000","last_change":"2024-06-14T22:34:46.495386+0000","last_active":"2024-06-14T22:35:27.631589+0000","last_peered":"2024-06-14T22:35:27.631589+0000","last_clean":"2024-06-14T22:35:27.631589+0000","last_became_active":"2024-06-14T22:34:46.495044+0000","last_became_peered":"2024-06-14T22:34:46.495044+0000","last_unstale":"2024-06-14T22:35:27.631589+0000","last_undegraded":"2024-06-14T22:35:27.631589+0000","last_fullsized":"2024-06-14T22:35:27.631589+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:14:25.297282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369891+0000","last_change":"2024-06-14T22:34:45.820658+0000","last_active":"2024-06-14T22:35:28.369891+0000","last_peered":"2024-06-14T22:35:28.369891+0000","last_clean":"2024-06-14T22:35:28.369891+0000","last_became_active":"2024-06-14T22:34:45.820398+0000","last_became_peered":"2024-06-14T22:34:45.820398+0000","last_unstale":"2024-06-14T22:35:28.369891+0000","last_undegraded":"2024-06-14T22:35:28.369891+0000","last_fullsized":"2024-06-14T22:35:28.369891+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:07:26.958484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-06-14T22:35:29.087859+0000","last_change":"2024-06-14T22:35:29.087859+0000","last_active":"2024-06-14T22:35:29.087859+0000","last_peered":"2024-06-14T22:35:29.087859+0000","last_clean":"2024-06-14T22:35:29.087859+0000","last_became_active":"2024-06-14T22:35:29.087470+0000","last_became_peered":"2024-06-14T22:35:29.087470+0000","last_unstale":"2024-06-14T22:35:29.087859+0000","last_undegraded":"2024-06-14T22:35:29.087859+0000","last_fullsized":"2024-06-14T22:35:29.087859+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:49:37.524268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631267+0000","last_change":"2024-06-14T22:34:45.811353+0000","last_active":"2024-06-14T22:35:27.631267+0000","last_peered":"2024-06-14T22:35:27.631267+0000","last_clean":"2024-06-14T22:35:27.631267+0000","last_became_active":"2024-06-14T22:34:45.811101+0000","last_became_peered":"2024-06-14T22:34:45.811101+0000","last_unstale":"2024-06-14T22:35:27.631267+0000","last_undegraded":"2024-06-14T22:35:27.631267+0000","last_fullsized":"2024-06-14T22:35:27.631267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:38.603420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631780+0000","last_change":"2024-06-14T22:34:46.494233+0000","last_active":"2024-06-14T22:35:27.631780+0000","last_peered":"2024-06-14T22:35:27.631780+0000","last_clean":"2024-06-14T22:35:27.631780+0000","last_became_active":"2024-06-14T22:34:46.493933+0000","last_became_peered":"2024-06-14T22:34:46.493933+0000","last_unstale":"2024-06-14T22:35:27.631780+0000","last_undegraded":"2024-06-14T22:35:27.631780+0000","last_fullsized":"2024-06-14T22:35:27.631780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:06:52.440503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632254+0000","last_change":"2024-06-14T22:34:45.820681+0000","last_active":"2024-06-14T22:35:27.632254+0000","last_peered":"2024-06-14T22:35:27.632254+0000","last_clean":"2024-06-14T22:35:27.632254+0000","last_became_active":"2024-06-14T22:34:45.820408+0000","last_became_peered":"2024-06-14T22:34:45.820408+0000","last_unstale":"2024-06-14T22:35:27.632254+0000","last_undegraded":"2024-06-14T22:35:27.632254+0000","last_fullsized":"2024-06-14T22:35:27.632254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:39:39.570801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630241+0000","last_change":"2024-06-14T22:34:46.941193+0000","last_active":"2024-06-14T22:35:27.630241+0000","last_peered":"2024-06-14T22:35:27.630241+0000","last_clean":"2024-06-14T22:35:27.630241+0000","last_became_active":"2024-06-14T22:34:46.940866+0000","last_became_peered":"2024-06-14T22:34:46.940866+0000","last_unstale":"2024-06-14T22:35:27.630241+0000","last_undegraded":"2024-06-14T22:35:27.630241+0000","last_fullsized":"2024-06-14T22:35:27.630241+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:09:58.642955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":277,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369696+0000","last_change":"2024-06-14T22:33:02.896315+0000","last_active":"2024-06-14T22:35:28.369696+0000","last_peered":"2024-06-14T22:35:28.369696+0000","last_clean":"2024-06-14T22:35:28.369696+0000","last_became_active":"2024-06-14T22:33:02.895971+0000","last_became_peered":"2024-06-14T22:33:02.895971+0000","last_unstale":"2024-06-14T22:35:28.369696+0000","last_undegraded":"2024-06-14T22:35:28.369696+0000","last_fullsized":"2024-06-14T22:35:28.369696+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:59:26.863559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627642+0000","last_change":"2024-06-14T22:33:32.889322+0000","last_active":"2024-06-14T22:35:27.627642+0000","last_peered":"2024-06-14T22:35:27.627642+0000","last_clean":"2024-06-14T22:35:27.627642+0000","last_became_active":"2024-06-14T22:33:32.888906+0000","last_became_peered":"2024-06-14T22:33:32.888906+0000","last_unstale":"2024-06-14T22:35:27.627642+0000","last_undegraded":"2024-06-14T22:35:27.627642+0000","last_fullsized":"2024-06-14T22:35:27.627642+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:44:26.407894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632553+0000","last_change":"2024-06-14T22:34:45.812679+0000","last_active":"2024-06-14T22:35:27.632553+0000","last_peered":"2024-06-14T22:35:27.632553+0000","last_clean":"2024-06-14T22:35:27.632553+0000","last_became_active":"2024-06-14T22:34:45.812413+0000","last_became_peered":"2024-06-14T22:34:45.812413+0000","last_unstale":"2024-06-14T22:35:27.632553+0000","last_undegraded":"2024-06-14T22:35:27.632553+0000","last_fullsized":"2024-06-14T22:35:27.632553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:17:20.857814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370118+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:35:28.370118+0000","last_peered":"2024-06-14T22:35:28.370118+0000","last_clean":"2024-06-14T22:35:28.370118+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:35:28.370118+0000","last_undegraded":"2024-06-14T22:35:28.370118+0000","last_fullsized":"2024-06-14T22:35:28.370118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369599+0000","last_change":"2024-06-14T22:33:32.890674+0000","last_active":"2024-06-14T22:35:28.369599+0000","last_peered":"2024-06-14T22:35:28.369599+0000","last_clean":"2024-06-14T22:35:28.369599+0000","last_became_active":"2024-06-14T22:33:32.890268+0000","last_became_peered":"2024-06-14T22:33:32.890268+0000","last_unstale":"2024-06-14T22:35:28.369599+0000","last_undegraded":"2024-06-14T22:35:28.369599+0000","last_fullsized":"2024-06-14T22:35:28.369599+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:19:58.775230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630480+0000","last_change":"2024-06-14T22:34:45.813620+0000","last_active":"2024-06-14T22:35:27.630480+0000","last_peered":"2024-06-14T22:35:27.630480+0000","last_clean":"2024-06-14T22:35:27.630480+0000","last_became_active":"2024-06-14T22:34:45.813320+0000","last_became_peered":"2024-06-14T22:34:45.813320+0000","last_unstale":"2024-06-14T22:35:27.630480+0000","last_undegraded":"2024-06-14T22:35:27.630480+0000","last_fullsized":"2024-06-14T22:35:27.630480+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:05:50.010677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-06-14T22:35:29.088026+0000","last_change":"2024-06-14T22:35:29.088026+0000","last_active":"2024-06-14T22:35:29.088026+0000","last_peered":"2024-06-14T22:35:29.088026+0000","last_clean":"2024-06-14T22:35:29.088026+0000","last_became_active":"2024-06-14T22:35:29.087671+0000","last_became_peered":"2024-06-14T22:35:29.087671+0000","last_unstale":"2024-06-14T22:35:29.088026+0000","last_undegraded":"2024-06-14T22:35:29.088026+0000","last_fullsized":"2024-06-14T22:35:29.088026+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:38:46.679989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632647+0000","last_change":"2024-06-14T22:33:33.332830+0000","last_active":"2024-06-14T22:35:27.632647+0000","last_peered":"2024-06-14T22:35:27.632647+0000","last_clean":"2024-06-14T22:35:27.632647+0000","last_became_active":"2024-06-14T22:33:33.332537+0000","last_became_peered":"2024-06-14T22:33:33.332537+0000","last_unstale":"2024-06-14T22:35:27.632647+0000","last_undegraded":"2024-06-14T22:35:27.632647+0000","last_fullsized":"2024-06-14T22:35:27.632647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:04:42.394657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629361+0000","last_change":"2024-06-14T22:34:45.825225+0000","last_active":"2024-06-14T22:35:27.629361+0000","last_peered":"2024-06-14T22:35:27.629361+0000","last_clean":"2024-06-14T22:35:27.629361+0000","last_became_active":"2024-06-14T22:34:45.824340+0000","last_became_peered":"2024-06-14T22:34:45.824340+0000","last_unstale":"2024-06-14T22:35:27.629361+0000","last_undegraded":"2024-06-14T22:35:27.629361+0000","last_fullsized":"2024-06-14T22:35:27.629361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:16.925338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630278+0000","last_change":"2024-06-14T22:34:46.825958+0000","last_active":"2024-06-14T22:35:27.630278+0000","last_peered":"2024-06-14T22:35:27.630278+0000","last_clean":"2024-06-14T22:35:27.630278+0000","last_became_active":"2024-06-14T22:34:46.825690+0000","last_became_peered":"2024-06-14T22:34:46.825690+0000","last_unstale":"2024-06-14T22:35:27.630278+0000","last_undegraded":"2024-06-14T22:35:27.630278+0000","last_fullsized":"2024-06-14T22:35:27.630278+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:05:12.707288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627588+0000","last_change":"2024-06-14T22:33:32.889827+0000","last_active":"2024-06-14T22:35:27.627588+0000","last_peered":"2024-06-14T22:35:27.627588+0000","last_clean":"2024-06-14T22:35:27.627588+0000","last_became_active":"2024-06-14T22:33:32.888965+0000","last_became_peered":"2024-06-14T22:33:32.888965+0000","last_unstale":"2024-06-14T22:35:27.627588+0000","last_undegraded":"2024-06-14T22:35:27.627588+0000","last_fullsized":"2024-06-14T22:35:27.627588+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:58:20.273165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631650+0000","last_change":"2024-06-14T22:34:45.818906+0000","last_active":"2024-06-14T22:35:27.631650+0000","last_peered":"2024-06-14T22:35:27.631650+0000","last_clean":"2024-06-14T22:35:27.631650+0000","last_became_active":"2024-06-14T22:34:45.813746+0000","last_became_peered":"2024-06-14T22:34:45.813746+0000","last_unstale":"2024-06-14T22:35:27.631650+0000","last_undegraded":"2024-06-14T22:35:27.631650+0000","last_fullsized":"2024-06-14T22:35:27.631650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:54:48.659260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629559+0000","last_change":"2024-06-14T22:34:46.826241+0000","last_active":"2024-06-14T22:35:27.629559+0000","last_peered":"2024-06-14T22:35:27.629559+0000","last_clean":"2024-06-14T22:35:27.629559+0000","last_became_active":"2024-06-14T22:34:46.825315+0000","last_became_peered":"2024-06-14T22:34:46.825315+0000","last_unstale":"2024-06-14T22:35:27.629559+0000","last_undegraded":"2024-06-14T22:35:27.629559+0000","last_fullsized":"2024-06-14T22:35:27.629559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:16:54.588027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627894+0000","last_change":"2024-06-14T22:34:45.814788+0000","last_active":"2024-06-14T22:35:27.627894+0000","last_peered":"2024-06-14T22:35:27.627894+0000","last_clean":"2024-06-14T22:35:27.627894+0000","last_became_active":"2024-06-14T22:34:45.814514+0000","last_became_peered":"2024-06-14T22:34:45.814514+0000","last_unstale":"2024-06-14T22:35:27.627894+0000","last_undegraded":"2024-06-14T22:35:27.627894+0000","last_fullsized":"2024-06-14T22:35:27.627894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:41:42.814577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629709+0000","last_change":"2024-06-14T22:33:32.889192+0000","last_active":"2024-06-14T22:35:27.629709+0000","last_peered":"2024-06-14T22:35:27.629709+0000","last_clean":"2024-06-14T22:35:27.629709+0000","last_became_active":"2024-06-14T22:33:32.888868+0000","last_became_peered":"2024-06-14T22:33:32.888868+0000","last_unstale":"2024-06-14T22:35:27.629709+0000","last_undegraded":"2024-06-14T22:35:27.629709+0000","last_fullsized":"2024-06-14T22:35:27.629709+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:44:45.963989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629404+0000","last_change":"2024-06-14T22:34:46.941210+0000","last_active":"2024-06-14T22:35:27.629404+0000","last_peered":"2024-06-14T22:35:27.629404+0000","last_clean":"2024-06-14T22:35:27.629404+0000","last_became_active":"2024-06-14T22:34:46.940895+0000","last_became_peered":"2024-06-14T22:34:46.940895+0000","last_unstale":"2024-06-14T22:35:27.629404+0000","last_undegraded":"2024-06-14T22:35:27.629404+0000","last_fullsized":"2024-06-14T22:35:27.629404+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:20:37.459295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629648+0000","last_change":"2024-06-14T22:34:45.826249+0000","last_active":"2024-06-14T22:35:27.629648+0000","last_peered":"2024-06-14T22:35:27.629648+0000","last_clean":"2024-06-14T22:35:27.629648+0000","last_became_active":"2024-06-14T22:34:45.825993+0000","last_became_peered":"2024-06-14T22:34:45.825993+0000","last_unstale":"2024-06-14T22:35:27.629648+0000","last_undegraded":"2024-06-14T22:35:27.629648+0000","last_fullsized":"2024-06-14T22:35:27.629648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:49:14.625774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370049+0000","last_change":"2024-06-14T22:33:33.356876+0000","last_active":"2024-06-14T22:35:28.370049+0000","last_peered":"2024-06-14T22:35:28.370049+0000","last_clean":"2024-06-14T22:35:28.370049+0000","last_became_active":"2024-06-14T22:33:33.356363+0000","last_became_peered":"2024-06-14T22:33:33.356363+0000","last_unstale":"2024-06-14T22:35:28.370049+0000","last_undegraded":"2024-06-14T22:35:28.370049+0000","last_fullsized":"2024-06-14T22:35:28.370049+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:44.687332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630691+0000","last_change":"2024-06-14T22:34:47.047386+0000","last_active":"2024-06-14T22:35:27.630691+0000","last_peered":"2024-06-14T22:35:27.630691+0000","last_clean":"2024-06-14T22:35:27.630691+0000","last_became_active":"2024-06-14T22:34:47.047037+0000","last_became_peered":"2024-06-14T22:34:47.047037+0000","last_unstale":"2024-06-14T22:35:27.630691+0000","last_undegraded":"2024-06-14T22:35:27.630691+0000","last_fullsized":"2024-06-14T22:35:27.630691+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:59:01.638170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.633685+0000","last_change":"2024-06-14T22:34:45.824114+0000","last_active":"2024-06-14T22:35:27.633685+0000","last_peered":"2024-06-14T22:35:27.633685+0000","last_clean":"2024-06-14T22:35:27.633685+0000","last_became_active":"2024-06-14T22:34:45.823632+0000","last_became_peered":"2024-06-14T22:34:45.823632+0000","last_unstale":"2024-06-14T22:35:27.633685+0000","last_undegraded":"2024-06-14T22:35:27.633685+0000","last_fullsized":"2024-06-14T22:35:27.633685+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:45.353525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629829+0000","last_change":"2024-06-14T22:33:33.356640+0000","last_active":"2024-06-14T22:35:27.629829+0000","last_peered":"2024-06-14T22:35:27.629829+0000","last_clean":"2024-06-14T22:35:27.629829+0000","last_became_active":"2024-06-14T22:33:33.356350+0000","last_became_peered":"2024-06-14T22:33:33.356350+0000","last_unstale":"2024-06-14T22:35:27.629829+0000","last_undegraded":"2024-06-14T22:35:27.629829+0000","last_fullsized":"2024-06-14T22:35:27.629829+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:11.719635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632162+0000","last_change":"2024-06-14T22:34:46.494728+0000","last_active":"2024-06-14T22:35:27.632162+0000","last_peered":"2024-06-14T22:35:27.632162+0000","last_clean":"2024-06-14T22:35:27.632162+0000","last_became_active":"2024-06-14T22:34:46.494329+0000","last_became_peered":"2024-06-14T22:34:46.494329+0000","last_unstale":"2024-06-14T22:35:27.632162+0000","last_undegraded":"2024-06-14T22:35:27.632162+0000","last_fullsized":"2024-06-14T22:35:27.632162+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:29:10.758774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631334+0000","last_change":"2024-06-14T22:34:45.822132+0000","last_active":"2024-06-14T22:35:27.631334+0000","last_peered":"2024-06-14T22:35:27.631334+0000","last_clean":"2024-06-14T22:35:27.631334+0000","last_became_active":"2024-06-14T22:34:45.821772+0000","last_became_peered":"2024-06-14T22:34:45.821772+0000","last_unstale":"2024-06-14T22:35:27.631334+0000","last_undegraded":"2024-06-14T22:35:27.631334+0000","last_fullsized":"2024-06-14T22:35:27.631334+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:09:15.977711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370324+0000","last_change":"2024-06-14T22:33:32.889977+0000","last_active":"2024-06-14T22:35:28.370324+0000","last_peered":"2024-06-14T22:35:28.370324+0000","last_clean":"2024-06-14T22:35:28.370324+0000","last_became_active":"2024-06-14T22:33:32.889626+0000","last_became_peered":"2024-06-14T22:33:32.889626+0000","last_unstale":"2024-06-14T22:35:28.370324+0000","last_undegraded":"2024-06-14T22:35:28.370324+0000","last_fullsized":"2024-06-14T22:35:28.370324+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:17:06.530667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631141+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:35:27.631141+0000","last_peered":"2024-06-14T22:35:27.631141+0000","last_clean":"2024-06-14T22:35:27.631141+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:35:27.631141+0000","last_undegraded":"2024-06-14T22:35:27.631141+0000","last_fullsized":"2024-06-14T22:35:27.631141+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631723+0000","last_change":"2024-06-14T22:34:46.493764+0000","last_active":"2024-06-14T22:35:27.631723+0000","last_peered":"2024-06-14T22:35:27.631723+0000","last_clean":"2024-06-14T22:35:27.631723+0000","last_became_active":"2024-06-14T22:34:46.493470+0000","last_became_peered":"2024-06-14T22:34:46.493470+0000","last_unstale":"2024-06-14T22:35:27.631723+0000","last_undegraded":"2024-06-14T22:35:27.631723+0000","last_fullsized":"2024-06-14T22:35:27.631723+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:56:25.184221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"68'6","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631589+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:35:27.631589+0000","last_peered":"2024-06-14T22:35:27.631589+0000","last_clean":"2024-06-14T22:35:27.631589+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:35:27.631589+0000","last_undegraded":"2024-06-14T22:35:27.631589+0000","last_fullsized":"2024-06-14T22:35:27.631589+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629413+0000","last_change":"2024-06-14T22:34:46.825597+0000","last_active":"2024-06-14T22:35:27.629413+0000","last_peered":"2024-06-14T22:35:27.629413+0000","last_clean":"2024-06-14T22:35:27.629413+0000","last_became_active":"2024-06-14T22:34:46.825326+0000","last_became_peered":"2024-06-14T22:34:46.825326+0000","last_unstale":"2024-06-14T22:35:27.629413+0000","last_undegraded":"2024-06-14T22:35:27.629413+0000","last_fullsized":"2024-06-14T22:35:27.629413+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:58:52.948909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.369808+0000","last_change":"2024-06-14T22:34:45.814302+0000","last_active":"2024-06-14T22:35:28.369808+0000","last_peered":"2024-06-14T22:35:28.369808+0000","last_clean":"2024-06-14T22:35:28.369808+0000","last_became_active":"2024-06-14T22:34:45.814024+0000","last_became_peered":"2024-06-14T22:34:45.814024+0000","last_unstale":"2024-06-14T22:35:28.369808+0000","last_undegraded":"2024-06-14T22:35:28.369808+0000","last_fullsized":"2024-06-14T22:35:28.369808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:01:48.646538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627713+0000","last_change":"2024-06-14T22:34:45.821858+0000","last_active":"2024-06-14T22:35:27.627713+0000","last_peered":"2024-06-14T22:35:27.627713+0000","last_clean":"2024-06-14T22:35:27.627713+0000","last_became_active":"2024-06-14T22:34:45.821478+0000","last_became_peered":"2024-06-14T22:34:45.821478+0000","last_unstale":"2024-06-14T22:35:27.627713+0000","last_undegraded":"2024-06-14T22:35:27.627713+0000","last_fullsized":"2024-06-14T22:35:27.627713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:02:38.862686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632453+0000","last_change":"2024-06-14T22:34:46.495167+0000","last_active":"2024-06-14T22:35:27.632453+0000","last_peered":"2024-06-14T22:35:27.632453+0000","last_clean":"2024-06-14T22:35:27.632453+0000","last_became_active":"2024-06-14T22:34:46.494841+0000","last_became_peered":"2024-06-14T22:34:46.494841+0000","last_unstale":"2024-06-14T22:35:27.632453+0000","last_undegraded":"2024-06-14T22:35:27.632453+0000","last_fullsized":"2024-06-14T22:35:27.632453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:29:14.557081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630948+0000","last_change":"2024-06-14T22:34:45.816388+0000","last_active":"2024-06-14T22:35:27.630948+0000","last_peered":"2024-06-14T22:35:27.630948+0000","last_clean":"2024-06-14T22:35:27.630948+0000","last_became_active":"2024-06-14T22:34:45.816110+0000","last_became_peered":"2024-06-14T22:34:45.816110+0000","last_unstale":"2024-06-14T22:35:27.630948+0000","last_undegraded":"2024-06-14T22:35:27.630948+0000","last_fullsized":"2024-06-14T22:35:27.630948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:10:34.839829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632114+0000","last_change":"2024-06-14T22:34:46.494774+0000","last_active":"2024-06-14T22:35:27.632114+0000","last_peered":"2024-06-14T22:35:27.632114+0000","last_clean":"2024-06-14T22:35:27.632114+0000","last_became_active":"2024-06-14T22:34:46.494419+0000","last_became_peered":"2024-06-14T22:34:46.494419+0000","last_unstale":"2024-06-14T22:35:27.632114+0000","last_undegraded":"2024-06-14T22:35:27.632114+0000","last_fullsized":"2024-06-14T22:35:27.632114+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:52:53.094366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:28.370119+0000","last_change":"2024-06-14T22:34:46.827532+0000","last_active":"2024-06-14T22:35:28.370119+0000","last_peered":"2024-06-14T22:35:28.370119+0000","last_clean":"2024-06-14T22:35:28.370119+0000","last_became_active":"2024-06-14T22:34:46.827236+0000","last_became_peered":"2024-06-14T22:34:46.827236+0000","last_unstale":"2024-06-14T22:35:28.370119+0000","last_undegraded":"2024-06-14T22:35:28.370119+0000","last_fullsized":"2024-06-14T22:35:28.370119+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:35:44.648613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632225+0000","last_change":"2024-06-14T22:34:45.822922+0000","last_active":"2024-06-14T22:35:27.632225+0000","last_peered":"2024-06-14T22:35:27.632225+0000","last_clean":"2024-06-14T22:35:27.632225+0000","last_became_active":"2024-06-14T22:34:45.822512+0000","last_became_peered":"2024-06-14T22:34:45.822512+0000","last_unstale":"2024-06-14T22:35:27.632225+0000","last_undegraded":"2024-06-14T22:35:27.632225+0000","last_fullsized":"2024-06-14T22:35:27.632225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:30.970649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630776+0000","last_change":"2024-06-14T22:34:47.047452+0000","last_active":"2024-06-14T22:35:27.630776+0000","last_peered":"2024-06-14T22:35:27.630776+0000","last_clean":"2024-06-14T22:35:27.630776+0000","last_became_active":"2024-06-14T22:34:47.047212+0000","last_became_peered":"2024-06-14T22:34:47.047212+0000","last_unstale":"2024-06-14T22:35:27.630776+0000","last_undegraded":"2024-06-14T22:35:27.630776+0000","last_fullsized":"2024-06-14T22:35:27.630776+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:59.291709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.629413+0000","last_change":"2024-06-14T22:34:45.829172+0000","last_active":"2024-06-14T22:35:27.629413+0000","last_peered":"2024-06-14T22:35:27.629413+0000","last_clean":"2024-06-14T22:35:27.629413+0000","last_became_active":"2024-06-14T22:34:45.828859+0000","last_became_peered":"2024-06-14T22:34:45.828859+0000","last_unstale":"2024-06-14T22:35:27.629413+0000","last_undegraded":"2024-06-14T22:35:27.629413+0000","last_fullsized":"2024-06-14T22:35:27.629413+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:29:46.335986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-06-14T22:35:29.086555+0000","last_change":"2024-06-14T22:35:29.086555+0000","last_active":"2024-06-14T22:35:29.086555+0000","last_peered":"2024-06-14T22:35:29.086555+0000","last_clean":"2024-06-14T22:35:29.086555+0000","last_became_active":"2024-06-14T22:35:29.086297+0000","last_became_peered":"2024-06-14T22:35:29.086297+0000","last_unstale":"2024-06-14T22:35:29.086555+0000","last_undegraded":"2024-06-14T22:35:29.086555+0000","last_fullsized":"2024-06-14T22:35:29.086555+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:33:26.603821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631441+0000","last_change":"2024-06-14T22:34:45.822913+0000","last_active":"2024-06-14T22:35:27.631441+0000","last_peered":"2024-06-14T22:35:27.631441+0000","last_clean":"2024-06-14T22:35:27.631441+0000","last_became_active":"2024-06-14T22:34:45.822330+0000","last_became_peered":"2024-06-14T22:34:45.822330+0000","last_unstale":"2024-06-14T22:35:27.631441+0000","last_undegraded":"2024-06-14T22:35:27.631441+0000","last_fullsized":"2024-06-14T22:35:27.631441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:17:22.638197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.632605+0000","last_change":"2024-06-14T22:34:45.836816+0000","last_active":"2024-06-14T22:35:27.632605+0000","last_peered":"2024-06-14T22:35:27.632605+0000","last_clean":"2024-06-14T22:35:27.632605+0000","last_became_active":"2024-06-14T22:34:45.836535+0000","last_became_peered":"2024-06-14T22:34:45.836535+0000","last_unstale":"2024-06-14T22:35:27.632605+0000","last_undegraded":"2024-06-14T22:35:27.632605+0000","last_fullsized":"2024-06-14T22:35:27.632605+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:53:49.296682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630820+0000","last_change":"2024-06-14T22:34:45.811823+0000","last_active":"2024-06-14T22:35:27.630820+0000","last_peered":"2024-06-14T22:35:27.630820+0000","last_clean":"2024-06-14T22:35:27.630820+0000","last_became_active":"2024-06-14T22:34:45.811440+0000","last_became_peered":"2024-06-14T22:34:45.811440+0000","last_unstale":"2024-06-14T22:35:27.630820+0000","last_undegraded":"2024-06-14T22:35:27.630820+0000","last_fullsized":"2024-06-14T22:35:27.630820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:03.265328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631557+0000","last_change":"2024-06-14T22:34:45.836936+0000","last_active":"2024-06-14T22:35:27.631557+0000","last_peered":"2024-06-14T22:35:27.631557+0000","last_clean":"2024-06-14T22:35:27.631557+0000","last_became_active":"2024-06-14T22:34:45.836687+0000","last_became_peered":"2024-06-14T22:34:45.836687+0000","last_unstale":"2024-06-14T22:35:27.631557+0000","last_undegraded":"2024-06-14T22:35:27.631557+0000","last_fullsized":"2024-06-14T22:35:27.631557+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:44:49.187441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627662+0000","last_change":"2024-06-14T22:34:45.816712+0000","last_active":"2024-06-14T22:35:27.627662+0000","last_peered":"2024-06-14T22:35:27.627662+0000","last_clean":"2024-06-14T22:35:27.627662+0000","last_became_active":"2024-06-14T22:34:45.816438+0000","last_became_peered":"2024-06-14T22:34:45.816438+0000","last_unstale":"2024-06-14T22:35:27.627662+0000","last_undegraded":"2024-06-14T22:35:27.627662+0000","last_fullsized":"2024-06-14T22:35:27.627662+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:02:31.452920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631363+0000","last_change":"2024-06-14T22:34:46.247253+0000","last_active":"2024-06-14T22:35:27.631363+0000","last_peered":"2024-06-14T22:35:27.631363+0000","last_clean":"2024-06-14T22:35:27.631363+0000","last_became_active":"2024-06-14T22:34:46.247020+0000","last_became_peered":"2024-06-14T22:34:46.247020+0000","last_unstale":"2024-06-14T22:35:27.631363+0000","last_undegraded":"2024-06-14T22:35:27.631363+0000","last_fullsized":"2024-06-14T22:35:27.631363+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:59:33.350586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.627837+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:35:27.627837+0000","last_peered":"2024-06-14T22:35:27.627837+0000","last_clean":"2024-06-14T22:35:27.627837+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:35:27.627837+0000","last_undegraded":"2024-06-14T22:35:27.627837+0000","last_fullsized":"2024-06-14T22:35:27.627837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.633132+0000","last_change":"2024-06-14T22:34:46.246908+0000","last_active":"2024-06-14T22:35:27.633132+0000","last_peered":"2024-06-14T22:35:27.633132+0000","last_clean":"2024-06-14T22:35:27.633132+0000","last_became_active":"2024-06-14T22:34:46.246554+0000","last_became_peered":"2024-06-14T22:34:46.246554+0000","last_unstale":"2024-06-14T22:35:27.633132+0000","last_undegraded":"2024-06-14T22:35:27.633132+0000","last_fullsized":"2024-06-14T22:35:27.633132+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:49.875407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.633568+0000","last_change":"2024-06-14T22:34:45.814299+0000","last_active":"2024-06-14T22:35:27.633568+0000","last_peered":"2024-06-14T22:35:27.633568+0000","last_clean":"2024-06-14T22:35:27.633568+0000","last_became_active":"2024-06-14T22:34:45.814017+0000","last_became_peered":"2024-06-14T22:34:45.814017+0000","last_unstale":"2024-06-14T22:35:27.633568+0000","last_undegraded":"2024-06-14T22:35:27.633568+0000","last_fullsized":"2024-06-14T22:35:27.633568+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:49:50.153404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.631064+0000","last_change":"2024-06-14T22:34:46.494835+0000","last_active":"2024-06-14T22:35:27.631064+0000","last_peered":"2024-06-14T22:35:27.631064+0000","last_clean":"2024-06-14T22:35:27.631064+0000","last_became_active":"2024-06-14T22:34:46.494588+0000","last_became_peered":"2024-06-14T22:34:46.494588+0000","last_unstale":"2024-06-14T22:35:27.631064+0000","last_undegraded":"2024-06-14T22:35:27.631064+0000","last_fullsized":"2024-06-14T22:35:27.631064+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:51:28.347865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-06-14T22:35:27.630922+0000","last_change":"2024-06-14T22:34:45.810611+0000","last_active":"2024-06-14T22:35:27.630922+0000","last_peered":"2024-06-14T22:35:27.630922+0000","last_clean":"2024-06-14T22:35:27.630922+0000","last_became_active":"2024-06-14T22:34:45.810339+0000","last_became_peered":"2024-06-14T22:34:45.810339+0000","last_unstale":"2024-06-14T22:35:27.630922+0000","last_undegraded":"2024-06-14T22:35:27.630922+0000","last_fullsized":"2024-06-14T22:35:27.630922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:10:25.525863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234020,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298668,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446484,"statfs":{"total":95995035648,"available":95689199616,"internally_reserved":0,"allocated":1339392,"data_stored":905786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.47799999999999998,"15min":0.46200000000000002},"min":{"1min":0.41499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.505,"15min":0.496},"min":{"1min":0.40799999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.58699999999999997}]},{"osd":1,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.50600000000000001,"15min":0.49399999999999999},"min":{"1min":0.42799999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.49299999999999999,"15min":0.48299999999999998},"min":{"1min":0.41999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.48199999999999998,"15min":0.47099999999999997},"min":{"1min":0.44500000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.49399999999999999,"15min":0.47899999999999998},"min":{"1min":0.43099999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.505},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.40799999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55500000000000005}]},{"osd":4,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.442,"15min":0.437},"min":{"1min":0.29399999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68600000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41099999999999998}]},{"osd":5,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72299999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51900000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.746,"5min":0.753,"15min":0.753},"last":0.438},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72399999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.56899999999999995}]}]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298652,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446500,"statfs":{"total":95995035648,"available":95689216000,"internally_reserved":0,"allocated":1323008,"data_stored":905317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.502},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66400000000000003,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.51500000000000001},"min":{"1min":0.35999999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65000000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35999999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.62,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.498},{"interface":"front","average":{"1min":0.5,"5min":0.51100000000000001,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.63600000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.376}]},{"osd":2,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.499,"15min":0.495},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70299999999999996,"5min":0.752,"15min":0.752},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.40400000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71799999999999997,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.40999999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51200000000000001,"15min":0.52400000000000002},"min":{"1min":0.38,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68600000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.438},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52000000000000002,"15min":0.52500000000000002},"min":{"1min":0.40400000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73999999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.42599999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.41099999999999998,"15min":0.40699999999999997},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.61599999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.29699999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.629,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.309}]},{"osd":5,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.44400000000000001,"15min":0.42999999999999999},"min":{"1min":0.34300000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.72199999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.47399999999999998,"15min":0.46400000000000002},"min":{"1min":0.32800000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70499999999999996,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.53800000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298064,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447088,"statfs":{"total":95995035648,"available":95689818112,"internally_reserved":0,"allocated":720896,"data_stored":313967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66900000000000004,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.41199999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68700000000000006,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.38200000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55400000000000005,"15min":0.55900000000000005},"min":{"1min":0.41799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69899999999999995,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.39100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67500000000000004,"5min":0.748,"15min":0.748},"last":0.44800000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.37,"15min":0.37},"max":{"1min":0.67600000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.621},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57599999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68300000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.53300000000000003}]},{"osd":3,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.53800000000000003},"min":{"1min":0.38700000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.497},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69999999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.47499999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.30399999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.42499999999999999,"15min":0.40300000000000002},"min":{"1min":0.312,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64900000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46800000000000003,"15min":0.45500000000000002},"min":{"1min":0.32000000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.47499999999999998,"15min":0.46000000000000002},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71399999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Jun 14 22:35:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.47599999999999998,"15min":0.47199999999999998},"min":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.52000000000000002}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298080,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447072,"statfs":{"total":95995035648,"available":95689801728,"internally_reserved":0,"allocated":737280,"data_stored":315083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67600000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.502},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.61599999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.48099999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.66100000000000003,"5min":0.755,"15min":0.755},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66800000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72299999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67400000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Jun 14 22:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54700000000000004,"15min":0.55900000000000005},"min":{"1min":0.376,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69399999999999995,"5min":0.748,"15min":0.748},"last":0.432},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65100000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.44900000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:35:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39700000000000002,"15min":0.39000000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.63,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36399999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42899999999999999,"15min":0.42499999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Jun 14 22:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.64100000000000001,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.48399999999999999,"15min":0.47799999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.57899999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298064,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447088,"statfs":{"total":95995035648,"available":95689818112,"internally_reserved":0,"allocated":720896,"data_stored":313740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.372,"15min":0.38300000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.45100000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.316},{"interface":"front","average":{"1min":0.318,"5min":0.36699999999999999,"15min":0.38500000000000001},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.49399999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.27000000000000002}]},{"osd":1,"last update":"Fri Jun 14 22:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.45400000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38200000000000001,"15min":0.39800000000000002},"min":{"1min":0.26700000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.46899999999999997,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.21299999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.27100000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51800000000000002,"5min":1.276,"15min":1.276},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.39500000000000002,"15min":0.39900000000000002},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53900000000000003,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.38,"5min":0.371,"15min":0.371},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.40500000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.39200000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.55800000000000005,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.68899999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.379}]},{"osd":6,"last update":"Fri Jun 14 22:34:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72799999999999998,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.39400000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.59199999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.60699999999999998,"15min":0.621},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76000000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.38700000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298064,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447088,"statfs":{"total":95995035648,"available":95689818112,"internally_reserved":0,"allocated":724992,"data_stored":314254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.375,"15min":0.36699999999999999},"min":{"1min":0.25900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55700000000000005,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.374,"15min":0.379},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.48599999999999999,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.255}]},{"osd":1,"last update":"Fri Jun 14 22:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.371,"15min":0.373},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45400000000000001,"5min":0.629,"15min":0.629},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.48799999999999999,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.30499999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.43099999999999999,"15min":0.44700000000000001},"min":{"1min":0.28399999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.51400000000000001,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.41099999999999998,"15min":0.442},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.498,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.317}]},{"osd":4,"last update":"Fri Jun 14 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.41099999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72599999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.36499999999999999}]},{"osd":5,"last update":"Fri Jun 14 22:35:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73499999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69499999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38200000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.54800000000000004},"min":{"1min":0.39600000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.745,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.40300000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60599999999999998,"15min":0.61299999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68200000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.433}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298052,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447100,"statfs":{"total":95995035648,"available":95689830400,"internally_reserved":0,"allocated":716800,"data_stored":313650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.33100000000000002,"15min":0.29499999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.51400000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.22},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.32800000000000001,"15min":0.28999999999999998},"min":{"1min":0.22600000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45000000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.315}]},{"osd":2,"last update":"Fri Jun 14 22:35:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36699999999999999,"15min":0.36799999999999999},"min":{"1min":0.23699999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.53900000000000003,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.25},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36299999999999999,"15min":0.35899999999999999},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.26300000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.43099999999999999,"15min":0.45500000000000002},"min":{"1min":0.29899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56200000000000006,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.42499999999999999,"15min":0.45200000000000001},"min":{"1min":0.27000000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.29099999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.66100000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38800000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70799999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.55500000000000005}]},{"osd":5,"last update":"Fri Jun 14 22:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55900000000000005,"15min":0.57699999999999996},"min":{"1min":0.39200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78300000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56200000000000006,"15min":0.57299999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.34399999999999997}]},{"osd":6,"last update":"Fri Jun 14 22:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.58499999999999996,"15min":0.59599999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.66000000000000003,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72599999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.372}]},{"osd":7,"last update":"Fri Jun 14 22:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.96299999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.69199999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.437}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298640,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446512,"statfs":{"total":95995035648,"available":95689228288,"internally_reserved":0,"allocated":1318912,"data_stored":904610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:35:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.33200000000000002,"15min":0.29299999999999998},"min":{"1min":0.19400000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.34499999999999997,"15min":0.312},"min":{"1min":0.216,"5min":0.183,"15min":0.183},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.28100000000000003}]},{"osd":2,"last update":"Fri Jun 14 22:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36699999999999999,"15min":0.35899999999999999},"min":{"1min":0.25600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.313},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36299999999999999,"15min":0.35099999999999998},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.20399999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:35:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.40799999999999997,"15min":0.42199999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.56100000000000005,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.38700000000000001,"15min":0.39700000000000002},"min":{"1min":0.27300000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.56399999999999995,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.24099999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.42599999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53700000000000003}]},{"osd":6,"last update":"Fri Jun 14 22:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.40999999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.56200000000000006}]},{"osd":7,"last update":"Fri Jun 14 22:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.439,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:35:49.385 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-14T22:35:49.386 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-14T22:35:49.386 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-14T22:35:49.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-14T22:35:49.634 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:49.744 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:49] "GET /metrics HTTP/1.1" 200 239164 "" "Prometheus/2.43.0" 2024-06-14T22:35:49.745 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:49 smithi086 ceph-mon[29889]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:35:50.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:49 smithi086 ceph-mon[38904]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:35:50.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:49 smithi137 ceph-mon[32951]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:35:50.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:50 smithi086 ceph-mon[29889]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:50.931 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:50 smithi086 ceph-mon[38904]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:50.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:50 smithi137 ceph-mon[32951]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:50.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:35:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:35:51.679 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:35:51.679 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-14T22:35:52.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:51 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3762127330' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-14T22:35:52.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:51 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3762127330' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-14T22:35:52.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:51 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3762127330' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-14T22:35:52.440 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-14T22:35:52.440 INFO:teuthology.run_tasks:Running task print... 2024-06-14T22:35:52.453 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-06-14T22:35:52.453 INFO:teuthology.run_tasks:Running task parallel... 2024-06-14T22:35:52.489 INFO:teuthology.task.parallel:starting parallel... 2024-06-14T22:35:52.489 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-06-14T22:35:52.490 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-06-14T22:35:52.491 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T22:35:52.493 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T22:35:52.493 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-06-14T22:35:52.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-06-14T22:35:52.493 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-06-14T22:35:52.494 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-06-14T22:35:52.494 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi086.front.sepia.ceph.com 2024-06-14T22:35:52.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-06-14T22:35:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:53.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:52 smithi086 ceph-mon[29889]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:53.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:52 smithi086 ceph-mon[38904]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:53.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:52 smithi137 ceph-mon[32951]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:54 smithi086 ceph-mon[29889]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:55.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:54 smithi086 ceph-mon[38904]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:54 smithi137 ceph-mon[32951]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:56.053 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:55 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:35:55] ENGINE Bus STOPPING 2024-06-14T22:35:56.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:35:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T22:35:56.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:35:56] ENGINE Bus STOPPED 2024-06-14T22:35:56.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:35:56] ENGINE Bus STARTING 2024-06-14T22:35:56.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:35:56] ENGINE Serving on http://:::9283 2024-06-14T22:35:56.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:35:56] ENGINE Bus STARTED 2024-06-14T22:35:56.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph config set global log_to_journald false --force' 2024-06-14T22:35:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:57.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[38904]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:57.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3055038397' entity='client.admin' 2024-06-14T22:35:57.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:35:57.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:35:57.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:35:57.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:57.193 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[29889]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:57.193 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3055038397' entity='client.admin' 2024-06-14T22:35:57.193 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:35:57.193 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:35:57.193 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:35:57.193 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:56 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:57.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:56 smithi137 ceph-mon[32951]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:57.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:56 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3055038397' entity='client.admin' 2024-06-14T22:35:57.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:56 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:35:57.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:56 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:35:57.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:56 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:35:57.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:56 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:35:57.905 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_0' created 2024-06-14T22:35:58.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:57 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1547941728' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-14T22:35:58.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:57 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-14T22:35:58.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:57 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:57 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1547941728' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-14T22:35:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:57 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-14T22:35:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:57 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:58.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:57 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1547941728' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-14T22:35:58.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:57 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-14T22:35:58.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:57 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:35:58.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-06-14T22:35:59.238 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:58 smithi086 ceph-mon[29889]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:59.238 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:58 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-06-14T22:35:59.238 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:58 smithi086 ceph-mon[29889]: osdmap e71: 8 total, 8 up, 8 in 2024-06-14T22:35:59.239 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:58 smithi086 ceph-mon[38904]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:59.239 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:58 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-06-14T22:35:59.239 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:58 smithi086 ceph-mon[38904]: osdmap e71: 8 total, 8 up, 8 in 2024-06-14T22:35:59.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:58 smithi137 ceph-mon[32951]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:35:59.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:58 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-06-14T22:35:59.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:58 smithi137 ceph-mon[32951]: osdmap e71: 8 total, 8 up, 8 in 2024-06-14T22:35:59.553 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:35:59.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:35:59 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:35:59] "GET /metrics HTTP/1.1" 200 239142 "" "Prometheus/2.43.0" 2024-06-14T22:36:00.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:35:59 smithi137 ceph-mon[32951]: osdmap e72: 8 total, 8 up, 8 in 2024-06-14T22:36:00.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:35:59 smithi086 ceph-mon[29889]: osdmap e72: 8 total, 8 up, 8 in 2024-06-14T22:36:00.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:35:59 smithi086 ceph-mon[38904]: osdmap e72: 8 total, 8 up, 8 in 2024-06-14T22:36:00.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-06-14T22:36:00.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:00 smithi137 ceph-mon[32951]: pgmap v92: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:00.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:00 smithi137 ceph-mon[32951]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:36:00.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:00 smithi137 ceph-mon[32951]: osdmap e73: 8 total, 8 up, 8 in 2024-06-14T22:36:00.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:36:00 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:36:01.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:00 smithi086 ceph-mon[29889]: pgmap v92: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:01.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:00 smithi086 ceph-mon[29889]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:36:01.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:00 smithi086 ceph-mon[29889]: osdmap e73: 8 total, 8 up, 8 in 2024-06-14T22:36:01.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:00 smithi086 ceph-mon[38904]: pgmap v92: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:01.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:00 smithi086 ceph-mon[38904]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:36:01.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:00 smithi086 ceph-mon[38904]: osdmap e73: 8 total, 8 up, 8 in 2024-06-14T22:36:02.119 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:36:02.922 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-06-14T22:36:03.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:02 smithi137 ceph-mon[32951]: pgmap v94: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:03.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:02 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4257976745' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:36:03.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:02 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:36:03.328 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:02 smithi086 ceph-mon[29889]: pgmap v94: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:03.328 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:02 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4257976745' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:36:03.328 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:02 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:36:03.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:02 smithi086 ceph-mon[38904]: pgmap v94: 89 pgs: 16 creating+peering, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:03.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:02 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4257976745' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:36:03.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:02 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:36:03.925 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:03 smithi086 ceph-mon[29889]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:36:03.925 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:03 smithi086 ceph-mon[29889]: osdmap e74: 8 total, 8 up, 8 in 2024-06-14T22:36:04.225 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:03 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:36:04.225 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:03 smithi086 ceph-mon[38904]: osdmap e74: 8 total, 8 up, 8 in 2024-06-14T22:36:04.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:03 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:36:04.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:03 smithi137 ceph-mon[32951]: osdmap e74: 8 total, 8 up, 8 in 2024-06-14T22:36:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:36:05.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:04 smithi137 ceph-mon[32951]: pgmap v96: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:05.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:04 smithi137 ceph-mon[32951]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:36:05.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:04 smithi137 ceph-mon[32951]: Cluster is now healthy 2024-06-14T22:36:05.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:04 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:05.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:04 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:36:05.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:04 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:36:05.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:04 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:36:05.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:04 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:05.256 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-06-14T22:36:05.259 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T22:36:05.259 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T22:36:05.303 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T22:36:05.304 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T22:36:05.304 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_83580 2024-06-14T22:36:05.304 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:36:05.304 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T22:36:05.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[38904]: pgmap v96: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:05.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[38904]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:36:05.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[38904]: Cluster is now healthy 2024-06-14T22:36:05.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:05.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:36:05.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:36:05.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:36:05.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:05.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[29889]: pgmap v96: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:05.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[29889]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:36:05.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[29889]: Cluster is now healthy 2024-06-14T22:36:05.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:05.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:36:05.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:36:05.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:36:05.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:04 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:05.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch ps' 2024-06-14T22:36:05.988 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:36:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:05 smithi137 ceph-mon[32951]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:36:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:05 smithi137 ceph-mon[32951]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:36:06.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:05 smithi137 ceph-mon[32951]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:36:06.257 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:05 smithi086 ceph-mon[29889]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:36:06.258 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:05 smithi086 ceph-mon[29889]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:36:06.258 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:05 smithi086 ceph-mon[29889]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:36:06.258 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:05 smithi086 ceph-mon[38904]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:36:06.258 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:05 smithi086 ceph-mon[38904]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:36:06.258 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:05 smithi086 ceph-mon[38904]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T22:36:06.305 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 3866 3850 240.612 240.625 0.00336336 0.00414214 2024-06-14T22:36:07.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:06 smithi137 ceph-mon[32951]: pgmap v97: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:07.304 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 16 6997 6981 218.141 195.688 0.00217751 0.00457174 2024-06-14T22:36:07.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:06 smithi086 ceph-mon[29889]: pgmap v97: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:07.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:06 smithi086 ceph-mon[38904]: pgmap v97: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:08.304 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 16 9829 9813 204.422 177 0.00168229 0.0047907 2024-06-14T22:36:08.897 INFO:teuthology.orchestra.run.smithi086.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:alertmanager.a smithi086 *:9093,9094 running (87s) 26s ago 2m 16.2M - 0.25.0 c8568f914cd2 3c3fa36d4b10 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:grafana.a smithi137 *:3000 running (2m) 55s ago 2m 82.4M - 9.4.7 954c08fa6188 a38b34e5bca6 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi086.nhdjdb smithi086 running (70s) 26s ago 70s 15.5M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 2bd94d6f47de 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi137.yjevpg smithi137 running (68s) 55s ago 68s 11.5M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4151acede02a 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.x smithi137 *:8443 running (6m) 55s ago 6m 398M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 2817a158009e 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.y smithi086 *:9283 running (7m) 26s ago 7m 442M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 3ed18014b7a8 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:mon.a smithi086 running (8m) 26s ago 8m 46.7M 2048M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 00d4288edc6f 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:mon.b smithi137 running (6m) 55s ago 6m 39.3M 2048M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 bd4aed27097c 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:mon.c smithi086 running (6m) 26s ago 6m 39.0M 2048M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 e02d9b886f41 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.a smithi086 *:9100 running (2m) 26s ago 119s 16.5M - 1.5.0 0da6a335fe13 d83ddcfcdc25 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.b smithi137 *:9100 running (116s) 55s ago 115s 8216k - 1.5.0 0da6a335fe13 6082a0b8781a 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0 smithi086 running (6m) 26s ago 6m 44.1M 4016M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 129bff24d5cc 2024-06-14T22:36:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1 smithi086 running (5m) 26s ago 5m 42.8M 4016M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 88baee12d7cb 2024-06-14T22:36:08.899 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2 smithi086 running (5m) 26s ago 5m 41.3M 4016M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 fbf5ce68d6eb 2024-06-14T22:36:08.899 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3 smithi086 running (4m) 26s ago 4m 41.2M 4016M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 bee260f44ae7 2024-06-14T22:36:08.899 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4 smithi137 running (4m) 55s ago 4m 43.2M 4016M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4b2729030303 2024-06-14T22:36:08.899 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5 smithi137 running (4m) 55s ago 4m 39.7M 4016M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 e01c2182297f 2024-06-14T22:36:08.899 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6 smithi137 running (3m) 55s ago 3m 40.3M 4016M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 7210ba0a183d 2024-06-14T22:36:08.899 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7 smithi137 running (3m) 55s ago 3m 41.7M 4016M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 a9ca11c000f2 2024-06-14T22:36:08.899 INFO:teuthology.orchestra.run.smithi086.stdout:prometheus.a smithi137 *:9095 running (82s) 55s ago 2m 35.3M - 2.43.0 a07b618ecd1d 4b95a815ba8c 2024-06-14T22:36:09.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:08 smithi137 ceph-mon[32951]: pgmap v98: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:09.306 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 16 12433 12417 193.991 162.75 0.00758497 0.00514716 2024-06-14T22:36:09.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:08 smithi086 ceph-mon[29889]: pgmap v98: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:09.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:08 smithi086 ceph-mon[38904]: pgmap v98: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:36:09.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:36:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:09] "GET /metrics HTTP/1.1" 200 239142 "" "Prometheus/2.43.0" 2024-06-14T22:36:10.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:09 smithi137 ceph-mon[32951]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:36:10.305 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 16 15098 15082 188.476 166.562 0.00858867 0.00529689 2024-06-14T22:36:10.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:09 smithi086 ceph-mon[38904]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:36:10.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:09 smithi086 ceph-mon[29889]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:36:10.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:10 smithi137 ceph-mon[32951]: pgmap v99: 89 pgs: 89 active+clean; 650 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-14T22:36:10.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:36:10 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:36:11.306 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 17972 17956 186.998 179.625 0.00444155 0.00534061 2024-06-14T22:36:11.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:10 smithi086 ceph-mon[38904]: pgmap v99: 89 pgs: 89 active+clean; 650 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-14T22:36:11.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:10 smithi086 ceph-mon[29889]: pgmap v99: 89 pgs: 89 active+clean; 650 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-14T22:36:11.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'echo wait for minority of mons to upgrade' 2024-06-14T22:36:12.306 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 15 20736 20721 184.958 172.812 0.00827223 0.00540066 2024-06-14T22:36:12.427 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:36:13.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:12 smithi137 ceph-mon[32951]: pgmap v100: 89 pgs: 89 active+clean; 650 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-14T22:36:13.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:12 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:13.307 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 23459 23443 183.09 170.125 0.00595038 0.00545653 2024-06-14T22:36:13.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:12 smithi086 ceph-mon[29889]: pgmap v100: 89 pgs: 89 active+clean; 650 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-14T22:36:13.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:12 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:13.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:12 smithi086 ceph-mon[38904]: pgmap v100: 89 pgs: 89 active+clean; 650 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-14T22:36:13.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:12 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:14.307 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 26034 26018 180.621 160.938 0.00718836 0.00553081 2024-06-14T22:36:15.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:14 smithi086 ceph-mon[38904]: pgmap v101: 89 pgs: 89 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-06-14T22:36:15.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:14 smithi086 ceph-mon[29889]: pgmap v101: 89 pgs: 89 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-06-14T22:36:15.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:14 smithi137 ceph-mon[32951]: pgmap v101: 89 pgs: 89 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-06-14T22:36:15.308 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 16 28484 28468 177.87 153.125 0.00688715 0.00561738 2024-06-14T22:36:15.508 INFO:teuthology.orchestra.run.smithi086.stdout:wait for minority of mons to upgrade 2024-06-14T22:36:16.307 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 30587 30571 173.648 131.438 0.00917019 0.00575338 2024-06-14T22:36:17.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:16 smithi137 ceph-mon[32951]: pgmap v102: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-06-14T22:36:17.307 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 16 32573 32557 169.521 124.125 0.00462248 0.005894 2024-06-14T22:36:17.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:16 smithi086 ceph-mon[29889]: pgmap v102: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-06-14T22:36:17.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:16 smithi086 ceph-mon[38904]: pgmap v102: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-06-14T22:36:18.308 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 34720 34704 166.793 134.188 0.00597789 0.00599066 2024-06-14T22:36:19.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:18 smithi137 ceph-mon[32951]: pgmap v103: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-06-14T22:36:19.308 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 36859 36843 164.427 133.688 0.00379896 0.00607128 2024-06-14T22:36:19.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:18 smithi086 ceph-mon[29889]: pgmap v103: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-06-14T22:36:19.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:18 smithi086 ceph-mon[38904]: pgmap v103: 89 pgs: 89 active+clean; 1.5 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-06-14T22:36:19.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:36:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:19] "GET /metrics HTTP/1.1" 200 242131 "" "Prometheus/2.43.0" 2024-06-14T22:36:20.308 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 38821 38805 161.64 122.625 0.00683605 0.00618064 2024-06-14T22:36:20.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:36:20 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:36:21.309 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 40662 40646 158.725 115.062 0.00788056 0.00629535 2024-06-14T22:36:21.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:21 smithi086 ceph-mon[29889]: pgmap v104: 89 pgs: 89 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 184 MiB/s wr, 2.94k op/s 2024-06-14T22:36:21.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:21 smithi086 ceph-mon[38904]: pgmap v104: 89 pgs: 89 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 184 MiB/s wr, 2.94k op/s 2024-06-14T22:36:21.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:21 smithi137 ceph-mon[32951]: pgmap v104: 89 pgs: 89 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 184 MiB/s wr, 2.94k op/s 2024-06-14T22:36:22.309 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 16 42561 42545 156.369 118.688 0.0151269 0.00639069 2024-06-14T22:36:23.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'while ! ceph mon versions | grep $sha1 ; do sleep 2 ; done' 2024-06-14T22:36:23.309 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 44482 44466 154.351 120.062 0.003128 0.00646951 2024-06-14T22:36:23.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:23 smithi086 ceph-mon[29889]: pgmap v105: 89 pgs: 89 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 129 MiB/s wr, 2.07k op/s 2024-06-14T22:36:23.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:23 smithi086 ceph-mon[38904]: pgmap v105: 89 pgs: 89 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 129 MiB/s wr, 2.07k op/s 2024-06-14T22:36:23.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:23 smithi137 ceph-mon[32951]: pgmap v105: 89 pgs: 89 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 129 MiB/s wr, 2.07k op/s 2024-06-14T22:36:23.628 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:36:24.309 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 46367 46351 152.428 117.812 0.0069861 0.00655558 2024-06-14T22:36:25.310 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:36:25.314924+0000 min lat: 0.00115898 max lat: 0.105712 avg lat: 0.00662961 2024-06-14T22:36:25.310 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:36:25.310 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 48257 48241 150.711 118.125 0.00745801 0.00662961 2024-06-14T22:36:25.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:25 smithi086 ceph-mon[29889]: pgmap v106: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-06-14T22:36:25.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:25 smithi086 ceph-mon[38904]: pgmap v106: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-06-14T22:36:25.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:25 smithi137 ceph-mon[32951]: pgmap v106: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 164 MiB/s wr, 2.62k op/s 2024-06-14T22:36:26.310 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 16 49865 49849 148.32 100.5 0.00414817 0.00673582 2024-06-14T22:36:27.310 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 51348 51332 145.791 92.6875 0.0122526 0.00685339 2024-06-14T22:36:27.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:27 smithi086 ceph-mon[29889]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-06-14T22:36:27.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:27 smithi086 ceph-mon[38904]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-06-14T22:36:27.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:27 smithi137 ceph-mon[32951]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-06-14T22:36:28.310 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 53083 53067 144.167 108.438 0.00778184 0.00693146 2024-06-14T22:36:28.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:28 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:28.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:28 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:28.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:28 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:29.310 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 54826 54810 142.698 108.938 0.0111573 0.00700294 2024-06-14T22:36:29.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:29 smithi086 ceph-mon[38904]: pgmap v108: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-06-14T22:36:29.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:29 smithi086 ceph-mon[29889]: pgmap v108: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-06-14T22:36:29.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:29 smithi137 ceph-mon[32951]: pgmap v108: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-06-14T22:36:29.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:36:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:29] "GET /metrics HTTP/1.1" 200 242530 "" "Prometheus/2.43.0" 2024-06-14T22:36:30.310 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 56506 56490 141.19 105 0.00854967 0.00707817 2024-06-14T22:36:30.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:30 smithi086 ceph-mon[29889]: pgmap v109: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-06-14T22:36:30.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:30 smithi086 ceph-mon[38904]: pgmap v109: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-06-14T22:36:30.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:30 smithi137 ceph-mon[32951]: pgmap v109: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-06-14T22:36:30.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:36:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:36:31.310 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 58055 58039 139.483 96.8125 0.00758332 0.00716463 2024-06-14T22:36:32.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 59717 59701 138.163 103.875 0.00736468 0.00723299 2024-06-14T22:36:32.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:32 smithi137 ceph-mon[32951]: pgmap v110: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T22:36:33.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:32 smithi086 ceph-mon[38904]: pgmap v110: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T22:36:33.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:32 smithi086 ceph-mon[29889]: pgmap v110: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T22:36:33.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 61299 61283 136.76 98.875 0.0148656 0.00730746 2024-06-14T22:36:34.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 62961 62945 135.626 103.875 0.00244204 0.00736768 2024-06-14T22:36:34.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:34 smithi137 ceph-mon[32951]: pgmap v111: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-06-14T22:36:35.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:34 smithi086 ceph-mon[38904]: pgmap v111: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-06-14T22:36:35.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:34 smithi086 ceph-mon[29889]: pgmap v111: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-06-14T22:36:35.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 64499 64483 134.309 96.125 0.0143573 0.00744159 2024-06-14T22:36:36.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 66204 66188 133.414 106.562 0.0114571 0.00749092 2024-06-14T22:36:36.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:36 smithi086 ceph-mon[29889]: pgmap v112: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-06-14T22:36:36.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:36 smithi086 ceph-mon[38904]: pgmap v112: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-06-14T22:36:36.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:36 smithi137 ceph-mon[32951]: pgmap v112: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-06-14T22:36:37.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 67849 67833 132.457 102.812 0.0160179 0.00754509 2024-06-14T22:36:38.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 16 69448 69432 131.472 99.9375 0.00817827 0.00760073 2024-06-14T22:36:38.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:38 smithi137 ceph-mon[32951]: pgmap v113: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-06-14T22:36:39.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:38 smithi086 ceph-mon[29889]: pgmap v113: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-06-14T22:36:39.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:38 smithi086 ceph-mon[38904]: pgmap v113: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-06-14T22:36:39.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 70928 70912 130.325 92.5 0.00670956 0.00766854 2024-06-14T22:36:39.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:39] "GET /metrics HTTP/1.1" 200 242530 "" "Prometheus/2.43.0" 2024-06-14T22:36:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[29889]: Upgrade: Target is version 18.2.2-1443-g6f4b0956 (unknown) 2024-06-14T22:36:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[29889]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0'] 2024-06-14T22:36:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:36:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2112792351' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[29889]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-14T22:36:39.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:39.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[38904]: Upgrade: Target is version 18.2.2-1443-g6f4b0956 (unknown) 2024-06-14T22:36:39.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[38904]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0'] 2024-06-14T22:36:39.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:36:39.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:39.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2112792351' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:39.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:39 smithi086 ceph-mon[38904]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-14T22:36:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:39 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:39 smithi137 ceph-mon[32951]: Upgrade: Target is version 18.2.2-1443-g6f4b0956 (unknown) 2024-06-14T22:36:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:39 smithi137 ceph-mon[32951]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0'] 2024-06-14T22:36:39.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:39 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:36:39.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:39 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:36:39.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:39 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2112792351' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:39.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:39 smithi137 ceph-mon[32951]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-14T22:36:40.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 72355 72339 129.15 89.1875 0.0258278 0.00773727 2024-06-14T22:36:40.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:40 smithi137 ceph-mon[32951]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 on smithi137 2024-06-14T22:36:40.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:40 smithi137 ceph-mon[32951]: pgmap v114: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-06-14T22:36:40.996 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:36:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:36:41.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:40 smithi086 ceph-mon[29889]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 on smithi137 2024-06-14T22:36:41.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:40 smithi086 ceph-mon[29889]: pgmap v114: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-06-14T22:36:41.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:40 smithi086 ceph-mon[38904]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 on smithi137 2024-06-14T22:36:41.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:40 smithi086 ceph-mon[38904]: pgmap v114: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-06-14T22:36:41.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 73499 73483 127.549 71.5 0.0063033 0.0078348 2024-06-14T22:36:41.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:41 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/51068004' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:42.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:41 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/51068004' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:42.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:41 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/51068004' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:42.311 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 74821 74805 126.335 82.625 0.00939822 0.00791083 2024-06-14T22:36:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:42 smithi137 ceph-mon[32951]: pgmap v115: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-14T22:36:42.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:42 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:43.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:42 smithi086 ceph-mon[29889]: pgmap v115: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-14T22:36:43.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:42 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:43.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:42 smithi086 ceph-mon[38904]: pgmap v115: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-14T22:36:43.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:42 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:43.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 76000 75984 124.949 73.6875 0.00907936 0.00799766 2024-06-14T22:36:44.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 16 77117 77101 123.535 69.8125 0.0182004 0.00808934 2024-06-14T22:36:44.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:44 smithi137 ceph-mon[32951]: pgmap v116: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T22:36:44.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:44 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/90617520' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:45.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:44 smithi086 ceph-mon[29889]: pgmap v116: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T22:36:45.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:44 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/90617520' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:45.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:44 smithi086 ceph-mon[38904]: pgmap v116: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T22:36:45.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:44 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/90617520' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:45.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:36:45.316993+0000 min lat: 0.00115898 max lat: 0.105712 avg lat: 0.00817972 2024-06-14T22:36:45.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:36:45.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 78219 78203 122.169 68.875 0.0314849 0.00817972 2024-06-14T22:36:46.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 79298 79282 120.834 67.4375 0.0144461 0.00827119 2024-06-14T22:36:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:46 smithi137 ceph-mon[32951]: pgmap v117: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-06-14T22:36:46.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:46 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1343231013' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:47.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:46 smithi086 ceph-mon[29889]: pgmap v117: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-06-14T22:36:47.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:46 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1343231013' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:46 smithi086 ceph-mon[38904]: pgmap v117: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-06-14T22:36:47.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:46 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1343231013' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:47.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 16 80552 80536 119.823 78.375 0.00883276 0.00834112 2024-06-14T22:36:48.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 81829 81813 118.892 79.8125 0.0186874 0.00840645 2024-06-14T22:36:48.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:48 smithi137 ceph-mon[32951]: pgmap v118: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T22:36:49.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:48 smithi086 ceph-mon[29889]: pgmap v118: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T22:36:49.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:48 smithi086 ceph-mon[38904]: pgmap v118: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T22:36:49.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 16 83075 83059 117.96 77.875 0.00389014 0.00847163 2024-06-14T22:36:49.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:49 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/174141435' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:36:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:49] "GET /metrics HTTP/1.1" 200 242601 "" "Prometheus/2.43.0" 2024-06-14T22:36:49.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:49 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/174141435' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:49.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:49 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/174141435' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:50.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 84328 84312 117.079 78.3125 0.023439 0.00853641 2024-06-14T22:36:50.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:50 smithi137 ceph-mon[32951]: pgmap v119: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-06-14T22:36:50.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:36:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:36:51.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:50 smithi086 ceph-mon[38904]: pgmap v119: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-06-14T22:36:51.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:50 smithi086 ceph-mon[29889]: pgmap v119: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-06-14T22:36:51.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 16 85521 85505 116.155 74.5625 0.00905815 0.00860436 2024-06-14T22:36:51.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:51 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/863492193' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:52.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:51 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/863492193' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:52.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:51 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/863492193' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:52.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 86728 86712 115.288 75.4375 0.0143335 0.00866835 2024-06-14T22:36:52.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:52 smithi137 ceph-mon[32951]: pgmap v120: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-14T22:36:53.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:52 smithi086 ceph-mon[29889]: pgmap v120: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-14T22:36:53.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:52 smithi086 ceph-mon[38904]: pgmap v120: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-14T22:36:53.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 87866 87850 114.368 71.125 0.0140407 0.00873885 2024-06-14T22:36:54.312 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 89033 89017 113.523 72.9375 0.0184914 0.00880385 2024-06-14T22:36:54.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:54 smithi137 ceph-mon[32951]: pgmap v121: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:36:54.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:54 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2990741158' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:54 smithi086 ceph-mon[29889]: pgmap v121: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:36:55.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:54 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2990741158' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:55.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:54 smithi086 ceph-mon[38904]: pgmap v121: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:36:55.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:54 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2990741158' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:55.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 90171 90155 112.675 71.125 0.0115949 0.00887017 2024-06-14T22:36:56.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 91271 91255 111.813 68.75 0.0183786 0.00893863 2024-06-14T22:36:56.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:56 smithi137 ceph-mon[32951]: pgmap v122: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-14T22:36:56.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:56 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3733963366' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:57.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:56 smithi086 ceph-mon[38904]: pgmap v122: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-14T22:36:57.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:56 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3733963366' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:57.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:56 smithi086 ceph-mon[29889]: pgmap v122: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-14T22:36:57.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:56 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3733963366' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:57.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 16 92396 92380 111.015 70.3125 0.00894993 0.0090028 2024-06-14T22:36:57.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:57 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:58.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:57 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:58.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:57 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:36:58.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 16 93568 93552 110.303 73.25 0.00735748 0.00906072 2024-06-14T22:36:58.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:58 smithi137 ceph-mon[32951]: pgmap v123: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-14T22:36:59.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:58 smithi086 ceph-mon[29889]: pgmap v123: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-14T22:36:59.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:58 smithi086 ceph-mon[38904]: pgmap v123: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-14T22:36:59.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 94659 94643 109.523 68.1875 0.00366189 0.00912258 2024-06-14T22:36:59.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:36:59 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/512024045' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:59.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:36:59 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/512024045' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:36:59.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:36:59 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:36:59] "GET /metrics HTTP/1.1" 200 242678 "" "Prometheus/2.43.0" 2024-06-14T22:37:00.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:36:59 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/512024045' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:00.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 95713 95697 108.729 65.875 0.0160871 0.00919205 2024-06-14T22:37:00.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:00 smithi137 ceph-mon[32951]: pgmap v124: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-06-14T22:37:00.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:00 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:37:01.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:00 smithi086 ceph-mon[29889]: pgmap v124: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-06-14T22:37:01.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:00 smithi086 ceph-mon[38904]: pgmap v124: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-06-14T22:37:01.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 96730 96714 107.922 63.5625 0.0113418 0.00926082 2024-06-14T22:37:02.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:01 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4013076305' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:02.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:01 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4013076305' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:02.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:01 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4013076305' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:02.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 97769 97753 107.168 64.9375 0.0158452 0.00932587 2024-06-14T22:37:03.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:02 smithi086 ceph-mon[29889]: pgmap v125: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s 2024-06-14T22:37:03.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:02 smithi086 ceph-mon[38904]: pgmap v125: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s 2024-06-14T22:37:03.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:02 smithi137 ceph-mon[32951]: pgmap v125: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s 2024-06-14T22:37:03.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 15 98835 98820 106.47 66.6875 0.0444961 0.00938718 2024-06-14T22:37:04.313 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 99869 99853 105.76 64.5625 0.0141584 0.00945051 2024-06-14T22:37:05.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:04 smithi137 ceph-mon[32951]: pgmap v126: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-14T22:37:05.043 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:04 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4152927112' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:05.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:04 smithi086 ceph-mon[29889]: pgmap v126: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-14T22:37:05.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:04 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4152927112' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:05.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:04 smithi086 ceph-mon[38904]: pgmap v126: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-14T22:37:05.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:04 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4152927112' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:05.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:37:05.318657+0000 min lat: 0.00115898 max lat: 0.105712 avg lat: 0.00951774 2024-06-14T22:37:05.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:37:05.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 100836 100820 105.005 60.4375 0.0449571 0.00951774 2024-06-14T22:37:06.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 101857 101841 104.329 63.8125 0.0174112 0.00958045 2024-06-14T22:37:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:06 smithi086 ceph-mon[29889]: pgmap v127: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-14T22:37:07.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:06 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3171071026' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:06 smithi086 ceph-mon[38904]: pgmap v127: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-14T22:37:07.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:06 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3171071026' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:07.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:06 smithi137 ceph-mon[32951]: pgmap v127: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-14T22:37:07.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:06 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3171071026' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:07.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 102868 102852 103.666 63.1875 0.00406551 0.00964102 2024-06-14T22:37:08.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 103919 103903 103.063 65.6875 0.0137708 0.00969862 2024-06-14T22:37:09.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:08 smithi086 ceph-mon[29889]: pgmap v128: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 910 op/s 2024-06-14T22:37:09.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:08 smithi086 ceph-mon[38904]: pgmap v128: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 910 op/s 2024-06-14T22:37:09.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:08 smithi137 ceph-mon[32951]: pgmap v128: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 910 op/s 2024-06-14T22:37:09.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 105067 105051 102.573 71.75 0.020997 0.00974481 2024-06-14T22:37:09.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:09 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2776363829' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:09.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:09] "GET /metrics HTTP/1.1" 200 242678 "" "Prometheus/2.43.0" 2024-06-14T22:37:09.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:09 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2776363829' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:10.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:09 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2776363829' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:10.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 106135 106119 102.022 66.75 0.0153122 0.00979732 2024-06-14T22:37:10.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:10 smithi137 ceph-mon[32951]: pgmap v129: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T22:37:10.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:10 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[35003]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:37:11.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:10 smithi086 ceph-mon[38904]: pgmap v129: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T22:37:11.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:10 smithi086 ceph-mon[29889]: pgmap v129: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T22:37:11.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 106985 106969 101.281 53.125 0.00567915 0.009867 2024-06-14T22:37:12.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:11 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4225703939' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:12.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:11 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4225703939' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:12.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:11 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4225703939' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:12.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 108154 108138 100.86 73.0625 0.0117985 0.00991049 2024-06-14T22:37:13.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:12 smithi086 ceph-mon[29889]: pgmap v130: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s 2024-06-14T22:37:13.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:12 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:12 smithi086 ceph-mon[38904]: pgmap v130: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s 2024-06-14T22:37:13.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:12 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:13.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:12 smithi137 ceph-mon[32951]: pgmap v130: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s 2024-06-14T22:37:13.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:12 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:13.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 109166 109150 100.307 63.25 0.0992322 0.00995373 2024-06-14T22:37:14.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 110244 110228 99.8296 67.375 0.00770768 0.010012 2024-06-14T22:37:15.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:14 smithi086 ceph-mon[38904]: pgmap v131: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s 2024-06-14T22:37:15.080 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:14 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1907153411' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:15.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:14 smithi086 ceph-mon[29889]: pgmap v131: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s 2024-06-14T22:37:15.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:14 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1907153411' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:15.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:14 smithi137 ceph-mon[32951]: pgmap v131: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s 2024-06-14T22:37:15.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:14 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1907153411' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:15.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 111270 111254 99.3195 64.125 0.0220678 0.0100632 2024-06-14T22:37:16.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 112358 112342 98.8783 68 0.0172239 0.0101087 2024-06-14T22:37:16.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:16 smithi137 ceph-mon[32951]: Upgrade: Updating mgr.x 2024-06-14T22:37:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:16 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:16 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:37:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:16 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:37:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:16 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:37:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:16 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:37:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:16 smithi137 ceph-mon[32951]: Deploying daemon mgr.x on smithi137 2024-06-14T22:37:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:16 smithi137 ceph-mon[32951]: pgmap v132: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T22:37:16.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[29889]: Upgrade: Updating mgr.x 2024-06-14T22:37:16.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:16.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:37:16.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:37:16.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:37:16.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:37:16.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[29889]: Deploying daemon mgr.x on smithi137 2024-06-14T22:37:16.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[29889]: pgmap v132: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T22:37:16.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[38904]: Upgrade: Updating mgr.x 2024-06-14T22:37:16.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:16.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:37:16.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:37:16.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:37:16.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:37:16.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[38904]: Deploying daemon mgr.x on smithi137 2024-06-14T22:37:16.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:16 smithi086 ceph-mon[38904]: pgmap v132: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T22:37:17.314 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 113424 113408 98.4303 66.625 0.00778123 0.0101543 2024-06-14T22:37:17.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:17 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/538855576' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:17 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/538855576' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:17.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:17 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/538855576' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:18.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 114529 114513 98.0279 69.0625 0.0136752 0.0101963 2024-06-14T22:37:18.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:18 smithi086 ceph-mon[29889]: pgmap v133: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s 2024-06-14T22:37:18.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:18 smithi086 ceph-mon[38904]: pgmap v133: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s 2024-06-14T22:37:18.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:18 smithi137 systemd[1]: Stopping Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:37:18.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:18 smithi137 ceph-mon[32951]: pgmap v133: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s 2024-06-14T22:37:19.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 16 115523 115507 97.5427 62.125 0.00896397 0.0102469 2024-06-14T22:37:19.326 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:19 smithi137 podman[68434]: 2024-06-14 22:37:19.020220005 +0000 UTC m=+0.171957282 container died 2817a158009e4f7062c5cc70d279647804d87e455bfd590ef50434c97327c2d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:37:19.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:19 smithi137 podman[68434]: 2024-06-14 22:37:19.436836021 +0000 UTC m=+0.588573300 container cleanup 2817a158009e4f7062c5cc70d279647804d87e455bfd590ef50434c97327c2d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240612) 2024-06-14T22:37:19.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:19 smithi137 bash[68434]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x 2024-06-14T22:37:19.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:19 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1946404162' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:19.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:19 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1946404162' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:19.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:19 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1946404162' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:19.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:19] "GET /metrics HTTP/1.1" 200 242692 "" "Prometheus/2.43.0" 2024-06-14T22:37:20.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 116467 116451 97.0288 59 0.0148707 0.0103008 2024-06-14T22:37:20.448 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:20 smithi137 podman[68450]: 2024-06-14 22:37:20.189275481 +0000 UTC m=+1.160060177 container remove 2817a158009e4f7062c5cc70d279647804d87e455bfd590ef50434c97327c2d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-14T22:37:20.448 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:20 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-06-14T22:37:20.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:20 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-06-14T22:37:20.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:20 smithi137 systemd[1]: Stopped Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:37:20.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:20 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.x.service: Consumed 10.693s CPU time. 2024-06-14T22:37:20.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:20 smithi137 ceph-mon[32951]: pgmap v134: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-14T22:37:20.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:20 smithi086 ceph-mon[29889]: pgmap v134: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-14T22:37:20.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:20 smithi086 ceph-mon[38904]: pgmap v134: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-14T22:37:21.126 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:20 smithi137 systemd[1]: Starting Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:37:21.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 117486 117470 96.5901 63.6875 0.0139541 0.0103484 2024-06-14T22:37:21.465 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:21 smithi137 podman[68570]: 2024-06-14 22:37:21.033959557 +0000 UTC m=+0.044777136 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:37:21.465 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:21 smithi137 podman[68570]: 2024-06-14 22:37:21.178533737 +0000 UTC m=+0.189351313 container create 41740301d000565aba71a1d956d0de19d0d8d0df5bafc6c239ccec4c1b1c62be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:37:21.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:21 smithi137 podman[68570]: 2024-06-14 22:37:21.468973335 +0000 UTC m=+0.479790930 container init 41740301d000565aba71a1d956d0de19d0d8d0df5bafc6c239ccec4c1b1c62be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-06-14T22:37:21.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:21 smithi137 podman[68570]: 2024-06-14 22:37:21.479480405 +0000 UTC m=+0.490297977 container start 41740301d000565aba71a1d956d0de19d0d8d0df5bafc6c239ccec4c1b1c62be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=) 2024-06-14T22:37:21.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:21 smithi137 ceph-mgr[68603]: -- 172.21.15.137:0/1203733191 <== mon.2 v2:172.21.15.86:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562d655a2d00 con 0x562d65577000 2024-06-14T22:37:21.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:21 smithi137 bash[68570]: 41740301d000565aba71a1d956d0de19d0d8d0df5bafc6c239ccec4c1b1c62be 2024-06-14T22:37:21.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:21 smithi137 systemd[1]: Started Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:37:21.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:21.694+0000 7f8c9d616040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:37:22.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:21.974+0000 7f8c9d616040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:37:22.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:22.110+0000 7f8c9d616040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:37:22.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 16 118377 118361 96.0588 55.6875 0.0122733 0.0104049 2024-06-14T22:37:22.829 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:22.564+0000 7f8c9d616040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:37:22.829 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:22.667+0000 7f8c9d616040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:37:22.829 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:22 smithi137 ceph-mon[32951]: pgmap v135: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 879 op/s 2024-06-14T22:37:22.829 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:22 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:22.829 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:22 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:22.829 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:22 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:37:22.829 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:22 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3725832534' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:23.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[29889]: pgmap v135: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 879 op/s 2024-06-14T22:37:23.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:23.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:23.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:37:23.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3725832534' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:23.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[38904]: pgmap v135: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 879 op/s 2024-06-14T22:37:23.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:23.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:23.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:37:23.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:22 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3725832534' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:23.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 119246 119230 95.5236 54.3125 0.0176763 0.0104624 2024-06-14T22:37:23.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:23.208+0000 7f8c9d616040 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:37:23.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:23.208+0000 7f8c9d616040 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:37:23.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:37:23.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: from .module import Module 2024-06-14T22:37:23.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:37:23.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:37:23.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:37:23.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:23.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:23.210+0000 7f8c9d616040 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:37:23.491 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:23.210+0000 7f8c9d616040 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:37:24.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 120134 120118 95.0169 55.5 0.0382846 0.0105195 2024-06-14T22:37:24.376 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:24.378+0000 7f8c9d616040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:37:24.688 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:24.588+0000 7f8c9d616040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:37:24.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:24.693+0000 7f8c9d616040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:37:24.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:24 smithi137 ceph-mon[32951]: pgmap v136: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s 2024-06-14T22:37:24.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:24 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3861688554' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:25.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:24 smithi086 ceph-mon[29889]: pgmap v136: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s 2024-06-14T22:37:25.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:24 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3861688554' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:25.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:24 smithi086 ceph-mon[38904]: pgmap v136: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s 2024-06-14T22:37:25.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:24 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3861688554' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:25.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:37:25.320340+0000 min lat: 0.00115898 max lat: 0.162806 avg lat: 0.0105774 2024-06-14T22:37:25.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:37:25.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 120993 120977 94.5003 53.6875 0.0143882 0.0105774 2024-06-14T22:37:25.329 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:25 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:25.093+0000 7f8c9d616040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:37:25.329 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:25 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:25.207+0000 7f8c9d616040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:37:25.592 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:25 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:25.329+0000 7f8c9d616040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:37:25.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:25 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:25.595+0000 7f8c9d616040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:37:26.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 121833 121817 93.9817 52.5 0.0255545 0.0106351 2024-06-14T22:37:26.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:26 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:26.352+0000 7f8c9d616040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:37:27.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:26 smithi137 ceph-mon[32951]: pgmap v137: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T22:37:27.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:26 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:27.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:26 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:27.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:26 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:26.921+0000 7f8c9d616040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:37:27.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:27.028+0000 7f8c9d616040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:37:27.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 122650 122634 93.4583 51.0625 0.0217982 0.010695 2024-06-14T22:37:27.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:26 smithi086 ceph-mon[29889]: pgmap v137: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T22:37:27.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:26 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:27.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:26 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:27.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:26 smithi086 ceph-mon[38904]: pgmap v137: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T22:37:27.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:26 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:27.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:26 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:27.563 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:27.272+0000 7f8c9d616040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:37:28.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:27.853+0000 7f8c9d616040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:37:28.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:27 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:27.968+0000 7f8c9d616040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:37:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:27 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1811476818' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:27 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:28.315 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 16 123508 123492 92.9783 53.625 0.0181338 0.0107502 2024-06-14T22:37:28.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:27 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1811476818' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:28.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:27 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:27 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1811476818' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:27 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:28.562 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:28 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:28.306+0000 7f8c9d616040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:37:28.832 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:28 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:28.566+0000 7f8c9d616040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-14T22:37:28.832 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:28 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:28.684+0000 7f8c9d616040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:37:29.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:28 smithi137 ceph-mon[32951]: pgmap v138: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 52 MiB/s wr, 835 op/s 2024-06-14T22:37:29.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:28 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:29.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:28 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:29.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:28 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:28.835+0000 7f8c9d616040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:37:29.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:28 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:28.943+0000 7f8c9d616040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:37:29.316 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 124336 124320 92.4875 51.75 0.0148385 0.0108075 2024-06-14T22:37:29.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:28 smithi086 ceph-mon[29889]: pgmap v138: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 52 MiB/s wr, 835 op/s 2024-06-14T22:37:29.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:28 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:29.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:28 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:28 smithi086 ceph-mon[38904]: pgmap v138: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 52 MiB/s wr, 835 op/s 2024-06-14T22:37:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:28 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:28 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:29.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:29] "GET /metrics HTTP/1.1" 200 242701 "" "Prometheus/2.43.0" 2024-06-14T22:37:30.126 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:30.127 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:29 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:29.947+0000 7f8c9d616040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:37:30.316 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 125171 125155 92.0133 52.1875 0.0279639 0.0108637 2024-06-14T22:37:30.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:30.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/586227' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:30.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-mon[32951]: pgmap v139: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-14T22:37:30.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:30.215+0000 7f8c9d616040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:37:30.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:30.215+0000 7f8c9d616040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:37:30.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:37:30] ENGINE Bus STARTING 2024-06-14T22:37:30.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: CherryPy Checker: 2024-06-14T22:37:30.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: The Application mounted at '' has an empty config. 2024-06-14T22:37:30.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:30.491 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:37:30] ENGINE Serving on http://:::9283 2024-06-14T22:37:30.491 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:37:30] ENGINE Bus STARTED 2024-06-14T22:37:30.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:30.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:30 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:30.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:30 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/586227' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:30.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:30 smithi086 ceph-mon[29889]: pgmap v139: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-14T22:37:30.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:30.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:30 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:30.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:30 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/586227' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:30.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:30 smithi086 ceph-mon[38904]: pgmap v139: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-14T22:37:30.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:37:31.316 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 126190 126174 91.6839 63.6875 0.00782149 0.0109025 2024-06-14T22:37:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:31 smithi137 ceph-mon[32951]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:37:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:31 smithi137 ceph-mon[32951]: Standby manager daemon x restarted 2024-06-14T22:37:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:31 smithi137 ceph-mon[32951]: Standby manager daemon x started 2024-06-14T22:37:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:31 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:37:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:31 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:37:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:31 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:37:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:31 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[29889]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[29889]: Standby manager daemon x restarted 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[29889]: Standby manager daemon x started 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[38904]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[38904]: Standby manager daemon x restarted 2024-06-14T22:37:31.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[38904]: Standby manager daemon x started 2024-06-14T22:37:31.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:37:31.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:37:31.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:37:31.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:31 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:37:32.316 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 127176 127160 91.3384 61.625 0.0135378 0.0109429 2024-06-14T22:37:32.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:32 smithi137 ceph-mon[32951]: mgrmap e21: y(active, since 4m), standbys: x 2024-06-14T22:37:32.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:32 smithi137 ceph-mon[32951]: pgmap v140: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 47 MiB/s wr, 746 op/s 2024-06-14T22:37:32.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:32 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2855705305' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:32.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:32 smithi086 ceph-mon[29889]: mgrmap e21: y(active, since 4m), standbys: x 2024-06-14T22:37:32.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:32 smithi086 ceph-mon[29889]: pgmap v140: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 47 MiB/s wr, 746 op/s 2024-06-14T22:37:32.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:32 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2855705305' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:32.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:32 smithi086 ceph-mon[38904]: mgrmap e21: y(active, since 4m), standbys: x 2024-06-14T22:37:32.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:32 smithi086 ceph-mon[38904]: pgmap v140: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 47 MiB/s wr, 746 op/s 2024-06-14T22:37:32.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:32 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2855705305' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:33.316 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 128143 128127 90.9872 60.4375 0.00638648 0.010986 2024-06-14T22:37:34.316 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 129111 129095 90.6446 60.5 0.0105439 0.0110271 2024-06-14T22:37:34.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:34 smithi137 ceph-mon[32951]: pgmap v141: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 770 op/s 2024-06-14T22:37:34.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:34 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2537191407' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:35.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:34 smithi086 ceph-mon[29889]: pgmap v141: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 770 op/s 2024-06-14T22:37:35.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:34 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2537191407' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:34 smithi086 ceph-mon[38904]: pgmap v141: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 770 op/s 2024-06-14T22:37:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:34 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2537191407' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:35.316 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0077 2024-06-14T22:37:35.316 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 130083 2024-06-14T22:37:35.316 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 90.3277 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 38.7229 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 240.625 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 51.0625 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1445 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 619.567 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 3850 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 817 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0110668 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.00805898 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 0.162806 2024-06-14T22:37:35.317 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00115898 2024-06-14T22:37:35.322 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T22:37:36.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:36 smithi137 ceph-mon[32951]: pgmap v142: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-06-14T22:37:37.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:36 smithi086 ceph-mon[29889]: pgmap v142: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-06-14T22:37:37.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:36 smithi086 ceph-mon[38904]: pgmap v142: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-06-14T22:37:37.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:37 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2269999092' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:37.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:37 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2269999092' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:37 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2269999092' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:38.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:38 smithi137 ceph-mon[32951]: pgmap v143: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 46 MiB/s wr, 742 op/s 2024-06-14T22:37:39.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:38 smithi086 ceph-mon[29889]: pgmap v143: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 46 MiB/s wr, 742 op/s 2024-06-14T22:37:39.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:38 smithi086 ceph-mon[38904]: pgmap v143: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 46 MiB/s wr, 742 op/s 2024-06-14T22:37:39.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:39 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/778196268' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:39.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:39] "GET /metrics HTTP/1.1" 200 242701 "" "Prometheus/2.43.0" 2024-06-14T22:37:39.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:39 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/778196268' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:39.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:39 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/778196268' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:40.912 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:40 smithi137 ceph-mon[32951]: pgmap v144: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 55 MiB/s wr, 1.14k op/s 2024-06-14T22:37:40.913 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:37:41.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:40 smithi086 ceph-mon[29889]: pgmap v144: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 55 MiB/s wr, 1.14k op/s 2024-06-14T22:37:41.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:40 smithi086 ceph-mon[38904]: pgmap v144: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 55 MiB/s wr, 1.14k op/s 2024-06-14T22:37:42.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:37:41] ENGINE Bus STOPPING 2024-06-14T22:37:42.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ignoring --setuser ceph since I am not root 2024-06-14T22:37:42.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ignoring --setgroup ceph since I am not root 2024-06-14T22:37:42.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mgr[30165]: -- 172.21.15.86:0/1461759100 <== mon.2 v2:172.21.15.86:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b8425b49c0 con 0x55b8425c0c00 2024-06-14T22:37:42.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:42.194+0000 7fd3b6b84ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:37:42.605 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:37:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T22:37:42.606 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:37:42] ENGINE Bus STOPPED 2024-06-14T22:37:42.606 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:37:42] ENGINE Bus STARTING 2024-06-14T22:37:42.733 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:42.394+0000 7fd3b6b84ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:37:42.733 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:42.486+0000 7fd3b6b84ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:37:42.863 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:37:42] ENGINE Serving on http://:::9283 2024-06-14T22:37:42.863 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:37:42] ENGINE Bus STARTED 2024-06-14T22:37:42.863 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.821+0000 7f8c04459640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.864 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.821+0000 7f8c04459640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.864 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.821+0000 7f8c04459640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.864 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.821+0000 7f8c04459640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.864 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.821+0000 7f8c04459640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.864 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.822+0000 7f8c0a465640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.864 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.822+0000 7f8c0a465640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.864 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.822+0000 7f8c0a465640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.864 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.822+0000 7f8c0a465640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.864 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:42.822+0000 7f8c0a465640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: Activating manager daemon x 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: osdmap e75: 8 total, 8 up, 8 in 2024-06-14T22:37:42.865 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: mgrmap e22: x(active, starting, since 0.0211576s) 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:37:42.866 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2505172664' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: Manager daemon x is now available 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-14T22:37:42.867 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:42 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-14T22:37:43.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:42.848+0000 7fd3b6b84ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:37:43.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:42.938+0000 7fd3b6b84ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: Activating manager daemon x 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: osdmap e75: 8 total, 8 up, 8 in 2024-06-14T22:37:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: mgrmap e22: x(active, starting, since 0.0211576s) 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:37:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2505172664' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: Manager daemon x is now available 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-14T22:37:43.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.14424 172.21.15.86:0/4096723409' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: Activating manager daemon x 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: osdmap e75: 8 total, 8 up, 8 in 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.14424 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-06-14T22:37:43.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: mgrmap e22: x(active, starting, since 0.0211576s) 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:37:43.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2505172664' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: Manager daemon x is now available 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-14T22:37:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:42 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:43.394+0000 7fd3b6b84ec0 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:43.394+0000 7fd3b6b84ec0 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from .module import Module 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:43.395+0000 7fd3b6b84ec0 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:37:43.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:43.395+0000 7fd3b6b84ec0 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:37:44.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:43 smithi137 ceph-mon[32951]: mgrmap e23: x(active, since 1.01738s) 2024-06-14T22:37:44.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:43 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:44.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:43 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:44.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:43 smithi137 ceph-mon[32951]: Deploying cephadm binary to smithi086 2024-06-14T22:37:44.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:43 smithi137 ceph-mon[32951]: Deploying cephadm binary to smithi137 2024-06-14T22:37:44.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[29889]: mgrmap e23: x(active, since 1.01738s) 2024-06-14T22:37:44.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:44.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:44.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[29889]: Deploying cephadm binary to smithi086 2024-06-14T22:37:44.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[29889]: Deploying cephadm binary to smithi137 2024-06-14T22:37:44.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[38904]: mgrmap e23: x(active, since 1.01738s) 2024-06-14T22:37:44.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:44.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:44.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[38904]: Deploying cephadm binary to smithi086 2024-06-14T22:37:44.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:43 smithi086 ceph-mon[38904]: Deploying cephadm binary to smithi137 2024-06-14T22:37:44.660 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:44.365+0000 7fd3b6b84ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:37:44.660 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:44.561+0000 7fd3b6b84ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:37:44.660 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:44.663+0000 7fd3b6b84ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:37:45.100 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-mon[32951]: pgmap v4: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail 2024-06-14T22:37:45.100 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/268261765' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:45.100 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:44.772+0000 7f8be7c20640 -1 log_channel(cephadm) log [ERR] : [14/Jun/2024:22:37:44] ENGINE Error in HTTPServer.serve 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: Traceback (most recent call last): 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: self._connections.run(self.expiration_interval) 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: self._run(expiration_interval) 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: s = self.context.wrap_socket( 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:37:45.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: return self.sslsocket_class._create( 2024-06-14T22:37:45.102 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:37:45.102 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: self.do_handshake() 2024-06-14T22:37:45.102 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:37:45.102 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: self._sslobj.do_handshake() 2024-06-14T22:37:45.102 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:37:45.102 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:37:45.212 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:45.000+0000 7fd3b6b84ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:37:45.212 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:45.099+0000 7fd3b6b84ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:37:45.212 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:45.215+0000 7fd3b6b84ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:37:45.212 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:44 smithi086 ceph-mon[38904]: pgmap v4: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail 2024-06-14T22:37:45.212 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:44 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/268261765' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:45.213 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:44 smithi086 ceph-mon[29889]: pgmap v4: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail 2024-06-14T22:37:45.213 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:44 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/268261765' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:45.580 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:45.444+0000 7fd3b6b84ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:37:46.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:45 smithi137 ceph-mon[32951]: [14/Jun/2024:22:37:44] ENGINE Bus STARTING 2024-06-14T22:37:46.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:45 smithi137 ceph-mon[32951]: [14/Jun/2024:22:37:44] ENGINE Serving on http://172.21.15.137:8765 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:45 smithi137 ceph-mon[32951]: [14/Jun/2024:22:37:44] ENGINE Serving on https://172.21.15.137:7150 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:45 smithi137 ceph-mon[32951]: [14/Jun/2024:22:37:44] ENGINE Bus STARTED 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:45 smithi137 ceph-mon[32951]: [14/Jun/2024:22:37:44] ENGINE Error in HTTPServer.serve 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: Traceback (most recent call last): 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._run(expiration_interval) 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:37:46.242 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:37:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout: s = self.context.wrap_socket( 2024-06-14T22:37:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:37:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout: return self.sslsocket_class._create( 2024-06-14T22:37:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:37:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout: self.do_handshake() 2024-06-14T22:37:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:37:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._sslobj.do_handshake() 2024-06-14T22:37:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:37:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:45 smithi137 ceph-mon[32951]: mgrmap e24: x(active, since 3s) 2024-06-14T22:37:46.245 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[29889]: [14/Jun/2024:22:37:44] ENGINE Bus STARTING 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[29889]: [14/Jun/2024:22:37:44] ENGINE Serving on http://172.21.15.137:8765 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[29889]: [14/Jun/2024:22:37:44] ENGINE Serving on https://172.21.15.137:7150 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[29889]: [14/Jun/2024:22:37:44] ENGINE Bus STARTED 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[29889]: [14/Jun/2024:22:37:44] ENGINE Error in HTTPServer.serve 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:37:46.246 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.a.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.a.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.a.smithi086.stdout: self.do_handshake() 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.a.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[29889]: mgrmap e24: x(active, since 3s) 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[38904]: [14/Jun/2024:22:37:44] ENGINE Bus STARTING 2024-06-14T22:37:46.247 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[38904]: [14/Jun/2024:22:37:44] ENGINE Serving on http://172.21.15.137:8765 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[38904]: [14/Jun/2024:22:37:44] ENGINE Serving on https://172.21.15.137:7150 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[38904]: [14/Jun/2024:22:37:44] ENGINE Bus STARTED 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[38904]: [14/Jun/2024:22:37:44] ENGINE Error in HTTPServer.serve 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:37:46.248 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:37:46.249 INFO:journalctl@ceph.mon.c.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:37:46.249 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:37:46.249 INFO:journalctl@ceph.mon.c.smithi086.stdout: self.do_handshake() 2024-06-14T22:37:46.249 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:37:46.249 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:37:46.249 INFO:journalctl@ceph.mon.c.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:37:46.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:45 smithi086 ceph-mon[38904]: mgrmap e24: x(active, since 3s) 2024-06-14T22:37:46.249 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:46.155+0000 7fd3b6b84ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:37:46.939 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:46.634+0000 7fd3b6b84ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:37:46.939 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:46.736+0000 7fd3b6b84ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:37:46.939 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:46.940+0000 7fd3b6b84ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:37:47.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:46 smithi137 ceph-mon[32951]: pgmap v5: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail 2024-06-14T22:37:47.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:46 smithi086 ceph-mon[29889]: pgmap v5: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail 2024-06-14T22:37:47.288 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:46 smithi086 ceph-mon[38904]: pgmap v5: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail 2024-06-14T22:37:47.580 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:47.430+0000 7fd3b6b84ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:37:47.581 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:47.531+0000 7fd3b6b84ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:37:48.184 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-mon[38904]: mgrmap e25: x(active, since 5s) 2024-06-14T22:37:48.184 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1404134949' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:48.184 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:48.184 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:48.184 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:47.849+0000 7fd3b6b84ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:37:48.184 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:47.948+0000 7fd3b6b84ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:37:48.185 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:48.090+0000 7fd3b6b84ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:37:48.185 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:48.186+0000 7fd3b6b84ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:37:48.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-mon[29889]: mgrmap e25: x(active, since 5s) 2024-06-14T22:37:48.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1404134949' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:48.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:48.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:47 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:48.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:47 smithi137 ceph-mon[32951]: mgrmap e25: x(active, since 5s) 2024-06-14T22:37:48.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:47 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1404134949' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:48.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:47 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:48.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:47 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: pgmap v6: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.099 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.100 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.100 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:48 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:49.132+0000 7fd3b6b84ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:37:49.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: pgmap v6: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail 2024-06-14T22:37:49.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: pgmap v6: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:49.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:48 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:49.363+0000 7fd3b6b84ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:37:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:49.364+0000 7fd3b6b84ec0 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:37:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:37:49] ENGINE Bus STARTING 2024-06-14T22:37:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: CherryPy Checker: 2024-06-14T22:37:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: The Application mounted at '' has an empty config. 2024-06-14T22:37:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: 2024-06-14T22:37:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:37:49] ENGINE Serving on http://:::9283 2024-06-14T22:37:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: [14/Jun/2024:22:37:49] ENGINE Bus STARTED 2024-06-14T22:37:49.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:37:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:49 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi086 to 2480M 2024-06-14T22:37:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:49 smithi137 ceph-mon[32951]: Standby manager daemon y started 2024-06-14T22:37:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:49 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-14T22:37:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:49 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:37:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:49 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-14T22:37:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:49 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:37:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:49 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:49 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:50.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi086 to 2480M 2024-06-14T22:37:50.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[29889]: Standby manager daemon y started 2024-06-14T22:37:50.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-14T22:37:50.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:37:50.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-14T22:37:50.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:37:50.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:50.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:50.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi086 to 2480M 2024-06-14T22:37:50.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[38904]: Standby manager daemon y started 2024-06-14T22:37:50.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-14T22:37:50.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:37:50.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-14T22:37:50.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.86:0/891985790' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:37:50.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:50.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:49 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:50.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:37:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:50] "GET /metrics HTTP/1.1" 200 33289 "" "Prometheus/2.43.0" 2024-06-14T22:37:51.322 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:50 smithi137 ceph-mon[32951]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:37:51.322 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:50 smithi137 ceph-mon[32951]: pgmap v7: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 604 op/s 2024-06-14T22:37:51.322 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:50 smithi137 ceph-mon[32951]: mgrmap e26: x(active, since 8s), standbys: y 2024-06-14T22:37:51.322 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:50 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:37:51.322 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:50 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1509424487' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:51.322 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:50 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:51.322 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:50 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:51.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[29889]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:37:51.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[29889]: pgmap v7: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 604 op/s 2024-06-14T22:37:51.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[29889]: mgrmap e26: x(active, since 8s), standbys: y 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1509424487' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[38904]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[38904]: pgmap v7: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 604 op/s 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[38904]: mgrmap e26: x(active, since 8s), standbys: y 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1509424487' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:51.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:50 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:53.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:52 smithi137 ceph-mon[32951]: pgmap v8: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 469 op/s 2024-06-14T22:37:53.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:52 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1245637854' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:53.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:52 smithi086 ceph-mon[29889]: pgmap v8: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 469 op/s 2024-06-14T22:37:53.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:52 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1245637854' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:53.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:52 smithi086 ceph-mon[38904]: pgmap v8: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 469 op/s 2024-06-14T22:37:53.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:52 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1245637854' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:55.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:55 smithi137 ceph-mon[32951]: pgmap v9: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 793 op/s 2024-06-14T22:37:55.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:55 smithi086 ceph-mon[29889]: pgmap v9: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 793 op/s 2024-06-14T22:37:55.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:55 smithi086 ceph-mon[38904]: pgmap v9: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 793 op/s 2024-06-14T22:37:56.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:56 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4011379860' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:56.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:56 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4011379860' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:56.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:56 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4011379860' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:57.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: pgmap v10: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 724 op/s 2024-06-14T22:37:57.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:57.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:57.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: Adjusting osd_memory_target on smithi137 to 2480M 2024-06-14T22:37:57.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:57.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:37:57.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:37:57.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:37:57.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:57 smithi137 ceph-mon[32951]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:37:57.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: pgmap v10: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 724 op/s 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: Adjusting osd_memory_target on smithi137 to 2480M 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: pgmap v10: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 724 op/s 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: Adjusting osd_memory_target on smithi137 to 2480M 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:37:57.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:57 smithi086 ceph-mon[29889]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:37:58.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:37:58.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:37:58.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:37:58.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:37:58.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:37:58.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3494189270' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:37:58.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3494189270' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:58 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3494189270' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:58.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:58 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:37:59.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:59 smithi137 ceph-mon[32951]: pgmap v11: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 724 op/s 2024-06-14T22:37:59.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:59 smithi137 ceph-mon[32951]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-14T22:37:59.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:37:59 smithi137 ceph-mon[32951]: Reconfiguring daemon alertmanager.a on smithi086 2024-06-14T22:37:59.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:59 smithi086 ceph-mon[29889]: pgmap v11: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 724 op/s 2024-06-14T22:37:59.547 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:59 smithi086 ceph-mon[29889]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-14T22:37:59.547 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:37:59 smithi086 ceph-mon[29889]: Reconfiguring daemon alertmanager.a on smithi086 2024-06-14T22:37:59.547 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:59 smithi086 ceph-mon[38904]: pgmap v11: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 724 op/s 2024-06-14T22:37:59.547 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:59 smithi086 ceph-mon[38904]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-14T22:37:59.547 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:37:59 smithi086 ceph-mon[38904]: Reconfiguring daemon alertmanager.a on smithi086 2024-06-14T22:37:59.830 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:37:59 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:37:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:38:00.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:00 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:38:00] "GET /metrics HTTP/1.1" 200 36229 "" "Prometheus/2.43.0" 2024-06-14T22:38:01.135 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:01 smithi086 ceph-mon[29889]: pgmap v12: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-06-14T22:38:01.136 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:01 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3910493479' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:01 smithi137 ceph-mon[32951]: pgmap v12: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-06-14T22:38:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:01 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3910493479' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:01.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:01 smithi086 ceph-mon[38904]: pgmap v12: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-06-14T22:38:01.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:01 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3910493479' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:03.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:03 smithi086 ceph-mon[29889]: pgmap v13: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-06-14T22:38:03.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:03 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/718219931' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:03.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:03 smithi086 ceph-mon[38904]: pgmap v13: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-06-14T22:38:03.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:03 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/718219931' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:03.332 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:03 smithi086 systemd[1]: Stopping Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:03.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:03 smithi137 ceph-mon[32951]: pgmap v13: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-06-14T22:38:03.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:03 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/718219931' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:04.054 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:03 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[75240]: ts=2024-06-14T22:38:03.661Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-06-14T22:38:04.054 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:03 smithi086 podman[87628]: 2024-06-14 22:38:03.734477433 +0000 UTC m=+0.217175499 container died 3c3fa36d4b10054f84174258c50f477108018943b6b5eefde88677273874da19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:04.330 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:04 smithi086 podman[87628]: 2024-06-14 22:38:04.256605348 +0000 UTC m=+0.739303415 container cleanup 3c3fa36d4b10054f84174258c50f477108018943b6b5eefde88677273874da19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:04.355 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:04 smithi086 bash[87628]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a 2024-06-14T22:38:04.830 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:04 smithi086 podman[87644]: 2024-06-14 22:38:04.483061728 +0000 UTC m=+0.740253157 container remove 3c3fa36d4b10054f84174258c50f477108018943b6b5eefde88677273874da19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:04.831 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:04 smithi086 podman[87644]: 2024-06-14 22:38:04.575611786 +0000 UTC m=+0.832803211 volume remove 43058b324957ee1cb23aeb5b4c53846bba8e9eaa4cfe93137444878a09197c7a 2024-06-14T22:38:05.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:05 smithi086 ceph-mon[29889]: pgmap v14: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-14T22:38:05.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:05 smithi086 ceph-mon[38904]: pgmap v14: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-14T22:38:05.331 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:05 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-06-14T22:38:05.331 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:05 smithi086 systemd[1]: Stopped Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:05.331 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:05 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@alertmanager.a.service: Consumed 1.462s CPU time. 2024-06-14T22:38:05.331 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:05 smithi086 systemd[1]: Starting Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:05.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:05 smithi137 ceph-mon[32951]: pgmap v14: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-14T22:38:05.747 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:05 smithi086 podman[87740]: 2024-06-14 22:38:05.295518963 +0000 UTC m=+0.028150398 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-06-14T22:38:06.080 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:05 smithi086 podman[87740]: 2024-06-14 22:38:05.751709052 +0000 UTC m=+0.484340471 volume create 2b944fe13be593311291ac67446c7b2832416ba066ab53ff48b15955c3d44bf6 2024-06-14T22:38:06.081 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:05 smithi086 podman[87740]: 2024-06-14 22:38:05.860527397 +0000 UTC m=+0.593158815 container create 1c9917382faac8a5d9d505bb607b6203aaeea0114aa2bc6fe83e757e75c03f30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:06.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:06 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2850082557' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:06.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2850082557' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:06.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2850082557' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:06.581 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 podman[87740]: 2024-06-14 22:38:06.255263419 +0000 UTC m=+0.987894842 container init 1c9917382faac8a5d9d505bb607b6203aaeea0114aa2bc6fe83e757e75c03f30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:06.581 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 podman[87740]: 2024-06-14 22:38:06.267521089 +0000 UTC m=+1.000152503 container start 1c9917382faac8a5d9d505bb607b6203aaeea0114aa2bc6fe83e757e75c03f30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:06.581 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:06.289Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-06-14T22:38:06.581 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:06.289Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-06-14T22:38:06.581 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:06.289Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.86 port=9094 2024-06-14T22:38:06.582 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:06.291Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-06-14T22:38:06.582 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:06.335Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-14T22:38:06.582 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:06.336Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-14T22:38:06.582 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:06.340Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-06-14T22:38:06.582 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:06.340Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-06-14T22:38:06.582 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 bash[87740]: 1c9917382faac8a5d9d505bb607b6203aaeea0114aa2bc6fe83e757e75c03f30 2024-06-14T22:38:06.582 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:06 smithi086 systemd[1]: Started Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:07.081 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:06 smithi086 systemd[1]: Stopping Ceph node-exporter.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:07.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[29889]: pgmap v15: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-06-14T22:38:07.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:07.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:07.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[29889]: Reconfiguring node-exporter.a (dependencies changed)... 2024-06-14T22:38:07.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[29889]: Reconfiguring daemon node-exporter.a on smithi086 2024-06-14T22:38:07.535 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[38904]: pgmap v15: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-06-14T22:38:07.535 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:07.535 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:07.535 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[38904]: Reconfiguring node-exporter.a (dependencies changed)... 2024-06-14T22:38:07.535 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:07 smithi086 ceph-mon[38904]: Reconfiguring daemon node-exporter.a on smithi086 2024-06-14T22:38:07.535 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:07 smithi086 podman[87905]: 2024-06-14 22:38:07.235741258 +0000 UTC m=+0.172098940 container died d83ddcfcdc25409dfba8b2471316d7907ab387c84b6cdbc17ae8059a0e314b52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:07.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:07 smithi137 ceph-mon[32951]: pgmap v15: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-06-14T22:38:07.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:07 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:07.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:07 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:07.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:07 smithi137 ceph-mon[32951]: Reconfiguring node-exporter.a (dependencies changed)... 2024-06-14T22:38:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:07 smithi137 ceph-mon[32951]: Reconfiguring daemon node-exporter.a on smithi086 2024-06-14T22:38:07.830 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:07 smithi086 podman[87905]: 2024-06-14 22:38:07.633790234 +0000 UTC m=+0.570147919 container cleanup d83ddcfcdc25409dfba8b2471316d7907ab387c84b6cdbc17ae8059a0e314b52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:07.831 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:07 smithi086 bash[87905]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a 2024-06-14T22:38:08.178 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:07 smithi086 podman[87921]: 2024-06-14 22:38:07.868526151 +0000 UTC m=+0.623388751 container remove d83ddcfcdc25409dfba8b2471316d7907ab387c84b6cdbc17ae8059a0e314b52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:08.178 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:07 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-06-14T22:38:08.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:08 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/35476829' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:08.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/35476829' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:08.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/35476829' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:08.581 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:08.291Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00032893s 2024-06-14T22:38:08.581 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-06-14T22:38:08.581 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 systemd[1]: Stopped Ceph node-exporter.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:08.581 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.a.service: Consumed 2.531s CPU time. 2024-06-14T22:38:08.582 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 systemd[1]: Starting Ceph node-exporter.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:08.942 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 podman[88025]: 2024-06-14 22:38:08.529310918 +0000 UTC m=+0.032251690 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-06-14T22:38:08.943 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 podman[88025]: 2024-06-14 22:38:08.661380151 +0000 UTC m=+0.164320914 container create d1de50e6087955ce1dabe399095a42d9ebac98b2290c1643769a421f423b99d6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:08.943 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 podman[88025]: 2024-06-14 22:38:08.946860112 +0000 UTC m=+0.449800884 container init d1de50e6087955ce1dabe399095a42d9ebac98b2290c1643769a421f423b99d6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 podman[88025]: 2024-06-14 22:38:08.959098547 +0000 UTC m=+0.462039317 container start d1de50e6087955ce1dabe399095a42d9ebac98b2290c1643769a421f423b99d6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.965Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.965Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.967Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.967Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.967Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.967Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=arp 2024-06-14T22:38:09.333 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=bcache 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=bonding 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=btrfs 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=conntrack 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=cpu 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=diskstats 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=dmi 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=edac 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=entropy 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=filefd 2024-06-14T22:38:09.334 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=filesystem 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=hwmon 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=infiniband 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=ipvs 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=loadavg 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=mdadm 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=meminfo 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=netclass 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=netdev 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=netstat 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=nfs 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=nfsd 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=nvme 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=os 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-06-14T22:38:09.335 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=pressure 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=rapl 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=schedstat 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=selinux 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=sockstat 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=softnet 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=stat 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=tapestats 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=textfile 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=time 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=uname 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=vmstat 2024-06-14T22:38:09.336 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=xfs 2024-06-14T22:38:09.337 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.968Z caller=node_exporter.go:117 level=info collector=zfs 2024-06-14T22:38:09.337 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.969Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-06-14T22:38:09.337 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a[88037]: ts=2024-06-14T22:38:08.969Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-06-14T22:38:09.337 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:09 smithi086 bash[88025]: d1de50e6087955ce1dabe399095a42d9ebac98b2290c1643769a421f423b99d6 2024-06-14T22:38:09.337 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 22:38:09 smithi086 systemd[1]: Started Ceph node-exporter.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:09.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:09 smithi137 ceph-mon[32951]: pgmap v16: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-06-14T22:38:09.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:09 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:09.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:09 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:09.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:09 smithi086 ceph-mon[29889]: pgmap v16: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-06-14T22:38:09.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:09 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:09.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:09 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:09.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[30142]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:38:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:38:09.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:09 smithi086 ceph-mon[38904]: pgmap v16: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-06-14T22:38:09.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:09 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:09.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:09 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:10.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:10 smithi137 ceph-mon[32951]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-14T22:38:10.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:10 smithi137 ceph-mon[32951]: Reconfiguring daemon prometheus.a on smithi137 2024-06-14T22:38:10.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:10 smithi086 ceph-mon[38904]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-14T22:38:10.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:10 smithi086 ceph-mon[38904]: Reconfiguring daemon prometheus.a on smithi137 2024-06-14T22:38:10.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:10 smithi086 ceph-mon[29889]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-14T22:38:10.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:10 smithi086 ceph-mon[29889]: Reconfiguring daemon prometheus.a on smithi137 2024-06-14T22:38:10.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:10 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:38:10] "GET /metrics HTTP/1.1" 200 36229 "" "Prometheus/2.43.0" 2024-06-14T22:38:11.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:11 smithi137 ceph-mon[32951]: pgmap v17: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-14T22:38:11.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:11 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/304419621' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:11.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:11 smithi086 ceph-mon[38904]: pgmap v17: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-14T22:38:11.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:11 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/304419621' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:11 smithi086 ceph-mon[29889]: pgmap v17: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-14T22:38:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:11 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/304419621' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:12.732 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:12 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:12 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:12.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:12 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:13.433 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 systemd[1]: Stopping Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:13.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-mon[32951]: pgmap v18: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 762 op/s 2024-06-14T22:38:13.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/838840559' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:13.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:13 smithi086 ceph-mon[29889]: pgmap v18: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 762 op/s 2024-06-14T22:38:13.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:13 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/838840559' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:13.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:13 smithi086 ceph-mon[38904]: pgmap v18: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 762 op/s 2024-06-14T22:38:13.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:13 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/838840559' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:14.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-14T22:38:14.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-06-14T22:38:14.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-06-14T22:38:14.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-14T22:38:14.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-06-14T22:38:14.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-06-14T22:38:14.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-06-14T22:38:14.241 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-06-14T22:38:14.241 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-06-14T22:38:14.241 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-06-14T22:38:14.241 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-06-14T22:38:14.241 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[65595]: ts=2024-06-14T22:38:13.789Z caller=main.go:1147 level=info msg="See you next time!" 2024-06-14T22:38:14.241 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:13 smithi137 podman[76959]: 2024-06-14 22:38:13.803083319 +0000 UTC m=+0.150987609 container died 4b95a815ba8cdfbd6e30b11b8b8f964dec63a58f773430ec39be740fa1f3052f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:14.683 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:14 smithi137 podman[76959]: 2024-06-14 22:38:14.393156792 +0000 UTC m=+0.741061060 container cleanup 4b95a815ba8cdfbd6e30b11b8b8f964dec63a58f773430ec39be740fa1f3052f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:14.683 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:14 smithi137 bash[76959]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a 2024-06-14T22:38:14.935 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:14 smithi137 podman[76975]: 2024-06-14 22:38:14.686828405 +0000 UTC m=+0.880440850 container remove 4b95a815ba8cdfbd6e30b11b8b8f964dec63a58f773430ec39be740fa1f3052f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:15.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:14 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-06-14T22:38:15.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:14 smithi137 systemd[1]: Stopped Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:15.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:14 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@prometheus.a.service: Consumed 2.109s CPU time. 2024-06-14T22:38:15.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:14 smithi137 systemd[1]: Starting Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:15.612 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-mon[32951]: pgmap v19: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:38:15.613 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 podman[77056]: 2024-06-14 22:38:15.181895323 +0000 UTC m=+0.044942115 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-06-14T22:38:15.613 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 podman[77056]: 2024-06-14 22:38:15.338195277 +0000 UTC m=+0.201242058 container create 93e3e52e4164a7c7925c095843b3bbf823aa9662204b2711be0d3b9b0fe041dc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:15.613 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 podman[77056]: 2024-06-14 22:38:15.616120349 +0000 UTC m=+0.479167134 container init 93e3e52e4164a7c7925c095843b3bbf823aa9662204b2711be0d3b9b0fe041dc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:15.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:15 smithi086 ceph-mon[29889]: pgmap v19: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:38:15.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:15 smithi086 ceph-mon[38904]: pgmap v19: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:38:15.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 podman[77056]: 2024-06-14 22:38:15.625634107 +0000 UTC m=+0.488680894 container start 93e3e52e4164a7c7925c095843b3bbf823aa9662204b2711be0d3b9b0fe041dc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.691Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.691Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.691Z caller=main.go:570 level=info host_details="(Linux 5.14.0-457.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024 x86_64 smithi137 (none))" 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.691Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.692Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.694Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.694Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.695Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.696Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.704Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.705Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.261µs 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.705Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.705Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 bash[77056]: 93e3e52e4164a7c7925c095843b3bbf823aa9662204b2711be0d3b9b0fe041dc 2024-06-14T22:38:15.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 systemd[1]: Started Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:15.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.761Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-06-14T22:38:15.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.762Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-06-14T22:38:15.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.762Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=39.309µs wal_replay_duration=56.870358ms wbl_replay_duration=137ns total_replay_duration=57.126426ms 2024-06-14T22:38:15.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.766Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-06-14T22:38:15.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.766Z caller=main.go:1029 level=info msg="TSDB started" 2024-06-14T22:38:15.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.766Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-14T22:38:15.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.830Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=64.003849ms db_storage=968ns remote_storage=1.544µs web_handler=490ns query_engine=1.258µs scrape=418.762µs scrape_sd=152.554µs notify=14.739µs notify_sd=15.542µs rules=62.83277ms tracing=8.049µs 2024-06-14T22:38:15.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.830Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-06-14T22:38:15.992 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:15.830Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-06-14T22:38:16.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:16 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1245111878' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:16.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:16 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:16.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:16 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:16.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:16 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1245111878' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:16.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:16 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:16.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:16 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:16.581 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:16 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:16.294Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002937057s 2024-06-14T22:38:16.640 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:16 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1245111878' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:16.640 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:16 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:16.640 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:16 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:17.682 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:17 smithi137 ceph-mon[32951]: Reconfiguring grafana.a (dependencies changed)... 2024-06-14T22:38:17.682 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:17 smithi137 ceph-mon[32951]: Reconfiguring daemon grafana.a on smithi137 2024-06-14T22:38:17.682 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:17 smithi137 ceph-mon[32951]: pgmap v20: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-06-14T22:38:17.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:17 smithi086 ceph-mon[38904]: Reconfiguring grafana.a (dependencies changed)... 2024-06-14T22:38:17.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:17 smithi086 ceph-mon[38904]: Reconfiguring daemon grafana.a on smithi137 2024-06-14T22:38:17.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:17 smithi086 ceph-mon[38904]: pgmap v20: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-06-14T22:38:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:17 smithi086 ceph-mon[29889]: Reconfiguring grafana.a (dependencies changed)... 2024-06-14T22:38:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:17 smithi086 ceph-mon[29889]: Reconfiguring daemon grafana.a on smithi137 2024-06-14T22:38:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:17 smithi086 ceph-mon[29889]: pgmap v20: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-06-14T22:38:18.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:18 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2632312260' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:18.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:18 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2632312260' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:18.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:18 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2632312260' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:19.458 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:19 smithi137 systemd[1]: Stopping Ceph grafana.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:19.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:19 smithi137 ceph-mon[32951]: pgmap v21: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-06-14T22:38:19.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:19 smithi086 ceph-mon[29889]: pgmap v21: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-06-14T22:38:19.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:19 smithi086 ceph-mon[38904]: pgmap v21: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 748 op/s 2024-06-14T22:38:20.242 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:19 smithi137 podman[77325]: 2024-06-14 22:38:19.947225753 +0000 UTC m=+0.487905814 container died a38b34e5bca6d227bf29d808fc97fc787503cedb7063d9d9999f35eb9a4fc078 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, architecture=x86_64, version=8.5, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, name=ubi8, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-06-14T22:38:20.739 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:20 smithi137 podman[77325]: 2024-06-14 22:38:20.56707513 +0000 UTC m=+1.107755191 container cleanup a38b34e5bca6d227bf29d808fc97fc787503cedb7063d9d9999f35eb9a4fc078 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, distribution-scope=public, maintainer=Paul Cuzner , vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.openshift.expose-services=, version=8.5, name=ubi8, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-06-14T22:38:20.740 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:20 smithi137 bash[77325]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a 2024-06-14T22:38:21.078 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:20 smithi137 podman[77341]: 2024-06-14 22:38:20.82853566 +0000 UTC m=+0.884961450 container remove a38b34e5bca6d227bf29d808fc97fc787503cedb7063d9d9999f35eb9a4fc078 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , release=239.1651231664, vcs-type=git, version=8.5, io.openshift.tags=base rhel8, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, distribution-scope=public, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, build-date=2022-04-29T11:48:11.055897) 2024-06-14T22:38:21.078 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:20 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-06-14T22:38:21.402 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-06-14T22:38:21.403 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 systemd[1]: Stopped Ceph grafana.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:21.403 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@grafana.a.service: Consumed 6.014s CPU time. 2024-06-14T22:38:21.403 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 systemd[1]: Starting Ceph grafana.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:21.720 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:21 smithi137 ceph-mon[32951]: pgmap v22: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-06-14T22:38:21.720 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:21 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2425414505' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:21.720 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 podman[77424]: 2024-06-14 22:38:21.309632209 +0000 UTC m=+0.027740351 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-06-14T22:38:21.720 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 podman[77424]: 2024-06-14 22:38:21.437396162 +0000 UTC m=+0.155504309 container create 12b0617a16e548189d76b0fdaeb60cfc07cf125520d1955aa756668e62f52a31 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, name=ubi8, architecture=x86_64, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-06-14T22:38:21.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:21 smithi086 ceph-mon[29889]: pgmap v22: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-06-14T22:38:21.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:21 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2425414505' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:21.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:21 smithi086 ceph-mon[38904]: pgmap v22: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-06-14T22:38:21.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:21 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2425414505' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:21.989 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 podman[77424]: 2024-06-14 22:38:21.724226165 +0000 UTC m=+0.442334322 container init 12b0617a16e548189d76b0fdaeb60cfc07cf125520d1955aa756668e62f52a31 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, vcs-type=git, version=8.5, io.openshift.tags=base rhel8, release=239.1651231664, io.buildah.version=1.32.0, distribution-scope=public, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, architecture=x86_64, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-06-14T22:38:21.990 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 podman[77424]: 2024-06-14 22:38:21.733402187 +0000 UTC m=+0.451510332 container start 12b0617a16e548189d76b0fdaeb60cfc07cf125520d1955aa756668e62f52a31 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.openshift.tags=base rhel8, name=ubi8, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc.) 2024-06-14T22:38:21.990 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 bash[77424]: 12b0617a16e548189d76b0fdaeb60cfc07cf125520d1955aa756668e62f52a31 2024-06-14T22:38:21.990 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:21 smithi137 systemd[1]: Started Ceph grafana.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:22.342 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.062096835Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-06-14T22:38:22.342 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.062898557Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.063094699Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.063207337Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.063311526Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.063414161Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.063518379Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.063616915Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.063725906Z level=info msg="Path Home" path=/usr/share/grafana 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.063833351Z level=info msg="Path Data" path=/var/lib/grafana 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.063945216Z level=info msg="Path Logs" path=/var/log/grafana 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.064055843Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.064106568Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=settings t=2024-06-14T22:38:22.064145178Z level=info msg="App mode production" 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=sqlstore t=2024-06-14T22:38:22.064259346Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-06-14T22:38:22.343 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=sqlstore t=2024-06-14T22:38:22.064317936Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-06-14T22:38:22.344 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=migrator t=2024-06-14T22:38:22.090026169Z level=info msg="Starting DB migrations" 2024-06-14T22:38:22.344 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=migrator t=2024-06-14T22:38:22.098080412Z level=info msg="migrations completed" performed=0 skipped=481 duration=705.93µs 2024-06-14T22:38:22.344 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=sqlstore t=2024-06-14T22:38:22.09888448Z level=info msg="Created default organization" 2024-06-14T22:38:22.344 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=plugin.loader t=2024-06-14T22:38:22.201288657Z level=info msg="Plugin registered" pluginID=input 2024-06-14T22:38:22.344 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=plugin.loader t=2024-06-14T22:38:22.245728522Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-06-14T22:38:22.344 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=plugin.loader t=2024-06-14T22:38:22.246008636Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-06-14T22:38:22.344 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=secrets t=2024-06-14T22:38:22.246194974Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-06-14T22:38:22.344 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=query_data t=2024-06-14T22:38:22.248808057Z level=info msg="Query Service initialization" 2024-06-14T22:38:22.344 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=live.push_http t=2024-06-14T22:38:22.256873814Z level=info msg="Live Push Gateway initialization" 2024-06-14T22:38:22.739 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:22 smithi137 systemd[1]: Stopping Ceph node-exporter.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:23.094 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:22 smithi137 podman[77588]: 2024-06-14 22:38:22.782825722 +0000 UTC m=+0.154989141 container died 6082a0b8781a5b874d805cfaeb6d84b6331b30f22b2994833b8fa5ec9e4c9f5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T22:38:23.095 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:23.095 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:23.095 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-mon[32951]: Reconfiguring node-exporter.b (dependencies changed)... 2024-06-14T22:38:23.095 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-mon[32951]: Reconfiguring daemon node-exporter.b on smithi137 2024-06-14T22:38:23.095 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:22 smithi137 ceph-mon[32951]: pgmap v23: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-06-14T22:38:23.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:23.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:23.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[38904]: Reconfiguring node-exporter.b (dependencies changed)... 2024-06-14T22:38:23.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[38904]: Reconfiguring daemon node-exporter.b on smithi137 2024-06-14T22:38:23.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[38904]: pgmap v23: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-06-14T22:38:23.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:23.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:23.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[29889]: Reconfiguring node-exporter.b (dependencies changed)... 2024-06-14T22:38:23.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[29889]: Reconfiguring daemon node-exporter.b on smithi137 2024-06-14T22:38:23.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:22 smithi086 ceph-mon[29889]: pgmap v23: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-06-14T22:38:23.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:23 smithi137 podman[77588]: 2024-06-14 22:38:23.279613991 +0000 UTC m=+0.651777415 container cleanup 6082a0b8781a5b874d805cfaeb6d84b6331b30f22b2994833b8fa5ec9e4c9f5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T22:38:23.490 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:23 smithi137 bash[77588]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b 2024-06-14T22:38:23.826 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:23 smithi137 podman[77603]: 2024-06-14 22:38:23.581751669 +0000 UTC m=+0.787912103 container remove 6082a0b8781a5b874d805cfaeb6d84b6331b30f22b2994833b8fa5ec9e4c9f5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T22:38:23.826 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:23 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=infra.usagestats.collector t=2024-06-14T22:38:23.540827867Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=provisioning.datasources t=2024-06-14T22:38:23.635144129Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=provisioning.datasources t=2024-06-14T22:38:23.635427471Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=provisioning.datasources t=2024-06-14T22:38:23.646154675Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=provisioning.alerting t=2024-06-14T22:38:23.750900785Z level=info msg="starting to provision alerting" 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=provisioning.alerting t=2024-06-14T22:38:23.750953316Z level=info msg="finished to provision alerting" 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=grafanaStorageLogger t=2024-06-14T22:38:23.752760346Z level=info msg="storage starting" 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=ngalert.state.manager t=2024-06-14T22:38:23.752966398Z level=info msg="Warming state cache for startup" 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=http.server t=2024-06-14T22:38:23.754312382Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=ngalert.state.manager t=2024-06-14T22:38:23.766872694Z level=info msg="State cache has been initialized" states=0 duration=13.904619ms 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=ticker t=2024-06-14T22:38:23.767724391Z level=info msg=starting first_tick=2024-06-14T22:38:30Z 2024-06-14T22:38:23.827 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=ngalert.multiorg.alertmanager t=2024-06-14T22:38:23.767752669Z level=info msg="starting MultiOrg Alertmanager" 2024-06-14T22:38:24.153 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:23 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1655640744' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:24.153 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:23 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-06-14T22:38:24.153 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:23 smithi137 systemd[1]: Stopped Ceph node-exporter.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:24.153 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:23 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.b.service: Consumed 2.601s CPU time. 2024-06-14T22:38:24.153 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:23 smithi137 systemd[1]: Starting Ceph node-exporter.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:24.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:23 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1655640744' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:24.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:23 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1655640744' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:24.484 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 podman[77682]: 2024-06-14 22:38:24.06122575 +0000 UTC m=+0.035379996 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-06-14T22:38:24.484 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 podman[77682]: 2024-06-14 22:38:24.341695579 +0000 UTC m=+0.315849817 container create b8c1542255f2b228a5cdb45bade4bab608b7a34f9b299291291e96755d7cbc98 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T22:38:24.741 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 podman[77682]: 2024-06-14 22:38:24.644363997 +0000 UTC m=+0.618518240 container init b8c1542255f2b228a5cdb45bade4bab608b7a34f9b299291291e96755d7cbc98 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 podman[77682]: 2024-06-14 22:38:24.657676782 +0000 UTC m=+0.631831017 container start b8c1542255f2b228a5cdb45bade4bab608b7a34f9b299291291e96755d7cbc98 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.663Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.663Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.663Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=arp 2024-06-14T22:38:24.742 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=bcache 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=bonding 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=btrfs 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=conntrack 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=cpu 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=diskstats 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=dmi 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=edac 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=entropy 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=filefd 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.664Z caller=node_exporter.go:117 level=info collector=filesystem 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=hwmon 2024-06-14T22:38:24.743 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=infiniband 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=ipvs 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=loadavg 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=mdadm 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=meminfo 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=netclass 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=netdev 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=netstat 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=nfs 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=nfsd 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=nvme 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=os 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=pressure 2024-06-14T22:38:24.744 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=rapl 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=schedstat 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=selinux 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=sockstat 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=softnet 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=stat 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=tapestats 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=textfile 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=time 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=uname 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=vmstat 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=xfs 2024-06-14T22:38:24.745 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=node_exporter.go:117 level=info collector=zfs 2024-06-14T22:38:24.746 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-06-14T22:38:24.746 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b[77694]: ts=2024-06-14T22:38:24.665Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-06-14T22:38:25.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[29889]: pgmap v24: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-06-14T22:38:25.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:25.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:25.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:38:25.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:38:25.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:38:25.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[38904]: pgmap v24: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-06-14T22:38:25.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:25.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:25.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:38:25.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:38:25.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:24 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:38:25.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-mon[32951]: pgmap v24: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-06-14T22:38:25.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:25.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:25.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:38:25.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:38:25.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:24 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:38:25.240 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 bash[77682]: b8c1542255f2b228a5cdb45bade4bab608b7a34f9b299291291e96755d7cbc98 2024-06-14T22:38:25.240 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 22:38:24 smithi137 systemd[1]: Started Ceph node-exporter.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:26.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:25 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:38:26.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:25 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:38:26.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:25 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:38:26.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:25 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:26.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:25 smithi137 ceph-mon[32951]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-14T22:38:26.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:25 smithi137 ceph-mon[32951]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-14T22:38:26.267 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:38:26.267 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:38:26.267 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:38:26.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:26.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[29889]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-14T22:38:26.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[29889]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-14T22:38:26.268 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-14T22:38:26.268 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:38:26.268 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-14T22:38:26.268 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:26.268 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[38904]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-14T22:38:26.268 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:25 smithi086 ceph-mon[38904]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-14T22:38:27.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:26 smithi137 ceph-mon[32951]: pgmap v25: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-06-14T22:38:27.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:26 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/450759212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:27.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:26 smithi086 ceph-mon[29889]: pgmap v25: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-06-14T22:38:27.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:26 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/450759212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:27.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:26 smithi086 ceph-mon[38904]: pgmap v25: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-06-14T22:38:27.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:26 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/450759212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:28.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:27 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:28.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:27 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:28.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:27 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[38904]: pgmap v26: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-06-14T22:38:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[38904]: Upgrade: Updating mgr.y 2024-06-14T22:38:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:38:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:38:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:38:29.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[38904]: Deploying daemon mgr.y on smithi086 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/585980518' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[29889]: pgmap v26: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[29889]: Upgrade: Updating mgr.y 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[29889]: Deploying daemon mgr.y on smithi086 2024-06-14T22:38:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:29 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/585980518' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:29.489 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:38:29] "GET /metrics HTTP/1.1" 200 36223 "" "Prometheus/2.43.0" 2024-06-14T22:38:29.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-mon[32951]: pgmap v26: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-06-14T22:38:29.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-mon[32951]: Upgrade: Updating mgr.y 2024-06-14T22:38:29.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:29.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:38:29.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:38:29.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:38:29.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:38:29.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-mon[32951]: Deploying daemon mgr.y on smithi086 2024-06-14T22:38:29.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:29 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/585980518' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:30.835 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:30 smithi086 systemd[1]: Stopping Ceph mgr.y for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:31.267 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:31 smithi086 ceph-mon[38904]: pgmap v27: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-14T22:38:31.267 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:31 smithi086 ceph-mon[29889]: pgmap v27: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-14T22:38:31.267 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:30 smithi086 podman[88614]: 2024-06-14 22:38:30.935838808 +0000 UTC m=+0.209661663 container died 3ed18014b7a87431fed59503ac4e324f5e6ddd075b6260868b45d9f503ba8241 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0) 2024-06-14T22:38:31.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:31 smithi137 ceph-mon[32951]: pgmap v27: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-14T22:38:31.580 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:31 smithi086 podman[88614]: 2024-06-14 22:38:31.476052076 +0000 UTC m=+0.749874928 container cleanup 3ed18014b7a87431fed59503ac4e324f5e6ddd075b6260868b45d9f503ba8241 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-06-14T22:38:31.581 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:31 smithi086 bash[88614]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y 2024-06-14T22:38:32.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:31 smithi086 podman[88649]: 2024-06-14 22:38:31.727725215 +0000 UTC m=+0.787161503 container remove 3ed18014b7a87431fed59503ac4e324f5e6ddd075b6260868b45d9f503ba8241 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:38:32.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:31 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-06-14T22:38:32.408 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:32 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3088786823' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:32.409 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:32 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3088786823' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:32.409 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:32 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-06-14T22:38:32.409 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:32 smithi086 systemd[1]: Stopped Ceph mgr.y for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:32.409 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:32 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y.service: Consumed 44.303s CPU time. 2024-06-14T22:38:32.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:32 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3088786823' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:32.739 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:32 smithi086 systemd[1]: Starting Ceph mgr.y for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:38:33.070 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:32 smithi086 podman[88777]: 2024-06-14 22:38:32.645532277 +0000 UTC m=+0.042840458 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:38:33.070 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:32 smithi086 podman[88777]: 2024-06-14 22:38:32.787442529 +0000 UTC m=+0.184750705 container create edafb7b2b182f1c704baab1d447c3cd128aff25ced52c1a50ce6c194e1b85cd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True) 2024-06-14T22:38:33.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:33 smithi086 ceph-mon[38904]: pgmap v28: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-06-14T22:38:33.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:33 smithi086 ceph-mon[29889]: pgmap v28: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-06-14T22:38:33.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:33 smithi086 podman[88777]: 2024-06-14 22:38:33.073785728 +0000 UTC m=+0.471093907 container init edafb7b2b182f1c704baab1d447c3cd128aff25ced52c1a50ce6c194e1b85cd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_CLEAN=True) 2024-06-14T22:38:33.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:33 smithi086 podman[88777]: 2024-06-14 22:38:33.085310863 +0000 UTC m=+0.482619034 container start edafb7b2b182f1c704baab1d447c3cd128aff25ced52c1a50ce6c194e1b85cd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-14T22:38:33.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:33 smithi086 ceph-mgr[88817]: -- 172.21.15.86:0/3641806059 <== mon.2 v2:172.21.15.86:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55559f396d00 con 0x55559f36a800 2024-06-14T22:38:33.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:33 smithi086 bash[88777]: edafb7b2b182f1c704baab1d447c3cd128aff25ced52c1a50ce6c194e1b85cd1 2024-06-14T22:38:33.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:33 smithi086 systemd[1]: Started Ceph mgr.y for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:38:33.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:33 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:33.307+0000 7fe3dc287040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:38:33.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:33 smithi137 ceph-mon[32951]: pgmap v28: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-06-14T22:38:33.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:33 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:33.603+0000 7fe3dc287040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:38:33.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:33 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:33.727+0000 7fe3dc287040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:38:33.831 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:33 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:33.510Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi086.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.86:8443: connect: connection refused" 2024-06-14T22:38:33.989 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:33 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:33.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:38:34.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:34 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:34.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:34 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:34.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:34 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:34.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:34 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1158661122' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:34.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:34.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:34.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:34.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1158661122' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:34.581 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:34.146+0000 7fe3dc287040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:38:34.581 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:34.253+0000 7fe3dc287040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:38:34.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:34.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:34.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:34.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1158661122' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:34.766+0000 7fe3dc287040 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:38:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:34.766+0000 7fe3dc287040 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:38:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:38:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: from .module import Module 2024-06-14T22:38:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:38:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:38:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:38:35.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:35.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:34.767+0000 7fe3dc287040 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:38:35.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:34.767+0000 7fe3dc287040 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:38:35.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:35 smithi137 ceph-mon[32951]: pgmap v29: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:38:35.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:35 smithi086 ceph-mon[29889]: pgmap v29: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:38:35.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:35 smithi086 ceph-mon[38904]: pgmap v29: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:38:36.158 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:35.856+0000 7fe3dc287040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:38:36.158 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:36 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:36.058+0000 7fe3dc287040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:38:36.546 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:36 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:36.160+0000 7fe3dc287040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:38:36.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:36 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:36.549+0000 7fe3dc287040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:38:36.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:36 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:36.661+0000 7fe3dc287040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:38:36.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:36 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:36.778+0000 7fe3dc287040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:38:37.203 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:37 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:37.031+0000 7fe3dc287040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:38:37.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:37 smithi137 ceph-mon[32951]: pgmap v30: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-06-14T22:38:37.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:37 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2599204372' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:37.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:37 smithi086 ceph-mon[29889]: pgmap v30: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-06-14T22:38:37.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:37 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2599204372' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:37.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:37 smithi086 ceph-mon[38904]: pgmap v30: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-06-14T22:38:37.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:37 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2599204372' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:38.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:37 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:37.734+0000 7fe3dc287040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:38:38.581 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:38 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:38.267+0000 7fe3dc287040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:38:38.581 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:38 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:38.381+0000 7fe3dc287040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:38:38.989 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:38.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi137:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:38:39.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:38 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:38.628+0000 7fe3dc287040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:38:39.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:38:39] "GET /metrics HTTP/1.1" 200 36223 "" "Prometheus/2.43.0" 2024-06-14T22:38:39.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:39 smithi137 ceph-mon[32951]: pgmap v31: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-06-14T22:38:39.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:39 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:39 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:39 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:39 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[29889]: pgmap v31: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-06-14T22:38:39.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.581 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:39.218+0000 7fe3dc287040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:38:39.582 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:39.333+0000 7fe3dc287040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:38:39.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[38904]: pgmap v31: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-06-14T22:38:39.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:39.939 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:39.660+0000 7fe3dc287040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:38:39.939 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:39.904+0000 7fe3dc287040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-14T22:38:40.203 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:40.016+0000 7fe3dc287040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:38:40.232 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:40.159+0000 7fe3dc287040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:38:40.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:40 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1257162142' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:40.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:40 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:40.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:40 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:40.507 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:40 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1257162142' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:40.507 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:40 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:40.507 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:40 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:40.508 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:40 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1257162142' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:40.508 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:40 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:40.508 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:40 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:40.508 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:40.262+0000 7fe3dc287040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:38:41.472 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-mon[38904]: pgmap v32: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:38:41.472 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-mon[29889]: pgmap v32: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:38:41.472 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:41.167+0000 7fe3dc287040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:38:41.734 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:41.474+0000 7fe3dc287040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:38:41.734 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:41.474+0000 7fe3dc287040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:38:41.734 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:38:41] ENGINE Bus STARTING 2024-06-14T22:38:41.734 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: CherryPy Checker: 2024-06-14T22:38:41.734 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: The Application mounted at '' has an empty config. 2024-06-14T22:38:41.734 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:41.734 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:38:41] ENGINE Serving on http://:::9283 2024-06-14T22:38:41.734 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:38:41] ENGINE Bus STARTED 2024-06-14T22:38:41.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:41 smithi137 ceph-mon[32951]: pgmap v32: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:38:42.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[29889]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:38:42.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[29889]: Standby manager daemon y restarted 2024-06-14T22:38:42.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[29889]: Standby manager daemon y started 2024-06-14T22:38:42.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-14T22:38:42.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:38:42.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-14T22:38:42.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:38:42.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/110307250' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:42.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[38904]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:38:42.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[38904]: Standby manager daemon y restarted 2024-06-14T22:38:42.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[38904]: Standby manager daemon y started 2024-06-14T22:38:42.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-14T22:38:42.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:38:42.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-14T22:38:42.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:38:42.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:42 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/110307250' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:42.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:42 smithi137 ceph-mon[32951]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:38:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:42 smithi137 ceph-mon[32951]: Standby manager daemon y restarted 2024-06-14T22:38:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:42 smithi137 ceph-mon[32951]: Standby manager daemon y started 2024-06-14T22:38:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:42 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-14T22:38:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:42 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:38:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:42 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-14T22:38:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:42 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:38:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:42 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/110307250' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:43.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:43 smithi086 ceph-mon[29889]: pgmap v33: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-06-14T22:38:43.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:43 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:43.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:43 smithi086 ceph-mon[29889]: mgrmap e27: x(active, since 60s), standbys: y 2024-06-14T22:38:43.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:43 smithi086 ceph-mon[38904]: pgmap v33: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-06-14T22:38:43.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:43 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:43 smithi086 ceph-mon[38904]: mgrmap e27: x(active, since 60s), standbys: y 2024-06-14T22:38:43.582 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 22:38:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T22:38:43.163Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=7 2024-06-14T22:38:43.739 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:43.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:38:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:43 smithi137 ceph-mon[32951]: pgmap v33: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-06-14T22:38:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:43 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:43 smithi137 ceph-mon[32951]: mgrmap e27: x(active, since 60s), standbys: y 2024-06-14T22:38:44.736 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:38:44] ENGINE Bus STOPPING 2024-06-14T22:38:44.736 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:38:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T22:38:44.736 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:38:44] ENGINE Bus STOPPED 2024-06-14T22:38:44.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:44.362+0000 7f8c985a8640 -1 mgr handle_mgr_map I was active but no longer am 2024-06-14T22:38:44.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ignoring --setuser ceph since I am not root 2024-06-14T22:38:44.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ignoring --setgroup ceph since I am not root 2024-06-14T22:38:44.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:44 smithi137 ceph-mgr[68603]: -- 172.21.15.137:0/218089682 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x558f24bc91e0 con 0x558f24b81000 2024-06-14T22:38:44.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:44.600+0000 7fed7cfe3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:38:45.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:38:44] ENGINE Bus STARTING 2024-06-14T22:38:45.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.953+0000 7fe34a0d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.953+0000 7fe34a0d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.953+0000 7fe34a0d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.953+0000 7fe34a0d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.953+0000 7fe34a0d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.961+0000 7fe3428c9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.961+0000 7fe3428c9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.961+0000 7fe3428c9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.962+0000 7fe3428c9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:44.962+0000 7fe3428c9640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:38:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:38:44] ENGINE Serving on http://:::9283 2024-06-14T22:38:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:38:44] ENGINE Bus STARTED 2024-06-14T22:38:45.099 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:44.842+0000 7fed7cfe3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:38:45.099 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:44.942+0000 7fed7cfe3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:38:45.441 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:45.333+0000 7fed7cfe3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:38:45.442 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:45.442 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:45.442 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:38:45.442 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:38:45.442 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:45.442 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:45.442 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-14T22:38:45.442 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-14T22:38:45.442 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: Activating manager daemon y 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: osdmap e76: 8 total, 8 up, 8 in 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.24574 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: mgrmap e28: y(active, starting, since 0.0142627s) 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:38:45.443 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/283423961' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: Manager daemon y is now available 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:38:45.444 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:38:45.445 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:38:45.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:45.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: Activating manager daemon y 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: osdmap e76: 8 total, 8 up, 8 in 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.24574 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: mgrmap e28: y(active, starting, since 0.0142627s) 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:38:45.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:38:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.24574 172.21.15.137:0/1717001832' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: Activating manager daemon y 2024-06-14T22:38:45.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: osdmap e76: 8 total, 8 up, 8 in 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.24574 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: mgrmap e28: y(active, starting, since 0.0142627s) 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:38:45.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/283423961' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: Manager daemon y is now available 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:38:45.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:38:45.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/283423961' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:45.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: Manager daemon y is now available 2024-06-14T22:38:45.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:38:45.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:38:45.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:38:45.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:38:45.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:38:45.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:45 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:38:45.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:45.444+0000 7fed7cfe3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:38:46.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:45.947+0000 7fed7cfe3040 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:38:46.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:45.948+0000 7fed7cfe3040 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:38:46.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:38:46.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: from .module import Module 2024-06-14T22:38:46.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:38:46.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:38:46.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:38:46.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:46.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:45.948+0000 7fed7cfe3040 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:38:46.241 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:45.948+0000 7fed7cfe3040 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:38:46.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:46 smithi137 ceph-mon[32951]: mgrmap e29: y(active, since 1.01362s) 2024-06-14T22:38:46.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:46 smithi137 ceph-mon[32951]: pgmap v3: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail 2024-06-14T22:38:46.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-mon[29889]: mgrmap e29: y(active, since 1.01362s) 2024-06-14T22:38:46.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-mon[29889]: pgmap v3: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail 2024-06-14T22:38:46.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-mon[38904]: mgrmap e29: y(active, since 1.01362s) 2024-06-14T22:38:46.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-mon[38904]: pgmap v3: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:46.405+0000 7fe32a899640 -1 log_channel(cephadm) log [ERR] : [14/Jun/2024:22:38:46] ENGINE Error in HTTPServer.serve 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: Traceback (most recent call last): 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: self._connections.run(self.expiration_interval) 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: self._run(expiration_interval) 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: s = self.context.wrap_socket( 2024-06-14T22:38:46.834 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:38:46.835 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: return self.sslsocket_class._create( 2024-06-14T22:38:46.835 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:38:46.835 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: self.do_handshake() 2024-06-14T22:38:46.835 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:38:46.835 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: self._sslobj.do_handshake() 2024-06-14T22:38:46.835 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:38:46.835 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:38:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:38:47.379 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:47.095+0000 7fed7cfe3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:38:47.379 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:47.304+0000 7fed7cfe3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:38:47.383 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-mon[32951]: [14/Jun/2024:22:38:46] ENGINE Bus STARTING 2024-06-14T22:38:47.383 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-mon[32951]: [14/Jun/2024:22:38:46] ENGINE Serving on http://172.21.15.86:8765 2024-06-14T22:38:47.383 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-mon[32951]: mgrmap e30: y(active, since 2s) 2024-06-14T22:38:47.383 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-mon[32951]: pgmap v4: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail 2024-06-14T22:38:47.383 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-mon[32951]: [14/Jun/2024:22:38:46] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:38:47.383 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-mon[32951]: [14/Jun/2024:22:38:46] ENGINE Bus STARTED 2024-06-14T22:38:47.682 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[29889]: [14/Jun/2024:22:38:46] ENGINE Bus STARTING 2024-06-14T22:38:47.682 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[29889]: [14/Jun/2024:22:38:46] ENGINE Serving on http://172.21.15.86:8765 2024-06-14T22:38:47.682 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[29889]: mgrmap e30: y(active, since 2s) 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[29889]: pgmap v4: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[29889]: [14/Jun/2024:22:38:46] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[29889]: [14/Jun/2024:22:38:46] ENGINE Bus STARTED 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[29889]: [14/Jun/2024:22:38:46] ENGINE Error in HTTPServer.serve 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:38:47.683 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.a.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.a.smithi086.stdout: self.do_handshake() 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.a.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1918537647' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[38904]: [14/Jun/2024:22:38:46] ENGINE Bus STARTING 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[38904]: [14/Jun/2024:22:38:46] ENGINE Serving on http://172.21.15.86:8765 2024-06-14T22:38:47.684 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[38904]: mgrmap e30: y(active, since 2s) 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[38904]: pgmap v4: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[38904]: [14/Jun/2024:22:38:46] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[38904]: [14/Jun/2024:22:38:46] ENGINE Bus STARTED 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[38904]: [14/Jun/2024:22:38:46] ENGINE Error in HTTPServer.serve 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:38:47.685 INFO:journalctl@ceph.mon.c.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:38:47.686 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:38:47.686 INFO:journalctl@ceph.mon.c.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:38:47.686 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:38:47.686 INFO:journalctl@ceph.mon.c.smithi086.stdout: self.do_handshake() 2024-06-14T22:38:47.686 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:38:47.686 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:38:47.686 INFO:journalctl@ceph.mon.c.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:38:47.686 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:47 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1918537647' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:47.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-mon[32951]: [14/Jun/2024:22:38:46] ENGINE Error in HTTPServer.serve 2024-06-14T22:38:47.739 INFO:journalctl@ceph.mon.b.smithi137.stdout: Traceback (most recent call last): 2024-06-14T22:38:47.739 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._run(expiration_interval) 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: s = self.context.wrap_socket( 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: return self.sslsocket_class._create( 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: self.do_handshake() 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._sslobj.do_handshake() 2024-06-14T22:38:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:38:47.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1918537647' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:47.741 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:47.413+0000 7fed7cfe3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:38:48.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:47.802+0000 7fed7cfe3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:38:48.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:47.913+0000 7fed7cfe3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:38:48.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:48.033+0000 7fed7cfe3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:38:48.583 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:48.294+0000 7fed7cfe3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:38:48.989 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:48.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi137:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:38:49.369 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:49.031+0000 7fed7cfe3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:38:49.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:49 smithi086 ceph-mon[29889]: pgmap v5: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail 2024-06-14T22:38:49.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:49 smithi086 ceph-mon[29889]: mgrmap e31: y(active, since 4s) 2024-06-14T22:38:49.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:49 smithi086 ceph-mon[38904]: pgmap v5: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail 2024-06-14T22:38:49.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:49 smithi086 ceph-mon[38904]: mgrmap e31: y(active, since 4s) 2024-06-14T22:38:49.737 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:49.618+0000 7fed7cfe3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:38:49.737 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:49 smithi137 ceph-mon[32951]: pgmap v5: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail 2024-06-14T22:38:49.737 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:49 smithi137 ceph-mon[32951]: mgrmap e31: y(active, since 4s) 2024-06-14T22:38:49.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:49.738+0000 7fed7cfe3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:38:50.396 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:50.023+0000 7fed7cfe3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:38:50.396 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:50 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/19720025' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:50.396 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:50 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:50.396 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:50 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:50.679 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:50.575+0000 7fed7cfe3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:38:50.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:50 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/19720025' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:50.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:50 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:50.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:50 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:50.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:50 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/19720025' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:50.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:50 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:50.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:50 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:50.970 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:50.681+0000 7fed7cfe3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:38:51.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:51.008+0000 7fed7cfe3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:38:51.525 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:51.259+0000 7fed7cfe3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-14T22:38:51.525 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:51.377+0000 7fed7cfe3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:38:51.525 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:51.527+0000 7fed7cfe3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:38:51.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:51.639+0000 7fed7cfe3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:38:52.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[29889]: pgmap v6: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 331 op/s 2024-06-14T22:38:52.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3218948123' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:52.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[38904]: pgmap v6: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 331 op/s 2024-06-14T22:38:52.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:52 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3218948123' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:52.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-mon[32951]: pgmap v6: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 331 op/s 2024-06-14T22:38:52.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:52.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3218948123' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:52.918 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:52.634+0000 7fed7cfe3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:38:52.919 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:52.920+0000 7fed7cfe3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:38:53.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:52.920+0000 7fed7cfe3040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:38:53.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:38:52] ENGINE Bus STARTING 2024-06-14T22:38:53.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: CherryPy Checker: 2024-06-14T22:38:53.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: The Application mounted at '' has an empty config. 2024-06-14T22:38:53.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: 2024-06-14T22:38:53.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:38:53] ENGINE Serving on http://:::9283 2024-06-14T22:38:53.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: [14/Jun/2024:22:38:53] ENGINE Bus STARTED 2024-06-14T22:38:53.558 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[29889]: pgmap v7: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 542 op/s 2024-06-14T22:38:53.558 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:53.558 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:53.558 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[29889]: Standby manager daemon x started 2024-06-14T22:38:53.558 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:38:53.558 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:38:53.558 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:38:53.558 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:38:53.559 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[38904]: pgmap v7: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 542 op/s 2024-06-14T22:38:53.559 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:53.559 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:53.559 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[38904]: Standby manager daemon x started 2024-06-14T22:38:53.559 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:38:53.559 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:38:53.559 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:38:53.559 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:53 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:38:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-mon[32951]: pgmap v7: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 542 op/s 2024-06-14T22:38:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-mon[32951]: Standby manager daemon x started 2024-06-14T22:38:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:38:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:38:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:38:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/3510221885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:38:53.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:38:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:38:53.508Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:38:54.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:54 smithi137 ceph-mon[32951]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:38:54.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:54 smithi137 ceph-mon[32951]: mgrmap e32: y(active, since 9s), standbys: x 2024-06-14T22:38:54.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:54 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:38:54.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:54 smithi086 ceph-mon[29889]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:38:54.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:54 smithi086 ceph-mon[29889]: mgrmap e32: y(active, since 9s), standbys: x 2024-06-14T22:38:54.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:54 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:38:54.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:54 smithi086 ceph-mon[38904]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:38:54.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:54 smithi086 ceph-mon[38904]: mgrmap e32: y(active, since 9s), standbys: x 2024-06-14T22:38:54.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:54 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:38:55.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:55 smithi137 ceph-mon[32951]: pgmap v8: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 421 op/s 2024-06-14T22:38:55.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:55 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/87706332' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:55.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:55 smithi086 ceph-mon[29889]: pgmap v8: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 421 op/s 2024-06-14T22:38:55.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:55 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/87706332' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:55.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:55 smithi086 ceph-mon[38904]: pgmap v8: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 421 op/s 2024-06-14T22:38:55.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:55 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/87706332' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:57.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:56 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:57.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:56 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:57.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:56 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:38:57.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:56 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:38:57.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:56 smithi137 ceph-mon[32951]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:38:57.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:56 smithi137 ceph-mon[32951]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:38:57.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:57.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:57.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:38:57.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:38:57.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[29889]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:38:57.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[29889]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:38:57.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:57.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:57.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:38:57.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:38:57.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[38904]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:38:57.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:56 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: pgmap v9: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 723 op/s 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2131491739' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: pgmap v9: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 723 op/s 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2131491739' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:58 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: pgmap v9: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 723 op/s 2024-06-14T22:38:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:38:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:38:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:38:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:38:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:38:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2131491739' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:38:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:58.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:58 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:38:59.078 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:59 smithi137 ceph-mon[32951]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:38:59.078 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:59 smithi137 ceph-mon[32951]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-14T22:38:59.078 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:38:59 smithi137 ceph-mon[32951]: Reconfiguring daemon prometheus.a on smithi137 2024-06-14T22:38:59.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:59 smithi086 ceph-mon[29889]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:38:59.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:59 smithi086 ceph-mon[29889]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-14T22:38:59.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:38:59 smithi086 ceph-mon[29889]: Reconfiguring daemon prometheus.a on smithi137 2024-06-14T22:38:59.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:59 smithi086 ceph-mon[38904]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:38:59.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:59 smithi086 ceph-mon[38904]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-14T22:38:59.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:38:59 smithi086 ceph-mon[38904]: Reconfiguring daemon prometheus.a on smithi137 2024-06-14T22:38:59.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:38:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[68581]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:38:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-06-14T22:39:00.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:00 smithi137 ceph-mon[32951]: pgmap v10: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 662 op/s 2024-06-14T22:39:00.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:00 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:00.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:00 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2349228928' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:00.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:00 smithi086 ceph-mon[29889]: pgmap v10: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 662 op/s 2024-06-14T22:39:00.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:00 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:00.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:00 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2349228928' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:00.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:00 smithi086 ceph-mon[38904]: pgmap v10: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 662 op/s 2024-06-14T22:39:00.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:00 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:00.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:00 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2349228928' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:02.126 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:02 smithi137 ceph-mon[32951]: pgmap v11: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 796 op/s 2024-06-14T22:39:02.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:02 smithi086 ceph-mon[29889]: pgmap v11: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 796 op/s 2024-06-14T22:39:02.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:02 smithi086 ceph-mon[38904]: pgmap v11: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 796 op/s 2024-06-14T22:39:03.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 systemd[1]: Stopping Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:39:03.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2088250915' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:03.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:03 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2088250915' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:03.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:03 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2088250915' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:03.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.472Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-14T22:39:03.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.472Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-06-14T22:39:03.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.472Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-06-14T22:39:03.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.472Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-14T22:39:03.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.472Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-06-14T22:39:03.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.472Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-06-14T22:39:03.741 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.472Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-06-14T22:39:03.741 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.472Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-06-14T22:39:03.741 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.473Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-06-14T22:39:03.741 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.473Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-06-14T22:39:03.741 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.473Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-06-14T22:39:03.741 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[77068]: ts=2024-06-14T22:39:03.473Z caller=main.go:1147 level=info msg="See you next time!" 2024-06-14T22:39:03.741 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 podman[80135]: 2024-06-14 22:39:03.486107284 +0000 UTC m=+0.167808276 container died 93e3e52e4164a7c7925c095843b3bbf823aa9662204b2711be0d3b9b0fe041dc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:39:04.172 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:04 smithi137 ceph-mon[32951]: pgmap v12: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 879 op/s 2024-06-14T22:39:04.172 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 podman[80135]: 2024-06-14 22:39:03.898863455 +0000 UTC m=+0.580564445 container cleanup 93e3e52e4164a7c7925c095843b3bbf823aa9662204b2711be0d3b9b0fe041dc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:39:04.172 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:03 smithi137 bash[80135]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a 2024-06-14T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:04 smithi086 ceph-mon[38904]: pgmap v12: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 879 op/s 2024-06-14T22:39:04.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:04 smithi086 ceph-mon[29889]: pgmap v12: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 879 op/s 2024-06-14T22:39:04.436 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:04 smithi137 podman[80149]: 2024-06-14 22:39:04.175820617 +0000 UTC m=+0.684772914 container remove 93e3e52e4164a7c7925c095843b3bbf823aa9662204b2711be0d3b9b0fe041dc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:39:04.739 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:04 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-06-14T22:39:04.739 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:04 smithi137 systemd[1]: Stopped Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:39:04.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:04 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@prometheus.a.service: Consumed 1.702s CPU time. 2024-06-14T22:39:04.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:04 smithi137 systemd[1]: Starting Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:39:05.085 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:04 smithi137 podman[80239]: 2024-06-14 22:39:04.691436201 +0000 UTC m=+0.031654753 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-06-14T22:39:05.085 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:04 smithi137 podman[80239]: 2024-06-14 22:39:04.818692378 +0000 UTC m=+0.158910930 container create 1604035f49dfb23dfcea6ed42b510e46166e35aaf5accf39bea89b2ca6c06089 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:39:05.490 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 podman[80239]: 2024-06-14 22:39:05.089236835 +0000 UTC m=+0.429455400 container init 1604035f49dfb23dfcea6ed42b510e46166e35aaf5accf39bea89b2ca6c06089 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:39:05.490 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 podman[80239]: 2024-06-14 22:39:05.098355455 +0000 UTC m=+0.438574011 container start 1604035f49dfb23dfcea6ed42b510e46166e35aaf5accf39bea89b2ca6c06089 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.161Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.161Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.162Z caller=main.go:570 level=info host_details="(Linux 5.14.0-457.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024 x86_64 smithi137 (none))" 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.162Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.162Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.165Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.166Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.168Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.169Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.170Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.170Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.181µs 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.170Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-14T22:39:05.491 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.173Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.210Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.226Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.226Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.226Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=37.198µs wal_replay_duration=55.656805ms wbl_replay_duration=126ns total_replay_duration=55.778567ms 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.231Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.231Z caller=main.go:1029 level=info msg="TSDB started" 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.231Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.260Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=29.220768ms db_storage=1.126µs remote_storage=1.456µs web_handler=432ns query_engine=945ns scrape=544.093µs scrape_sd=132.566µs notify=17.314µs notify_sd=14.45µs rules=27.994582ms tracing=6.003µs 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.260Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-06-14T22:39:05.492 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:05.260Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-06-14T22:39:06.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 bash[80239]: 1604035f49dfb23dfcea6ed42b510e46166e35aaf5accf39bea89b2ca6c06089 2024-06-14T22:39:06.240 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:05 smithi137 systemd[1]: Started Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:39:06.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:06 smithi137 ceph-mon[32951]: pgmap v13: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-06-14T22:39:06.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:06 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4202532002' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:06.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:06 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:06.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:06 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:06.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:06 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:39:06.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:06 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[29889]: pgmap v13: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/4202532002' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[38904]: pgmap v13: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 700 op/s 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4202532002' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:06.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:06.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:39:06.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:06 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:07.122 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:07 smithi137 ceph-mon[32951]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:39:07.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:07 smithi086 ceph-mon[29889]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:39:07.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:07 smithi086 ceph-mon[38904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-14T22:39:08.112 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:08 smithi137 ceph-mon[32951]: pgmap v14: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-14T22:39:08.112 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:08 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1956669216' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:08.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:08 smithi086 ceph-mon[29889]: pgmap v14: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-14T22:39:08.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:08 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1956669216' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:08.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:08 smithi086 ceph-mon[38904]: pgmap v14: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-14T22:39:08.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:08 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1956669216' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:09 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:39:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:09 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:39:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:09 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:39:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:09 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:09.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:09 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:39:09.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:09 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:39:09.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:09 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:39:09.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:09 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:09.341 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:09 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:39:09.341 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:09 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:39:09.341 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:09 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:39:09.341 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:09 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:10.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:10 smithi086 ceph-mon[29889]: Upgrade: Updating mgr.x 2024-06-14T22:39:10.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:10 smithi086 ceph-mon[29889]: Deploying daemon mgr.x on smithi137 2024-06-14T22:39:10.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:10 smithi086 ceph-mon[29889]: pgmap v15: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-06-14T22:39:10.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:10 smithi086 ceph-mon[38904]: Upgrade: Updating mgr.x 2024-06-14T22:39:10.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:10 smithi086 ceph-mon[38904]: Deploying daemon mgr.x on smithi137 2024-06-14T22:39:10.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:10 smithi086 ceph-mon[38904]: pgmap v15: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-06-14T22:39:10.445 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:10 smithi137 ceph-mon[32951]: Upgrade: Updating mgr.x 2024-06-14T22:39:10.445 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:10 smithi137 ceph-mon[32951]: Deploying daemon mgr.x on smithi137 2024-06-14T22:39:10.445 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:10 smithi137 ceph-mon[32951]: pgmap v15: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-06-14T22:39:10.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:10 smithi137 systemd[1]: Stopping Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:39:11.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:11 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2274204589' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:11.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:10 smithi137 podman[80630]: 2024-06-14 22:39:10.960659026 +0000 UTC m=+0.205503984 container died 41740301d000565aba71a1d956d0de19d0d8d0df5bafc6c239ccec4c1b1c62be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:39:11.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:11 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2274204589' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:11.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:11 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2274204589' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:11.637 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:11 smithi137 podman[80630]: 2024-06-14 22:39:11.363842045 +0000 UTC m=+0.608687005 container cleanup 41740301d000565aba71a1d956d0de19d0d8d0df5bafc6c239ccec4c1b1c62be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=) 2024-06-14T22:39:11.637 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:11 smithi137 bash[80630]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x 2024-06-14T22:39:11.905 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:11 smithi137 podman[80646]: 2024-06-14 22:39:11.640867242 +0000 UTC m=+0.677001941 container remove 41740301d000565aba71a1d956d0de19d0d8d0df5bafc6c239ccec4c1b1c62be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-14T22:39:11.905 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:11 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-06-14T22:39:12.222 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:12 smithi137 ceph-mon[32951]: pgmap v16: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 744 op/s 2024-06-14T22:39:12.222 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:11 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-06-14T22:39:12.222 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:11 smithi137 systemd[1]: Stopped Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:39:12.222 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:11 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.x.service: Consumed 25.470s CPU time. 2024-06-14T22:39:12.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:12 smithi086 ceph-mon[29889]: pgmap v16: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 744 op/s 2024-06-14T22:39:12.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:12 smithi086 ceph-mon[38904]: pgmap v16: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 744 op/s 2024-06-14T22:39:12.489 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:12 smithi137 systemd[1]: Starting Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:39:12.908 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:12 smithi137 podman[80764]: 2024-06-14 22:39:12.47774573 +0000 UTC m=+0.044473661 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:39:12.908 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:12 smithi137 podman[80764]: 2024-06-14 22:39:12.633802349 +0000 UTC m=+0.200530253 container create 086e4706077c6f7bf553ce7380ebdf262b12e5dc3233dcb5bedb3e748149cfb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-14T22:39:12.908 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:12 smithi137 podman[80764]: 2024-06-14 22:39:12.91224995 +0000 UTC m=+0.478977877 container init 086e4706077c6f7bf553ce7380ebdf262b12e5dc3233dcb5bedb3e748149cfb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=) 2024-06-14T22:39:13.174 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:13 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2189662363' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:13.174 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:12 smithi137 podman[80764]: 2024-06-14 22:39:12.927354504 +0000 UTC m=+0.494082419 container start 086e4706077c6f7bf553ce7380ebdf262b12e5dc3233dcb5bedb3e748149cfb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-14T22:39:13.175 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:13 smithi137 bash[80764]: 086e4706077c6f7bf553ce7380ebdf262b12e5dc3233dcb5bedb3e748149cfb1 2024-06-14T22:39:13.175 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:13 smithi137 systemd[1]: Started Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:39:13.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:13 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2189662363' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:13.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:13 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2189662363' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:13.443 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:13.176+0000 7fc208629040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:39:13.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:13.445+0000 7fc208629040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:39:13.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:13.568+0000 7fc208629040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:39:13.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:13.504Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:39:14.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:14.001+0000 7fc208629040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:39:14.490 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:14.106+0000 7fc208629040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:39:14.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-mon[32951]: pgmap v17: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T22:39:14.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:14.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[29889]: pgmap v17: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[38904]: pgmap v17: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:14.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:14.605+0000 7fc208629040 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:14.605+0000 7fc208629040 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: from .module import Module 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:14.606+0000 7fc208629040 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:39:14.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:14 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:14.606+0000 7fc208629040 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:39:15.126 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:39:14] "GET /metrics HTTP/1.1" 200 36223 "" "Prometheus/2.43.0" 2024-06-14T22:39:15.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:15 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:15.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:15 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:15.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:15 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:16.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:15.781+0000 7fc208629040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:39:16.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:15 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:15.995+0000 7fc208629040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:39:16.101 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:16 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:16.102+0000 7fc208629040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:39:16.392 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:16 smithi137 ceph-mon[32951]: pgmap v18: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-06-14T22:39:16.392 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:16 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1242985338' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:16.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:16 smithi086 ceph-mon[29889]: pgmap v18: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-06-14T22:39:16.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:16 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1242985338' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:16.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:16 smithi086 ceph-mon[38904]: pgmap v18: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-06-14T22:39:16.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:16 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1242985338' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:16.721 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:16 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:16.497+0000 7fc208629040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:39:16.722 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:16 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:16.606+0000 7fc208629040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:39:16.991 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:16 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:16.723+0000 7fc208629040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:39:17.306 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:17 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:17.000+0000 7fc208629040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:39:17.989 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:17 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:17.766+0000 7fc208629040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:39:18.384 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:18 smithi137 ceph-mon[32951]: pgmap v19: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:39:18.384 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:18 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1125359741' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:18.385 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:18 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:18.346+0000 7fc208629040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:39:18.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:18 smithi086 ceph-mon[38904]: pgmap v19: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:39:18.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:18 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1125359741' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:18.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:18 smithi086 ceph-mon[29889]: pgmap v19: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:39:18.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:18 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1125359741' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:18.656 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:18 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:18.466+0000 7fc208629040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:39:18.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:18 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:18.739+0000 7fc208629040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:39:18.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:18 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:18.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:39:19.742 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:19 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:19.305+0000 7fc208629040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:39:19.742 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:19 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:19.419+0000 7fc208629040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:39:20.013 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:19 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:19.748+0000 7fc208629040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:39:20.290 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:20 smithi137 ceph-mon[32951]: pgmap v20: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 685 op/s 2024-06-14T22:39:20.291 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:20 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.291 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:20 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.291 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:20 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.291 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:20 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.291 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:20 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:20.016+0000 7fc208629040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-14T22:39:20.291 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:20 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:20.134+0000 7fc208629040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:39:20.291 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:20 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:20.291+0000 7fc208629040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:39:20.330 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[38904]: pgmap v20: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 685 op/s 2024-06-14T22:39:20.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[29889]: pgmap v20: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 685 op/s 2024-06-14T22:39:20.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:20 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:20.616 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:20 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:20.407+0000 7fc208629040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:39:21.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:21 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/1595153492' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:21.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:21 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1595153492' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:21.368 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1595153492' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:21.656 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:21.371+0000 7fc208629040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:39:21.656 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:21.659+0000 7fc208629040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:39:21.656 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:21.659+0000 7fc208629040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:39:21.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: [14/Jun/2024:22:39:21] ENGINE Bus STARTING 2024-06-14T22:39:21.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: CherryPy Checker: 2024-06-14T22:39:21.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: The Application mounted at '' has an empty config. 2024-06-14T22:39:21.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:21.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: [14/Jun/2024:22:39:21] ENGINE Serving on http://:::9283 2024-06-14T22:39:21.990 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:21 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: [14/Jun/2024:22:39:21] ENGINE Bus STARTED 2024-06-14T22:39:22.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[29889]: pgmap v21: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 705 op/s 2024-06-14T22:39:22.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[29889]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:39:22.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[29889]: Standby manager daemon x restarted 2024-06-14T22:39:22.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[29889]: Standby manager daemon x started 2024-06-14T22:39:22.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:39:22.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:39:22.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:39:22.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[29889]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:39:22.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[38904]: pgmap v21: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 705 op/s 2024-06-14T22:39:22.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[38904]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:39:22.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[38904]: Standby manager daemon x restarted 2024-06-14T22:39:22.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[38904]: Standby manager daemon x started 2024-06-14T22:39:22.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:39:22.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:39:22.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:39:22.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:22 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:39:22.400 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:22 smithi137 ceph-mon[32951]: pgmap v21: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 705 op/s 2024-06-14T22:39:22.400 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:22 smithi137 ceph-mon[32951]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:39:22.400 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:22 smithi137 ceph-mon[32951]: Standby manager daemon x restarted 2024-06-14T22:39:22.400 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:22 smithi137 ceph-mon[32951]: Standby manager daemon x started 2024-06-14T22:39:22.400 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:22 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:39:22.400 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:22 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:39:22.400 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:22 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:39:22.400 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:22 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/3707792507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:39:23.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:23 smithi086 ceph-mon[29889]: mgrmap e33: y(active, since 37s), standbys: x 2024-06-14T22:39:23.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:23 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3009693081' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:23.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:23 smithi086 ceph-mon[38904]: mgrmap e33: y(active, since 37s), standbys: x 2024-06-14T22:39:23.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:23 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3009693081' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:23.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:23 smithi137 ceph-mon[32951]: mgrmap e33: y(active, since 37s), standbys: x 2024-06-14T22:39:23.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:23 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3009693081' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:23.989 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:23.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:39:24.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:24 smithi086 ceph-mon[29889]: pgmap v22: 89 pgs: 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 973 op/s 2024-06-14T22:39:24.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:24 smithi086 ceph-mon[38904]: pgmap v22: 89 pgs: 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 973 op/s 2024-06-14T22:39:24.403 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:24 smithi137 ceph-mon[32951]: pgmap v22: 89 pgs: 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 973 op/s 2024-06-14T22:39:25.232 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:39:24] "GET /metrics HTTP/1.1" 200 36223 "" "Prometheus/2.43.0" 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: pgmap v23: 89 pgs: 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 648 op/s 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: Upgrade: Setting container_image for all mgr 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:25.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-14T22:39:25.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-14T22:39:25.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-06-14T22:39:25.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-14T22:39:25.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-14T22:39:25.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-06-14T22:39:25.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-14T22:39:25.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-06-14T22:39:25.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: Upgrade: It appears safe to stop mon.a 2024-06-14T22:39:25.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3096101463' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: pgmap v23: 89 pgs: 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 648 op/s 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: Upgrade: Setting container_image for all mgr 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-14T22:39:25.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-14T22:39:25.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-06-14T22:39:25.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-14T22:39:25.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-06-14T22:39:25.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: Upgrade: It appears safe to stop mon.a 2024-06-14T22:39:25.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:25 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/3096101463' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:26.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: pgmap v23: 89 pgs: 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 648 op/s 2024-06-14T22:39:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:39:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:39:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: Upgrade: Setting container_image for all mgr 2024-06-14T22:39:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: Upgrade: It appears safe to stop mon.a 2024-06-14T22:39:26.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:25 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3096101463' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[38904]: pgmap v24: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1k op/s 2024-06-14T22:39:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:39:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:39:28.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:28.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[29889]: pgmap v24: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1k op/s 2024-06-14T22:39:28.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[29889]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:28.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:39:28.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:39:28.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:27 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:28.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:27 smithi137 ceph-mon[32951]: pgmap v24: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1k op/s 2024-06-14T22:39:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:27 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='mgr.y' 2024-06-14T22:39:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:27 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:39:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:27 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:39:28.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:27 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:28.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:28 smithi137 ceph-mon[32951]: Upgrade: Updating mon.a 2024-06-14T22:39:28.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:28 smithi137 ceph-mon[32951]: Deploying daemon mon.a on smithi086 2024-06-14T22:39:28.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:28 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2696426249' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:28.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:28 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:28.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:39:29.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:28 smithi086 ceph-mon[29889]: Upgrade: Updating mon.a 2024-06-14T22:39:29.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:28 smithi086 ceph-mon[29889]: Deploying daemon mon.a on smithi086 2024-06-14T22:39:29.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:28 smithi086 ceph-mon[29889]: from='client.? 172.21.15.86:0/2696426249' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:29.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:28 smithi086 ceph-mon[38904]: Upgrade: Updating mon.a 2024-06-14T22:39:29.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:28 smithi086 ceph-mon[38904]: Deploying daemon mon.a on smithi086 2024-06-14T22:39:29.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:28 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2696426249' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:30.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:29 smithi086 ceph-mon[29889]: pgmap v25: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-06-14T22:39:30.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:29 smithi086 ceph-mon[29889]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:30.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:29 smithi086 ceph-mon[38904]: pgmap v25: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-06-14T22:39:30.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:29 smithi086 ceph-mon[38904]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:30.239 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:29 smithi137 ceph-mon[32951]: pgmap v25: 89 pgs: 89 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-06-14T22:39:30.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:29 smithi137 ceph-mon[32951]: from='mgr.15084 172.21.15.86:0/2204318876' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:30.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:30 smithi086 systemd[1]: Stopping Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:39:31.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:30 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[29866]: 2024-06-14T22:39:30.638+0000 7f96079d8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T22:39:31.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:30 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[29866]: 2024-06-14T22:39:30.638+0000 7f96079d8640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-06-14T22:39:31.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:30 smithi086 podman[97720]: 2024-06-14 22:39:30.846313586 +0000 UTC m=+0.377536118 container died 00d4288edc6f73fc20c932a25a48c22be87315f4bb97313ae8e9b3b0faeac61d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0) 2024-06-14T22:39:31.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:31 smithi086 podman[97720]: 2024-06-14 22:39:31.324466869 +0000 UTC m=+0.855689404 container cleanup 00d4288edc6f73fc20c932a25a48c22be87315f4bb97313ae8e9b3b0faeac61d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-14T22:39:31.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:31 smithi086 bash[97720]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a 2024-06-14T22:39:32.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:31 smithi086 podman[97740]: 2024-06-14 22:39:31.659737429 +0000 UTC m=+0.803456085 container remove 00d4288edc6f73fc20c932a25a48c22be87315f4bb97313ae8e9b3b0faeac61d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-14T22:39:32.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:31 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-14T22:39:32.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:31 smithi086 systemd[1]: Stopped Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:39:32.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:31 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a.service: Consumed 12.877s CPU time. 2024-06-14T22:39:32.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 systemd[1]: Starting Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:39:32.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 podman[97857]: 2024-06-14 22:39:32.449276143 +0000 UTC m=+0.047003983 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:39:32.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 podman[97857]: 2024-06-14 22:39:32.585742089 +0000 UTC m=+0.183469924 container create 0fe78616d3a277d695dd048c54f444bcc0f0ca52558a870e0dccbb4a29247d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:39:33.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 podman[97857]: 2024-06-14 22:39:32.864128414 +0000 UTC m=+0.461856271 container init 0fe78616d3a277d695dd048c54f444bcc0f0ca52558a870e0dccbb4a29247d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 podman[97857]: 2024-06-14 22:39:32.880455225 +0000 UTC m=+0.478183060 container start 0fe78616d3a277d695dd048c54f444bcc0f0ca52558a870e0dccbb4a29247d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, io.buildah.version=1.35.3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True) 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: set uid:gid to 167:167 (ceph:ceph) 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable), process ceph-mon, pid 7 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: pidfile_write: ignore empty --pid-file 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: load: jerasure load: lrc 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: RocksDB version: 7.9.2 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Git sha 0 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Compile date 2024-06-08 16:28:17 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: DB SUMMARY 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: DB Session ID: UWHD6535FWRRU7EKY60A 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: CURRENT file: CURRENT 2024-06-14T22:39:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: IDENTITY file: IDENTITY 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1168 Bytes 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000033.sst 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000031.log size: 58794 ; 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.error_if_exists: 0 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.create_if_missing: 0 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.paranoid_checks: 1 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.env: 0x55c5fa5dbba0 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.fs: PosixFileSystem 2024-06-14T22:39:33.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.info_log: 0x55c5fba96be0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_file_opening_threads: 16 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.statistics: (nil) 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.use_fsync: 0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_log_file_size: 0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.keep_log_file_num: 1000 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.recycle_log_file_num: 0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.allow_fallocate: 1 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.allow_mmap_reads: 0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.allow_mmap_writes: 0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.use_direct_reads: 0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.create_missing_column_families: 0 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.db_log_dir: 2024-06-14T22:39:33.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.wal_dir: 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.advise_random_on_open: 1 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.db_write_buffer_size: 0 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.write_buffer_manager: 0x55c5fba68640 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.rate_limiter: (nil) 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.wal_recovery_mode: 2 2024-06-14T22:39:33.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.enable_thread_tracking: 0 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.enable_pipelined_write: 0 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.unordered_write: 0 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.row_cache: None 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.wal_filter: None 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.allow_ingest_behind: 0 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.two_write_queues: 0 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.manual_wal_flush: 0 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.wal_compression: 0 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.atomic_flush: 0 2024-06-14T22:39:33.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.log_readahead_size: 0 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.best_efforts_recovery: 0 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.allow_data_in_errors: 0 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.db_host_id: __hostname__ 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_background_jobs: 2 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_background_compactions: -1 2024-06-14T22:39:33.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_subcompactions: 1 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_total_wal_size: 0 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_open_files: -1 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bytes_per_sync: 0 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_readahead_size: 0 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_background_flushes: -1 2024-06-14T22:39:33.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Compression algorithms supported: 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: kZSTD supported: 0 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: kXpressCompression supported: 0 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: kBZip2Compression supported: 0 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: kLZ4Compression supported: 1 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: kZlibCompression supported: 1 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: kLZ4HCCompression supported: 1 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: kSnappyCompression supported: 1 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.merge_operator: 2024-06-14T22:39:33.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_filter: None 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_filter_factory: None 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.sst_partitioner_factory: None 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c5fba96820) 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout: index_type: 0 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout: data_block_index_type: 0 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout: index_shortening: 1 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-14T22:39:33.345 INFO:journalctl@ceph.mon.a.smithi086.stdout: checksum: 4 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: no_block_cache: 0 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache: 0x55c5fba8d350 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache_options: 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: capacity : 536870912 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: num_shard_bits : 4 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: strict_capacity_limit : 0 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache_compressed: (nil) 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: persistent_cache: (nil) 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_size: 4096 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_size_deviation: 10 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_restart_interval: 16 2024-06-14T22:39:33.346 INFO:journalctl@ceph.mon.a.smithi086.stdout: index_block_restart_interval: 1 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: metadata_block_size: 4096 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: partition_filters: 0 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: use_delta_encoding: 1 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: filter_policy: bloomfilter 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: whole_key_filtering: 1 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: verify_compression: 0 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: format_version: 5 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: enable_index_compression: 1 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_align: 0 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: max_auto_readahead_size: 262144 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: prepopulate_block_cache: 0 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: initial_auto_readahead_size: 8192 2024-06-14T22:39:33.347 INFO:journalctl@ceph.mon.a.smithi086.stdout: num_file_reads_for_auto_readahead: 2 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.write_buffer_size: 33554432 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_write_buffer_number: 2 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression: NoCompression 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression: Disabled 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.prefix_extractor: nullptr 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.num_levels: 7 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-14T22:39:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression_opts.level: 32767 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression_opts.strategy: 0 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression_opts.enabled: false 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-14T22:39:33.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.target_file_size_base: 67108864 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-14T22:39:33.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.arena_block_size: 1048576 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.disable_auto_compactions: 0 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.table_properties_collectors: 2024-06-14T22:39:33.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.inplace_update_support: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.bloom_locality: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.max_successive_merges: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.paranoid_file_checks: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.force_consistency_checks: 1 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.report_bg_io_stats: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.ttl: 2592000 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-14T22:39:33.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.enable_blob_files: false 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.min_blob_size: 0 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.blob_file_size: 268435456 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.blob_file_starting_level: 0 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 12187, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0a7968b0-5381-4231-8672-4adf5c5c9dde 2024-06-14T22:39:33.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404772928155, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404772929225, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 59229, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12188, "largest_seqno": 12266, "table_properties": {"data_size": 57809, "index_size": 216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 197, "raw_key_size": 1464, "raw_average_key_size": 26, "raw_value_size": 56620, "raw_average_value_size": 1011, "num_data_blocks": 9, "num_entries": 56, "num_filter_entries": 56, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1718404772, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0a7968b0-5381-4231-8672-4adf5c5c9dde", "db_session_id": "UWHD6535FWRRU7EKY60A", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404772931660, "job": 1, "event": "recovery_finished"} 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c5fbab6e00 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: DB pointer 0x55c5fbbaa000 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** DB Stats ** 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-14T22:39:33.354 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: L0 1/0 57.84 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: L6 1/0 7.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: Sum 2/0 7.40 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 70.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-14T22:39:33.355 INFO:journalctl@ceph.mon.a.smithi086.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 70.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative compaction: 0.00 GB write, 5.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval compaction: 0.00 GB write, 5.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: Block cache BinnedLRUCache@0x55c5fba8d350#7 capacity: 512.00 MB usage: 0.53 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.8e-05 secs_since: 0 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.23 KB,4.47035e-05%) IndexBlock(1,0.30 KB,5.66244e-05%) Misc(1,0.00 KB,0%) 2024-06-14T22:39:33.356 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: starting mon.a rank 0 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: mon.a@-1(???) e3 preinit fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: mon.a@-1(???).mds e9 new map 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: mon.a@-1(???).mds e9 print_map 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: e9 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: legacy client fscid: 1 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: Filesystem 'foo' (1) 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: fs_name foo 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: epoch 8 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-06-14T22:39:33.357 INFO:journalctl@ceph.mon.a.smithi086.stdout: created 2024-06-14T22:34:45.808786+0000 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: modified 2024-06-14T22:35:03.607395+0000 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: tableserver 0 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: root 0 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: session_timeout 60 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: session_autoclose 300 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: max_file_size 1099511627776 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: max_xattr_size 65536 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: required_client_features {} 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: last_failure 0 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: last_failure_osd_epoch 0 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: max_mds 1 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: in 0 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: up {0=24548} 2024-06-14T22:39:33.358 INFO:journalctl@ceph.mon.a.smithi086.stdout: failed 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: damaged 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: stopped 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: data_pools [4] 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: metadata_pool 3 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: inline_data disabled 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: balancer 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: bal_rank_mask -1 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: standby_count_wanted 1 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: [mds.foo.smithi086.nhdjdb{0:24548} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] compat {c=[1],r=[1],i=[7ff]}] 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: Standby daemons: 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout: [mds.foo.smithi137.yjevpg{-1:24445} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.137:6832/1266342281,v1:172.21.15.137:6833/1266342281] compat {c=[1],r=[1],i=[7ff]}] 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: mon.a@-1(???).osd e76 crush map has features 3314933000854323200, adjusting msgr requires 2024-06-14T22:39:33.359 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: mon.a@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-06-14T22:39:33.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: mon.a@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-06-14T22:39:33.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: mon.a@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-06-14T22:39:33.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 ceph-mon[97913]: mon.a@-1(???).paxosservice(auth 1..18) refresh upgraded, format 0 -> 3 2024-06-14T22:39:33.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 bash[97857]: 0fe78616d3a277d695dd048c54f444bcc0f0ca52558a870e0dccbb4a29247d7e 2024-06-14T22:39:33.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:32 smithi086 systemd[1]: Started Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:39:33.989 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:33 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:33.507Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:39:34.642 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: pgmap v26: 89 pgs: 89 active+clean; 2.2 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 685 op/s 2024-06-14T22:39:34.642 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3722804776' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:34.642 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: pgmap v27: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-06-14T22:39:34.642 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1343920672' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:34.642 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: mon.a calling monitor election 2024-06-14T22:39:34.642 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:39:34.642 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: monmap e3: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:39:34.642 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:39:34.643 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: osdmap e76: 8 total, 8 up, 8 in 2024-06-14T22:39:34.643 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: mgrmap e33: y(active, since 50s), standbys: x 2024-06-14T22:39:34.643 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: overall HEALTH_OK 2024-06-14T22:39:34.643 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: mgrmap e34: y(active, since 50s), standbys: x 2024-06-14T22:39:34.643 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[38904]: from='mgr.15084 ' entity='' 2024-06-14T22:39:34.643 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ignoring --setuser ceph since I am not root 2024-06-14T22:39:34.643 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ignoring --setgroup ceph since I am not root 2024-06-14T22:39:34.643 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mgr[88817]: -- 172.21.15.86:0/1202311898 <== mon.2 v2:172.21.15.86:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5556ab098d00 con 0x5556ab06d000 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: pgmap v26: 89 pgs: 89 active+clean; 2.2 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 685 op/s 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3722804776' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: pgmap v27: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1343920672' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: mon.a calling monitor election 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: monmap e3: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: osdmap e76: 8 total, 8 up, 8 in 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: mgrmap e33: y(active, since 50s), standbys: x 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: overall HEALTH_OK 2024-06-14T22:39:34.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: mgrmap e34: y(active, since 50s), standbys: x 2024-06-14T22:39:34.645 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-mon[97913]: from='mgr.15084 ' entity='' 2024-06-14T22:39:34.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: ignoring --setuser ceph since I am not root 2024-06-14T22:39:34.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: ignoring --setgroup ceph since I am not root 2024-06-14T22:39:34.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:34.665+0000 7fc030540040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:39:34.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: pgmap v26: 89 pgs: 89 active+clean; 2.2 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 685 op/s 2024-06-14T22:39:34.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3722804776' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:34.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: pgmap v27: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-06-14T22:39:34.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1343920672' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:34.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: mon.a calling monitor election 2024-06-14T22:39:34.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:39:34.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: monmap e3: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:39:34.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:39:34.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: osdmap e76: 8 total, 8 up, 8 in 2024-06-14T22:39:34.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: mgrmap e33: y(active, since 50s), standbys: x 2024-06-14T22:39:34.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: overall HEALTH_OK 2024-06-14T22:39:34.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: mgrmap e34: y(active, since 50s), standbys: x 2024-06-14T22:39:34.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-mon[32951]: from='mgr.15084 ' entity='' 2024-06-14T22:39:34.993 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:34.643+0000 7f494961f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-14T22:39:34.993 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:34.889+0000 7f494961f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:39:35.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:34.927+0000 7fc030540040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:39:35.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:35 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:35.038+0000 7fc030540040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:39:35.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:34.995+0000 7f494961f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-14T22:39:35.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:35 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:35.449+0000 7fc030540040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:39:35.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:35 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:35.554+0000 7fc030540040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:39:35.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:35.383+0000 7f494961f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-14T22:39:35.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:35.484+0000 7f494961f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-14T22:39:35.933 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable)": 1 2024-06-14T22:39:36.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:35 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1783408601' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:36.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:36.009+0000 7fc030540040 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:39:36.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:36.009+0000 7fc030540040 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:39:36.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:39:36.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: from .module import Module 2024-06-14T22:39:36.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:39:36.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:39:36.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:39:36.241 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:36.241 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:36.010+0000 7fc030540040 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:39:36.241 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:36.010+0000 7fc030540040 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:39:36.280 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1783408601' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:36.280 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1783408601' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:39:36.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:35.968+0000 7f494961f040 -1 mgr[py] Module not found: 'k8sevents' 2024-06-14T22:39:36.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:35.969+0000 7f494961f040 -1 mgr[py] Traceback (most recent call last): 2024-06-14T22:39:36.281 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/usr/share/ceph/mgr/k8sevents/__init__.py", line 1, in 2024-06-14T22:39:36.281 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: from .module import Module 2024-06-14T22:39:36.281 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/usr/share/ceph/mgr/k8sevents/module.py", line 70, in 2024-06-14T22:39:36.281 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: from kubernetes.client.models.v1_event import V1Event 2024-06-14T22:39:36.281 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ModuleNotFoundError: No module named 'kubernetes.client.models.v1_event' 2024-06-14T22:39:36.281 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:36.281 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:35.971+0000 7f494961f040 -1 mgr[py] Class not found in module 'k8sevents' 2024-06-14T22:39:36.281 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:35 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:35.971+0000 7f494961f040 -1 mgr[py] Error loading module 'k8sevents': (2) No such file or directory 2024-06-14T22:39:36.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch ps' 2024-06-14T22:39:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:39:37.302 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:37 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:37.005+0000 7f494961f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:39:37.302 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:37 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:37.208+0000 7f494961f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:39:37.302 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:37 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:37.304+0000 7f494961f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:39:37.398 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:37.107+0000 7fc030540040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-14T22:39:37.398 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:37.303+0000 7fc030540040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-14T22:39:37.666 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:37.401+0000 7fc030540040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-14T22:39:37.932 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:37.792+0000 7fc030540040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:39:37.932 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:37.903+0000 7fc030540040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:39:38.073 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:37 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:37.652+0000 7f494961f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-14T22:39:38.073 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:37 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:37.743+0000 7f494961f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-14T22:39:38.073 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:37 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:37.852+0000 7f494961f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:39:38.239 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:38.017+0000 7fc030540040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-14T22:39:38.330 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:38 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:38.075+0000 7f494961f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:39:38.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:38.276+0000 7fc030540040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-14T22:39:38.881 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:38 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:38.707+0000 7f494961f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:39:39.489 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:39.007+0000 7fc030540040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-14T22:39:39.560 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:39.198+0000 7f494961f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:39:39.561 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:39.309+0000 7f494961f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:39:39.561 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:39.560+0000 7f494961f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:39:39.938 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:39.554+0000 7fc030540040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-14T22:39:39.938 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:39.668+0000 7fc030540040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-14T22:39:40.240 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:39.941+0000 7fc030540040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-14T22:39:40.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:40.080+0000 7f494961f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:39:40.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:40.179+0000 7f494961f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:39:40.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:40.480+0000 7fc030540040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-14T22:39:40.740 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:40.583+0000 7fc030540040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-14T22:39:40.803 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:40.468+0000 7f494961f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:39:40.803 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:40.703+0000 7f494961f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-14T22:39:41.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:40.806+0000 7f494961f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:39:41.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:40.942+0000 7f494961f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:39:41.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:41.042+0000 7f494961f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:39:41.156 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:40.909+0000 7fc030540040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-14T22:39:41.408 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:41.159+0000 7fc030540040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-14T22:39:41.408 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:41.268+0000 7fc030540040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-14T22:39:41.408 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:41.410+0000 7fc030540040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-14T22:39:41.739 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:41.512+0000 7fc030540040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-14T22:39:42.189 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:41.925+0000 7f494961f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:39:42.189 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.190+0000 7f494961f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:39:42.189 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.190+0000 7f494961f040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:39:42.456 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: Active manager daemon y restarted 2024-06-14T22:39:42.456 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: Activating manager daemon y 2024-06-14T22:39:42.456 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: osdmap e77: 8 total, 8 up, 8 in 2024-06-14T22:39:42.456 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: mgrmap e35: y(active, starting, since 0.00757397s), standbys: x 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:39:42.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:39:42.458 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:39:42.458 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:39:42.458 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:39:42.458 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[97913]: Manager daemon y is now available 2024-06-14T22:39:42.458 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:39:42] ENGINE Bus STARTING 2024-06-14T22:39:42.458 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: CherryPy Checker: 2024-06-14T22:39:42.458 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: The Application mounted at '' has an empty config. 2024-06-14T22:39:42.458 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.458 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:39:42] ENGINE Serving on http://:::9283 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: Active manager daemon y restarted 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: Activating manager daemon y 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: osdmap e77: 8 total, 8 up, 8 in 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: mgrmap e35: y(active, starting, since 0.00757397s), standbys: x 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:39:42.459 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:39:42.460 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-mon[38904]: Manager daemon y is now available 2024-06-14T22:39:42.671 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:42.434+0000 7fc030540040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:42.673+0000 7fc030540040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: Active manager daemon y restarted 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: Activating manager daemon y 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: osdmap e77: 8 total, 8 up, 8 in 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: mgrmap e35: y(active, starting, since 0.00757397s), standbys: x 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi137.yjevpg"}]: dispatch 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi086.nhdjdb"}]: dispatch 2024-06-14T22:39:42.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-14T22:39:42.673 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-mon[32951]: Manager daemon y is now available 2024-06-14T22:39:42.708 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:39:42] ENGINE Bus STARTED 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.540+0000 7f48b3368640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.540+0000 7f48b3368640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.540+0000 7f48b3368640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.540+0000 7f48b3368640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.540+0000 7f48b3368640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.554+0000 7f48b1364640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.554+0000 7f48b1364640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.554+0000 7f48b1364640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.554+0000 7f48b1364640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.709 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:42.554+0000 7f48b1364640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-14T22:39:42.932 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:42.673+0000 7fc030540040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:39:42.932 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: [14/Jun/2024:22:39:42] ENGINE Bus STARTING 2024-06-14T22:39:42.932 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: CherryPy Checker: 2024-06-14T22:39:42.932 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: The Application mounted at '' has an empty config. 2024-06-14T22:39:42.932 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: 2024-06-14T22:39:42.932 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: [14/Jun/2024:22:39:42] ENGINE Serving on http://:::9283 2024-06-14T22:39:42.932 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 22:39:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x[80777]: [14/Jun/2024:22:39:42] ENGINE Bus STARTED 2024-06-14T22:39:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-14T22:39:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:alertmanager.a smithi086 *:9093,9094 running (97s) 52s ago 5m 14.9M - 0.25.0 c8568f914cd2 1c9917382faa 2024-06-14T22:39:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:grafana.a smithi137 *:3000 running (81s) 24s ago 5m 80.0M - 9.4.7 954c08fa6188 12b0617a16e5 2024-06-14T22:39:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi086.nhdjdb smithi086 running (4m) 52s ago 4m 16.3M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 2bd94d6f47de 2024-06-14T22:39:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi137.yjevpg smithi137 running (4m) 24s ago 4m 13.0M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4151acede02a 2024-06-14T22:39:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.x smithi137 *:8443,9283,8765 running (30s) 24s ago 9m 45.5M - 18.2.2-1443-g6f4b0956 e7282a3bffba 086e4706077c 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.y smithi086 *:8443,9283,8765 running (70s) 52s ago 11m 462M - 18.2.2-1443-g6f4b0956 e7282a3bffba edafb7b2b182 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:mon.a smithi086 running (11m) 52s ago 11m 50.0M 2048M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 00d4288edc6f 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:mon.b smithi137 running (10m) 24s ago 10m 43.3M 2048M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 bd4aed27097c 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:mon.c smithi086 running (10m) 52s ago 10m 47.3M 2048M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 e02d9b886f41 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.a smithi086 *:9100 running (94s) 52s ago 5m 7377k - 1.5.0 0da6a335fe13 d1de50e60879 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.b smithi137 *:9100 running (79s) 24s ago 5m 8858k - 1.5.0 0da6a335fe13 b8c1542255f2 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0 smithi086 running (9m) 52s ago 9m 367M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 129bff24d5cc 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1 smithi086 running (9m) 52s ago 9m 244M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 88baee12d7cb 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2 smithi086 running (8m) 52s ago 8m 306M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 fbf5ce68d6eb 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3 smithi086 running (8m) 52s ago 8m 165M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 bee260f44ae7 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4 smithi137 running (8m) 24s ago 8m 369M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4b2729030303 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5 smithi137 running (7m) 24s ago 7m 404M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 e01c2182297f 2024-06-14T22:39:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6 smithi137 running (7m) 24s ago 7m 207M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 7210ba0a183d 2024-06-14T22:39:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7 smithi137 running (6m) 24s ago 6m 251M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 a9ca11c000f2 2024-06-14T22:39:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:prometheus.a smithi137 *:9095 running (38s) 24s ago 6m 46.0M - 2.43.0 a07b618ecd1d 1604035f49df 2024-06-14T22:39:43.581 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:43.224+0000 7f48e8f21640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: Standby manager daemon x restarted 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: Standby manager daemon x started 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:39:43.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[38904]: mgrmap e36: y(active, since 1.02087s), standbys: x 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: Standby manager daemon x restarted 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: Standby manager daemon x started 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:39:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:39:43.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:43 smithi086 ceph-mon[97913]: mgrmap e36: y(active, since 1.02087s), standbys: x 2024-06-14T22:39:43.739 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:43.504Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: Standby manager daemon x restarted 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: Standby manager daemon x started 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-14T22:39:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-14T22:39:43.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-14T22:39:43.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: from='mgr.? 172.21.15.137:0/407422492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-14T22:39:43.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:43 smithi137 ceph-mon[32951]: mgrmap e36: y(active, since 1.02087s), standbys: x 2024-06-14T22:39:44.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch upgrade pause' 2024-06-14T22:39:44.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:44 smithi137 ceph-mon[32951]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:39:44.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:44 smithi137 ceph-mon[32951]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:39:44.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:44 smithi137 ceph-mon[32951]: pgmap v3: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-06-14T22:39:44.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:44 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:44.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:44 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:44.693 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[97913]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:39:44.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[97913]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:39:44.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[97913]: pgmap v3: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-06-14T22:39:44.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:44.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:44.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[38904]: Failed to load ceph-mgr modules: k8sevents 2024-06-14T22:39:44.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[38904]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:39:44.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[38904]: pgmap v3: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-06-14T22:39:44.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:44.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:44.534+0000 7f489c5bb640 -1 log_channel(cephadm) log [ERR] : [14/Jun/2024:22:39:44] ENGINE Error in HTTPServer.serve 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: Traceback (most recent call last): 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: self._connections.run(self.expiration_interval) 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: self._run(expiration_interval) 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:39:44.695 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:39:44.696 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: s = self.context.wrap_socket( 2024-06-14T22:39:44.696 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:39:44.696 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: return self.sslsocket_class._create( 2024-06-14T22:39:44.696 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:39:44.696 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: self.do_handshake() 2024-06-14T22:39:44.696 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:39:44.696 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: self._sslobj.do_handshake() 2024-06-14T22:39:44.696 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:39:44.696 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:39:44.955 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:39:44] "GET /metrics HTTP/1.1" 200 33499 "" "Prometheus/2.43.0" 2024-06-14T22:39:45.236 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:39:45.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:45 smithi086 ceph-mon[97913]: [14/Jun/2024:22:39:44] ENGINE Bus STARTING 2024-06-14T22:39:45.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:45 smithi086 ceph-mon[97913]: mgrmap e37: y(active, since 2s), standbys: x 2024-06-14T22:39:45.731 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:45 smithi086 ceph-mon[38904]: [14/Jun/2024:22:39:44] ENGINE Bus STARTING 2024-06-14T22:39:45.731 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:45 smithi086 ceph-mon[38904]: mgrmap e37: y(active, since 2s), standbys: x 2024-06-14T22:39:45.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:45 smithi137 ceph-mon[32951]: [14/Jun/2024:22:39:44] ENGINE Bus STARTING 2024-06-14T22:39:45.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:45 smithi137 ceph-mon[32951]: mgrmap e37: y(active, since 2s), standbys: x 2024-06-14T22:39:46.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:46 smithi086 ceph-mon[38904]: [14/Jun/2024:22:39:44] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:39:46.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:46 smithi086 ceph-mon[38904]: [14/Jun/2024:22:39:44] ENGINE Error in HTTPServer.serve 2024-06-14T22:39:46.581 INFO:journalctl@ceph.mon.c.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:39:46.581 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:39:46.581 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:39:46.581 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:39:46.581 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:39:46.581 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:39:46.581 INFO:journalctl@ceph.mon.c.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: self.do_handshake() 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:46 smithi086 ceph-mon[38904]: [14/Jun/2024:22:39:44] ENGINE Serving on http://172.21.15.86:8765 2024-06-14T22:39:46.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:46 smithi086 ceph-mon[38904]: [14/Jun/2024:22:39:44] ENGINE Bus STARTED 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:46 smithi086 ceph-mon[97913]: [14/Jun/2024:22:39:44] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:46 smithi086 ceph-mon[97913]: [14/Jun/2024:22:39:44] ENGINE Error in HTTPServer.serve 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout: Traceback (most recent call last): 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._run(expiration_interval) 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:39:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout: s = self.context.wrap_socket( 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout: return self.sslsocket_class._create( 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout: self.do_handshake() 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._sslobj.do_handshake() 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:46 smithi086 ceph-mon[97913]: [14/Jun/2024:22:39:44] ENGINE Serving on http://172.21.15.86:8765 2024-06-14T22:39:46.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:46 smithi086 ceph-mon[97913]: [14/Jun/2024:22:39:44] ENGINE Bus STARTED 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:46 smithi137 ceph-mon[32951]: [14/Jun/2024:22:39:44] ENGINE Serving on https://172.21.15.86:7150 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:46 smithi137 ceph-mon[32951]: [14/Jun/2024:22:39:44] ENGINE Error in HTTPServer.serve 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout: Traceback (most recent call last): 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._run(expiration_interval) 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-14T22:39:46.593 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: s = self.context.wrap_socket( 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: return self.sslsocket_class._create( 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: self.do_handshake() 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: self._sslobj.do_handshake() 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:46 smithi137 ceph-mon[32951]: [14/Jun/2024:22:39:44] ENGINE Serving on http://172.21.15.86:8765 2024-06-14T22:39:46.594 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:46 smithi137 ceph-mon[32951]: [14/Jun/2024:22:39:44] ENGINE Bus STARTED 2024-06-14T22:39:47.553 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:47 smithi137 ceph-mon[32951]: pgmap v5: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-06-14T22:39:47.553 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:47 smithi137 ceph-mon[32951]: mgrmap e38: y(active, since 4s), standbys: x 2024-06-14T22:39:47.704 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:47 smithi086 ceph-mon[38904]: pgmap v5: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-06-14T22:39:47.704 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:47 smithi086 ceph-mon[38904]: mgrmap e38: y(active, since 4s), standbys: x 2024-06-14T22:39:47.704 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:47 smithi086 ceph-mon[97913]: pgmap v5: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-06-14T22:39:47.704 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:47 smithi086 ceph-mon[97913]: mgrmap e38: y(active, since 4s), standbys: x 2024-06-14T22:39:48.241 INFO:teuthology.orchestra.run.smithi086.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:39:48.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:48.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:39:49.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:49 smithi137 ceph-mon[32951]: pgmap v6: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-06-14T22:39:49.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:49 smithi137 ceph-mon[32951]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:39:49.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:49 smithi137 ceph-mon[32951]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:39:49.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:49 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:49.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:49 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:49.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:49 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:49.492 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[38904]: pgmap v6: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[38904]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[38904]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[97913]: pgmap v6: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[97913]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[97913]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:49.493 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:49 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:50.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'sleep 60' 2024-06-14T22:39:51.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:50 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:51.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:50 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:51.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:50 smithi086 ceph-mon[38904]: pgmap v7: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 605 op/s 2024-06-14T22:39:51.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:50 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:51.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:50 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:51.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:50 smithi086 ceph-mon[97913]: pgmap v7: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 605 op/s 2024-06-14T22:39:51.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:50 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:51.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:50 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:51.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:50 smithi137 ceph-mon[32951]: pgmap v7: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 605 op/s 2024-06-14T22:39:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:39:52.478 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:52 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.479 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:52 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.479 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:52 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.479 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:52 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:52 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:52 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:52 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:52 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:52 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:52 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:52 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:52.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:52 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:53.518 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:53 smithi086 ceph-mon[97913]: pgmap v8: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 470 op/s 2024-06-14T22:39:53.518 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:53 smithi086 ceph-mon[38904]: pgmap v8: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 470 op/s 2024-06-14T22:39:53.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:53.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:39:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:53 smithi137 ceph-mon[32951]: pgmap v8: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 470 op/s 2024-06-14T22:39:55.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:39:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:39:54] "GET /metrics HTTP/1.1" 200 33499 "" "Prometheus/2.43.0" 2024-06-14T22:39:55.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:55 smithi137 ceph-mon[32951]: pgmap v9: 89 pgs: 89 active+clean; 839 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 768 op/s 2024-06-14T22:39:55.751 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:55 smithi086 ceph-mon[97913]: pgmap v9: 89 pgs: 89 active+clean; 839 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 768 op/s 2024-06-14T22:39:55.751 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:55 smithi086 ceph-mon[38904]: pgmap v9: 89 pgs: 89 active+clean; 839 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 768 op/s 2024-06-14T22:39:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:56 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:56 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:56 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:56 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:39:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:56 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:56 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:56.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:39:56.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:56 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:56.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:56 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:56.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:56 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:56 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:39:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:39:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:39:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:39:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:39:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: pgmap v10: 89 pgs: 89 active+clean; 839 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 700 op/s 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:39:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: pgmap v10: 89 pgs: 89 active+clean; 839 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 700 op/s 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:57 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: Updating smithi086:/etc/ceph/ceph.conf 2024-06-14T22:39:57.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: Updating smithi137:/etc/ceph/ceph.conf 2024-06-14T22:39:57.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:39:57.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-14T22:39:57.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: pgmap v10: 89 pgs: 89 active+clean; 839 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 700 op/s 2024-06-14T22:39:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:39:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-06-14T22:39:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:39:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:57 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[97913]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:39:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[97913]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:39:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:39:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[97913]: pgmap v11: 89 pgs: 89 active+clean; 839 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 700 op/s 2024-06-14T22:39:58.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[38904]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:39:58.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[38904]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:39:58.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:58.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:58.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:39:58.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:58.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:39:58 smithi086 ceph-mon[38904]: pgmap v11: 89 pgs: 89 active+clean; 839 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 700 op/s 2024-06-14T22:39:58.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:39:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:39:58.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:39:58.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:58 smithi137 ceph-mon[32951]: Updating smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:39:58.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:58 smithi137 ceph-mon[32951]: Updating smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-14T22:39:58.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:58 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:39:58.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:58 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:39:58.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:58 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:39:58.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:58 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:39:58.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:39:58 smithi137 ceph-mon[32951]: pgmap v11: 89 pgs: 89 active+clean; 839 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 700 op/s 2024-06-14T22:40:00.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:00 smithi086 ceph-mon[97913]: overall HEALTH_OK 2024-06-14T22:40:00.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:00 smithi086 ceph-mon[38904]: overall HEALTH_OK 2024-06-14T22:40:00.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:00 smithi137 ceph-mon[32951]: overall HEALTH_OK 2024-06-14T22:40:01.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:01 smithi086 ceph-mon[97913]: pgmap v12: 89 pgs: 89 active+clean; 574 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-06-14T22:40:01.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:01 smithi086 ceph-mon[38904]: pgmap v12: 89 pgs: 89 active+clean; 574 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-06-14T22:40:01.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:01 smithi137 ceph-mon[32951]: pgmap v12: 89 pgs: 89 active+clean; 574 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-06-14T22:40:03.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:03 smithi086 ceph-mon[97913]: pgmap v13: 89 pgs: 89 active+clean; 574 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-06-14T22:40:03.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:03 smithi086 ceph-mon[38904]: pgmap v13: 89 pgs: 89 active+clean; 574 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-06-14T22:40:03.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:03 smithi137 ceph-mon[32951]: pgmap v13: 89 pgs: 89 active+clean; 574 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-06-14T22:40:03.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:03.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:05.284 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:40:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:40:04] "GET /metrics HTTP/1.1" 200 36213 "" "Prometheus/2.43.0" 2024-06-14T22:40:05.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:05 smithi086 ceph-mon[38904]: pgmap v14: 89 pgs: 89 active+clean; 283 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:40:05.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:05 smithi086 ceph-mon[97913]: pgmap v14: 89 pgs: 89 active+clean; 283 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:40:05.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:05 smithi137 ceph-mon[32951]: pgmap v14: 89 pgs: 89 active+clean; 283 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:40:07.542 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:07 smithi137 ceph-mon[32951]: pgmap v15: 89 pgs: 89 active+clean; 283 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-06-14T22:40:07.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:07 smithi086 ceph-mon[97913]: pgmap v15: 89 pgs: 89 active+clean; 283 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-06-14T22:40:07.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:07 smithi086 ceph-mon[38904]: pgmap v15: 89 pgs: 89 active+clean; 283 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-06-14T22:40:08.164 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 130083 objects 2024-06-14T22:40:08.164 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :152.842 2024-06-14T22:40:08.963 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:08 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:08.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:09.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:08 smithi137 ceph-mon[32951]: pgmap v16: 89 pgs: 89 active+clean; 283 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-06-14T22:40:09.312 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-06-14T22:40:09.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-06-14T22:40:09.330 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:08 smithi086 ceph-mon[97913]: pgmap v16: 89 pgs: 89 active+clean; 283 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-06-14T22:40:09.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:08 smithi086 ceph-mon[38904]: pgmap v16: 89 pgs: 89 active+clean; 283 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-06-14T22:40:09.579 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:40:11.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:11 smithi086 ceph-mon[97913]: pgmap v17: 89 pgs: 89 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-14T22:40:11.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:11 smithi086 ceph-mon[38904]: pgmap v17: 89 pgs: 89 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-14T22:40:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:11 smithi137 ceph-mon[32951]: pgmap v17: 89 pgs: 89 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-14T22:40:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_0' removed 2024-06-14T22:40:12.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1220568389' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:40:12.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:12 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:40:12.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:12 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/1220568389' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:40:12.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:12 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:40:12.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:12 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/1220568389' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:40:12.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:12 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:40:13.174 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T22:40:13.175 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T22:40:13.175 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-06-14T22:40:13.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-06-14T22:40:13.483 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:40:13.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:13 smithi086 ceph-mon[97913]: pgmap v18: 89 pgs: 89 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 764 op/s 2024-06-14T22:40:13.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:13 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:40:13.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:13 smithi086 ceph-mon[97913]: osdmap e78: 8 total, 8 up, 8 in 2024-06-14T22:40:13.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:13.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:13 smithi086 ceph-mon[38904]: pgmap v18: 89 pgs: 89 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 764 op/s 2024-06-14T22:40:13.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:13 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:40:13.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:13 smithi086 ceph-mon[38904]: osdmap e78: 8 total, 8 up, 8 in 2024-06-14T22:40:13.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:13 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:13.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:13 smithi137 ceph-mon[32951]: pgmap v18: 89 pgs: 89 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 764 op/s 2024-06-14T22:40:13.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:13 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:40:13.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:13 smithi137 ceph-mon[32951]: osdmap e78: 8 total, 8 up, 8 in 2024-06-14T22:40:13.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:13 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:13.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:13.512Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:15.117 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:40:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:40:14] "GET /metrics HTTP/1.1" 200 33859 "" "Prometheus/2.43.0" 2024-06-14T22:40:15.454 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:15 smithi086 ceph-mon[97913]: pgmap v20: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:15.454 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:15 smithi086 ceph-mon[38904]: pgmap v20: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:15 smithi137 ceph-mon[32951]: pgmap v20: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:16.301 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_1' created 2024-06-14T22:40:16.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/597126394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-06-14T22:40:16.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:16 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/597126394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-06-14T22:40:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:16 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/597126394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-06-14T22:40:17.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-06-14T22:40:17.295 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:40:17.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:17 smithi086 ceph-mon[97913]: pgmap v21: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:17.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/597126394' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-06-14T22:40:17.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:17 smithi086 ceph-mon[97913]: osdmap e79: 8 total, 8 up, 8 in 2024-06-14T22:40:17.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:17 smithi086 ceph-mon[38904]: pgmap v21: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:17.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:17 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/597126394' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-06-14T22:40:17.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:17 smithi086 ceph-mon[38904]: osdmap e79: 8 total, 8 up, 8 in 2024-06-14T22:40:17.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:17 smithi137 ceph-mon[32951]: pgmap v21: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:17.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:17 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/597126394' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-06-14T22:40:17.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:17 smithi137 ceph-mon[32951]: osdmap e79: 8 total, 8 up, 8 in 2024-06-14T22:40:18.473 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:18 smithi086 ceph-mon[97913]: osdmap e80: 8 total, 8 up, 8 in 2024-06-14T22:40:18.473 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:18 smithi086 ceph-mon[38904]: osdmap e80: 8 total, 8 up, 8 in 2024-06-14T22:40:18.656 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:18 smithi137 ceph-mon[32951]: osdmap e80: 8 total, 8 up, 8 in 2024-06-14T22:40:18.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:18 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:18.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:19.325 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-06-14T22:40:19.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:19 smithi137 ceph-mon[32951]: pgmap v24: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:19.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:19 smithi137 ceph-mon[32951]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:40:19.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:19 smithi137 ceph-mon[32951]: osdmap e81: 8 total, 8 up, 8 in 2024-06-14T22:40:19.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:19 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3089128445' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:40:19.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:19 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:40:19.580 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[97913]: pgmap v24: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:19.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[97913]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:40:19.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[97913]: osdmap e81: 8 total, 8 up, 8 in 2024-06-14T22:40:19.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3089128445' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:40:19.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:40:19.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[38904]: pgmap v24: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:19.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[38904]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:40:19.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[38904]: osdmap e81: 8 total, 8 up, 8 in 2024-06-14T22:40:19.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/3089128445' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:40:19.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:19 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:40:20.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:20 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:40:20.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:20 smithi086 ceph-mon[97913]: osdmap e82: 8 total, 8 up, 8 in 2024-06-14T22:40:20.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:20 smithi086 ceph-mon[38904]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:40:20.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:20 smithi086 ceph-mon[38904]: osdmap e82: 8 total, 8 up, 8 in 2024-06-14T22:40:20.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:20 smithi137 ceph-mon[32951]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:40:20.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:20 smithi137 ceph-mon[32951]: osdmap e82: 8 total, 8 up, 8 in 2024-06-14T22:40:21.080 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-06-14T22:40:21.083 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T22:40:21.083 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T22:40:21.135 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T22:40:21.135 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T22:40:21.135 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_100957 2024-06-14T22:40:21.136 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:40:21.136 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T22:40:21.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:21 smithi086 ceph-mon[97913]: pgmap v27: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:21.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:21 smithi086 ceph-mon[97913]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:40:21.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:21 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:40:21.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:21 smithi086 ceph-mon[38904]: pgmap v27: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:21.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:21 smithi086 ceph-mon[38904]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:40:21.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:21 smithi086 ceph-mon[38904]: Cluster is now healthy 2024-06-14T22:40:21.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:21 smithi137 ceph-mon[32951]: pgmap v27: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:21 smithi137 ceph-mon[32951]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:40:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:21 smithi137 ceph-mon[32951]: Cluster is now healthy 2024-06-14T22:40:22.136 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 4362 4346 271.609 271.625 0.00263555 0.00366862 2024-06-14T22:40:23.136 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 15 8417 8402 262.544 253.5 0.00687624 0.00380083 2024-06-14T22:40:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:23 smithi137 ceph-mon[32951]: pgmap v28: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:23.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:23.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:23.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:23 smithi086 ceph-mon[97913]: pgmap v28: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:23.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:23 smithi086 ceph-mon[38904]: pgmap v28: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:40:24.136 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 16 12120 12104 252.144 231.375 0.00279823 0.0039591 2024-06-14T22:40:25.136 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 16 15555 15539 242.775 214.688 0.00556919 0.00411353 2024-06-14T22:40:25.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:40:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:40:24] "GET /metrics HTTP/1.1" 200 33859 "" "Prometheus/2.43.0" 2024-06-14T22:40:25.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:25 smithi137 ceph-mon[32951]: pgmap v29: 89 pgs: 89 active+clean; 558 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-14T22:40:25.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:25 smithi086 ceph-mon[97913]: pgmap v29: 89 pgs: 89 active+clean; 558 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-14T22:40:25.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:25 smithi086 ceph-mon[38904]: pgmap v29: 89 pgs: 89 active+clean; 558 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-14T22:40:26.136 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 16 18804 18788 234.829 203.062 0.00392453 0.00425254 2024-06-14T22:40:27.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 21555 21539 224.344 171.938 0.00547305 0.00445206 2024-06-14T22:40:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:27 smithi137 ceph-mon[32951]: pgmap v30: 89 pgs: 89 active+clean; 558 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T22:40:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:27 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:27.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:27 smithi086 ceph-mon[38904]: pgmap v30: 89 pgs: 89 active+clean; 558 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T22:40:27.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:27 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:27 smithi086 ceph-mon[97913]: pgmap v30: 89 pgs: 89 active+clean; 558 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T22:40:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:28.136 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 15 24111 24096 215.123 159.812 0.0108157 0.00464246 2024-06-14T22:40:28.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:28 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:28.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:29.136 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 26841 26825 209.551 170.562 0.00440481 0.0047671 2024-06-14T22:40:29.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:29 smithi137 ceph-mon[32951]: pgmap v31: 89 pgs: 89 active+clean; 558 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 56 MiB/s wr, 898 op/s 2024-06-14T22:40:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:29 smithi086 ceph-mon[97913]: pgmap v31: 89 pgs: 89 active+clean; 558 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 56 MiB/s wr, 898 op/s 2024-06-14T22:40:29.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:29 smithi086 ceph-mon[38904]: pgmap v31: 89 pgs: 89 active+clean; 558 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 56 MiB/s wr, 898 op/s 2024-06-14T22:40:30.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 29507 29491 204.78 166.625 0.00445417 0.00487827 2024-06-14T22:40:31.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 15 32073 32058 200.344 160.438 0.0090859 0.00498603 2024-06-14T22:40:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:31 smithi137 ceph-mon[32951]: pgmap v32: 89 pgs: 89 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-06-14T22:40:31.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:31 smithi086 ceph-mon[97913]: pgmap v32: 89 pgs: 89 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-06-14T22:40:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:31 smithi086 ceph-mon[38904]: pgmap v32: 89 pgs: 89 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-06-14T22:40:32.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 34529 34513 196.079 153.438 0.00408187 0.00509459 2024-06-14T22:40:33.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 16 36531 36515 190.165 125.125 0.00492969 0.00525362 2024-06-14T22:40:33.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:33 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:33.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:33 smithi137 ceph-mon[32951]: pgmap v33: 89 pgs: 89 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s wr, 2.04k op/s 2024-06-14T22:40:33.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:33 smithi086 ceph-mon[97913]: pgmap v33: 89 pgs: 89 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s wr, 2.04k op/s 2024-06-14T22:40:33.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:33 smithi086 ceph-mon[38904]: pgmap v33: 89 pgs: 89 active+clean; 1.5 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s wr, 2.04k op/s 2024-06-14T22:40:34.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 38846 38830 186.666 144.688 0.00418799 0.00535227 2024-06-14T22:40:35.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 40886 40870 182.439 127.5 0.00512015 0.00547554 2024-06-14T22:40:35.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:40:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:40:34] "GET /metrics HTTP/1.1" 200 36206 "" "Prometheus/2.43.0" 2024-06-14T22:40:35.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:35 smithi137 ceph-mon[32951]: pgmap v34: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 192 MiB/s wr, 3.07k op/s 2024-06-14T22:40:35.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:35 smithi086 ceph-mon[97913]: pgmap v34: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 192 MiB/s wr, 3.07k op/s 2024-06-14T22:40:35.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:35 smithi086 ceph-mon[38904]: pgmap v34: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 192 MiB/s wr, 3.07k op/s 2024-06-14T22:40:36.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 42157 42141 175.572 79.4375 0.00619496 0.00569122 2024-06-14T22:40:37.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 44196 44180 172.562 127.438 0.00669501 0.00578988 2024-06-14T22:40:37.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:37 smithi137 ceph-mon[32951]: pgmap v35: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 145 MiB/s wr, 2.33k op/s 2024-06-14T22:40:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:37 smithi086 ceph-mon[38904]: pgmap v35: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 145 MiB/s wr, 2.33k op/s 2024-06-14T22:40:37.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:37 smithi086 ceph-mon[97913]: pgmap v35: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 145 MiB/s wr, 2.33k op/s 2024-06-14T22:40:38.137 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 16 46244 46228 169.94 128 0.00424792 0.00587967 2024-06-14T22:40:38.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:38.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:39.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 48229 48213 167.391 124.062 0.00532647 0.00596913 2024-06-14T22:40:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:39 smithi137 ceph-mon[32951]: pgmap v36: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 145 MiB/s wr, 2.33k op/s 2024-06-14T22:40:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:39 smithi086 ceph-mon[97913]: pgmap v36: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 145 MiB/s wr, 2.33k op/s 2024-06-14T22:40:39.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:39 smithi086 ceph-mon[38904]: pgmap v36: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 145 MiB/s wr, 2.33k op/s 2024-06-14T22:40:40.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 50199 50183 165.061 123.125 0.00677219 0.00605314 2024-06-14T22:40:41.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:40:41.142279+0000 min lat: 0.00116587 max lat: 0.439427 avg lat: 0.00613345 2024-06-14T22:40:41.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:40:41.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 52155 52139 162.92 122.25 0.00902353 0.00613345 2024-06-14T22:40:41.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:41 smithi137 ceph-mon[32951]: pgmap v37: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-06-14T22:40:41.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:41 smithi086 ceph-mon[97913]: pgmap v37: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-06-14T22:40:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:41 smithi086 ceph-mon[38904]: pgmap v37: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 196 MiB/s wr, 3.13k op/s 2024-06-14T22:40:42.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 16 54056 54040 160.819 118.812 0.0034517 0.0062129 2024-06-14T22:40:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:42.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:42 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:43.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 55895 55879 158.733 114.938 0.00857936 0.00629493 2024-06-14T22:40:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:43 smithi137 ceph-mon[32951]: pgmap v38: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-06-14T22:40:43.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:43.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:43.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:43 smithi086 ceph-mon[97913]: pgmap v38: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-06-14T22:40:43.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:43 smithi086 ceph-mon[38904]: pgmap v38: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-06-14T22:40:44.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 57691 57675 156.711 112.25 0.00688485 0.0063766 2024-06-14T22:40:45.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 59455 59439 154.775 110.25 0.0128768 0.00645587 2024-06-14T22:40:45.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:40:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:40:44] "GET /metrics HTTP/1.1" 200 36211 "" "Prometheus/2.43.0" 2024-06-14T22:40:45.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:45 smithi137 ceph-mon[32951]: pgmap v39: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 165 MiB/s wr, 2.64k op/s 2024-06-14T22:40:45.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:45 smithi086 ceph-mon[97913]: pgmap v39: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 165 MiB/s wr, 2.64k op/s 2024-06-14T22:40:45.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:45 smithi086 ceph-mon[38904]: pgmap v39: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 165 MiB/s wr, 2.64k op/s 2024-06-14T22:40:46.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 61200 61184 152.946 109.062 0.0086511 0.00653394 2024-06-14T22:40:47.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 62903 62887 151.157 106.438 0.00784125 0.00661064 2024-06-14T22:40:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:47 smithi137 ceph-mon[32951]: pgmap v40: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-06-14T22:40:47.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:47 smithi086 ceph-mon[97913]: pgmap v40: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-06-14T22:40:47.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:47 smithi086 ceph-mon[38904]: pgmap v40: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-06-14T22:40:48.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 64587 64571 149.456 105.25 0.00600476 0.00668666 2024-06-14T22:40:48.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:48.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:49.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 66215 66199 147.752 101.75 0.0184039 0.00676322 2024-06-14T22:40:49.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:49 smithi137 ceph-mon[32951]: pgmap v41: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-06-14T22:40:49.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:49 smithi086 ceph-mon[97913]: pgmap v41: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-06-14T22:40:49.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:49 smithi086 ceph-mon[38904]: pgmap v41: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-06-14T22:40:50.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 67822 67806 146.12 100.438 0.00339828 0.00683838 2024-06-14T22:40:51.138 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 69375 69359 144.485 97.0625 0.0117734 0.0069164 2024-06-14T22:40:51.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:51 smithi137 ceph-mon[32951]: pgmap v42: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-06-14T22:40:51.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:51 smithi086 ceph-mon[97913]: pgmap v42: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-06-14T22:40:51.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:51 smithi086 ceph-mon[38904]: pgmap v42: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-06-14T22:40:52.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 70940 70924 142.979 97.8125 0.00609508 0.00698892 2024-06-14T22:40:53.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 72487 72471 141.532 96.6875 0.0116795 0.00706107 2024-06-14T22:40:53.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:53.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:53 smithi137 ceph-mon[32951]: pgmap v43: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T22:40:53.759 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:53 smithi086 ceph-mon[97913]: pgmap v43: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T22:40:53.759 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:53 smithi086 ceph-mon[38904]: pgmap v43: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T22:40:54.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 16 73962 73946 140.037 92.1875 0.010518 0.00713623 2024-06-14T22:40:54.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch upgrade resume' 2024-06-14T22:40:54.718 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:40:55.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 75310 75294 138.396 84.25 0.0202281 0.00722079 2024-06-14T22:40:55.232 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:40:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:40:54] "GET /metrics HTTP/1.1" 200 36211 "" "Prometheus/2.43.0" 2024-06-14T22:40:55.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:55 smithi086 ceph-mon[97913]: pgmap v44: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-06-14T22:40:55.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:55 smithi086 ceph-mon[38904]: pgmap v44: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-06-14T22:40:55.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:55 smithi137 ceph-mon[32951]: pgmap v44: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-06-14T22:40:56.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 76746 76730 137.005 89.75 0.00801164 0.00729388 2024-06-14T22:40:57.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 78073 78057 135.503 82.9375 0.00622377 0.00737514 2024-06-14T22:40:57.258 INFO:teuthology.orchestra.run.smithi086.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:40:57.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:57 smithi086 ceph-mon[38904]: pgmap v45: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T22:40:57.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:57 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:40:57.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:57 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:40:57.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:57 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:57.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:57 smithi086 ceph-mon[97913]: pgmap v45: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T22:40:57.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:57 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:40:57.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:40:57.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:57 smithi137 ceph-mon[32951]: pgmap v45: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T22:40:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:57 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:40:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:57 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:40:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:57 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:40:58.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 79177 79161 133.706 69 0.015827 0.00747365 2024-06-14T22:40:58.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:58 smithi137 ceph-mon[32951]: from='client.25211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:40:58.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:58 smithi137 ceph-mon[32951]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:40:58.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:40:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:40:58.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:40:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:58 smithi086 ceph-mon[97913]: from='client.25211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:40:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:58 smithi086 ceph-mon[97913]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:40:58.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:58 smithi086 ceph-mon[38904]: from='client.25211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:40:58.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:58 smithi086 ceph-mon[38904]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:40:59.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 80255 80239 131.96 67.375 0.00690025 0.00757319 2024-06-14T22:40:59.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:40:59 smithi137 ceph-mon[32951]: pgmap v46: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T22:40:59.495 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:40:59 smithi086 ceph-mon[97913]: pgmap v46: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T22:40:59.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:40:59 smithi086 ceph-mon[38904]: pgmap v46: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T22:40:59.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'echo wait for majority of mons to upgrade' 2024-06-14T22:41:00.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 16 81422 81406 130.447 72.9375 0.00573675 0.00766093 2024-06-14T22:41:01.140 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:41:01.144097+0000 min lat: 0.00116587 max lat: 0.439427 avg lat: 0.00774023 2024-06-14T22:41:01.140 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:41:01.140 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 82653 82637 129.109 76.9375 0.0192799 0.00774023 2024-06-14T22:41:01.159 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:41:01.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:01 smithi086 ceph-mon[38904]: pgmap v47: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-06-14T22:41:01.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:01 smithi086 ceph-mon[97913]: pgmap v47: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-06-14T22:41:01.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:01 smithi137 ceph-mon[32951]: pgmap v47: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-06-14T22:41:02.139 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 83778 83762 127.674 70.3125 0.00683213 0.00782698 2024-06-14T22:41:03.140 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 16 84974 84958 126.414 74.75 0.00725728 0.00790599 2024-06-14T22:41:03.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:03 smithi086 ceph-mon[97913]: pgmap v48: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-14T22:41:03.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:03 smithi086 ceph-mon[38904]: pgmap v48: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-14T22:41:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:03 smithi137 ceph-mon[32951]: pgmap v48: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-14T22:41:03.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:03.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:03.763 INFO:teuthology.orchestra.run.smithi086.stdout:wait for majority of mons to upgrade 2024-06-14T22:41:04.140 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 86082 86066 125.085 69.25 0.00381004 0.00798615 2024-06-14T22:41:04.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'while ! ceph mon versions | grep $sha1 | egrep '"'"': [23]'"'"' ; do sleep 2 ; done' 2024-06-14T22:41:05.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:41:04] "GET /metrics HTTP/1.1" 200 36210 "" "Prometheus/2.43.0" 2024-06-14T22:41:05.142 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 16 87244 87228 123.886 72.625 0.0220044 0.00806676 2024-06-14T22:41:05.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:05 smithi137 ceph-mon[32951]: pgmap v49: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-06-14T22:41:05.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:05 smithi086 ceph-mon[38904]: pgmap v49: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-06-14T22:41:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:05 smithi086 ceph-mon[97913]: pgmap v49: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-06-14T22:41:05.979 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:41:06.143 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 88412 88396 122.752 73 0.0120834 0.00814143 2024-06-14T22:41:06.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:06 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:06.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:06 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:06.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:06 smithi086 ceph-mon[97913]: pgmap v50: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-14T22:41:06.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:06 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:06.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:06 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:06.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:06 smithi086 ceph-mon[38904]: pgmap v50: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-14T22:41:06.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:06 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:06.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:06 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:06.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:06 smithi137 ceph-mon[32951]: pgmap v50: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-14T22:41:07.143 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 15 89650 89635 121.767 77.4375 0.0168326 0.00820815 2024-06-14T22:41:08.145 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 90892 90876 120.824 77.5625 0.00714189 0.00827193 2024-06-14T22:41:08.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:08 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:08.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:09.144 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 91875 91859 119.587 61.4375 0.0104265 0.00835631 2024-06-14T22:41:09.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: pgmap v51: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2889459207' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-14T22:41:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: pgmap v51: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/2889459207' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-14T22:41:09.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:09 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-06-14T22:41:09.595 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: pgmap v51: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-14T22:41:09.595 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:09.595 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:09.595 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:09.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:41:09.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:09.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/2889459207' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:09.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:41:09.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:41:09.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-14T22:41:09.596 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:09 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-06-14T22:41:10.145 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 93017 93001 118.601 71.375 0.0184476 0.00842648 2024-06-14T22:41:10.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:10 smithi086 ceph-mon[38904]: Upgrade: It appears safe to stop mon.c 2024-06-14T22:41:10.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:10 smithi086 ceph-mon[97913]: Upgrade: It appears safe to stop mon.c 2024-06-14T22:41:10.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:10 smithi137 ceph-mon[32951]: Upgrade: It appears safe to stop mon.c 2024-06-14T22:41:11.145 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 94120 94104 117.608 68.9375 0.0060716 0.00849771 2024-06-14T22:41:11.572 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:11 smithi086 ceph-mon[97913]: pgmap v52: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-14T22:41:11.572 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:11 smithi086 ceph-mon[38904]: pgmap v52: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-14T22:41:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:11 smithi137 ceph-mon[32951]: pgmap v52: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-14T22:41:12.145 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 95181 95165 116.602 66.3125 0.0140611 0.00857164 2024-06-14T22:41:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[38904]: Upgrade: Updating mon.c 2024-06-14T22:41:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[38904]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:41:12.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:41:12.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:12.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[38904]: Deploying daemon mon.c on smithi086 2024-06-14T22:41:12.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[38904]: from='client.? 172.21.15.86:0/4051991735' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:12.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[38904]: pgmap v53: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s 2024-06-14T22:41:12.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[38904]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[97913]: Upgrade: Updating mon.c 2024-06-14T22:41:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[97913]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:41:12.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:41:12.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:12.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[97913]: Deploying daemon mon.c on smithi086 2024-06-14T22:41:12.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4051991735' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:12.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[97913]: pgmap v53: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s 2024-06-14T22:41:12.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:12 smithi137 ceph-mon[32951]: Upgrade: Updating mon.c 2024-06-14T22:41:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:12 smithi137 ceph-mon[32951]: from='mgr.25130 ' entity='mgr.y' 2024-06-14T22:41:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:12 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:41:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:12 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:41:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:12 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:12 smithi137 ceph-mon[32951]: Deploying daemon mon.c on smithi086 2024-06-14T22:41:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:12 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/4051991735' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:12 smithi137 ceph-mon[32951]: pgmap v53: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s 2024-06-14T22:41:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:12 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:13.145 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 15 96389 96374 115.813 75.5625 0.0246828 0.00862992 2024-06-14T22:41:13.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:13.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:14.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 16 97425 97409 114.848 64.6875 0.0311911 0.0087016 2024-06-14T22:41:14.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:13 smithi086 systemd[1]: Stopping Ceph mon.c for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:41:15.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:41:14] ENGINE Bus STOPPING 2024-06-14T22:41:15.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:41:14] "GET /metrics HTTP/1.1" 200 36213 "" "Prometheus/2.43.0" 2024-06-14T22:41:15.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:41:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T22:41:15.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:41:14] ENGINE Bus STOPPED 2024-06-14T22:41:15.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:41:14] ENGINE Bus STARTING 2024-06-14T22:41:15.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:41:14] ENGINE Serving on http://:::9283 2024-06-14T22:41:15.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:22:41:14] ENGINE Bus STARTED 2024-06-14T22:41:15.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c[38881]: 2024-06-14T22:41:14.748+0000 7f45e6909640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T22:41:15.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c[38881]: 2024-06-14T22:41:14.748+0000 7f45e6909640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-06-14T22:41:15.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:14 smithi086 podman[102544]: 2024-06-14 22:41:14.837281013 +0000 UTC m=+0.663973289 container died e02d9b886f41e1da45f684fb73c25a7bd7086eb9bf94b02aac4aa937f900af12 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=) 2024-06-14T22:41:15.145 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 98524 98508 113.993 68.6875 0.0185788 0.00876781 2024-06-14T22:41:15.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:15 smithi086 podman[102544]: 2024-06-14 22:41:15.411975127 +0000 UTC m=+1.238667410 container cleanup e02d9b886f41e1da45f684fb73c25a7bd7086eb9bf94b02aac4aa937f900af12 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-06-14T22:41:15.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:15 smithi086 bash[102544]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c 2024-06-14T22:41:16.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:15 smithi086 podman[102562]: 2024-06-14 22:41:15.689201224 +0000 UTC m=+0.849810254 container remove e02d9b886f41e1da45f684fb73c25a7bd7086eb9bf94b02aac4aa937f900af12 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-14T22:41:16.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 99659 99643 113.211 70.9375 0.01019 0.00882843 2024-06-14T22:41:16.472 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:16 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.c.service: Deactivated successfully. 2024-06-14T22:41:16.472 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:16 smithi086 systemd[1]: Stopped Ceph mon.c for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:41:16.472 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:16 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.c.service: Consumed 8.911s CPU time. 2024-06-14T22:41:16.826 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:16 smithi086 systemd[1]: Starting Ceph mon.c for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:41:17.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:16 smithi086 podman[102716]: 2024-06-14 22:41:16.73921523 +0000 UTC m=+0.042575598 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:41:17.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:16 smithi086 podman[102716]: 2024-06-14 22:41:16.881552801 +0000 UTC m=+0.184913160 container create 1c4e71d265f00f50c1d0b7e3a7f4f1c3dfeb1ab75de74b98ded9b1b91b7dd71b (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.35.3) 2024-06-14T22:41:17.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 100629 100613 112.271 60.625 0.0167627 0.00890203 2024-06-14T22:41:17.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 podman[102716]: 2024-06-14 22:41:17.151061836 +0000 UTC m=+0.454422200 container init 1c4e71d265f00f50c1d0b7e3a7f4f1c3dfeb1ab75de74b98ded9b1b91b7dd71b (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, maintainer=Guillaume Abrioux ) 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 podman[102716]: 2024-06-14 22:41:17.161257574 +0000 UTC m=+0.464617938 container start 1c4e71d265f00f50c1d0b7e3a7f4f1c3dfeb1ab75de74b98ded9b1b91b7dd71b (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: set uid:gid to 167:167 (ceph:ceph) 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable), process ceph-mon, pid 7 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: pidfile_write: ignore empty --pid-file 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: load: jerasure load: lrc 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: RocksDB version: 7.9.2 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Git sha 0 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Compile date 2024-06-08 16:28:17 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: DB SUMMARY 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: DB Session ID: 2Q1MFJ8XD0IOC8XDMWRH 2024-06-14T22:41:17.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: CURRENT file: CURRENT 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: IDENTITY file: IDENTITY 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1093 Bytes 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 4202348 ; 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.error_if_exists: 0 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.create_if_missing: 0 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.paranoid_checks: 1 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.env: 0x556e7b913ba0 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.fs: PosixFileSystem 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.info_log: 0x556e7e004be0 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_file_opening_threads: 16 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.statistics: (nil) 2024-06-14T22:41:17.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.use_fsync: 0 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_log_file_size: 0 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.keep_log_file_num: 1000 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.recycle_log_file_num: 0 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.allow_fallocate: 1 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.allow_mmap_reads: 0 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.allow_mmap_writes: 0 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.use_direct_reads: 0 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.create_missing_column_families: 0 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.db_log_dir: 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.wal_dir: 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-14T22:41:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.advise_random_on_open: 1 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.db_write_buffer_size: 0 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.write_buffer_manager: 0x556e7dfd6640 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.rate_limiter: (nil) 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.wal_recovery_mode: 2 2024-06-14T22:41:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.enable_thread_tracking: 0 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.enable_pipelined_write: 0 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.unordered_write: 0 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.row_cache: None 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.wal_filter: None 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.allow_ingest_behind: 0 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.two_write_queues: 0 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.manual_wal_flush: 0 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.wal_compression: 0 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.atomic_flush: 0 2024-06-14T22:41:17.591 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.log_readahead_size: 0 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.best_efforts_recovery: 0 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.allow_data_in_errors: 0 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.db_host_id: __hostname__ 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_background_jobs: 2 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_background_compactions: -1 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_subcompactions: 1 2024-06-14T22:41:17.592 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_total_wal_size: 0 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_open_files: -1 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bytes_per_sync: 0 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_readahead_size: 0 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_background_flushes: -1 2024-06-14T22:41:17.593 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Compression algorithms supported: 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: kZSTD supported: 0 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: kXpressCompression supported: 0 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: kBZip2Compression supported: 0 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: kLZ4Compression supported: 1 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: kZlibCompression supported: 1 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: kLZ4HCCompression supported: 1 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: kSnappyCompression supported: 1 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-14T22:41:17.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.merge_operator: 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_filter: None 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_filter_factory: None 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.sst_partitioner_factory: None 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556e7e004820) 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout: index_type: 0 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout: data_block_index_type: 0 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout: index_shortening: 1 2024-06-14T22:41:17.595 INFO:journalctl@ceph.mon.c.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: checksum: 4 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: no_block_cache: 0 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_cache: 0x556e7dffb350 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_cache_options: 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: capacity : 536870912 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: num_shard_bits : 4 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: strict_capacity_limit : 0 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_cache_compressed: (nil) 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: persistent_cache: (nil) 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_size: 4096 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_size_deviation: 10 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_restart_interval: 16 2024-06-14T22:41:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout: index_block_restart_interval: 1 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: metadata_block_size: 4096 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: partition_filters: 0 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: use_delta_encoding: 1 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: filter_policy: bloomfilter 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: whole_key_filtering: 1 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: verify_compression: 0 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: format_version: 5 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: enable_index_compression: 1 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: block_align: 0 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: max_auto_readahead_size: 262144 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: prepopulate_block_cache: 0 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: initial_auto_readahead_size: 8192 2024-06-14T22:41:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout: num_file_reads_for_auto_readahead: 2 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.write_buffer_size: 33554432 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_write_buffer_number: 2 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression: NoCompression 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression: Disabled 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.prefix_extractor: nullptr 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.num_levels: 7 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-14T22:41:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression_opts.level: 32767 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression_opts.strategy: 0 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-14T22:41:17.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression_opts.enabled: false 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.target_file_size_base: 67108864 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-14T22:41:17.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.arena_block_size: 1048576 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.disable_auto_compactions: 0 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-14T22:41:17.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.table_properties_collectors: 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.inplace_update_support: 0 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.bloom_locality: 0 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.max_successive_merges: 0 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.paranoid_file_checks: 0 2024-06-14T22:41:17.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.force_consistency_checks: 1 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.report_bg_io_stats: 0 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.ttl: 2592000 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.enable_blob_files: false 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.min_blob_size: 0 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.blob_file_size: 268435456 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-14T22:41:17.603 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.blob_file_starting_level: 0 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12785, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f10911cb-4818-4a57-ad5f-5985f7538716 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404877209483, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404877225100, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 2674292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12790, "largest_seqno": 14207, "table_properties": {"data_size": 2667665, "index_size": 4136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1477, "raw_key_size": 12745, "raw_average_key_size": 22, "raw_value_size": 2655284, "raw_average_value_size": 4691, "num_data_blocks": 187, "num_entries": 566, "num_filter_entries": 566, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1718404877, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f10911cb-4818-4a57-ad5f-5985f7538716", "db_session_id": "2Q1MFJ8XD0IOC8XDMWRH", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404877225559, "job": 1, "event": "recovery_finished"} 2024-06-14T22:41:17.604 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556e7e024e00 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: DB pointer 0x556e7e118000 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout: ** DB Stats ** 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-14T22:41:17.605 INFO:journalctl@ceph.mon.c.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: L0 1/0 2.55 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 375.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: L6 1/0 7.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: Sum 2/0 9.89 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 375.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 375.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: ** Compaction Stats [default] ** 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-14T22:41:17.606 INFO:journalctl@ceph.mon.c.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 375.1 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: Cumulative compaction: 0.00 GB write, 101.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: Interval compaction: 0.00 GB write, 101.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-14T22:41:17.607 INFO:journalctl@ceph.mon.c.smithi086.stdout: Block cache BinnedLRUCache@0x556e7dffb350#7 capacity: 512.00 MB usage: 61.53 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: Block cache entry stats(count,size,portion): DataBlock(4,30.31 KB,0.00578165%) FilterBlock(2,10.56 KB,0.00201464%) IndexBlock(2,20.66 KB,0.00393987%) Misc(1,0.00 KB,0%) 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: starting mon.c rank 2 at public addrs [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] at bind addrs [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: mon.c@-1(???) e3 preinit fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: mon.c@-1(???).mds e9 new map 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: mon.c@-1(???).mds e9 print_map 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: e9 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: legacy client fscid: 1 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: Filesystem 'foo' (1) 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: fs_name foo 2024-06-14T22:41:17.608 INFO:journalctl@ceph.mon.c.smithi086.stdout: epoch 8 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: created 2024-06-14T22:34:45.808786+0000 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: modified 2024-06-14T22:35:03.607395+0000 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: tableserver 0 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: root 0 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: session_timeout 60 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: session_autoclose 300 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: max_file_size 1099511627776 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: max_xattr_size 65536 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: required_client_features {} 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: last_failure 0 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: last_failure_osd_epoch 0 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: max_mds 1 2024-06-14T22:41:17.609 INFO:journalctl@ceph.mon.c.smithi086.stdout: in 0 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: up {0=24548} 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: failed 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: damaged 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: stopped 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: data_pools [4] 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: metadata_pool 3 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: inline_data disabled 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: balancer 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: bal_rank_mask -1 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: standby_count_wanted 1 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: [mds.foo.smithi086.nhdjdb{0:24548} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] compat {c=[1],r=[1],i=[7ff]}] 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: Standby daemons: 2024-06-14T22:41:17.610 INFO:journalctl@ceph.mon.c.smithi086.stdout: 2024-06-14T22:41:17.611 INFO:journalctl@ceph.mon.c.smithi086.stdout: [mds.foo.smithi137.yjevpg{-1:24445} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.137:6832/1266342281,v1:172.21.15.137:6833/1266342281] compat {c=[1],r=[1],i=[7ff]}] 2024-06-14T22:41:17.611 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: mon.c@-1(???).osd e82 crush map has features 3314933000854323200, adjusting msgr requires 2024-06-14T22:41:17.611 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: mon.c@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-06-14T22:41:17.611 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: mon.c@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-06-14T22:41:17.611 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: mon.c@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-06-14T22:41:17.611 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 ceph-mon[102754]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-06-14T22:41:17.611 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 bash[102716]: 1c4e71d265f00f50c1d0b7e3a7f4f1c3dfeb1ab75de74b98ded9b1b91b7dd71b 2024-06-14T22:41:17.611 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:17 smithi086 systemd[1]: Started Ceph mon.c for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:41:18.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 101771 101755 111.554 71.375 0.0157457 0.00895955 2024-06-14T22:41:18.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:18 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:18.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:19.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 16 102920 102904 110.869 71.8125 0.0171088 0.00901458 2024-06-14T22:41:20.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 104069 104053 110.207 71.8125 0.0102752 0.00906868 2024-06-14T22:41:21.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:41:21.150715+0000 min lat: 0.00116587 max lat: 0.439427 avg lat: 0.00912161 2024-06-14T22:41:21.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:41:21.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 105221 105205 109.57 72 0.020421 0.00912161 2024-06-14T22:41:22.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 106338 106322 108.918 69.8125 0.0126397 0.00917469 2024-06-14T22:41:22.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:22 smithi086 ceph-mon[97913]: pgmap v54: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-14T22:41:22.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:22 smithi086 ceph-mon[97913]: mgrmap e39: y(active, since 92s), standbys: x 2024-06-14T22:41:22.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3733400538' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:22.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:22 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:41:22.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:22 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:41:22.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:22 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:41:22.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:22 smithi137 ceph-mon[32951]: pgmap v54: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-14T22:41:22.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:22 smithi137 ceph-mon[32951]: mgrmap e39: y(active, since 92s), standbys: x 2024-06-14T22:41:22.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:22 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3733400538' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:22.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:22 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:41:22.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:22 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:41:22.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:22 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:41:23.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 107304 107288 108.135 60.375 0.0167708 0.00924278 2024-06-14T22:41:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: mon.c calling monitor election 2024-06-14T22:41:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: mon.b calling monitor election 2024-06-14T22:41:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: mon.a calling monitor election 2024-06-14T22:41:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: monmap e3: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: osdmap e82: 8 total, 8 up, 8 in 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: mgrmap e39: y(active, since 101s), standbys: x 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[97913]: overall HEALTH_OK 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: pgmap v54: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: mgrmap e39: y(active, since 92s), standbys: x 2024-06-14T22:41:23.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3733400538' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: mon.c calling monitor election 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: mon.b calling monitor election 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: mon.a calling monitor election 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: monmap e3: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: osdmap e82: 8 total, 8 up, 8 in 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: mgrmap e39: y(active, since 101s), standbys: x 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:41:23.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:23 smithi086 ceph-mon[102754]: overall HEALTH_OK 2024-06-14T22:41:23.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:23.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: mon.c calling monitor election 2024-06-14T22:41:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: mon.b calling monitor election 2024-06-14T22:41:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: mon.a calling monitor election 2024-06-14T22:41:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:41:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: monmap e3: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:41:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:41:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: osdmap e82: 8 total, 8 up, 8 in 2024-06-14T22:41:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: mgrmap e39: y(active, since 101s), standbys: x 2024-06-14T22:41:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-06-14T22:41:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: Cluster is now healthy 2024-06-14T22:41:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:23 smithi137 ceph-mon[32951]: overall HEALTH_OK 2024-06-14T22:41:24.146 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 108324 108308 107.43 63.75 0.01231 0.00930319 2024-06-14T22:41:25.147 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 109339 109323 106.743 63.4375 0.0124159 0.0093622 2024-06-14T22:41:25.311 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:41:24] "GET /metrics HTTP/1.1" 200 36213 "" "Prometheus/2.43.0" 2024-06-14T22:41:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable)": 2 2024-06-14T22:41:25.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:25 smithi086 ceph-mon[97913]: pgmap v59: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-06-14T22:41:25.562 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:25 smithi086 ceph-mon[102754]: pgmap v59: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-06-14T22:41:25.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:25 smithi137 ceph-mon[32951]: pgmap v59: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-06-14T22:41:26.147 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 110208 110192 105.936 54.3125 0.00624356 0.00943505 2024-06-14T22:41:26.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch ps' 2024-06-14T22:41:26.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3684633267' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:26.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3684633267' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:26.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:26 smithi137 ceph-mon[32951]: from='client.? 172.21.15.86:0/3684633267' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:41:27.148 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 111043 111027 105.12 52.1875 0.0328656 0.00950827 2024-06-14T22:41:27.408 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:27 smithi086 ceph-mon[97913]: pgmap v60: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 908 op/s 2024-06-14T22:41:27.408 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:27.409 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:27 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:41:27.264+0000 7f48e8f21640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-14T22:41:27.409 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:27 smithi086 ceph-mon[102754]: pgmap v60: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 908 op/s 2024-06-14T22:41:27.409 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:27.666 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:41:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:27 smithi137 ceph-mon[32951]: pgmap v60: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 908 op/s 2024-06-14T22:41:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:27 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:28.148 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 112023 112007 104.465 61.25 0.0069568 0.00956821 2024-06-14T22:41:28.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:28 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:28.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:29.148 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 113067 113051 103.888 65.25 0.0077727 0.00962057 2024-06-14T22:41:29.476 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:29 smithi086 ceph-mon[97913]: pgmap v61: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 908 op/s 2024-06-14T22:41:29.476 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:29 smithi086 ceph-mon[102754]: pgmap v61: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 908 op/s 2024-06-14T22:41:29.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:29 smithi137 ceph-mon[32951]: pgmap v61: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 908 op/s 2024-06-14T22:41:30.148 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 114060 114044 103.282 62.0625 0.0361012 0.00967724 2024-06-14T22:41:30.680 INFO:teuthology.orchestra.run.smithi086.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:alertmanager.a smithi086 *:9093,9094 running (3m) 0s ago 7m 26.4M - 0.25.0 c8568f914cd2 1c9917382faa 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:grafana.a smithi137 *:3000 running (3m) 25s ago 7m 88.4M - 9.4.7 954c08fa6188 12b0617a16e5 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi086.nhdjdb smithi086 running (6m) 0s ago 6m 16.9M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 2bd94d6f47de 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi137.yjevpg smithi137 running (6m) 25s ago 6m 13.4M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4151acede02a 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.x smithi137 *:8443,9283,8765 running (2m) 25s ago 11m 439M - 18.2.2-1443-g6f4b0956 e7282a3bffba 086e4706077c 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.y smithi086 *:8443,9283,8765 running (2m) 0s ago 13m 485M - 18.2.2-1443-g6f4b0956 e7282a3bffba edafb7b2b182 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:mon.a smithi086 running (118s) 0s ago 13m 40.2M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 0fe78616d3a2 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:mon.b smithi137 running (12m) 25s ago 12m 46.4M 2048M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 bd4aed27097c 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:mon.c smithi086 running (13s) 0s ago 11m 22.2M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 1c4e71d265f0 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.a smithi086 *:9100 running (3m) 0s ago 7m 15.1M - 1.5.0 0da6a335fe13 d1de50e60879 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.b smithi137 *:9100 running (3m) 25s ago 7m 8493k - 1.5.0 0da6a335fe13 b8c1542255f2 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0 smithi086 running (11m) 0s ago 11m 355M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 129bff24d5cc 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1 smithi086 running (10m) 0s ago 10m 293M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 88baee12d7cb 2024-06-14T22:41:30.681 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2 smithi086 running (10m) 0s ago 10m 400M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 fbf5ce68d6eb 2024-06-14T22:41:30.682 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3 smithi086 running (10m) 0s ago 10m 406M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 bee260f44ae7 2024-06-14T22:41:30.682 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4 smithi137 running (9m) 25s ago 9m 343M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4b2729030303 2024-06-14T22:41:30.682 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5 smithi137 running (9m) 25s ago 9m 371M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 e01c2182297f 2024-06-14T22:41:30.682 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6 smithi137 running (9m) 25s ago 9m 226M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 7210ba0a183d 2024-06-14T22:41:30.682 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7 smithi137 running (8m) 25s ago 8m 289M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 a9ca11c000f2 2024-06-14T22:41:30.682 INFO:teuthology.orchestra.run.smithi086.stdout:prometheus.a smithi137 *:9095 running (2m) 25s ago 8m 51.2M - 2.43.0 a07b618ecd1d 1604035f49df 2024-06-14T22:41:31.149 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 114937 114921 102.589 54.8125 0.0104838 0.00974283 2024-06-14T22:41:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:31 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:31 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:31.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:31 smithi086 ceph-mon[97913]: pgmap v62: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-14T22:41:31.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:31 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:31.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:31 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:31.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:31 smithi086 ceph-mon[102754]: pgmap v62: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-14T22:41:31.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch upgrade pause' 2024-06-14T22:41:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:31 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:31 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:31.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:31 smithi137 ceph-mon[32951]: pgmap v62: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-14T22:41:32.149 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 115862 115846 101.959 57.8125 0.0167416 0.0098028 2024-06-14T22:41:32.437 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:32 smithi086 ceph-mon[97913]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:41:32.437 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:32.437 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:32.438 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:32 smithi086 ceph-mon[102754]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:41:32.438 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:32.438 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:32.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:32 smithi137 ceph-mon[32951]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:41:32.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:32 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:32.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:32 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:33.149 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 116833 116817 101.385 60.6875 0.0192061 0.00985483 2024-06-14T22:41:33.485 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:41:33.489 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:33 smithi137 ceph-mon[32951]: pgmap v63: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s 2024-06-14T22:41:33.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:33 smithi086 ceph-mon[97913]: pgmap v63: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s 2024-06-14T22:41:33.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:33 smithi086 ceph-mon[102754]: pgmap v63: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s 2024-06-14T22:41:33.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:33 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:33.504Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:34.150 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 117823 117807 100.844 61.875 0.0146015 0.00991141 2024-06-14T22:41:35.149 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 16 118795 118779 100.302 60.75 0.0108155 0.0099651 2024-06-14T22:41:35.191 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:41:34] "GET /metrics HTTP/1.1" 200 36210 "" "Prometheus/2.43.0" 2024-06-14T22:41:35.443 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:35 smithi086 ceph-mon[102754]: pgmap v64: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T22:41:35.444 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:35 smithi086 ceph-mon[97913]: pgmap v64: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T22:41:35.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:35 smithi137 ceph-mon[32951]: pgmap v64: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T22:41:36.149 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 119707 119691 99.7244 57 0.0137653 0.0100224 2024-06-14T22:41:37.048 INFO:teuthology.orchestra.run.smithi086.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:41:37.150 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 120599 120583 99.1458 55.75 0.0101997 0.0100808 2024-06-14T22:41:37.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:37 smithi086 ceph-mon[102754]: pgmap v65: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 50 MiB/s wr, 794 op/s 2024-06-14T22:41:37.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:37 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:37.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:37 smithi086 ceph-mon[97913]: pgmap v65: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 50 MiB/s wr, 794 op/s 2024-06-14T22:41:37.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:37 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:37.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:37 smithi137 ceph-mon[32951]: pgmap v65: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 50 MiB/s wr, 794 op/s 2024-06-14T22:41:37.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:37 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:37.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'sleep 60' 2024-06-14T22:41:38.150 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 16 121535 121519 98.6179 58.5 0.0174927 0.0101349 2024-06-14T22:41:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:38 smithi086 ceph-mon[97913]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:41:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:38 smithi086 ceph-mon[97913]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:41:38.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:38 smithi086 ceph-mon[102754]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:41:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:38 smithi086 ceph-mon[102754]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:41:38.641 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:38 smithi137 ceph-mon[32951]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:41:38.641 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:38 smithi137 ceph-mon[32951]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:41:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:41:38.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:38.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:39.150 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 122426 122410 98.0675 55.6875 0.0220017 0.0101917 2024-06-14T22:41:39.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: pgmap v66: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 50 MiB/s wr, 794 op/s 2024-06-14T22:41:39.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:39.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:41:39.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:41:39.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:39.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: pgmap v66: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 50 MiB/s wr, 794 op/s 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:41:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: pgmap v66: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 50 MiB/s wr, 794 op/s 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:41:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:41:40.150 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 123420 123404 97.6119 62.125 0.0216216 0.0102395 2024-06-14T22:41:41.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:41:41.155006+0000 min lat: 0.00116587 max lat: 0.439427 avg lat: 0.0102855 2024-06-14T22:41:41.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:41:41.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 124430 124414 97.1808 63.125 0.0120401 0.0102855 2024-06-14T22:41:41.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:41 smithi137 ceph-mon[32951]: pgmap v67: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-14T22:41:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:41 smithi086 ceph-mon[102754]: pgmap v67: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-14T22:41:41.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:41 smithi086 ceph-mon[97913]: pgmap v67: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-14T22:41:42.150 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 125431 125415 96.7534 62.5625 0.014476 0.0103306 2024-06-14T22:41:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:42.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:42.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:43.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 126399 126383 96.3112 60.5 0.012105 0.0103778 2024-06-14T22:41:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:43 smithi137 ceph-mon[32951]: pgmap v68: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 792 op/s 2024-06-14T22:41:43.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:43.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:43.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:43 smithi086 ceph-mon[97913]: pgmap v68: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 792 op/s 2024-06-14T22:41:43.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:43 smithi086 ceph-mon[102754]: pgmap v68: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 792 op/s 2024-06-14T22:41:44.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 16 127403 127387 95.9068 62.75 0.0265299 0.0104216 2024-06-14T22:41:45.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 128396 128380 95.5039 62.0625 0.0154123 0.0104661 2024-06-14T22:41:45.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:41:44] "GET /metrics HTTP/1.1" 200 36213 "" "Prometheus/2.43.0" 2024-06-14T22:41:45.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:45 smithi137 ceph-mon[32951]: pgmap v69: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-14T22:41:45.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:45 smithi086 ceph-mon[97913]: pgmap v69: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-14T22:41:45.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:45 smithi086 ceph-mon[102754]: pgmap v69: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-14T22:41:46.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 129392 129376 95.1127 62.25 0.00758143 0.0105085 2024-06-14T22:41:47.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 130369 130353 94.7167 61.0625 0.0114814 0.0105527 2024-06-14T22:41:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:47 smithi137 ceph-mon[32951]: pgmap v70: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 50 MiB/s wr, 797 op/s 2024-06-14T22:41:47.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:47 smithi086 ceph-mon[97913]: pgmap v70: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 50 MiB/s wr, 797 op/s 2024-06-14T22:41:47.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:47 smithi086 ceph-mon[102754]: pgmap v70: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 50 MiB/s wr, 797 op/s 2024-06-14T22:41:48.154 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 131337 131321 94.3234 60.5 0.0118088 0.010597 2024-06-14T22:41:48.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:48.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:49.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 132291 132275 93.929 59.625 0.00852776 0.0106412 2024-06-14T22:41:49.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:49 smithi137 ceph-mon[32951]: pgmap v71: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 50 MiB/s wr, 797 op/s 2024-06-14T22:41:49.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:49 smithi086 ceph-mon[97913]: pgmap v71: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 50 MiB/s wr, 797 op/s 2024-06-14T22:41:49.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:49 smithi086 ceph-mon[102754]: pgmap v71: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 50 MiB/s wr, 797 op/s 2024-06-14T22:41:50.151 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 133259 133243 93.5534 60.5 0.0145044 0.0106846 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0063 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 134215 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 93.1983 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 46.5352 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 271.625 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 52.1875 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1491 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 744.563 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 4346 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 835 2024-06-14T22:41:51.152 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0107257 2024-06-14T22:41:51.153 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.00880053 2024-06-14T22:41:51.153 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 0.439427 2024-06-14T22:41:51.153 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00116587 2024-06-14T22:41:51.157 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T22:41:51.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:51 smithi137 ceph-mon[32951]: pgmap v72: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-14T22:41:51.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:51 smithi086 ceph-mon[97913]: pgmap v72: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-14T22:41:51.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:51 smithi086 ceph-mon[102754]: pgmap v72: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-14T22:41:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:53 smithi137 ceph-mon[32951]: pgmap v73: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 817 op/s 2024-06-14T22:41:53.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:53.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:53.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:53 smithi086 ceph-mon[97913]: pgmap v73: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 817 op/s 2024-06-14T22:41:53.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:53 smithi086 ceph-mon[102754]: pgmap v73: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 51 MiB/s wr, 817 op/s 2024-06-14T22:41:55.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:41:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:41:54] "GET /metrics HTTP/1.1" 200 36213 "" "Prometheus/2.43.0" 2024-06-14T22:41:55.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:55 smithi137 ceph-mon[32951]: pgmap v74: 89 pgs: 89 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 65 MiB/s wr, 1.23k op/s 2024-06-14T22:41:55.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:55 smithi086 ceph-mon[97913]: pgmap v74: 89 pgs: 89 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 65 MiB/s wr, 1.23k op/s 2024-06-14T22:41:55.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:55 smithi086 ceph-mon[102754]: pgmap v74: 89 pgs: 89 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 65 MiB/s wr, 1.23k op/s 2024-06-14T22:41:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:57 smithi137 ceph-mon[32951]: pgmap v75: 89 pgs: 89 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 40 MiB/s wr, 819 op/s 2024-06-14T22:41:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:57 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:57 smithi086 ceph-mon[97913]: pgmap v75: 89 pgs: 89 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 40 MiB/s wr, 819 op/s 2024-06-14T22:41:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:57.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:57 smithi086 ceph-mon[102754]: pgmap v75: 89 pgs: 89 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 40 MiB/s wr, 819 op/s 2024-06-14T22:41:57.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:41:58.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:41:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:41:58.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:41:59.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:41:59 smithi137 ceph-mon[32951]: pgmap v76: 89 pgs: 89 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 40 MiB/s wr, 819 op/s 2024-06-14T22:41:59.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:41:59 smithi086 ceph-mon[97913]: pgmap v76: 89 pgs: 89 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 40 MiB/s wr, 819 op/s 2024-06-14T22:41:59.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:41:59 smithi086 ceph-mon[102754]: pgmap v76: 89 pgs: 89 active+clean; 8.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 40 MiB/s wr, 819 op/s 2024-06-14T22:42:01.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:01 smithi137 ceph-mon[32951]: pgmap v77: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 40 MiB/s wr, 1.21k op/s 2024-06-14T22:42:01.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:01 smithi086 ceph-mon[97913]: pgmap v77: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 40 MiB/s wr, 1.21k op/s 2024-06-14T22:42:01.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:01 smithi086 ceph-mon[102754]: pgmap v77: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 40 MiB/s wr, 1.21k op/s 2024-06-14T22:42:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:03 smithi137 ceph-mon[32951]: pgmap v78: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 14 MiB/s wr, 797 op/s 2024-06-14T22:42:03.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:03.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:03.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:03 smithi086 ceph-mon[97913]: pgmap v78: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 14 MiB/s wr, 797 op/s 2024-06-14T22:42:03.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:03 smithi086 ceph-mon[102754]: pgmap v78: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 14 MiB/s wr, 797 op/s 2024-06-14T22:42:05.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:42:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:42:04] "GET /metrics HTTP/1.1" 200 36228 "" "Prometheus/2.43.0" 2024-06-14T22:42:05.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:05 smithi137 ceph-mon[32951]: pgmap v79: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 14 MiB/s wr, 1.21k op/s 2024-06-14T22:42:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:05 smithi086 ceph-mon[97913]: pgmap v79: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 14 MiB/s wr, 1.21k op/s 2024-06-14T22:42:05.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:05 smithi086 ceph-mon[102754]: pgmap v79: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 14 MiB/s wr, 1.21k op/s 2024-06-14T22:42:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:07 smithi137 ceph-mon[32951]: pgmap v80: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-06-14T22:42:07.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:07 smithi086 ceph-mon[102754]: pgmap v80: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-06-14T22:42:07.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:07 smithi086 ceph-mon[97913]: pgmap v80: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-06-14T22:42:08.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:08 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:08.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:09.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:09 smithi137 ceph-mon[32951]: pgmap v81: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-06-14T22:42:09.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:09 smithi086 ceph-mon[102754]: pgmap v81: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-06-14T22:42:09.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:09 smithi086 ceph-mon[97913]: pgmap v81: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-06-14T22:42:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:11 smithi137 ceph-mon[32951]: pgmap v82: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-06-14T22:42:11.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:11 smithi086 ceph-mon[102754]: pgmap v82: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-06-14T22:42:11.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:11 smithi086 ceph-mon[97913]: pgmap v82: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-06-14T22:42:12.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:12 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:12.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:13.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:13 smithi137 ceph-mon[32951]: pgmap v83: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 807 op/s 2024-06-14T22:42:13.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:13 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:13.504Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:13.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:13 smithi086 ceph-mon[102754]: pgmap v83: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 807 op/s 2024-06-14T22:42:13.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:13 smithi086 ceph-mon[97913]: pgmap v83: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 807 op/s 2024-06-14T22:42:15.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:42:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:42:14] "GET /metrics HTTP/1.1" 200 36226 "" "Prometheus/2.43.0" 2024-06-14T22:42:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:15 smithi137 ceph-mon[32951]: pgmap v84: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-06-14T22:42:15.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:15 smithi086 ceph-mon[102754]: pgmap v84: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-06-14T22:42:15.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:15 smithi086 ceph-mon[97913]: pgmap v84: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-06-14T22:42:17.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:17 smithi137 ceph-mon[32951]: pgmap v85: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-06-14T22:42:17.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:17 smithi086 ceph-mon[102754]: pgmap v85: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-06-14T22:42:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:17 smithi086 ceph-mon[97913]: pgmap v85: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-06-14T22:42:18.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:18 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:18.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:19.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:19 smithi137 ceph-mon[32951]: pgmap v86: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-06-14T22:42:19.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:19 smithi086 ceph-mon[102754]: pgmap v86: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-06-14T22:42:19.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:19 smithi086 ceph-mon[97913]: pgmap v86: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-06-14T22:42:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:21 smithi137 ceph-mon[32951]: pgmap v87: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-06-14T22:42:21.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:21 smithi086 ceph-mon[102754]: pgmap v87: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-06-14T22:42:21.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:21 smithi086 ceph-mon[97913]: pgmap v87: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-06-14T22:42:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:23 smithi137 ceph-mon[32951]: pgmap v88: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-06-14T22:42:23.740 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:23.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:23.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:23 smithi086 ceph-mon[102754]: pgmap v88: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-06-14T22:42:23.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:23 smithi086 ceph-mon[97913]: pgmap v88: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-06-14T22:42:25.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:42:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:42:24] "GET /metrics HTTP/1.1" 200 36226 "" "Prometheus/2.43.0" 2024-06-14T22:42:25.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:25 smithi137 ceph-mon[32951]: pgmap v89: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.19k op/s 2024-06-14T22:42:25.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:25 smithi086 ceph-mon[97913]: pgmap v89: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.19k op/s 2024-06-14T22:42:25.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:25 smithi086 ceph-mon[102754]: pgmap v89: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.19k op/s 2024-06-14T22:42:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:27 smithi137 ceph-mon[32951]: pgmap v90: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 791 op/s 2024-06-14T22:42:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:27 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:27 smithi086 ceph-mon[97913]: pgmap v90: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 791 op/s 2024-06-14T22:42:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:27 smithi086 ceph-mon[102754]: pgmap v90: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 791 op/s 2024-06-14T22:42:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:28.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:28 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:28.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:29.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:29 smithi137 ceph-mon[32951]: pgmap v91: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 791 op/s 2024-06-14T22:42:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:29 smithi086 ceph-mon[97913]: pgmap v91: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 791 op/s 2024-06-14T22:42:29.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:29 smithi086 ceph-mon[102754]: pgmap v91: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 791 op/s 2024-06-14T22:42:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:31 smithi137 ceph-mon[32951]: pgmap v92: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-06-14T22:42:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:31 smithi086 ceph-mon[102754]: pgmap v92: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-06-14T22:42:31.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:31 smithi086 ceph-mon[97913]: pgmap v92: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-06-14T22:42:33.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:33 smithi137 ceph-mon[32951]: pgmap v93: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 786 op/s 2024-06-14T22:42:33.742 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:33 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:33.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:33.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:33 smithi086 ceph-mon[102754]: pgmap v93: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 786 op/s 2024-06-14T22:42:33.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:33 smithi086 ceph-mon[97913]: pgmap v93: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 786 op/s 2024-06-14T22:42:35.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:42:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:42:34] "GET /metrics HTTP/1.1" 200 36227 "" "Prometheus/2.43.0" 2024-06-14T22:42:35.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:35 smithi137 ceph-mon[32951]: pgmap v94: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-06-14T22:42:35.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:35 smithi086 ceph-mon[97913]: pgmap v94: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-06-14T22:42:35.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:35 smithi086 ceph-mon[102754]: pgmap v94: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-06-14T22:42:37.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:37 smithi137 ceph-mon[32951]: pgmap v95: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 785 op/s 2024-06-14T22:42:37.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:37 smithi086 ceph-mon[97913]: pgmap v95: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 785 op/s 2024-06-14T22:42:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:37 smithi086 ceph-mon[102754]: pgmap v95: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 785 op/s 2024-06-14T22:42:38.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:38.663Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:39 smithi137 ceph-mon[32951]: pgmap v96: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 785 op/s 2024-06-14T22:42:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:39 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:42:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:39 smithi086 ceph-mon[97913]: pgmap v96: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 785 op/s 2024-06-14T22:42:39.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:42:39.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:39 smithi086 ceph-mon[102754]: pgmap v96: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 785 op/s 2024-06-14T22:42:39.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:42:41.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch upgrade resume' 2024-06-14T22:42:41.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:41 smithi137 ceph-mon[32951]: pgmap v97: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-06-14T22:42:41.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:41 smithi086 ceph-mon[97913]: pgmap v97: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-06-14T22:42:41.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:41 smithi086 ceph-mon[102754]: pgmap v97: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-06-14T22:42:42.606 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:42:42.759 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:42.759 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:42 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:43.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:43 smithi086 ceph-mon[102754]: pgmap v98: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 784 op/s 2024-06-14T22:42:43.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:43 smithi086 ceph-mon[97913]: pgmap v98: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 784 op/s 2024-06-14T22:42:43.890 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:43 smithi137 ceph-mon[32951]: pgmap v98: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 784 op/s 2024-06-14T22:42:43.891 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:43.504Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:45.121 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:42:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:42:44] "GET /metrics HTTP/1.1" 200 36225 "" "Prometheus/2.43.0" 2024-06-14T22:42:45.631 INFO:teuthology.orchestra.run.smithi086.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:42:45.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:45 smithi086 ceph-mon[97913]: pgmap v99: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:42:45.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:45 smithi086 ceph-mon[102754]: pgmap v99: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:42:45.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:45 smithi137 ceph-mon[32951]: pgmap v99: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:42:46.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'echo wait for all mons to upgrade' 2024-06-14T22:42:46.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[97913]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:42:46.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[97913]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[97913]: pgmap v100: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[102754]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[102754]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:46.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:46 smithi086 ceph-mon[102754]: pgmap v100: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-06-14T22:42:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:46 smithi137 ceph-mon[32951]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:42:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:46 smithi137 ceph-mon[32951]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:42:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:46 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:46 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:46 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:46 smithi137 ceph-mon[32951]: pgmap v100: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-06-14T22:42:47.326 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:42:48.656 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:48.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:48.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:42:48.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:42:48.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:48.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:42:48.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:42:48.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-14T22:42:48.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-06-14T22:42:48.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:48.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:48.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:42:48.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:42:48.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:48.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:42:48.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:42:48.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-14T22:42:48.702 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-06-14T22:42:48.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:48.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:48.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:42:48.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:42:48.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:48.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:42:48.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:42:48.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-14T22:42:48.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-06-14T22:42:48.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:48.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:49.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:49 smithi137 ceph-mon[32951]: Upgrade: It appears safe to stop mon.b 2024-06-14T22:42:49.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:49 smithi137 ceph-mon[32951]: pgmap v101: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-06-14T22:42:49.749 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:49 smithi086 ceph-mon[102754]: Upgrade: It appears safe to stop mon.b 2024-06-14T22:42:49.749 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:49 smithi086 ceph-mon[102754]: pgmap v101: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-06-14T22:42:49.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:49 smithi086 ceph-mon[97913]: Upgrade: It appears safe to stop mon.b 2024-06-14T22:42:49.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:49 smithi086 ceph-mon[97913]: pgmap v101: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-06-14T22:42:50.413 INFO:teuthology.orchestra.run.smithi086.stdout:wait for all mons to upgrade 2024-06-14T22:42:50.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:50 smithi137 ceph-mon[32951]: Upgrade: Updating mon.b 2024-06-14T22:42:50.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:50 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:50.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:50 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:42:50.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:50 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:42:50.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:50 smithi137 ceph-mon[32951]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:42:50.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:50 smithi137 ceph-mon[32951]: Deploying daemon mon.b on smithi137 2024-06-14T22:42:50.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:50 smithi137 ceph-mon[32951]: pgmap v102: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-06-14T22:42:51.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[97913]: Upgrade: Updating mon.b 2024-06-14T22:42:51.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:51.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:42:51.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:42:51.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:42:51.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[97913]: Deploying daemon mon.b on smithi137 2024-06-14T22:42:51.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[97913]: pgmap v102: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-06-14T22:42:51.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[102754]: Upgrade: Updating mon.b 2024-06-14T22:42:51.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:51.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:42:51.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:42:51.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:42:51.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[102754]: Deploying daemon mon.b on smithi137 2024-06-14T22:42:51.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:50 smithi086 ceph-mon[102754]: pgmap v102: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-06-14T22:42:51.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'while ! ceph mon versions | grep $sha1 | grep '"'"': 3'"'"' ; do sleep 2 ; done' 2024-06-14T22:42:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:42:52.413 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:52 smithi137 systemd[1]: Stopping Ceph mon.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:42:52.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b[32916]: 2024-06-14T22:42:52.412+0000 7fc555e4c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T22:42:52.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b[32916]: 2024-06-14T22:42:52.412+0000 7fc555e4c640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-06-14T22:42:52.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:52 smithi137 podman[90457]: 2024-06-14 22:42:52.53276641 +0000 UTC m=+0.254034664 container died bd4aed27097c3e761c223fe064ebccaee77a0e06b62d1f222bbbcadb7da718cc (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, CEPH_POINT_RELEASE=) 2024-06-14T22:42:53.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:53 smithi137 podman[90457]: 2024-06-14 22:42:53.222996771 +0000 UTC m=+0.944265024 container cleanup bd4aed27097c3e761c223fe064ebccaee77a0e06b62d1f222bbbcadb7da718cc (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-14T22:42:53.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:53 smithi137 bash[90457]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b 2024-06-14T22:42:53.991 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:53.504Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:53 smithi137 podman[90474]: 2024-06-14 22:42:53.524972883 +0000 UTC m=+0.986736360 container remove bd4aed27097c3e761c223fe064ebccaee77a0e06b62d1f222bbbcadb7da718cc (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-14T22:42:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:53 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.b.service: Deactivated successfully. 2024-06-14T22:42:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:53 smithi137 systemd[1]: Stopped Ceph mon.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:42:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:53 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.b.service: Consumed 9.990s CPU time. 2024-06-14T22:42:54.292 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 systemd[1]: Starting Ceph mon.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:42:54.624 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 podman[90599]: 2024-06-14 22:42:54.205049698 +0000 UTC m=+0.037044849 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:42:54.625 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 podman[90599]: 2024-06-14 22:42:54.350357614 +0000 UTC m=+0.182352747 container create 74ce1e795ab2fef3fa15149bb35f25a1ca9aa8f16b8098d2abcadfc30b9b4ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-14T22:42:54.683 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 podman[90599]: 2024-06-14 22:42:54.628366632 +0000 UTC m=+0.460361773 container init 74ce1e795ab2fef3fa15149bb35f25a1ca9aa8f16b8098d2abcadfc30b9b4ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, ceph=True, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528) 2024-06-14T22:42:54.683 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 podman[90599]: 2024-06-14 22:42:54.636858465 +0000 UTC m=+0.468853598 container start 74ce1e795ab2fef3fa15149bb35f25a1ca9aa8f16b8098d2abcadfc30b9b4ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-14T22:42:54.683 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: set uid:gid to 167:167 (ceph:ceph) 2024-06-14T22:42:54.683 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable), process ceph-mon, pid 6 2024-06-14T22:42:54.683 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: pidfile_write: ignore empty --pid-file 2024-06-14T22:42:54.683 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: load: jerasure load: lrc 2024-06-14T22:42:54.683 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: RocksDB version: 7.9.2 2024-06-14T22:42:54.683 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Git sha 0 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Compile date 2024-06-08 16:28:17 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: DB SUMMARY 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: DB Session ID: WFZICKK355ZB99779Q8W 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: CURRENT file: CURRENT 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: IDENTITY file: IDENTITY 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1093 Bytes 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 5810984 ; 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.error_if_exists: 0 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.create_if_missing: 0 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.paranoid_checks: 1 2024-06-14T22:42:54.684 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.env: 0x55c271409ba0 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.fs: PosixFileSystem 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.info_log: 0x55c273914be0 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_file_opening_threads: 16 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.statistics: (nil) 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.use_fsync: 0 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_log_file_size: 0 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.keep_log_file_num: 1000 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.recycle_log_file_num: 0 2024-06-14T22:42:54.685 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.allow_fallocate: 1 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.allow_mmap_reads: 0 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.allow_mmap_writes: 0 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.use_direct_reads: 0 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.create_missing_column_families: 0 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.db_log_dir: 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.wal_dir: 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-14T22:42:54.686 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.advise_random_on_open: 1 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.db_write_buffer_size: 0 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.write_buffer_manager: 0x55c2738e6640 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.rate_limiter: (nil) 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.wal_recovery_mode: 2 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.enable_thread_tracking: 0 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.enable_pipelined_write: 0 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.unordered_write: 0 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-14T22:42:54.687 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.row_cache: None 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.wal_filter: None 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.allow_ingest_behind: 0 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.two_write_queues: 0 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.manual_wal_flush: 0 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.wal_compression: 0 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.atomic_flush: 0 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-14T22:42:54.688 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.log_readahead_size: 0 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.best_efforts_recovery: 0 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.allow_data_in_errors: 0 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.db_host_id: __hostname__ 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_background_jobs: 2 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_background_compactions: -1 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_subcompactions: 1 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-14T22:42:54.689 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_total_wal_size: 0 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_open_files: -1 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bytes_per_sync: 0 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_readahead_size: 0 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_background_flushes: -1 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Compression algorithms supported: 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: kZSTD supported: 0 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: kXpressCompression supported: 0 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: kBZip2Compression supported: 0 2024-06-14T22:42:54.690 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: kLZ4Compression supported: 1 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: kZlibCompression supported: 1 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: kLZ4HCCompression supported: 1 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: kSnappyCompression supported: 1 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.merge_operator: 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_filter: None 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_filter_factory: None 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.sst_partitioner_factory: None 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-14T22:42:54.691 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c273914820) 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: cache_index_and_filter_blocks: 1 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: pin_top_level_index_and_filter: 1 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: index_type: 0 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: data_block_index_type: 0 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: index_shortening: 1 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: checksum: 4 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: no_block_cache: 0 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: block_cache: 0x55c27390b350 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: block_cache_name: BinnedLRUCache 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: block_cache_options: 2024-06-14T22:42:54.692 INFO:journalctl@ceph.mon.b.smithi137.stdout: capacity : 536870912 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: num_shard_bits : 4 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: strict_capacity_limit : 0 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: high_pri_pool_ratio: 0.000 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: block_cache_compressed: (nil) 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: persistent_cache: (nil) 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: block_size: 4096 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: block_size_deviation: 10 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: block_restart_interval: 16 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: index_block_restart_interval: 1 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: metadata_block_size: 4096 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: partition_filters: 0 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: use_delta_encoding: 1 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: filter_policy: bloomfilter 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: whole_key_filtering: 1 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: verify_compression: 0 2024-06-14T22:42:54.693 INFO:journalctl@ceph.mon.b.smithi137.stdout: read_amp_bytes_per_bit: 0 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout: format_version: 5 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout: enable_index_compression: 1 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout: block_align: 0 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout: max_auto_readahead_size: 262144 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout: prepopulate_block_cache: 0 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout: initial_auto_readahead_size: 8192 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout: num_file_reads_for_auto_readahead: 2 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.write_buffer_size: 33554432 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_write_buffer_number: 2 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression: NoCompression 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression: Disabled 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.prefix_extractor: nullptr 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-14T22:42:54.694 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.num_levels: 7 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-14T22:42:54.695 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression_opts.level: 32767 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression_opts.strategy: 0 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression_opts.enabled: false 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.target_file_size_base: 67108864 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-14T22:42:54.696 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-14T22:42:54.697 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-14T22:42:54.697 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-14T22:42:54.697 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-14T22:42:54.697 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-14T22:42:54.697 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-14T22:42:54.697 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-14T22:42:54.697 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-14T22:42:54.697 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.arena_block_size: 1048576 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.disable_auto_compactions: 0 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-14T22:42:54.698 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.table_properties_collectors: 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.inplace_update_support: 0 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.bloom_locality: 0 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.max_successive_merges: 0 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.paranoid_file_checks: 0 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.force_consistency_checks: 1 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.report_bg_io_stats: 0 2024-06-14T22:42:54.699 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.ttl: 2592000 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.enable_blob_files: false 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.min_blob_size: 0 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.blob_file_size: 268435456 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.blob_file_starting_level: 0 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-06-14T22:42:54.700 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-14T22:42:54.701 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12865, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-14T22:42:54.701 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-06-14T22:42:54.701 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 75c95a72-bb7d-4999-881a-59c6d1dab1a0 2024-06-14T22:42:54.701 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404974686991, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-06-14T22:42:54.701 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404974708668, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3643425, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12870, "largest_seqno": 15600, "table_properties": {"data_size": 3632527, "index_size": 6933, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2949, "raw_key_size": 24459, "raw_average_key_size": 20, "raw_value_size": 3609293, "raw_average_value_size": 3084, "num_data_blocks": 313, "num_entries": 1170, "num_filter_entries": 1170, "num_deletions": 251, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1718404974, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "75c95a72-bb7d-4999-881a-59c6d1dab1a0", "db_session_id": "WFZICKK355ZB99779Q8W", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718404974708934, "job": 1, "event": "recovery_finished"} 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c273934e00 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: DB pointer 0x55c273a28000 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout: ** DB Stats ** 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-14T22:42:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: ** Compaction Stats [default] ** 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: L0 1/0 3.47 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 368.2 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: L6 1/0 7.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: Sum 2/0 10.82 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 368.2 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 368.2 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: ** Compaction Stats [default] ** 2024-06-14T22:42:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 368.2 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: Cumulative compaction: 0.00 GB write, 88.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: Interval compaction: 0.00 GB write, 88.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-14T22:42:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout: Block cache BinnedLRUCache@0x55c27390b350#6 capacity: 512.00 MB usage: 11.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,3.03 KB,0.000578165%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: starting mon.b rank 1 at public addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] at bind addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: mon.b@-1(???) e3 preinit fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: mon.b@-1(???).mds e9 new map 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: mon.b@-1(???).mds e9 print_map 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: e9 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: legacy client fscid: 1 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: Filesystem 'foo' (1) 2024-06-14T22:42:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout: fs_name foo 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: epoch 8 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: created 2024-06-14T22:34:45.808786+0000 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: modified 2024-06-14T22:35:03.607395+0000 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: tableserver 0 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: root 0 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: session_timeout 60 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: session_autoclose 300 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: max_file_size 1099511627776 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: max_xattr_size 65536 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: required_client_features {} 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: last_failure 0 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: last_failure_osd_epoch 0 2024-06-14T22:42:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: max_mds 1 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: in 0 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: up {0=24548} 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: failed 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: damaged 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: stopped 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: data_pools [4] 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: metadata_pool 3 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: inline_data disabled 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: balancer 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: bal_rank_mask -1 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: standby_count_wanted 1 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: [mds.foo.smithi086.nhdjdb{0:24548} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.86:6834/2283984434,v1:172.21.15.86:6835/2283984434] compat {c=[1],r=[1],i=[7ff]}] 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: 2024-06-14T22:42:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout: Standby daemons: 2024-06-14T22:42:54.997 INFO:journalctl@ceph.mon.b.smithi137.stdout: 2024-06-14T22:42:54.997 INFO:journalctl@ceph.mon.b.smithi137.stdout: [mds.foo.smithi137.yjevpg{-1:24445} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.137:6832/1266342281,v1:172.21.15.137:6833/1266342281] compat {c=[1],r=[1],i=[7ff]}] 2024-06-14T22:42:54.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: mon.b@-1(???).osd e82 crush map has features 3314933000854323200, adjusting msgr requires 2024-06-14T22:42:54.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: mon.b@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-06-14T22:42:54.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: mon.b@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-06-14T22:42:54.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: mon.b@-1(???).osd e82 crush map has features 432629239337189376, adjusting msgr requires 2024-06-14T22:42:54.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 ceph-mon[90634]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-06-14T22:42:54.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 bash[90599]: 74ce1e795ab2fef3fa15149bb35f25a1ca9aa8f16b8098d2abcadfc30b9b4ae6 2024-06-14T22:42:54.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:54 smithi137 systemd[1]: Started Ceph mon.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:42:55.233 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:42:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:42:54] "GET /metrics HTTP/1.1" 200 36225 "" "Prometheus/2.43.0" 2024-06-14T22:42:55.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:55 smithi086 ceph-mon[102754]: pgmap v103: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-06-14T22:42:55.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:55 smithi086 ceph-mon[97913]: pgmap v103: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-06-14T22:42:55.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:55 smithi137 ceph-mon[90634]: pgmap v103: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-06-14T22:42:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: mon.b calling monitor election 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: mon.c calling monitor election 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: mon.a calling monitor election 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: monmap e4: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: osdmap e82: 8 total, 8 up, 8 in 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: mgrmap e39: y(active, since 3m), standbys: x 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: overall HEALTH_OK 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:42:56.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: mon.b calling monitor election 2024-06-14T22:42:56.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: mon.c calling monitor election 2024-06-14T22:42:56.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: mon.a calling monitor election 2024-06-14T22:42:56.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: monmap e4: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: osdmap e82: 8 total, 8 up, 8 in 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: mgrmap e39: y(active, since 3m), standbys: x 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: overall HEALTH_OK 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: mon.b calling monitor election 2024-06-14T22:42:56.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: mon.c calling monitor election 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: mon.a calling monitor election 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: monmap e4: 3 mons at {a=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0],b=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],c=[v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: fsmap foo:1 {0=foo.smithi086.nhdjdb=up:active} 1 up:standby 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: osdmap e82: 8 total, 8 up, 8 in 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: mgrmap e39: y(active, since 3m), standbys: x 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: overall HEALTH_OK 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:42:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:42:57.175 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable)": 3 2024-06-14T22:42:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:57 smithi086 ceph-mon[97913]: pgmap v105: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:42:57.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/902376888' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:42:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:57 smithi086 ceph-mon[102754]: pgmap v105: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:42:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/902376888' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:42:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:57.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:57 smithi137 ceph-mon[90634]: pgmap v105: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:42:57.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/902376888' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-06-14T22:42:57.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:42:58.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch ps' 2024-06-14T22:42:58.654 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:42:58.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:42:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:42:58.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:42:59.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:42:59 smithi086 ceph-mon[102754]: pgmap v106: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:42:59.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:42:59 smithi086 ceph-mon[97913]: pgmap v106: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:42:59.636 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:42:59 smithi137 ceph-mon[90634]: pgmap v106: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:43:00.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:42:59 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:42:59.730+0000 7f48e8f21640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-06-14T22:43:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:00 smithi137 ceph-mon[90634]: pgmap v107: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-06-14T22:43:01.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:00 smithi086 ceph-mon[97913]: pgmap v107: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-06-14T22:43:01.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:00 smithi086 ceph-mon[102754]: pgmap v107: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-06-14T22:43:01.255 INFO:teuthology.orchestra.run.smithi086.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:alertmanager.a smithi086 *:9093,9094 running (4m) 13s ago 9m 26.0M - 0.25.0 c8568f914cd2 1c9917382faa 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:grafana.a smithi137 *:3000 running (4m) 0s ago 9m 83.0M - 9.4.7 954c08fa6188 12b0617a16e5 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi086.nhdjdb smithi086 running (8m) 13s ago 8m 17.2M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 2bd94d6f47de 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi137.yjevpg smithi137 running (8m) 0s ago 8m 13.8M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4151acede02a 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.x smithi137 *:8443,9283,8765 running (3m) 0s ago 13m 441M - 18.2.2-1443-g6f4b0956 e7282a3bffba 086e4706077c 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.y smithi086 *:8443,9283,8765 running (4m) 13s ago 14m 486M - 18.2.2-1443-g6f4b0956 e7282a3bffba edafb7b2b182 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:mon.a smithi086 running (3m) 13s ago 14m 41.2M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 0fe78616d3a2 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:mon.b smithi137 running (7s) 0s ago 13m 22.6M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 74ce1e795ab2 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:mon.c smithi086 running (104s) 13s ago 13m 34.2M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 1c4e71d265f0 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.a smithi086 *:9100 running (4m) 13s ago 8m 15.6M - 1.5.0 0da6a335fe13 d1de50e60879 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.b smithi137 *:9100 running (4m) 0s ago 8m 15.8M - 1.5.0 0da6a335fe13 b8c1542255f2 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0 smithi086 running (12m) 13s ago 12m 385M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 129bff24d5cc 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1 smithi086 running (12m) 13s ago 12m 340M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 88baee12d7cb 2024-06-14T22:43:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2 smithi086 running (12m) 13s ago 12m 474M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 fbf5ce68d6eb 2024-06-14T22:43:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3 smithi086 running (11m) 13s ago 11m 498M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 bee260f44ae7 2024-06-14T22:43:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4 smithi137 running (11m) 0s ago 11m 416M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4b2729030303 2024-06-14T22:43:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5 smithi137 running (10m) 0s ago 10m 445M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 e01c2182297f 2024-06-14T22:43:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6 smithi137 running (10m) 0s ago 10m 311M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 7210ba0a183d 2024-06-14T22:43:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7 smithi137 running (10m) 0s ago 10m 392M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 a9ca11c000f2 2024-06-14T22:43:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:prometheus.a smithi137 *:9095 running (3m) 0s ago 9m 63.0M - 2.43.0 a07b618ecd1d 1604035f49df 2024-06-14T22:43:02.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[97913]: from='client.54100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:43:02.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[102754]: from='client.54100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:43:02.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.196 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.196 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.197 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:01 smithi137 ceph-mon[90634]: from='client.54100 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:43:02.197 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.197 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:02.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch upgrade pause' 2024-06-14T22:43:02.798 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:43:03.173 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:02 smithi086 ceph-mon[97913]: pgmap v108: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 710 op/s 2024-06-14T22:43:03.173 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:02 smithi086 ceph-mon[102754]: pgmap v108: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 710 op/s 2024-06-14T22:43:03.196 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:02 smithi137 ceph-mon[90634]: pgmap v108: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 710 op/s 2024-06-14T22:43:03.978 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:43:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:43:03.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi086\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi086\", job=\"node\", machine=\"x86_64\", nodename=\"smithi086\", release=\"5.14.0-457.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu May 30 22:48:58 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:43:05.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:43:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:43:04] "GET /metrics HTTP/1.1" 200 36153 "" "Prometheus/2.43.0" 2024-06-14T22:43:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:43:05.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:05 smithi137 ceph-mon[90634]: pgmap v109: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-14T22:43:05.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:05 smithi086 ceph-mon[97913]: pgmap v109: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-14T22:43:05.747 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:05 smithi086 ceph-mon[102754]: pgmap v109: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-14T22:43:06.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'sleep 60' 2024-06-14T22:43:06.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='client.44135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:43:06.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:43:06.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:06.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:43:06.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:43:06.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:43:06.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:06.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='client.44135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:43:06.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:43:06.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:06.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:43:06.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='client.44135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:43:06.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:43:06.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:07.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:07 smithi086 ceph-mon[102754]: pgmap v110: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-06-14T22:43:07.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:07 smithi086 ceph-mon[102754]: Reconfiguring mon.a (monmap changed)... 2024-06-14T22:43:07.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:07 smithi086 ceph-mon[102754]: Reconfiguring daemon mon.a on smithi086 2024-06-14T22:43:07.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:07 smithi086 ceph-mon[97913]: pgmap v110: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-06-14T22:43:07.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:07 smithi086 ceph-mon[97913]: Reconfiguring mon.a (monmap changed)... 2024-06-14T22:43:07.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:07 smithi086 ceph-mon[97913]: Reconfiguring daemon mon.a on smithi086 2024-06-14T22:43:07.633 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:43:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:07 smithi137 ceph-mon[90634]: pgmap v110: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-06-14T22:43:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:07 smithi137 ceph-mon[90634]: Reconfiguring mon.a (monmap changed)... 2024-06-14T22:43:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:07 smithi137 ceph-mon[90634]: Reconfiguring daemon mon.a on smithi086 2024-06-14T22:43:08.990 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 22:43:08 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T22:43:08.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)\", cluster=\"44b2848a-2a9d-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.86\", device_class=\"ssd\", hostname=\"smithi086\", instance=\"smithi086:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.86\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-14T22:43:09.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[102754]: pgmap v111: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-06-14T22:43:09.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:09.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:09.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[102754]: Reconfiguring mgr.y (monmap changed)... 2024-06-14T22:43:09.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:43:09.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:43:09.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:09.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[102754]: Reconfiguring daemon mgr.y on smithi086 2024-06-14T22:43:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[97913]: pgmap v111: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-06-14T22:43:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[97913]: Reconfiguring mgr.y (monmap changed)... 2024-06-14T22:43:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:43:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:43:09.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:09.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:09 smithi086 ceph-mon[97913]: Reconfiguring daemon mgr.y on smithi086 2024-06-14T22:43:09.617 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:09 smithi137 ceph-mon[90634]: pgmap v111: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-06-14T22:43:09.617 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:09 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:09.617 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:09 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:09.617 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:09 smithi137 ceph-mon[90634]: Reconfiguring mgr.y (monmap changed)... 2024-06-14T22:43:09.617 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:09 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:43:09.617 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:09 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:43:09.617 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:09 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:09.618 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:09 smithi137 ceph-mon[90634]: Reconfiguring daemon mgr.y on smithi086 2024-06-14T22:43:11.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:11 smithi086 ceph-mon[97913]: pgmap v112: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:43:11.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:11 smithi086 ceph-mon[102754]: pgmap v112: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:43:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:11 smithi137 ceph-mon[90634]: pgmap v112: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:43:13.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:13.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[97913]: pgmap v113: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 711 op/s 2024-06-14T22:43:13.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:13.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[97913]: Reconfiguring mon.c (monmap changed)... 2024-06-14T22:43:13.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:43:13.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:43:13.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:13.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[97913]: Reconfiguring daemon mon.c on smithi086 2024-06-14T22:43:13.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:13.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:13.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[102754]: pgmap v113: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 711 op/s 2024-06-14T22:43:13.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:13.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[102754]: Reconfiguring mon.c (monmap changed)... 2024-06-14T22:43:13.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:43:13.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:43:13.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:13.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[102754]: Reconfiguring daemon mon.c on smithi086 2024-06-14T22:43:13.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:13.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:13 smithi137 ceph-mon[90634]: pgmap v113: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 711 op/s 2024-06-14T22:43:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:13 smithi137 ceph-mon[90634]: Reconfiguring mon.c (monmap changed)... 2024-06-14T22:43:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:43:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:43:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:13 smithi137 ceph-mon[90634]: Reconfiguring daemon mon.c on smithi086 2024-06-14T22:43:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:15.190 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:43:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:43:14] "GET /metrics HTTP/1.1" 200 36219 "" "Prometheus/2.43.0" 2024-06-14T22:43:15.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[102754]: pgmap v114: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:43:15.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:15.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:15.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[102754]: Reconfiguring osd.0 (monmap changed)... 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[102754]: Reconfiguring daemon osd.0 on smithi086 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[97913]: pgmap v114: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[97913]: Reconfiguring osd.0 (monmap changed)... 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:15.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:15 smithi086 ceph-mon[97913]: Reconfiguring daemon osd.0 on smithi086 2024-06-14T22:43:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:15 smithi137 ceph-mon[90634]: pgmap v114: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:43:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:15 smithi137 ceph-mon[90634]: Reconfiguring osd.0 (monmap changed)... 2024-06-14T22:43:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-14T22:43:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:15.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:15 smithi137 ceph-mon[90634]: Reconfiguring daemon osd.0 on smithi086 2024-06-14T22:43:17.482 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:17.482 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:17.482 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[97913]: Reconfiguring osd.1 (monmap changed)... 2024-06-14T22:43:17.482 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-14T22:43:17.482 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:17.482 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[97913]: Reconfiguring daemon osd.1 on smithi086 2024-06-14T22:43:17.482 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[97913]: pgmap v115: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:43:17.483 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:17.483 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:17.483 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[102754]: Reconfiguring osd.1 (monmap changed)... 2024-06-14T22:43:17.483 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-14T22:43:17.483 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:17.483 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[102754]: Reconfiguring daemon osd.1 on smithi086 2024-06-14T22:43:17.483 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:17 smithi086 ceph-mon[102754]: pgmap v115: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:43:17.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:17.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:17.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:17 smithi137 ceph-mon[90634]: Reconfiguring osd.1 (monmap changed)... 2024-06-14T22:43:17.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-14T22:43:17.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:17.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:17 smithi137 ceph-mon[90634]: Reconfiguring daemon osd.1 on smithi086 2024-06-14T22:43:17.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:17 smithi137 ceph-mon[90634]: pgmap v115: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:43:19.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[102754]: pgmap v116: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:43:19.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:19.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:19.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-14T22:43:19.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:19.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[97913]: pgmap v116: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:43:19.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:19.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:19.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-14T22:43:19.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:19 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:19.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:19 smithi137 ceph-mon[90634]: pgmap v116: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T22:43:19.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:19 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:19.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:19 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:19.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:19 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-14T22:43:19.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:19 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:20.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:20 smithi137 ceph-mon[90634]: Reconfiguring osd.2 (monmap changed)... 2024-06-14T22:43:20.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:20 smithi137 ceph-mon[90634]: Reconfiguring daemon osd.2 on smithi086 2024-06-14T22:43:20.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:20 smithi086 ceph-mon[97913]: Reconfiguring osd.2 (monmap changed)... 2024-06-14T22:43:20.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:20 smithi086 ceph-mon[97913]: Reconfiguring daemon osd.2 on smithi086 2024-06-14T22:43:20.777 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:20 smithi086 ceph-mon[102754]: Reconfiguring osd.2 (monmap changed)... 2024-06-14T22:43:20.777 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:20 smithi086 ceph-mon[102754]: Reconfiguring daemon osd.2 on smithi086 2024-06-14T22:43:21.622 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[97913]: pgmap v117: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:43:21.622 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:21.623 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:21.623 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-14T22:43:21.623 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:21.623 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[102754]: pgmap v117: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:43:21.623 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:21.623 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:21.623 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-14T22:43:21.623 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:21 smithi137 ceph-mon[90634]: pgmap v117: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:43:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-14T22:43:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:22.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:22 smithi137 ceph-mon[90634]: Reconfiguring osd.3 (monmap changed)... 2024-06-14T22:43:22.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:22 smithi137 ceph-mon[90634]: Reconfiguring daemon osd.3 on smithi086 2024-06-14T22:43:22.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:22 smithi086 ceph-mon[97913]: Reconfiguring osd.3 (monmap changed)... 2024-06-14T22:43:22.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:22 smithi086 ceph-mon[97913]: Reconfiguring daemon osd.3 on smithi086 2024-06-14T22:43:22.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:22 smithi086 ceph-mon[102754]: Reconfiguring osd.3 (monmap changed)... 2024-06-14T22:43:22.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:22 smithi086 ceph-mon[102754]: Reconfiguring daemon osd.3 on smithi086 2024-06-14T22:43:23.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:23 smithi086 ceph-mon[97913]: pgmap v118: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-06-14T22:43:23.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:23 smithi086 ceph-mon[102754]: pgmap v118: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-06-14T22:43:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:23 smithi137 ceph-mon[90634]: pgmap v118: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-06-14T22:43:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[102754]: Reconfiguring mds.foo.smithi086.nhdjdb (monmap changed)... 2024-06-14T22:43:24.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:43:24.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:24.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[102754]: Reconfiguring daemon mds.foo.smithi086.nhdjdb on smithi086 2024-06-14T22:43:24.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:24.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:24.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[97913]: Reconfiguring mds.foo.smithi086.nhdjdb (monmap changed)... 2024-06-14T22:43:24.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:43:24.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:24.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-mon[97913]: Reconfiguring daemon mds.foo.smithi086.nhdjdb on smithi086 2024-06-14T22:43:24.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:24 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:24.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:24 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:24.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:24 smithi137 ceph-mon[90634]: Reconfiguring mds.foo.smithi086.nhdjdb (monmap changed)... 2024-06-14T22:43:24.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:24 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi086.nhdjdb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:43:24.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:24 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:24.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:24 smithi137 ceph-mon[90634]: Reconfiguring daemon mds.foo.smithi086.nhdjdb on smithi086 2024-06-14T22:43:25.011 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:43:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:43:24] "GET /metrics HTTP/1.1" 200 36219 "" "Prometheus/2.43.0" 2024-06-14T22:43:25.717 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:25 smithi137 ceph-mon[90634]: pgmap v119: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:43:25.717 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:25.717 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:25.717 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:43:25.718 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:43:25.718 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:25.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[97913]: pgmap v119: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:43:25.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[102754]: pgmap v119: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-14T22:43:25.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:26.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:26 smithi137 ceph-mon[90634]: Reconfiguring mon.b (monmap changed)... 2024-06-14T22:43:26.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:26 smithi137 ceph-mon[90634]: Reconfiguring daemon mon.b on smithi137 2024-06-14T22:43:26.769 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:26 smithi086 ceph-mon[97913]: Reconfiguring mon.b (monmap changed)... 2024-06-14T22:43:26.769 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:26 smithi086 ceph-mon[97913]: Reconfiguring daemon mon.b on smithi137 2024-06-14T22:43:26.770 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:26 smithi086 ceph-mon[102754]: Reconfiguring mon.b (monmap changed)... 2024-06-14T22:43:26.770 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:26 smithi086 ceph-mon[102754]: Reconfiguring daemon mon.b on smithi137 2024-06-14T22:43:27.727 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:27 smithi137 ceph-mon[90634]: pgmap v120: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-06-14T22:43:27.727 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:27.727 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:27.727 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:43:27.727 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:43:27.728 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:27.728 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[97913]: pgmap v120: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-06-14T22:43:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:43:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:43:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[102754]: pgmap v120: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-06-14T22:43:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-14T22:43:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-14T22:43:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:28.708 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:28 smithi137 ceph-mon[90634]: Reconfiguring mgr.x (monmap changed)... 2024-06-14T22:43:28.708 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:28 smithi137 ceph-mon[90634]: Reconfiguring daemon mgr.x on smithi137 2024-06-14T22:43:28.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:28 smithi086 ceph-mon[102754]: Reconfiguring mgr.x (monmap changed)... 2024-06-14T22:43:28.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:28 smithi086 ceph-mon[102754]: Reconfiguring daemon mgr.x on smithi137 2024-06-14T22:43:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:28 smithi086 ceph-mon[97913]: Reconfiguring mgr.x (monmap changed)... 2024-06-14T22:43:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:28 smithi086 ceph-mon[97913]: Reconfiguring daemon mgr.x on smithi137 2024-06-14T22:43:29.587 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:29 smithi137 ceph-mon[90634]: pgmap v121: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-06-14T22:43:29.587 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:29 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:29.587 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:29 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:29.587 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:29 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-14T22:43:29.587 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:29 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[97913]: pgmap v121: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-06-14T22:43:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-14T22:43:29.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[102754]: pgmap v121: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-06-14T22:43:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-14T22:43:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:29 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:30.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:30 smithi137 ceph-mon[90634]: Reconfiguring osd.4 (monmap changed)... 2024-06-14T22:43:30.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:30 smithi137 ceph-mon[90634]: Reconfiguring daemon osd.4 on smithi137 2024-06-14T22:43:30.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:30 smithi086 ceph-mon[97913]: Reconfiguring osd.4 (monmap changed)... 2024-06-14T22:43:30.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:30 smithi086 ceph-mon[97913]: Reconfiguring daemon osd.4 on smithi137 2024-06-14T22:43:30.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:30 smithi086 ceph-mon[102754]: Reconfiguring osd.4 (monmap changed)... 2024-06-14T22:43:30.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:30 smithi086 ceph-mon[102754]: Reconfiguring daemon osd.4 on smithi137 2024-06-14T22:43:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:31 smithi137 ceph-mon[90634]: pgmap v122: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:43:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:31 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:31 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:31 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-14T22:43:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:31 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[102754]: pgmap v122: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:43:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-14T22:43:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:31.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[97913]: pgmap v122: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:43:31.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:31.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:31.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-14T22:43:31.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:31 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:32.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:32 smithi137 ceph-mon[90634]: Reconfiguring osd.5 (monmap changed)... 2024-06-14T22:43:32.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:32 smithi137 ceph-mon[90634]: Reconfiguring daemon osd.5 on smithi137 2024-06-14T22:43:32.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:32 smithi086 ceph-mon[102754]: Reconfiguring osd.5 (monmap changed)... 2024-06-14T22:43:32.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:32 smithi086 ceph-mon[102754]: Reconfiguring daemon osd.5 on smithi137 2024-06-14T22:43:32.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:32 smithi086 ceph-mon[97913]: Reconfiguring osd.5 (monmap changed)... 2024-06-14T22:43:32.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:32 smithi086 ceph-mon[97913]: Reconfiguring daemon osd.5 on smithi137 2024-06-14T22:43:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:33 smithi137 ceph-mon[90634]: pgmap v123: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 713 op/s 2024-06-14T22:43:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:33 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:33 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:33 smithi137 ceph-mon[90634]: Reconfiguring osd.6 (monmap changed)... 2024-06-14T22:43:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:33 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-14T22:43:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:33 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:33.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:33 smithi137 ceph-mon[90634]: Reconfiguring daemon osd.6 on smithi137 2024-06-14T22:43:33.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[97913]: pgmap v123: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 713 op/s 2024-06-14T22:43:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[97913]: Reconfiguring osd.6 (monmap changed)... 2024-06-14T22:43:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-14T22:43:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:33.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[97913]: Reconfiguring daemon osd.6 on smithi137 2024-06-14T22:43:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[102754]: pgmap v123: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 713 op/s 2024-06-14T22:43:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:33.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[102754]: Reconfiguring osd.6 (monmap changed)... 2024-06-14T22:43:33.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-14T22:43:33.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:33.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:33 smithi086 ceph-mon[102754]: Reconfiguring daemon osd.6 on smithi137 2024-06-14T22:43:35.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:43:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:43:34] "GET /metrics HTTP/1.1" 200 36217 "" "Prometheus/2.43.0" 2024-06-14T22:43:35.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:35 smithi137 ceph-mon[90634]: pgmap v124: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:43:35.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:35.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:35.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-14T22:43:35.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:35.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[102754]: pgmap v124: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:43:35.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:35.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:35.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-14T22:43:35.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:35.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[97913]: pgmap v124: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:43:35.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:35.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:35.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-14T22:43:35.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:36.774 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:36 smithi137 ceph-mon[90634]: Reconfiguring osd.7 (monmap changed)... 2024-06-14T22:43:36.774 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:36 smithi137 ceph-mon[90634]: Reconfiguring daemon osd.7 on smithi137 2024-06-14T22:43:36.774 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:36 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:36.774 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:36 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:36.774 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:36 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:43:36.777 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:36 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:36.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[97913]: Reconfiguring osd.7 (monmap changed)... 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[97913]: Reconfiguring daemon osd.7 on smithi137 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[102754]: Reconfiguring osd.7 (monmap changed)... 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[102754]: Reconfiguring daemon osd.7 on smithi137 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi137.yjevpg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-14T22:43:36.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:36 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:37.808 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:37 smithi137 ceph-mon[90634]: Reconfiguring mds.foo.smithi137.yjevpg (monmap changed)... 2024-06-14T22:43:37.808 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:37 smithi137 ceph-mon[90634]: Reconfiguring daemon mds.foo.smithi137.yjevpg on smithi137 2024-06-14T22:43:37.808 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:37 smithi137 ceph-mon[90634]: pgmap v125: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-06-14T22:43:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:37 smithi086 ceph-mon[102754]: Reconfiguring mds.foo.smithi137.yjevpg (monmap changed)... 2024-06-14T22:43:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:37 smithi086 ceph-mon[102754]: Reconfiguring daemon mds.foo.smithi137.yjevpg on smithi137 2024-06-14T22:43:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:37 smithi086 ceph-mon[102754]: pgmap v125: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-06-14T22:43:37.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:37 smithi086 ceph-mon[97913]: Reconfiguring mds.foo.smithi137.yjevpg (monmap changed)... 2024-06-14T22:43:37.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:37 smithi086 ceph-mon[97913]: Reconfiguring daemon mds.foo.smithi137.yjevpg on smithi137 2024-06-14T22:43:37.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:37 smithi086 ceph-mon[97913]: pgmap v125: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-06-14T22:43:39.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:38 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:39.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:38 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:39.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:38 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:43:39.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:38 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:39.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:38 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:43:39.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:38 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:39.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:38 smithi137 ceph-mon[90634]: pgmap v126: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-06-14T22:43:39.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:39.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:39.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:43:39.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:39.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:43:39.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:39.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[102754]: pgmap v126: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-06-14T22:43:39.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:39.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:39.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:43:39.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:43:39.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:43:39.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:43:39.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:38 smithi086 ceph-mon[97913]: pgmap v126: 89 pgs: 89 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-06-14T22:43:41.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:41 smithi137 ceph-mon[90634]: pgmap v127: 89 pgs: 89 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-06-14T22:43:41.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:41 smithi086 ceph-mon[102754]: pgmap v127: 89 pgs: 89 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-06-14T22:43:41.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:41 smithi086 ceph-mon[97913]: pgmap v127: 89 pgs: 89 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-06-14T22:43:42.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:42.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:43.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:43 smithi137 ceph-mon[90634]: pgmap v128: 89 pgs: 89 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 758 op/s 2024-06-14T22:43:43.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:43 smithi086 ceph-mon[97913]: pgmap v128: 89 pgs: 89 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 758 op/s 2024-06-14T22:43:43.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:43 smithi086 ceph-mon[102754]: pgmap v128: 89 pgs: 89 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 758 op/s 2024-06-14T22:43:45.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:43:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:43:44] "GET /metrics HTTP/1.1" 200 36210 "" "Prometheus/2.43.0" 2024-06-14T22:43:45.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:45 smithi137 ceph-mon[90634]: pgmap v129: 89 pgs: 89 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:43:45.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:45 smithi086 ceph-mon[97913]: pgmap v129: 89 pgs: 89 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:43:45.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:45 smithi086 ceph-mon[102754]: pgmap v129: 89 pgs: 89 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:43:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:47 smithi137 ceph-mon[90634]: pgmap v130: 89 pgs: 89 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:43:47.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:47 smithi086 ceph-mon[102754]: pgmap v130: 89 pgs: 89 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:43:47.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:47 smithi086 ceph-mon[97913]: pgmap v130: 89 pgs: 89 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:43:49.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:49 smithi137 ceph-mon[90634]: pgmap v131: 89 pgs: 89 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:43:49.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:49 smithi086 ceph-mon[102754]: pgmap v131: 89 pgs: 89 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:43:49.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:49 smithi086 ceph-mon[97913]: pgmap v131: 89 pgs: 89 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:43:51.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:51 smithi137 ceph-mon[90634]: pgmap v132: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:43:51.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:51 smithi086 ceph-mon[102754]: pgmap v132: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:43:51.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:51 smithi086 ceph-mon[97913]: pgmap v132: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:43:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:53 smithi137 ceph-mon[90634]: pgmap v133: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 768 op/s 2024-06-14T22:43:53.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:53 smithi086 ceph-mon[102754]: pgmap v133: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 768 op/s 2024-06-14T22:43:53.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:53 smithi086 ceph-mon[97913]: pgmap v133: 89 pgs: 89 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 768 op/s 2024-06-14T22:43:55.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:43:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:43:54] "GET /metrics HTTP/1.1" 200 36210 "" "Prometheus/2.43.0" 2024-06-14T22:43:55.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:55 smithi137 ceph-mon[90634]: pgmap v134: 89 pgs: 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:43:55.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:55 smithi086 ceph-mon[97913]: pgmap v134: 89 pgs: 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:43:55.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:55 smithi086 ceph-mon[102754]: pgmap v134: 89 pgs: 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:43:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:57 smithi137 ceph-mon[90634]: pgmap v135: 89 pgs: 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-06-14T22:43:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:57 smithi086 ceph-mon[97913]: pgmap v135: 89 pgs: 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-06-14T22:43:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:57 smithi086 ceph-mon[102754]: pgmap v135: 89 pgs: 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-06-14T22:43:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:43:59.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:43:59 smithi137 ceph-mon[90634]: pgmap v136: 89 pgs: 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-06-14T22:43:59.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:43:59 smithi086 ceph-mon[97913]: pgmap v136: 89 pgs: 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-06-14T22:43:59.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:43:59 smithi086 ceph-mon[102754]: pgmap v136: 89 pgs: 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-06-14T22:44:01.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:01 smithi137 ceph-mon[90634]: pgmap v137: 89 pgs: 89 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:44:01.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:01 smithi086 ceph-mon[97913]: pgmap v137: 89 pgs: 89 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:44:01.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:01 smithi086 ceph-mon[102754]: pgmap v137: 89 pgs: 89 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:44:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:03 smithi137 ceph-mon[90634]: pgmap v138: 89 pgs: 89 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:44:03.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:03 smithi086 ceph-mon[97913]: pgmap v138: 89 pgs: 89 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:44:03.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:03 smithi086 ceph-mon[102754]: pgmap v138: 89 pgs: 89 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:44:05.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:44:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:44:04] "GET /metrics HTTP/1.1" 200 36216 "" "Prometheus/2.43.0" 2024-06-14T22:44:05.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:05 smithi137 ceph-mon[90634]: pgmap v139: 89 pgs: 89 active+clean; 886 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:44:05.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:05 smithi086 ceph-mon[102754]: pgmap v139: 89 pgs: 89 active+clean; 886 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:44:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:05 smithi086 ceph-mon[97913]: pgmap v139: 89 pgs: 89 active+clean; 886 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:44:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:07 smithi137 ceph-mon[90634]: pgmap v140: 89 pgs: 89 active+clean; 886 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:44:07.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:07 smithi086 ceph-mon[97913]: pgmap v140: 89 pgs: 89 active+clean; 886 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:44:07.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:07 smithi086 ceph-mon[102754]: pgmap v140: 89 pgs: 89 active+clean; 886 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:44:09.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:09 smithi086 ceph-mon[102754]: pgmap v141: 89 pgs: 89 active+clean; 886 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:44:09.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:09 smithi086 ceph-mon[97913]: pgmap v141: 89 pgs: 89 active+clean; 886 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:44:09.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:09 smithi137 ceph-mon[90634]: pgmap v141: 89 pgs: 89 active+clean; 886 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-06-14T22:44:11.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:11 smithi086 ceph-mon[97913]: pgmap v142: 89 pgs: 89 active+clean; 601 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:44:11.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:11 smithi086 ceph-mon[102754]: pgmap v142: 89 pgs: 89 active+clean; 601 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:44:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:11 smithi137 ceph-mon[90634]: pgmap v142: 89 pgs: 89 active+clean; 601 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-06-14T22:44:12.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch upgrade resume' 2024-06-14T22:44:12.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:12.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:12.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:44:13.498 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:13 smithi086 ceph-mon[97913]: pgmap v143: 89 pgs: 89 active+clean; 601 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 763 op/s 2024-06-14T22:44:13.499 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:13 smithi086 ceph-mon[102754]: pgmap v143: 89 pgs: 89 active+clean; 601 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 763 op/s 2024-06-14T22:44:13.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:13 smithi137 ceph-mon[90634]: pgmap v143: 89 pgs: 89 active+clean; 601 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 763 op/s 2024-06-14T22:44:15.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:44:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:44:14] "GET /metrics HTTP/1.1" 200 36206 "" "Prometheus/2.43.0" 2024-06-14T22:44:15.633 INFO:teuthology.orchestra.run.smithi086.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:44:15.660 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:15 smithi086 ceph-mon[97913]: pgmap v144: 89 pgs: 89 active+clean; 319 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:44:15.660 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:15 smithi086 ceph-mon[102754]: pgmap v144: 89 pgs: 89 active+clean; 319 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:44:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:15 smithi137 ceph-mon[90634]: pgmap v144: 89 pgs: 89 active+clean; 319 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T22:44:16.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'echo wait for half of osds to upgrade' 2024-06-14T22:44:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:16 smithi137 ceph-mon[90634]: from='client.54109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:44:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:16 smithi137 ceph-mon[90634]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:44:16.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:16 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:16.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:16 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:16.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:16 smithi137 ceph-mon[90634]: pgmap v145: 89 pgs: 89 active+clean; 319 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-06-14T22:44:17.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[97913]: from='client.54109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:44:17.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[97913]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:44:17.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:17.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:17.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[97913]: pgmap v145: 89 pgs: 89 active+clean; 319 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-06-14T22:44:17.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[102754]: from='client.54109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:44:17.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[102754]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:44:17.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:17.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:17.039 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:16 smithi086 ceph-mon[102754]: pgmap v145: 89 pgs: 89 active+clean; 319 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-06-14T22:44:17.623 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:44:19.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:19 smithi137 ceph-mon[90634]: pgmap v146: 89 pgs: 89 active+clean; 319 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-06-14T22:44:19.573 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:19 smithi086 ceph-mon[97913]: pgmap v146: 89 pgs: 89 active+clean; 319 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-06-14T22:44:19.574 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:19 smithi086 ceph-mon[102754]: pgmap v146: 89 pgs: 89 active+clean; 319 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-06-14T22:44:19.682 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 134215 objects 2024-06-14T22:44:19.682 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :148.525 2024-06-14T22:44:19.885 INFO:teuthology.orchestra.run.smithi086.stdout:wait for half of osds to upgrade 2024-06-14T22:44:20.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'while ! ceph osd versions | grep $sha1 | egrep '"'"': [45678]'"'"'; do sleep 2 ; done' 2024-06-14T22:44:20.807 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:44:21.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:21 smithi086 ceph-mon[102754]: pgmap v147: 89 pgs: 89 active+clean; 60 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-06-14T22:44:21.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:21.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:21.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:21 smithi086 ceph-mon[97913]: pgmap v147: 89 pgs: 89 active+clean; 60 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-06-14T22:44:21.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:21.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:21 smithi137 ceph-mon[90634]: pgmap v147: 89 pgs: 89 active+clean; 60 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-06-14T22:44:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:23.402 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:23 smithi086 ceph-mon[97913]: pgmap v148: 89 pgs: 89 active+clean; 60 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 720 op/s 2024-06-14T22:44:23.402 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:23 smithi086 ceph-mon[102754]: pgmap v148: 89 pgs: 89 active+clean; 60 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 720 op/s 2024-06-14T22:44:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:23 smithi137 ceph-mon[90634]: pgmap v148: 89 pgs: 89 active+clean; 60 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 720 op/s 2024-06-14T22:44:24.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:24 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2748223303' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:24.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:24 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2748223303' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:24.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:24 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2748223303' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:25.141 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-06-14T22:44:25.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-06-14T22:44:25.164 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:44:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:44:24] "GET /metrics HTTP/1.1" 200 36206 "" "Prometheus/2.43.0" 2024-06-14T22:44:25.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: pgmap v149: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-06-14T22:44:25.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:44:25.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:44:25.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:25.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:25.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: pgmap v149: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-06-14T22:44:25.417 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-06-14T22:44:25.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-06-14T22:44:25.419 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-06-14T22:44:25.419 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:25.419 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.419 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-14T22:44:25.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: pgmap v149: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-06-14T22:44:25.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-06-14T22:44:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-06-14T22:44:25.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-06-14T22:44:25.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:25.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:25.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-14T22:44:26.136 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:44:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[97913]: Upgrade: Setting container_image for all mon 2024-06-14T22:44:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[97913]: Upgrade: Setting container_image for all crash 2024-06-14T22:44:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-14T22:44:26.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[97913]: Upgrade: osd.0 is safe to restart 2024-06-14T22:44:26.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2172873472' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:26.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[102754]: Upgrade: Setting container_image for all mon 2024-06-14T22:44:26.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[102754]: Upgrade: Setting container_image for all crash 2024-06-14T22:44:26.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-14T22:44:26.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[102754]: Upgrade: osd.0 is safe to restart 2024-06-14T22:44:26.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2172873472' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:26.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:26 smithi137 ceph-mon[90634]: Upgrade: Setting container_image for all mon 2024-06-14T22:44:26.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:26 smithi137 ceph-mon[90634]: Upgrade: Setting container_image for all crash 2024-06-14T22:44:26.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:26 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-14T22:44:26.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:26 smithi137 ceph-mon[90634]: Upgrade: osd.0 is safe to restart 2024-06-14T22:44:26.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2172873472' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:27.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[97913]: pgmap v150: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 425 op/s 2024-06-14T22:44:27.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:27.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-14T22:44:27.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:44:27.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:27.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[102754]: pgmap v150: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 425 op/s 2024-06-14T22:44:27.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:27.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-14T22:44:27.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:44:27.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:27 smithi137 ceph-mon[90634]: pgmap v150: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 425 op/s 2024-06-14T22:44:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-14T22:44:27.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:44:27.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:28.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:28 smithi086 ceph-mon[97913]: Upgrade: Updating osd.0 2024-06-14T22:44:28.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:28 smithi086 ceph-mon[97913]: Deploying daemon osd.0 on smithi086 2024-06-14T22:44:28.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2443283313' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:28.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:28 smithi086 ceph-mon[102754]: Upgrade: Updating osd.0 2024-06-14T22:44:28.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:28 smithi086 ceph-mon[102754]: Deploying daemon osd.0 on smithi086 2024-06-14T22:44:28.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2443283313' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:28.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:28 smithi137 ceph-mon[90634]: Upgrade: Updating osd.0 2024-06-14T22:44:28.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:28 smithi137 ceph-mon[90634]: Deploying daemon osd.0 on smithi086 2024-06-14T22:44:28.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2443283313' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:29.559 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:29 smithi086 ceph-mon[97913]: pgmap v151: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 425 op/s 2024-06-14T22:44:29.559 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:29 smithi086 ceph-mon[102754]: pgmap v151: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 425 op/s 2024-06-14T22:44:29.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:29 smithi137 ceph-mon[90634]: pgmap v151: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 425 op/s 2024-06-14T22:44:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_1' removed 2024-06-14T22:44:30.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/252126188' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:44:30.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/238339156' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:30.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/252126188' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:44:30.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/238339156' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:30.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/252126188' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:44:30.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/238339156' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:31.475 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:31 smithi086 ceph-mon[102754]: pgmap v152: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 425 op/s 2024-06-14T22:44:31.475 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/252126188' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:44:31.475 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:31 smithi086 ceph-mon[102754]: osdmap e83: 8 total, 8 up, 8 in 2024-06-14T22:44:31.476 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:31 smithi086 ceph-mon[97913]: pgmap v152: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 425 op/s 2024-06-14T22:44:31.476 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/252126188' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:44:31.476 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:31 smithi086 ceph-mon[97913]: osdmap e83: 8 total, 8 up, 8 in 2024-06-14T22:44:31.476 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:31 smithi086 systemd[1]: Stopping Ceph osd.0 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:44:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:31 smithi137 ceph-mon[90634]: pgmap v152: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 425 op/s 2024-06-14T22:44:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/252126188' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:44:31.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:31 smithi137 ceph-mon[90634]: osdmap e83: 8 total, 8 up, 8 in 2024-06-14T22:44:31.853 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T22:44:31.854 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T22:44:31.855 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-06-14T22:44:31.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-06-14T22:44:32.174 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:44:32.402 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:32 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[43191]: 2024-06-14T22:44:32.108+0000 7f3cc9dfa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T22:44:32.402 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:32 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[43191]: 2024-06-14T22:44:32.108+0000 7f3cc9dfa640 -1 osd.0 83 *** Got signal Terminated *** 2024-06-14T22:44:32.402 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:32 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[43191]: 2024-06-14T22:44:32.108+0000 7f3cc9dfa640 -1 osd.0 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T22:44:32.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:32 smithi086 ceph-mon[97913]: osd.0 marked itself down and dead 2024-06-14T22:44:32.720 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:32 smithi086 ceph-mon[102754]: osd.0 marked itself down and dead 2024-06-14T22:44:32.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:32 smithi137 ceph-mon[90634]: osd.0 marked itself down and dead 2024-06-14T22:44:33.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:33 smithi086 ceph-mon[97913]: pgmap v154: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:44:33.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:33 smithi086 ceph-mon[97913]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:44:33.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:33 smithi086 ceph-mon[97913]: osdmap e84: 8 total, 7 up, 8 in 2024-06-14T22:44:33.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/364727617' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:33.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:33 smithi086 ceph-mon[102754]: pgmap v154: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:44:33.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:33 smithi086 ceph-mon[102754]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:44:33.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:33 smithi086 ceph-mon[102754]: osdmap e84: 8 total, 7 up, 8 in 2024-06-14T22:44:33.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/364727617' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:33.582 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:33 smithi086 podman[111903]: 2024-06-14 22:44:33.32019829 +0000 UTC m=+1.333751429 container died 129bff24d5cce66a88b08645ead829589fe677fbaadfd0a28266cca88a7a57d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, ceph=True, GIT_CLEAN=True) 2024-06-14T22:44:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:33 smithi137 ceph-mon[90634]: pgmap v154: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:44:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:33 smithi137 ceph-mon[90634]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:44:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:33 smithi137 ceph-mon[90634]: osdmap e84: 8 total, 7 up, 8 in 2024-06-14T22:44:33.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/364727617' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:33.952 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:33 smithi086 podman[111903]: 2024-06-14 22:44:33.721191918 +0000 UTC m=+1.734745055 container cleanup 129bff24d5cce66a88b08645ead829589fe677fbaadfd0a28266cca88a7a57d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-14T22:44:33.952 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:33 smithi086 bash[111903]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0 2024-06-14T22:44:34.285 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:33 smithi086 podman[112046]: 2024-06-14 22:44:33.956030581 +0000 UTC m=+1.061223275 container remove 129bff24d5cce66a88b08645ead829589fe677fbaadfd0a28266cca88a7a57d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-06-14T22:44:34.564 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:34 smithi086 ceph-mon[97913]: osdmap e85: 8 total, 7 up, 8 in 2024-06-14T22:44:34.565 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:34 smithi086 ceph-mon[102754]: osdmap e85: 8 total, 7 up, 8 in 2024-06-14T22:44:34.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:34 smithi137 ceph-mon[90634]: osdmap e85: 8 total, 7 up, 8 in 2024-06-14T22:44:35.151 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:44:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:44:34] "GET /metrics HTTP/1.1" 200 36164 "" "Prometheus/2.43.0" 2024-06-14T22:44:35.152 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:34 smithi086 podman[112229]: 2024-06-14 22:44:34.815670531 +0000 UTC m=+0.018054842 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:44:35.152 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:34 smithi086 podman[112229]: 2024-06-14 22:44:34.927455418 +0000 UTC m=+0.129839724 container create a09930b4223672437482348cacbe7f21d7c76661390523fa9b5b0ee03f18aa3f (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, ceph=True, GIT_CLEAN=True) 2024-06-14T22:44:35.409 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:35 smithi086 podman[112229]: 2024-06-14 22:44:35.155044513 +0000 UTC m=+0.357428836 container init a09930b4223672437482348cacbe7f21d7c76661390523fa9b5b0ee03f18aa3f (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-14T22:44:35.409 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:35 smithi086 podman[112229]: 2024-06-14 22:44:35.160441427 +0000 UTC m=+0.362825741 container start a09930b4223672437482348cacbe7f21d7c76661390523fa9b5b0ee03f18aa3f (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-14T22:44:35.410 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:35 smithi086 podman[112229]: 2024-06-14 22:44:35.229807713 +0000 UTC m=+0.432192032 container attach a09930b4223672437482348cacbe7f21d7c76661390523fa9b5b0ee03f18aa3f (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:44:35.424 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_2' created 2024-06-14T22:44:35.667 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[97913]: pgmap v157: 73 pgs: 15 active+undersized, 5 peering, 6 active+undersized+degraded, 47 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8/72 objects degraded (11.111%) 2024-06-14T22:44:35.667 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[97913]: Health check failed: Degraded data redundancy: 8/72 objects degraded (11.111%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/182377366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/922205527' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[102754]: pgmap v157: 73 pgs: 15 active+undersized, 5 peering, 6 active+undersized+degraded, 47 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8/72 objects degraded (11.111%) 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[102754]: Health check failed: Degraded data redundancy: 8/72 objects degraded (11.111%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/182377366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-14T22:44:35.668 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/922205527' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:35.669 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:35 smithi086 podman[112229]: 2024-06-14 22:44:35.413690443 +0000 UTC m=+0.616074757 container died a09930b4223672437482348cacbe7f21d7c76661390523fa9b5b0ee03f18aa3f (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0) 2024-06-14T22:44:35.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:35 smithi137 ceph-mon[90634]: pgmap v157: 73 pgs: 15 active+undersized, 5 peering, 6 active+undersized+degraded, 47 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8/72 objects degraded (11.111%) 2024-06-14T22:44:35.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:35 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T22:44:35.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:35 smithi137 ceph-mon[90634]: Health check failed: Degraded data redundancy: 8/72 objects degraded (11.111%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:35.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/182377366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-14T22:44:35.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:35 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-14T22:44:35.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/922205527' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:36.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-06-14T22:44:36.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:36 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-06-14T22:44:36.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:36 smithi137 ceph-mon[90634]: osdmap e86: 8 total, 7 up, 8 in 2024-06-14T22:44:36.809 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:36 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-06-14T22:44:36.809 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:36 smithi086 ceph-mon[97913]: osdmap e86: 8 total, 7 up, 8 in 2024-06-14T22:44:36.810 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:36 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-06-14T22:44:36.810 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:36 smithi086 ceph-mon[102754]: osdmap e86: 8 total, 7 up, 8 in 2024-06-14T22:44:36.897 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:44:37.081 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:36 smithi086 podman[112275]: 2024-06-14 22:44:36.813194384 +0000 UTC m=+1.392831503 container remove a09930b4223672437482348cacbe7f21d7c76661390523fa9b5b0ee03f18aa3f (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, ceph=True, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS) 2024-06-14T22:44:37.081 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:36 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.0.service: Deactivated successfully. 2024-06-14T22:44:37.081 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:36 smithi086 systemd[1]: Stopped Ceph osd.0 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:44:37.081 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:36 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.0.service: Consumed 12min 31.581s CPU time. 2024-06-14T22:44:37.082 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:37 smithi086 systemd[1]: Starting Ceph osd.0 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:44:37.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:37 smithi137 ceph-mon[90634]: pgmap v159: 89 pgs: 16 unknown, 15 active+undersized, 5 peering, 6 active+undersized+degraded, 47 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8/72 objects degraded (11.111%) 2024-06-14T22:44:37.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:37 smithi137 ceph-mon[90634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:44:37.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:37 smithi137 ceph-mon[90634]: osdmap e87: 8 total, 7 up, 8 in 2024-06-14T22:44:37.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/526556836' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:37.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:37 smithi086 ceph-mon[97913]: pgmap v159: 89 pgs: 16 unknown, 15 active+undersized, 5 peering, 6 active+undersized+degraded, 47 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8/72 objects degraded (11.111%) 2024-06-14T22:44:37.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:37 smithi086 ceph-mon[97913]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:44:37.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:37 smithi086 ceph-mon[97913]: osdmap e87: 8 total, 7 up, 8 in 2024-06-14T22:44:37.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/526556836' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:37 smithi086 ceph-mon[102754]: pgmap v159: 89 pgs: 16 unknown, 15 active+undersized, 5 peering, 6 active+undersized+degraded, 47 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8/72 objects degraded (11.111%) 2024-06-14T22:44:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:37 smithi086 ceph-mon[102754]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:44:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:37 smithi086 ceph-mon[102754]: osdmap e87: 8 total, 7 up, 8 in 2024-06-14T22:44:37.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/526556836' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:38.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:38 smithi086 ceph-mon[97913]: osdmap e88: 8 total, 7 up, 8 in 2024-06-14T22:44:38.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:38 smithi086 ceph-mon[102754]: osdmap e88: 8 total, 7 up, 8 in 2024-06-14T22:44:38.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:38 smithi137 ceph-mon[90634]: osdmap e88: 8 total, 7 up, 8 in 2024-06-14T22:44:39.017 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:38 smithi086 podman[112535]: 2024-06-14 22:44:38.759697516 +0000 UTC m=+0.549855250 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:44:39.017 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:38 smithi086 podman[112535]: 2024-06-14 22:44:38.897487621 +0000 UTC m=+0.687645330 container create eed5a0673ef2df4b505e1f8c1c22348807d2180838d66f7b2ba2bcc4a7ed6940 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:44:39.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:39 smithi086 ceph-mon[97913]: pgmap v162: 89 pgs: 16 unknown, 15 active+undersized, 5 peering, 6 active+undersized+degraded, 47 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8/72 objects degraded (11.111%) 2024-06-14T22:44:39.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:39 smithi086 ceph-mon[102754]: pgmap v162: 89 pgs: 16 unknown, 15 active+undersized, 5 peering, 6 active+undersized+degraded, 47 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8/72 objects degraded (11.111%) 2024-06-14T22:44:39.568 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 podman[112535]: 2024-06-14 22:44:39.394656242 +0000 UTC m=+1.184813967 container init eed5a0673ef2df4b505e1f8c1c22348807d2180838d66f7b2ba2bcc4a7ed6940 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-14T22:44:39.568 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 podman[112535]: 2024-06-14 22:44:39.403160486 +0000 UTC m=+1.193318213 container start eed5a0673ef2df4b505e1f8c1c22348807d2180838d66f7b2ba2bcc4a7ed6940 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:44:39.568 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 podman[112535]: 2024-06-14 22:44:39.486047214 +0000 UTC m=+1.276204934 container attach eed5a0673ef2df4b505e1f8c1c22348807d2180838d66f7b2ba2bcc4a7ed6940 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-14T22:44:39.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:39 smithi137 ceph-mon[90634]: pgmap v162: 89 pgs: 16 unknown, 15 active+undersized, 5 peering, 6 active+undersized+degraded, 47 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8/72 objects degraded (11.111%) 2024-06-14T22:44:39.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate[112574]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-14T22:44:39.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 bash[112535]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-14T22:44:39.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate[112574]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-14T22:44:39.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 bash[112535]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-14T22:44:39.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate[112574]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-14T22:44:39.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 bash[112535]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-14T22:44:39.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate[112574]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-14T22:44:39.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 bash[112535]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-14T22:44:39.823 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate[112574]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-06-14T22:44:39.823 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 bash[112535]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-06-14T22:44:39.823 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate[112574]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-14T22:44:39.823 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 bash[112535]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-14T22:44:39.823 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate[112574]: --> ceph-volume raw activate successful for osd ID: 0 2024-06-14T22:44:39.823 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 bash[112535]: --> ceph-volume raw activate successful for osd ID: 0 2024-06-14T22:44:40.081 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:39 smithi086 podman[112535]: 2024-06-14 22:44:39.825797768 +0000 UTC m=+1.615955510 container died eed5a0673ef2df4b505e1f8c1c22348807d2180838d66f7b2ba2bcc4a7ed6940 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240528, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-14T22:44:40.447 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-06-14T22:44:40.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3203663368' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:40.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1047805255' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:44:40.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:40 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:44:40.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3203663368' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:40.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1047805255' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:44:40.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:40 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:44:40.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3203663368' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:40.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1047805255' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:44:40.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:40 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:44:40.833 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:40 smithi086 podman[112780]: 2024-06-14 22:44:40.415626945 +0000 UTC m=+0.584354927 container remove eed5a0673ef2df4b505e1f8c1c22348807d2180838d66f7b2ba2bcc4a7ed6940 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-activate, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-06-14T22:44:41.440 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:41 smithi086 podman[112866]: 2024-06-14 22:44:41.208875958 +0000 UTC m=+0.019262668 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:44:41.440 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:41 smithi086 podman[112866]: 2024-06-14 22:44:41.320757912 +0000 UTC m=+0.131144665 container create 62732bd1f8028eee8a271d30772f3307e979d37d7611e1c18be7d4e18ea67b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, org.label-schema.license=GPLv2) 2024-06-14T22:44:41.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:41 smithi137 ceph-mon[90634]: pgmap v163: 89 pgs: 27 active+undersized, 7 active+undersized+degraded, 55 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12/72 objects degraded (16.667%) 2024-06-14T22:44:41.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:41 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 12/72 objects degraded (16.667%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:41.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:41 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T22:44:41.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:41 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:44:41.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:41 smithi137 ceph-mon[90634]: osdmap e89: 8 total, 7 up, 8 in 2024-06-14T22:44:41.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[97913]: pgmap v163: 89 pgs: 27 active+undersized, 7 active+undersized+degraded, 55 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12/72 objects degraded (16.667%) 2024-06-14T22:44:41.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 12/72 objects degraded (16.667%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:41.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T22:44:41.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:44:41.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[97913]: osdmap e89: 8 total, 7 up, 8 in 2024-06-14T22:44:41.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[102754]: pgmap v163: 89 pgs: 27 active+undersized, 7 active+undersized+degraded, 55 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12/72 objects degraded (16.667%) 2024-06-14T22:44:41.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 12/72 objects degraded (16.667%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:41.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T22:44:41.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:44:41.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-mon[102754]: osdmap e89: 8 total, 7 up, 8 in 2024-06-14T22:44:41.832 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:41 smithi086 podman[112866]: 2024-06-14 22:44:41.539888302 +0000 UTC m=+0.350275014 container init 62732bd1f8028eee8a271d30772f3307e979d37d7611e1c18be7d4e18ea67b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240528) 2024-06-14T22:44:41.833 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:41 smithi086 podman[112866]: 2024-06-14 22:44:41.545861658 +0000 UTC m=+0.356248370 container start 62732bd1f8028eee8a271d30772f3307e979d37d7611e1c18be7d4e18ea67b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-14T22:44:41.833 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:41 smithi086 ceph-osd[112904]: -- 172.21.15.86:0/537849403 <== mon.2 v2:172.21.15.86:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a44f0651e0 con 0x55a44f003c00 2024-06-14T22:44:41.833 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:41 smithi086 bash[112866]: 62732bd1f8028eee8a271d30772f3307e979d37d7611e1c18be7d4e18ea67b45 2024-06-14T22:44:41.833 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:41 smithi086 systemd[1]: Started Ceph osd.0 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:44:42.198 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-06-14T22:44:42.201 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T22:44:42.201 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T22:44:42.246 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T22:44:42.246 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T22:44:42.246 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_113105 2024-06-14T22:44:42.246 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:44:42.246 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T22:44:42.440 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[112881]: 2024-06-14T22:44:42.087+0000 7f3220810640 -1 Falling back to public interface 2024-06-14T22:44:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:42.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1082769185' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[102754]: pgmap v165: 89 pgs: 27 active+undersized, 7 active+undersized+degraded, 55 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12/72 objects degraded (16.667%) 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1082769185' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[97913]: pgmap v165: 89 pgs: 27 active+undersized, 7 active+undersized+degraded, 55 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12/72 objects degraded (16.667%) 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:42 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1082769185' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:42.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:42 smithi137 ceph-mon[90634]: pgmap v165: 89 pgs: 27 active+undersized, 7 active+undersized+degraded, 55 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12/72 objects degraded (16.667%) 2024-06-14T22:44:42.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:42.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:43.246 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 4633 4617 288.527 288.562 0.00307423 0.00345484 2024-06-14T22:44:43.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:43 smithi086 ceph-mon[97913]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:44:43.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:43 smithi086 ceph-mon[102754]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:44:43.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:43 smithi137 ceph-mon[90634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:44:44.246 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 16 8844 8828 275.846 263.188 0.00178989 0.00361727 2024-06-14T22:44:44.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:44 smithi137 ceph-mon[90634]: pgmap v166: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 389 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s wr, 787 op/s; 3025/18720 objects degraded (16.159%) 2024-06-14T22:44:44.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2115963924' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:45.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:44 smithi086 ceph-mon[102754]: pgmap v166: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 389 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s wr, 787 op/s; 3025/18720 objects degraded (16.159%) 2024-06-14T22:44:45.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2115963924' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:45.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:44 smithi086 ceph-mon[97913]: pgmap v166: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 389 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s wr, 787 op/s; 3025/18720 objects degraded (16.159%) 2024-06-14T22:44:45.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2115963924' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:44:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:44:44] "GET /metrics HTTP/1.1" 200 36352 "" "Prometheus/2.43.0" 2024-06-14T22:44:45.247 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 16 12489 12473 259.809 227.812 0.00488357 0.00384319 2024-06-14T22:44:46.247 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 16 15937 15921 248.728 215.5 0.0106792 0.00401212 2024-06-14T22:44:47.247 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 15 18903 18888 236.067 185.438 0.00624452 0.0042289 2024-06-14T22:44:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:47 smithi137 ceph-mon[90634]: pgmap v167: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 389 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 44 MiB/s wr, 699 op/s; 3025/18720 objects degraded (16.159%) 2024-06-14T22:44:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:47 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:47 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:47.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3648610403' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:47.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:47 smithi086 ceph-mon[97913]: pgmap v167: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 389 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 44 MiB/s wr, 699 op/s; 3025/18720 objects degraded (16.159%) 2024-06-14T22:44:47.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:47 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:47.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:47 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:47.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3648610403' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:47.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:47 smithi086 ceph-mon[102754]: pgmap v167: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 389 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 44 MiB/s wr, 699 op/s; 3025/18720 objects degraded (16.159%) 2024-06-14T22:44:47.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:47 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:47.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:47 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:47.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3648610403' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:48.247 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 21736 21720 226.22 177 0.00823846 0.00441441 2024-06-14T22:44:48.968 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:48.968 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:48.968 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 3025/18720 objects degraded (16.159%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:48.968 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:48 smithi086 ceph-mon[102754]: pgmap v168: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 389 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 39 MiB/s wr, 621 op/s; 3025/18720 objects degraded (16.159%) 2024-06-14T22:44:48.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:48.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:48.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 3025/18720 objects degraded (16.159%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:48.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:48 smithi086 ceph-mon[97913]: pgmap v168: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 389 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 39 MiB/s wr, 621 op/s; 3025/18720 objects degraded (16.159%) 2024-06-14T22:44:49.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:48 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:49.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:48 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:49.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 3025/18720 objects degraded (16.159%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:49.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:48 smithi137 ceph-mon[90634]: pgmap v168: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 389 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 39 MiB/s wr, 621 op/s; 3025/18720 objects degraded (16.159%) 2024-06-14T22:44:49.248 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 16 24624 24608 219.655 180.5 0.00593704 0.00454758 2024-06-14T22:44:49.331 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:48 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[112881]: 2024-06-14T22:44:48.969+0000 7f3220810640 -1 osd.0 83 log_to_monitors true 2024-06-14T22:44:50.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:49 smithi086 ceph-mon[102754]: from='osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-14T22:44:50.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2461471911' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:50.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:49 smithi086 ceph-mon[97913]: from='osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-14T22:44:50.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2461471911' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:50.082 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 22:44:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[112881]: 2024-06-14T22:44:49.814+0000 7f32185ba640 -1 osd.0 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:44:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:49 smithi137 ceph-mon[90634]: from='osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-14T22:44:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2461471911' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:50.248 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 27548 27532 215.04 182.75 0.00429665 0.00464555 2024-06-14T22:44:51.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:50 smithi086 ceph-mon[97913]: from='osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-14T22:44:51.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:50 smithi086 ceph-mon[97913]: osdmap e90: 8 total, 7 up, 8 in 2024-06-14T22:44:51.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:50 smithi086 ceph-mon[97913]: from='osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:44:51.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:50 smithi086 ceph-mon[97913]: pgmap v170: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s; 11311/68337 objects degraded (16.552%) 2024-06-14T22:44:51.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:50 smithi086 ceph-mon[102754]: from='osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-14T22:44:51.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:50 smithi086 ceph-mon[102754]: osdmap e90: 8 total, 7 up, 8 in 2024-06-14T22:44:51.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:50 smithi086 ceph-mon[102754]: from='osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:44:51.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:50 smithi086 ceph-mon[102754]: pgmap v170: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s; 11311/68337 objects degraded (16.552%) 2024-06-14T22:44:51.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:50 smithi137 ceph-mon[90634]: from='osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-14T22:44:51.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:50 smithi137 ceph-mon[90634]: osdmap e90: 8 total, 7 up, 8 in 2024-06-14T22:44:51.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:50 smithi137 ceph-mon[90634]: from='osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:44:51.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:50 smithi137 ceph-mon[90634]: pgmap v170: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s; 11311/68337 objects degraded (16.552%) 2024-06-14T22:44:51.248 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 28873 28857 200.348 82.8125 0.00207296 0.00479157 2024-06-14T22:44:52.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[97913]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:44:52.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[97913]: osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428] boot 2024-06-14T22:44:52.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[97913]: osdmap e91: 8 total, 8 up, 8 in 2024-06-14T22:44:52.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:44:52.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1379368033' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:52.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[102754]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:44:52.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[102754]: osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428] boot 2024-06-14T22:44:52.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[102754]: osdmap e91: 8 total, 8 up, 8 in 2024-06-14T22:44:52.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:44:52.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1379368033' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:52.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:51 smithi137 ceph-mon[90634]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:44:52.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:51 smithi137 ceph-mon[90634]: osd.0 [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428] boot 2024-06-14T22:44:52.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:51 smithi137 ceph-mon[90634]: osdmap e91: 8 total, 8 up, 8 in 2024-06-14T22:44:52.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-14T22:44:52.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1379368033' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:52.248 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 16 28873 28857 180.315 0 - 0.00479157 2024-06-14T22:44:53.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:52 smithi086 ceph-mon[97913]: osdmap e92: 8 total, 8 up, 8 in 2024-06-14T22:44:53.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:52 smithi086 ceph-mon[97913]: pgmap v173: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 172 MiB/s wr, 2.75k op/s; 11311/68337 objects degraded (16.552%) 2024-06-14T22:44:53.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:52 smithi086 ceph-mon[102754]: osdmap e92: 8 total, 8 up, 8 in 2024-06-14T22:44:53.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:52 smithi086 ceph-mon[102754]: pgmap v173: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 172 MiB/s wr, 2.75k op/s; 11311/68337 objects degraded (16.552%) 2024-06-14T22:44:53.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:52 smithi137 ceph-mon[90634]: osdmap e92: 8 total, 8 up, 8 in 2024-06-14T22:44:53.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:52 smithi137 ceph-mon[90634]: pgmap v173: 89 pgs: 19 active+undersized, 15 active+undersized+degraded, 55 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 172 MiB/s wr, 2.75k op/s; 11311/68337 objects degraded (16.552%) 2024-06-14T22:44:53.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 29357 29341 166.674 15.125 0.0237887 0.00599216 2024-06-14T22:44:54.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:53 smithi086 ceph-mon[102754]: osdmap e93: 8 total, 8 up, 8 in 2024-06-14T22:44:54.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 11311/68337 objects degraded (16.552%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:54.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:53 smithi086 ceph-mon[97913]: osdmap e93: 8 total, 8 up, 8 in 2024-06-14T22:44:54.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 11311/68337 objects degraded (16.552%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:54.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:53 smithi137 ceph-mon[90634]: osdmap e93: 8 total, 8 up, 8 in 2024-06-14T22:44:54.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 11311/68337 objects degraded (16.552%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:54.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 16 30870 30854 160.664 94.5625 0.0118285 0.00621746 2024-06-14T22:44:55.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:44:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:44:54] "GET /metrics HTTP/1.1" 200 36352 "" "Prometheus/2.43.0" 2024-06-14T22:44:55.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3171367905' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:55.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:54 smithi086 ceph-mon[102754]: pgmap v175: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized, 2 active+undersized+degraded, 70 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s; 14385/89964 objects degraded (15.990%); 479 KiB/s, 7 objects/s recovering 2024-06-14T22:44:55.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3171367905' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:55.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:54 smithi086 ceph-mon[97913]: pgmap v175: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized, 2 active+undersized+degraded, 70 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s; 14385/89964 objects degraded (15.990%); 479 KiB/s, 7 objects/s recovering 2024-06-14T22:44:55.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:54 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3171367905' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:55.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:54 smithi137 ceph-mon[90634]: pgmap v175: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 9 active+undersized, 2 active+undersized+degraded, 70 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 99 MiB/s wr, 1.59k op/s; 14385/89964 objects degraded (15.990%); 479 KiB/s, 7 objects/s recovering 2024-06-14T22:44:55.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 32431 32415 155.81 97.5625 0.0231323 0.00641114 2024-06-14T22:44:55.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:44:55 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:44:55.561+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-06-14T22:44:56.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 34027 34011 151.805 99.75 0.00623547 0.00658127 2024-06-14T22:44:56.573 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:56.573 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:56.573 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:44:56.573 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:44:56.574 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:56.574 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:56.574 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:56.574 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:56.574 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:56.574 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:44:56.574 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:44:56.574 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-06-14T22:44:56.574 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:44:56.575 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:56 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-06-14T22:44:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:44:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:44:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:44:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:44:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:44:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:44:56.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:56 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-06-14T22:44:57.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 35384 35368 147.339 84.8125 0.00748879 0.00678046 2024-06-14T22:44:57.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:57 smithi086 ceph-mon[102754]: pgmap v176: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 14382/89964 objects degraded (15.986%); 363 KiB/s, 5 objects/s recovering 2024-06-14T22:44:57.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/209625563' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:57.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:57 smithi086 ceph-mon[97913]: pgmap v176: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 14382/89964 objects degraded (15.986%); 363 KiB/s, 5 objects/s recovering 2024-06-14T22:44:57.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/209625563' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:57 smithi137 ceph-mon[90634]: pgmap v176: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 14382/89964 objects degraded (15.986%); 363 KiB/s, 5 objects/s recovering 2024-06-14T22:44:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/209625563' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:58.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 36755 36739 143.486 85.6875 0.0241544 0.00696174 2024-06-14T22:44:58.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:58.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:58.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 14382/89964 objects degraded (15.986%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:58.639 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:58.639 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:58.639 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 14382/89964 objects degraded (15.986%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:58.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:44:58.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:44:58.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 14382/89964 objects degraded (15.986%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:44:59.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 15 38122 38107 140.075 85.5 0.0252561 0.00713189 2024-06-14T22:44:59.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:59 smithi137 ceph-mon[90634]: pgmap v177: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s; 14382/89964 objects degraded (15.986%); 289 KiB/s, 4 objects/s recovering 2024-06-14T22:44:59.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:44:59 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2691581693' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:59.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:59 smithi086 ceph-mon[102754]: pgmap v177: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s; 14382/89964 objects degraded (15.986%); 289 KiB/s, 4 objects/s recovering 2024-06-14T22:44:59.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:44:59 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2691581693' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:44:59.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:59 smithi086 ceph-mon[97913]: pgmap v177: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s; 14382/89964 objects degraded (15.986%); 289 KiB/s, 4 objects/s recovering 2024-06-14T22:44:59.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:44:59 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2691581693' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:00.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 39636 39620 137.546 94.5625 0.00507305 0.00726127 2024-06-14T22:45:01.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 41194 41178 135.431 97.375 0.0109684 0.00737883 2024-06-14T22:45:01.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:01 smithi137 ceph-mon[90634]: pgmap v178: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.3 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s; 14024/111912 objects degraded (12.531%); 2.9 MiB/s, 45 objects/s recovering 2024-06-14T22:45:01.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:01 smithi086 ceph-mon[102754]: pgmap v178: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.3 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s; 14024/111912 objects degraded (12.531%); 2.9 MiB/s, 45 objects/s recovering 2024-06-14T22:45:01.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:01 smithi086 ceph-mon[97913]: pgmap v178: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.3 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s; 14024/111912 objects degraded (12.531%); 2.9 MiB/s, 45 objects/s recovering 2024-06-14T22:45:02.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:45:02.253753+0000 min lat: 0.00124087 max lat: 2.06531 avg lat: 0.00750754 2024-06-14T22:45:02.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:45:02.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 42609 42593 133.081 88.4375 0.00466457 0.00750754 2024-06-14T22:45:02.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2490225110' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:02.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2490225110' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:02.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2490225110' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:03.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 16 44070 44054 131.092 91.3125 0.0144375 0.00762202 2024-06-14T22:45:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:03 smithi137 ceph-mon[90634]: pgmap v179: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.3 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s; 14024/111912 objects degraded (12.531%); 2.4 MiB/s, 39 objects/s recovering 2024-06-14T22:45:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 14024/111912 objects degraded (12.531%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:03.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:03 smithi086 ceph-mon[102754]: pgmap v179: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.3 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s; 14024/111912 objects degraded (12.531%); 2.4 MiB/s, 39 objects/s recovering 2024-06-14T22:45:03.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 14024/111912 objects degraded (12.531%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:03.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:03 smithi086 ceph-mon[97913]: pgmap v179: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.3 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s; 14024/111912 objects degraded (12.531%); 2.4 MiB/s, 39 objects/s recovering 2024-06-14T22:45:03.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 14024/111912 objects degraded (12.531%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:04.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 45480 45464 129.139 88.125 0.0214022 0.00773714 2024-06-14T22:45:04.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:04 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4196864658' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:04.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:04 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4196864658' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:04.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:04 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4196864658' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:05.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 46866 46850 127.29 86.625 0.00828238 0.007848 2024-06-14T22:45:05.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:45:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:45:04] "GET /metrics HTTP/1.1" 200 36425 "" "Prometheus/2.43.0" 2024-06-14T22:45:05.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:05 smithi137 ceph-mon[90634]: pgmap v180: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s; 13621/133968 objects degraded (10.167%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:45:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:05 smithi086 ceph-mon[97913]: pgmap v180: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s; 13621/133968 objects degraded (10.167%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:45:05.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:05 smithi086 ceph-mon[102754]: pgmap v180: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s; 13621/133968 objects degraded (10.167%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:45:06.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 48284 48268 125.679 88.625 0.00293642 0.00795081 2024-06-14T22:45:07.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 49570 49554 123.867 80.375 0.0121319 0.00806744 2024-06-14T22:45:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:07 smithi137 ceph-mon[90634]: pgmap v181: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s; 13621/133968 objects degraded (10.167%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:45:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1773490973' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:07.746 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:07 smithi086 ceph-mon[102754]: pgmap v181: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s; 13621/133968 objects degraded (10.167%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:45:07.746 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1773490973' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:07.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:07 smithi086 ceph-mon[97913]: pgmap v181: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s; 13621/133968 objects degraded (10.167%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:45:07.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1773490973' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:08.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 50935 50919 122.383 85.3125 0.0190014 0.00816479 2024-06-14T22:45:08.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 13621/133968 objects degraded (10.167%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:08.756 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 13621/133968 objects degraded (10.167%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:08.756 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 13621/133968 objects degraded (10.167%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:09.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 52215 52199 120.814 80 0.0074991 0.00827071 2024-06-14T22:45:09.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:09 smithi137 ceph-mon[90634]: pgmap v182: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s; 13621/133968 objects degraded (10.167%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:45:09.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1111486736' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:09.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:09 smithi086 ceph-mon[102754]: pgmap v182: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s; 13621/133968 objects degraded (10.167%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:45:09.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1111486736' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:09.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:09 smithi086 ceph-mon[97913]: pgmap v182: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s; 13621/133968 objects degraded (10.167%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:45:09.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1111486736' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:10.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 53516 53500 119.403 81.3125 0.00362015 0.00836841 2024-06-14T22:45:10.831 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:45:10 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:45:10.581+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:45:11.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 54833 54817 118.124 82.3125 0.00389949 0.00845847 2024-06-14T22:45:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:11 smithi137 ceph-mon[90634]: pgmap v183: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s; 13184/154509 objects degraded (8.533%); 6.2 MiB/s, 99 objects/s recovering 2024-06-14T22:45:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:11 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:11 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:45:11.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:11 smithi086 ceph-mon[102754]: pgmap v183: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s; 13184/154509 objects degraded (8.533%); 6.2 MiB/s, 99 objects/s recovering 2024-06-14T22:45:11.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:11.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:11 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:11.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:11 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:45:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:11 smithi086 ceph-mon[97913]: pgmap v183: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s; 13184/154509 objects degraded (8.533%); 6.2 MiB/s, 99 objects/s recovering 2024-06-14T22:45:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:11 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:11 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:45:12.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 56043 56027 116.707 75.625 0.00811933 0.00856248 2024-06-14T22:45:12.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2806325470' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:12.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2806325470' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2806325470' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:13.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 57319 57303 115.515 79.75 0.00442341 0.00865019 2024-06-14T22:45:13.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:13 smithi137 ceph-mon[90634]: pgmap v184: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 13184/154509 objects degraded (8.533%); 4.4 MiB/s, 69 objects/s recovering 2024-06-14T22:45:13.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 13184/154509 objects degraded (8.533%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:13.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:13 smithi086 ceph-mon[97913]: pgmap v184: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 13184/154509 objects degraded (8.533%); 4.4 MiB/s, 69 objects/s recovering 2024-06-14T22:45:13.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 13184/154509 objects degraded (8.533%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:13.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:13 smithi086 ceph-mon[102754]: pgmap v184: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 13184/154509 objects degraded (8.533%); 4.4 MiB/s, 69 objects/s recovering 2024-06-14T22:45:13.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 13184/154509 objects degraded (8.533%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:14.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 58544 58528 114.297 76.5625 0.0143704 0.00874209 2024-06-14T22:45:15.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 16 58935 58919 111.574 24.4375 0.00254522 0.00876839 2024-06-14T22:45:15.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:45:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:45:14] "GET /metrics HTTP/1.1" 200 36434 "" "Prometheus/2.43.0" 2024-06-14T22:45:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:15 smithi137 ceph-mon[90634]: pgmap v185: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 12690/173484 objects degraded (7.315%); 6.9 MiB/s, 111 objects/s recovering 2024-06-14T22:45:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:15 smithi137 ceph-mon[90634]: osdmap e94: 8 total, 8 up, 8 in 2024-06-14T22:45:15.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3367220109' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:15.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:15 smithi086 ceph-mon[97913]: pgmap v185: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 12690/173484 objects degraded (7.315%); 6.9 MiB/s, 111 objects/s recovering 2024-06-14T22:45:15.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:15 smithi086 ceph-mon[97913]: osdmap e94: 8 total, 8 up, 8 in 2024-06-14T22:45:15.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3367220109' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:15.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:15 smithi086 ceph-mon[102754]: pgmap v185: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 12690/173484 objects degraded (7.315%); 6.9 MiB/s, 111 objects/s recovering 2024-06-14T22:45:15.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:15 smithi086 ceph-mon[102754]: osdmap e94: 8 total, 8 up, 8 in 2024-06-14T22:45:15.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3367220109' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:16.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 59863 59847 109.998 58 0.00410145 0.00908497 2024-06-14T22:45:16.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:16 smithi137 ceph-mon[90634]: osdmap e95: 8 total, 8 up, 8 in 2024-06-14T22:45:16.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:16 smithi086 ceph-mon[97913]: osdmap e95: 8 total, 8 up, 8 in 2024-06-14T22:45:16.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:16 smithi086 ceph-mon[102754]: osdmap e95: 8 total, 8 up, 8 in 2024-06-14T22:45:17.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 61004 60988 108.893 71.3125 0.0333757 0.00917707 2024-06-14T22:45:17.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:17 smithi137 ceph-mon[90634]: pgmap v188: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 103 MiB/s wr, 1.64k op/s; 12690/173484 objects degraded (7.315%); 7.3 MiB/s, 116 objects/s recovering 2024-06-14T22:45:17.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/168456953' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:17 smithi086 ceph-mon[97913]: pgmap v188: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 103 MiB/s wr, 1.64k op/s; 12690/173484 objects degraded (7.315%); 7.3 MiB/s, 116 objects/s recovering 2024-06-14T22:45:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/168456953' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:17.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:17 smithi086 ceph-mon[102754]: pgmap v188: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 103 MiB/s wr, 1.64k op/s; 12690/173484 objects degraded (7.315%); 7.3 MiB/s, 116 objects/s recovering 2024-06-14T22:45:17.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/168456953' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:18.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 62197 62181 107.939 74.5625 0.00404891 0.00925826 2024-06-14T22:45:18.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 12690/173484 objects degraded (7.315%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:18.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 12690/173484 objects degraded (7.315%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:18.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 12690/173484 objects degraded (7.315%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:19.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 63257 63241 106.812 66.25 0.0165139 0.00935589 2024-06-14T22:45:19.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:19 smithi137 ceph-mon[90634]: pgmap v189: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s; 12690/173484 objects degraded (7.315%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T22:45:19.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:19 smithi086 ceph-mon[97913]: pgmap v189: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s; 12690/173484 objects degraded (7.315%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T22:45:19.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:19 smithi086 ceph-mon[102754]: pgmap v189: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s; 12690/173484 objects degraded (7.315%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T22:45:20.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 64425 64409 105.922 73 0.00882775 0.00943419 2024-06-14T22:45:20.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2332733296' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:20.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2332733296' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:20.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2332733296' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:21.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 16 65615 65599 105.113 74.375 0.015622 0.00950712 2024-06-14T22:45:21.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:21 smithi086 ceph-mon[97913]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 12273/187812 objects degraded (6.535%); 7.1 MiB/s, 113 objects/s recovering 2024-06-14T22:45:21.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:21 smithi086 ceph-mon[102754]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 12273/187812 objects degraded (6.535%); 7.1 MiB/s, 113 objects/s recovering 2024-06-14T22:45:21.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:21 smithi137 ceph-mon[90634]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 12273/187812 objects degraded (6.535%); 7.1 MiB/s, 113 objects/s recovering 2024-06-14T22:45:22.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:45:22.255470+0000 min lat: 0.00124087 max lat: 2.06531 avg lat: 0.00959267 2024-06-14T22:45:22.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:45:22.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 66697 66681 104.176 67.625 0.0118148 0.00959267 2024-06-14T22:45:22.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2795653804' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:22.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2795653804' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:22.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2795653804' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:23.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 67858 67842 103.405 72.5625 0.00829386 0.0096645 2024-06-14T22:45:23.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:23 smithi086 ceph-mon[102754]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 37 MiB/s wr, 596 op/s; 12273/187812 objects degraded (6.535%); 3.3 MiB/s, 52 objects/s recovering 2024-06-14T22:45:23.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 12273/187812 objects degraded (6.535%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:23.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:23 smithi086 ceph-mon[97913]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 37 MiB/s wr, 596 op/s; 12273/187812 objects degraded (6.535%); 3.3 MiB/s, 52 objects/s recovering 2024-06-14T22:45:23.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 12273/187812 objects degraded (6.535%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:23.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:23 smithi137 ceph-mon[90634]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 37 MiB/s wr, 596 op/s; 12273/187812 objects degraded (6.535%); 3.3 MiB/s, 52 objects/s recovering 2024-06-14T22:45:23.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 12273/187812 objects degraded (6.535%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:24.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 16 68975 68959 102.605 69.8125 0.015237 0.00973976 2024-06-14T22:45:25.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 70086 70070 101.833 69.4375 0.0230781 0.00981373 2024-06-14T22:45:25.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:45:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:45:24] "GET /metrics HTTP/1.1" 200 36434 "" "Prometheus/2.43.0" 2024-06-14T22:45:25.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:25 smithi086 ceph-mon[97913]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 12052/205017 objects degraded (5.879%); 4.0 MiB/s, 64 objects/s recovering 2024-06-14T22:45:25.596 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/180169060' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:25.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:25 smithi086 ceph-mon[102754]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 12052/205017 objects degraded (5.879%); 4.0 MiB/s, 64 objects/s recovering 2024-06-14T22:45:25.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/180169060' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:25.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:25 smithi137 ceph-mon[90634]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 12052/205017 objects degraded (5.879%); 4.0 MiB/s, 64 objects/s recovering 2024-06-14T22:45:25.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/180169060' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:26.081 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:45:25 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:45:25.598+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:45:26.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 16 71225 71209 101.137 71.1875 0.0437784 0.00988019 2024-06-14T22:45:26.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:26 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:26.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:26 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:26.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:26 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:45:26.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:26 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:26.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:26 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:26.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:26 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:45:26.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:26 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:26.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:26 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:26.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:26 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:45:27.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 72281 72265 100.356 66 0.00474306 0.00995758 2024-06-14T22:45:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:27 smithi086 ceph-mon[97913]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s; 12052/205017 objects degraded (5.879%); 3.7 MiB/s, 58 objects/s recovering 2024-06-14T22:45:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/750739330' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:27 smithi086 ceph-mon[102754]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s; 12052/205017 objects degraded (5.879%); 3.7 MiB/s, 58 objects/s recovering 2024-06-14T22:45:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/750739330' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:27.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:27 smithi137 ceph-mon[90634]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s; 12052/205017 objects degraded (5.879%); 3.7 MiB/s, 58 objects/s recovering 2024-06-14T22:45:27.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/750739330' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:27.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:28.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 16 73415 73399 99.7149 70.875 0.0136236 0.0100228 2024-06-14T22:45:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 12052/205017 objects degraded (5.879%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:28.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 12052/205017 objects degraded (5.879%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:28.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 12052/205017 objects degraded (5.879%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:29.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 74488 74472 99.0201 67.0625 0.0212564 0.0100931 2024-06-14T22:45:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:29 smithi086 ceph-mon[97913]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 12052/205017 objects degraded (5.879%); 3.3 MiB/s, 53 objects/s recovering 2024-06-14T22:45:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:29 smithi086 ceph-mon[102754]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 12052/205017 objects degraded (5.879%); 3.3 MiB/s, 53 objects/s recovering 2024-06-14T22:45:29.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:29 smithi137 ceph-mon[90634]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 12052/205017 objects degraded (5.879%); 3.3 MiB/s, 53 objects/s recovering 2024-06-14T22:45:30.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 75531 75515 98.3151 65.1875 0.00718255 0.0101641 2024-06-14T22:45:30.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1568129066' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:30.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1568129066' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1568129066' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:31.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 76624 76608 97.7027 68.3125 0.00865268 0.0102288 2024-06-14T22:45:31.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:31 smithi086 ceph-mon[97913]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 11827/221691 objects degraded (5.335%); 4.5 MiB/s, 71 objects/s recovering 2024-06-14T22:45:31.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:31 smithi086 ceph-mon[102754]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 11827/221691 objects degraded (5.335%); 4.5 MiB/s, 71 objects/s recovering 2024-06-14T22:45:31.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:31 smithi137 ceph-mon[90634]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 11827/221691 objects degraded (5.335%); 4.5 MiB/s, 71 objects/s recovering 2024-06-14T22:45:32.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 77647 77631 97.0273 63.9375 0.0167376 0.0102997 2024-06-14T22:45:32.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1751153040' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:32.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1751153040' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:32.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1751153040' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:33.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 78706 78690 96.4225 66.1875 0.0090638 0.0103647 2024-06-14T22:45:33.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:33 smithi086 ceph-mon[102754]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 59 MiB/s wr, 940 op/s; 11827/221691 objects degraded (5.335%); 2.3 MiB/s, 37 objects/s recovering 2024-06-14T22:45:33.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 11827/221691 objects degraded (5.335%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:33 smithi086 ceph-mon[97913]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 59 MiB/s wr, 940 op/s; 11827/221691 objects degraded (5.335%); 2.3 MiB/s, 37 objects/s recovering 2024-06-14T22:45:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 11827/221691 objects degraded (5.335%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:33.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:33 smithi137 ceph-mon[90634]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 59 MiB/s wr, 940 op/s; 11827/221691 objects degraded (5.335%); 2.3 MiB/s, 37 objects/s recovering 2024-06-14T22:45:33.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 11827/221691 objects degraded (5.335%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:34.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 16 79771 79755 95.8482 66.5625 0.0139873 0.0104271 2024-06-14T22:45:35.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 15 80751 80736 95.1965 61.3125 0.0111192 0.0104985 2024-06-14T22:45:35.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:45:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:45:34] "GET /metrics HTTP/1.1" 200 36436 "" "Prometheus/2.43.0" 2024-06-14T22:45:35.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:35 smithi137 ceph-mon[90634]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 11586/237570 objects degraded (4.877%); 3.6 MiB/s, 57 objects/s recovering 2024-06-14T22:45:35.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1294177404' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:36.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:35 smithi086 ceph-mon[97913]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 11586/237570 objects degraded (4.877%); 3.6 MiB/s, 57 objects/s recovering 2024-06-14T22:45:36.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1294177404' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:36.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:35 smithi086 ceph-mon[102754]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 11586/237570 objects degraded (4.877%); 3.6 MiB/s, 57 objects/s recovering 2024-06-14T22:45:36.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1294177404' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:36.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 81805 81789 94.6522 65.8125 0.0101775 0.0105583 2024-06-14T22:45:37.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 82786 82770 94.046 61.3125 0.00720361 0.0106264 2024-06-14T22:45:37.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:37 smithi137 ceph-mon[90634]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s; 11586/237570 objects degraded (4.877%); 2.4 MiB/s, 38 objects/s recovering 2024-06-14T22:45:37.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2348551497' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:38.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:37 smithi086 ceph-mon[97913]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s; 11586/237570 objects degraded (4.877%); 2.4 MiB/s, 38 objects/s recovering 2024-06-14T22:45:38.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2348551497' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:38.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:37 smithi086 ceph-mon[102754]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s; 11586/237570 objects degraded (4.877%); 2.4 MiB/s, 38 objects/s recovering 2024-06-14T22:45:38.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2348551497' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:38.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 83832 83816 93.5339 65.375 0.0104322 0.0106862 2024-06-14T22:45:38.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 11586/237570 objects degraded (4.877%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:39.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 11586/237570 objects degraded (4.877%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:39.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 11586/237570 objects degraded (4.877%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:39.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 84864 84848 93.0244 64.5 0.0129666 0.0107433 2024-06-14T22:45:39.891 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:39 smithi137 ceph-mon[90634]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s; 11586/237570 objects degraded (4.877%); 2.4 MiB/s, 38 objects/s recovering 2024-06-14T22:45:40.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:39 smithi086 ceph-mon[102754]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s; 11586/237570 objects degraded (4.877%); 2.4 MiB/s, 38 objects/s recovering 2024-06-14T22:45:40.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:39 smithi086 ceph-mon[97913]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s; 11586/237570 objects degraded (4.877%); 2.4 MiB/s, 38 objects/s recovering 2024-06-14T22:45:40.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 16 85831 85815 92.4625 60.4375 0.00703508 0.0108087 2024-06-14T22:45:40.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1574334866' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:41.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1574334866' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:41.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1574334866' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:41.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:45:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:45:40.619+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:45:41.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 86850 86834 91.9747 63.6875 0.00976151 0.0108668 2024-06-14T22:45:41.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:41 smithi137 ceph-mon[90634]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 11292/252882 objects degraded (4.465%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:45:41.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:42.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:41 smithi086 ceph-mon[97913]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 11292/252882 objects degraded (4.465%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:45:42.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:42.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:41 smithi086 ceph-mon[102754]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 11292/252882 objects degraded (4.465%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:45:42.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:42.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:45:42.257276+0000 min lat: 0.00124087 max lat: 2.06531 avg lat: 0.01093 2024-06-14T22:45:42.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:45:42.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 87809 87793 91.4407 59.9375 0.0218317 0.01093 2024-06-14T22:45:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:42 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:42 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:45:42.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:42 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/570203088' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:42.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:43.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:42 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:43.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:42 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:45:43.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:42 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/570203088' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:43.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:42 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:45:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:42 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:45:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:42 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/570203088' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:43.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 88812 88796 90.9692 62.6875 0.00928437 0.0109864 2024-06-14T22:45:43.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:43 smithi137 ceph-mon[90634]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 54 MiB/s wr, 865 op/s; 11292/252882 objects degraded (4.465%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:45:43.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 11292/252882 objects degraded (4.465%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:44.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:43 smithi086 ceph-mon[97913]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 54 MiB/s wr, 865 op/s; 11292/252882 objects degraded (4.465%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:45:44.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 11292/252882 objects degraded (4.465%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:44.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:43 smithi086 ceph-mon[102754]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 54 MiB/s wr, 865 op/s; 11292/252882 objects degraded (4.465%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:45:44.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 11292/252882 objects degraded (4.465%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:44.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 89828 89812 90.5261 63.5 0.00934749 0.0110399 2024-06-14T22:45:44.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:44 smithi137 ceph-mon[90634]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 11006/267786 objects degraded (4.110%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:45:45.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:44 smithi086 ceph-mon[97913]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 11006/267786 objects degraded (4.110%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:45:45.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:44 smithi086 ceph-mon[102754]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 11006/267786 objects degraded (4.110%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:45:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:45:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:45:44] "GET /metrics HTTP/1.1" 200 36434 "" "Prometheus/2.43.0" 2024-06-14T22:45:45.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 90778 90762 90.0315 59.375 0.0192282 0.0111015 2024-06-14T22:45:45.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1619879048' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:46.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1619879048' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:46.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1619879048' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:46.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 91718 91702 89.5427 58.75 0.022179 0.0111615 2024-06-14T22:45:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:46 smithi137 ceph-mon[90634]: pgmap v203: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 52 MiB/s wr, 838 op/s; 11006/267786 objects degraded (4.110%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:45:47.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:46 smithi086 ceph-mon[97913]: pgmap v203: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 52 MiB/s wr, 838 op/s; 11006/267786 objects degraded (4.110%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:45:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:46 smithi086 ceph-mon[102754]: pgmap v203: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 52 MiB/s wr, 838 op/s; 11006/267786 objects degraded (4.110%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:45:47.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 92304 92288 88.7285 36.625 0.00358657 0.0111803 2024-06-14T22:45:47.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:47 smithi137 ceph-mon[90634]: osdmap e96: 8 total, 8 up, 8 in 2024-06-14T22:45:47.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3702664808' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:48.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:47 smithi086 ceph-mon[97913]: osdmap e96: 8 total, 8 up, 8 in 2024-06-14T22:45:48.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3702664808' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:48.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:47 smithi086 ceph-mon[102754]: osdmap e96: 8 total, 8 up, 8 in 2024-06-14T22:45:48.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3702664808' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:48.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 92876 92860 87.9258 35.75 0.0235683 0.011367 2024-06-14T22:45:48.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:48 smithi137 ceph-mon[90634]: osdmap e97: 8 total, 8 up, 8 in 2024-06-14T22:45:48.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 11006/267786 objects degraded (4.110%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:48.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:48 smithi137 ceph-mon[90634]: pgmap v206: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 39 MiB/s wr, 619 op/s; 11006/267786 objects degraded (4.110%); 2.2 MiB/s, 35 objects/s recovering 2024-06-14T22:45:49.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:48 smithi086 ceph-mon[102754]: osdmap e97: 8 total, 8 up, 8 in 2024-06-14T22:45:49.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 11006/267786 objects degraded (4.110%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:49.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:48 smithi086 ceph-mon[102754]: pgmap v206: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 39 MiB/s wr, 619 op/s; 11006/267786 objects degraded (4.110%); 2.2 MiB/s, 35 objects/s recovering 2024-06-14T22:45:49.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:48 smithi086 ceph-mon[97913]: osdmap e97: 8 total, 8 up, 8 in 2024-06-14T22:45:49.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 11006/267786 objects degraded (4.110%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:49.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:48 smithi086 ceph-mon[97913]: pgmap v206: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 39 MiB/s wr, 619 op/s; 11006/267786 objects degraded (4.110%); 2.2 MiB/s, 35 objects/s recovering 2024-06-14T22:45:49.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 93832 93816 87.5052 59.75 0.0276805 0.0114217 2024-06-14T22:45:49.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/452017732' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:50.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/452017732' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:50.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/452017732' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:50.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 94730 94714 87.0437 56.125 0.010745 0.0114818 2024-06-14T22:45:50.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:50 smithi137 ceph-mon[90634]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 10670/280041 objects degraded (3.810%); 4.9 MiB/s, 77 objects/s recovering 2024-06-14T22:45:51.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:50 smithi086 ceph-mon[97913]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 10670/280041 objects degraded (3.810%); 4.9 MiB/s, 77 objects/s recovering 2024-06-14T22:45:51.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:50 smithi086 ceph-mon[102754]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 10670/280041 objects degraded (3.810%); 4.9 MiB/s, 77 objects/s recovering 2024-06-14T22:45:51.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 95681 95665 86.6435 59.4375 0.00751076 0.0115355 2024-06-14T22:45:52.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 96566 96550 86.1959 55.3125 0.0148709 0.0115948 2024-06-14T22:45:52.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3159022368' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:52.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3159022368' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:52.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3159022368' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:53.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 97526 97510 85.8268 60 0.0162411 0.0116457 2024-06-14T22:45:53.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:53 smithi137 ceph-mon[90634]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 32 MiB/s wr, 510 op/s; 10670/280041 objects degraded (3.810%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:45:53.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10670/280041 objects degraded (3.810%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:53.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:53 smithi086 ceph-mon[97913]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 32 MiB/s wr, 510 op/s; 10670/280041 objects degraded (3.810%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:45:53.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10670/280041 objects degraded (3.810%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:53.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:53 smithi086 ceph-mon[102754]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 32 MiB/s wr, 510 op/s; 10670/280041 objects degraded (3.810%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:45:53.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10670/280041 objects degraded (3.810%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:45:54.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 98453 98437 85.4394 57.9375 0.00934517 0.0116982 2024-06-14T22:45:55.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 99326 99310 85.0164 54.5625 0.012934 0.0117564 2024-06-14T22:45:55.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:45:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:45:54] "GET /metrics HTTP/1.1" 200 36434 "" "Prometheus/2.43.0" 2024-06-14T22:45:55.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:55 smithi137 ceph-mon[90634]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 10483/293160 objects degraded (3.576%); 4.1 MiB/s, 65 objects/s recovering 2024-06-14T22:45:55.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2111003522' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:55.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:55 smithi086 ceph-mon[102754]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 10483/293160 objects degraded (3.576%); 4.1 MiB/s, 65 objects/s recovering 2024-06-14T22:45:55.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:55 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2111003522' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:55.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:55 smithi086 ceph-mon[97913]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 10483/293160 objects degraded (3.576%); 4.1 MiB/s, 65 objects/s recovering 2024-06-14T22:45:55.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:55 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2111003522' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:56.255 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 16 100188 100172 84.5951 53.875 0.0117649 0.011815 2024-06-14T22:45:56.566 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:45:56.773 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:45:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:45:57.256 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 100960 100944 84.1094 48.25 0.0271886 0.0118831 2024-06-14T22:45:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:57 smithi137 ceph-mon[90634]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 56 MiB/s wr, 898 op/s; 10483/294000 objects degraded (3.566%); 3.4 MiB/s, 53 objects/s recovering 2024-06-14T22:45:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4161385394' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:57.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:57 smithi086 ceph-mon[97913]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 56 MiB/s wr, 898 op/s; 10483/294000 objects degraded (3.566%); 3.4 MiB/s, 53 objects/s recovering 2024-06-14T22:45:57.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4161385394' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:57.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:57.767 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:57 smithi086 ceph-mon[102754]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 56 MiB/s wr, 898 op/s; 10483/294000 objects degraded (3.566%); 3.4 MiB/s, 53 objects/s recovering 2024-06-14T22:45:57.767 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4161385394' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:45:57.767 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:45:58.256 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 101792 101776 83.6869 52 0.02994 0.0119431 2024-06-14T22:45:58.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10483/294000 objects degraded (3.566%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:45:58.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10483/294000 objects degraded (3.566%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:45:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10483/294000 objects degraded (3.566%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:45:59.256 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 16 102610 102594 83.2635 51.125 0.0174566 0.0120037 2024-06-14T22:45:59.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:45:59 smithi086 ceph-mon[102754]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 51 MiB/s wr, 814 op/s; 10483/294000 objects degraded (3.566%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:45:59.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:45:59 smithi086 ceph-mon[97913]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 51 MiB/s wr, 814 op/s; 10483/294000 objects degraded (3.566%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:45:59.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:45:59 smithi137 ceph-mon[90634]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 51 MiB/s wr, 814 op/s; 10483/294000 objects degraded (3.566%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:46:00.257 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 103402 103386 82.8303 49.5 0.0240143 0.0120669 2024-06-14T22:46:00.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/449011891' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:00.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/449011891' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:00.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/449011891' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:01.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 104237 104221 82.4415 52.1875 0.0223044 0.012124 2024-06-14T22:46:01.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:01 smithi137 ceph-mon[90634]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 10289/306639 objects degraded (3.355%); 3.7 MiB/s, 59 objects/s recovering 2024-06-14T22:46:01.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[97913]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 10289/306639 objects degraded (3.355%); 3.7 MiB/s, 59 objects/s recovering 2024-06-14T22:46:01.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[102754]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 10289/306639 objects degraded (3.355%); 3.7 MiB/s, 59 objects/s recovering 2024-06-14T22:46:01.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:01.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:02.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:46:02.261847+0000 min lat: 0.00124087 max lat: 2.06531 avg lat: 0.0121878 2024-06-14T22:46:02.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:46:02.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 105007 104991 82.0126 48.125 0.00883818 0.0121878 2024-06-14T22:46:02.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:01 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:46:01.952+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:46:02.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:46:02.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:46:02.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:02.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:46:02.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:46:02.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:46:02.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:46:02.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:02.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2640969807' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:02.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2640969807' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:46:02.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:46:02.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:46:02.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:46:02.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:46:02.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:46:02.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:46:02.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:02.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2640969807' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:03.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 105879 105863 81.6729 54.5 0.0178802 0.0122382 2024-06-14T22:46:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:03 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:03 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:46:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:03 smithi137 ceph-mon[90634]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 738 op/s; 10289/306639 objects degraded (3.355%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T22:46:03.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10289/306639 objects degraded (3.355%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:03.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:03 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:03.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:03 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:46:03.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:03 smithi086 ceph-mon[97913]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 738 op/s; 10289/306639 objects degraded (3.355%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T22:46:03.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10289/306639 objects degraded (3.355%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:03.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:03 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:03.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:03 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:46:03.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:03 smithi086 ceph-mon[102754]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 738 op/s; 10289/306639 objects degraded (3.355%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T22:46:03.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10289/306639 objects degraded (3.355%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:04.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 106780 106764 81.3636 56.3125 0.0353153 0.0122841 2024-06-14T22:46:05.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 15 107613 107598 81.0112 52.125 0.0342113 0.012338 2024-06-14T22:46:05.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:46:04] "GET /metrics HTTP/1.1" 200 36434 "" "Prometheus/2.43.0" 2024-06-14T22:46:05.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:05 smithi137 ceph-mon[90634]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 10079/318276 objects degraded (3.167%); 3.1 MiB/s, 49 objects/s recovering 2024-06-14T22:46:05.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2927680941' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:05 smithi086 ceph-mon[97913]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 10079/318276 objects degraded (3.167%); 3.1 MiB/s, 49 objects/s recovering 2024-06-14T22:46:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2927680941' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:05.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:05 smithi086 ceph-mon[102754]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 10079/318276 objects degraded (3.167%); 3.1 MiB/s, 49 objects/s recovering 2024-06-14T22:46:05.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2927680941' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:06.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 108487 108471 80.6963 54.5625 0.0136334 0.0123856 2024-06-14T22:46:07.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 109307 109291 80.3499 51.25 0.0177836 0.0124391 2024-06-14T22:46:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:07 smithi137 ceph-mon[90634]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 45 MiB/s wr, 719 op/s; 10079/319059 objects degraded (3.159%); 2.1 MiB/s, 33 objects/s recovering 2024-06-14T22:46:07.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1636798257' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:07.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:07 smithi086 ceph-mon[97913]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 45 MiB/s wr, 719 op/s; 10079/319059 objects degraded (3.159%); 2.1 MiB/s, 33 objects/s recovering 2024-06-14T22:46:07.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1636798257' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:07.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:07 smithi086 ceph-mon[102754]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 45 MiB/s wr, 719 op/s; 10079/319059 objects degraded (3.159%); 2.1 MiB/s, 33 objects/s recovering 2024-06-14T22:46:07.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1636798257' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:08.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 110169 110153 80.042 53.875 0.0140173 0.0124868 2024-06-14T22:46:08.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10079/319059 objects degraded (3.159%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:08.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10079/319059 objects degraded (3.159%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:08.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10079/319059 objects degraded (3.159%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:09.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 111036 111020 79.7448 54.1875 0.0171408 0.0125336 2024-06-14T22:46:09.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:09 smithi137 ceph-mon[90634]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 43 MiB/s wr, 695 op/s; 10079/319059 objects degraded (3.159%); 2.1 MiB/s, 33 objects/s recovering 2024-06-14T22:46:09.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:09 smithi086 ceph-mon[97913]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 43 MiB/s wr, 695 op/s; 10079/319059 objects degraded (3.159%); 2.1 MiB/s, 33 objects/s recovering 2024-06-14T22:46:09.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:09 smithi086 ceph-mon[102754]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 43 MiB/s wr, 695 op/s; 10079/319059 objects degraded (3.159%); 2.1 MiB/s, 33 objects/s recovering 2024-06-14T22:46:10.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 111849 111833 79.4159 50.8125 0.0199283 0.012586 2024-06-14T22:46:10.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/798108750' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:10.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/798108750' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:10.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/798108750' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:11.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 112705 112689 79.1247 53.5 0.012181 0.012632 2024-06-14T22:46:11.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:11 smithi086 ceph-mon[97913]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 9844/331938 objects degraded (2.966%); 3.3 MiB/s, 53 objects/s recovering 2024-06-14T22:46:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:11 smithi086 ceph-mon[102754]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 9844/331938 objects degraded (2.966%); 3.3 MiB/s, 53 objects/s recovering 2024-06-14T22:46:11.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:11 smithi137 ceph-mon[90634]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 9844/331938 objects degraded (2.966%); 3.3 MiB/s, 53 objects/s recovering 2024-06-14T22:46:12.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 90 2 113512 113510 78.8157 51.3125 0.0184863 0.0126822 2024-06-14T22:46:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3830460697' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3830460697' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3830460697' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0149 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 113512 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 78.8147 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 47.5464 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 288.562 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 0 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1261 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 760.742 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 4617 2024-06-14T22:46:13.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 0 2024-06-14T22:46:13.260 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0126823 2024-06-14T22:46:13.260 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.0300374 2024-06-14T22:46:13.260 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 2.06531 2024-06-14T22:46:13.260 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00124087 2024-06-14T22:46:13.265 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T22:46:13.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:13 smithi086 ceph-mon[97913]: pgmap v218: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 44 MiB/s wr, 702 op/s; 9844/331938 objects degraded (2.966%); 2.3 MiB/s, 37 objects/s recovering 2024-06-14T22:46:13.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 9844/331938 objects degraded (2.966%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:13.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:13 smithi086 ceph-mon[102754]: pgmap v218: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 44 MiB/s wr, 702 op/s; 9844/331938 objects degraded (2.966%); 2.3 MiB/s, 37 objects/s recovering 2024-06-14T22:46:13.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 9844/331938 objects degraded (2.966%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:13.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:13 smithi137 ceph-mon[90634]: pgmap v218: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 44 MiB/s wr, 702 op/s; 9844/331938 objects degraded (2.966%); 2.3 MiB/s, 37 objects/s recovering 2024-06-14T22:46:13.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 9844/331938 objects degraded (2.966%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:15.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:46:14] "GET /metrics HTTP/1.1" 200 36437 "" "Prometheus/2.43.0" 2024-06-14T22:46:15.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:15 smithi086 ceph-mon[97913]: pgmap v219: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 58 MiB/s wr, 966 op/s; 9322/339051 objects degraded (2.749%); 5.0 MiB/s, 80 objects/s recovering 2024-06-14T22:46:15.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1213011939' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:15.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:15 smithi086 ceph-mon[102754]: pgmap v219: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 58 MiB/s wr, 966 op/s; 9322/339051 objects degraded (2.749%); 5.0 MiB/s, 80 objects/s recovering 2024-06-14T22:46:15.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1213011939' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:15.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:15 smithi137 ceph-mon[90634]: pgmap v219: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 58 MiB/s wr, 966 op/s; 9322/339051 objects degraded (2.749%); 5.0 MiB/s, 80 objects/s recovering 2024-06-14T22:46:15.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1213011939' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:17.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:16 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:46:16.971+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:46:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:17 smithi086 ceph-mon[97913]: pgmap v220: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 39 MiB/s wr, 660 op/s; 9322/339168 objects degraded (2.748%); 3.9 MiB/s, 63 objects/s recovering 2024-06-14T22:46:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:17.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1733723063' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:17.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:17 smithi086 ceph-mon[102754]: pgmap v220: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 39 MiB/s wr, 660 op/s; 9322/339168 objects degraded (2.748%); 3.9 MiB/s, 63 objects/s recovering 2024-06-14T22:46:17.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:17.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1733723063' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:17.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:17 smithi137 ceph-mon[90634]: pgmap v220: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 39 MiB/s wr, 660 op/s; 9322/339168 objects degraded (2.748%); 3.9 MiB/s, 63 objects/s recovering 2024-06-14T22:46:17.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:17.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1733723063' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:18.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:18 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:18.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:18 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:46:18.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 9322/339168 objects degraded (2.748%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:18.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:18 smithi086 ceph-mon[97913]: osdmap e98: 8 total, 8 up, 8 in 2024-06-14T22:46:18.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:18 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:18.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:18 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:46:18.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 9322/339168 objects degraded (2.748%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:18.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:18 smithi086 ceph-mon[102754]: osdmap e98: 8 total, 8 up, 8 in 2024-06-14T22:46:18.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:18 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:18.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:18 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:46:18.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 9322/339168 objects degraded (2.748%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:18.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:18 smithi137 ceph-mon[90634]: osdmap e98: 8 total, 8 up, 8 in 2024-06-14T22:46:19.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:19 smithi086 ceph-mon[97913]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 102 B/s rd, 45 MiB/s wr, 766 op/s; 9322/339168 objects degraded (2.748%); 4.7 MiB/s, 75 objects/s recovering 2024-06-14T22:46:19.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:19 smithi086 ceph-mon[97913]: osdmap e99: 8 total, 8 up, 8 in 2024-06-14T22:46:19.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:19 smithi086 ceph-mon[102754]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 102 B/s rd, 45 MiB/s wr, 766 op/s; 9322/339168 objects degraded (2.748%); 4.7 MiB/s, 75 objects/s recovering 2024-06-14T22:46:19.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:19 smithi086 ceph-mon[102754]: osdmap e99: 8 total, 8 up, 8 in 2024-06-14T22:46:19.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:19 smithi137 ceph-mon[90634]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 102 B/s rd, 45 MiB/s wr, 766 op/s; 9322/339168 objects degraded (2.748%); 4.7 MiB/s, 75 objects/s recovering 2024-06-14T22:46:19.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:19 smithi137 ceph-mon[90634]: osdmap e99: 8 total, 8 up, 8 in 2024-06-14T22:46:20.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4020132465' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:20.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4020132465' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:20.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4020132465' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:21.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:21 smithi137 ceph-mon[90634]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 127 B/s rd, 23 MiB/s wr, 921 op/s; 9023/327162 objects degraded (2.758%); 6.4 MiB/s, 102 objects/s recovering 2024-06-14T22:46:21.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:21 smithi086 ceph-mon[97913]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 127 B/s rd, 23 MiB/s wr, 921 op/s; 9023/327162 objects degraded (2.758%); 6.4 MiB/s, 102 objects/s recovering 2024-06-14T22:46:21.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:21 smithi086 ceph-mon[102754]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 127 B/s rd, 23 MiB/s wr, 921 op/s; 9023/327162 objects degraded (2.758%); 6.4 MiB/s, 102 objects/s recovering 2024-06-14T22:46:22.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2039028797' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:22.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2039028797' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:22.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2039028797' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:23.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:23 smithi086 ceph-mon[97913]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 992 KiB/s wr, 526 op/s; 9023/327162 objects degraded (2.758%); 2.3 MiB/s, 37 objects/s recovering 2024-06-14T22:46:23.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 9023/327162 objects degraded (2.758%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:23.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:23 smithi086 ceph-mon[102754]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 992 KiB/s wr, 526 op/s; 9023/327162 objects degraded (2.758%); 2.3 MiB/s, 37 objects/s recovering 2024-06-14T22:46:23.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 9023/327162 objects degraded (2.758%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:23.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:23 smithi137 ceph-mon[90634]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 992 KiB/s wr, 526 op/s; 9023/327162 objects degraded (2.758%); 2.3 MiB/s, 37 objects/s recovering 2024-06-14T22:46:23.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 9023/327162 objects degraded (2.758%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:25.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:46:24] "GET /metrics HTTP/1.1" 200 36437 "" "Prometheus/2.43.0" 2024-06-14T22:46:25.760 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:25 smithi086 ceph-mon[102754]: pgmap v226: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 8689/316722 objects degraded (2.743%); 4.9 MiB/s, 79 objects/s recovering 2024-06-14T22:46:25.760 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1463499354' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:25.760 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:25 smithi086 ceph-mon[97913]: pgmap v226: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 8689/316722 objects degraded (2.743%); 4.9 MiB/s, 79 objects/s recovering 2024-06-14T22:46:25.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1463499354' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:25.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:25 smithi137 ceph-mon[90634]: pgmap v226: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 8689/316722 objects degraded (2.743%); 4.9 MiB/s, 79 objects/s recovering 2024-06-14T22:46:25.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1463499354' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:27.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:27 smithi086 ceph-mon[102754]: pgmap v227: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 962 op/s; 8689/315081 objects degraded (2.758%); 4.7 MiB/s, 75 objects/s recovering 2024-06-14T22:46:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2149301353' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:27 smithi086 ceph-mon[97913]: pgmap v227: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 962 op/s; 8689/315081 objects degraded (2.758%); 4.7 MiB/s, 75 objects/s recovering 2024-06-14T22:46:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2149301353' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:27.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:27 smithi137 ceph-mon[90634]: pgmap v227: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 962 op/s; 8689/315081 objects degraded (2.758%); 4.7 MiB/s, 75 objects/s recovering 2024-06-14T22:46:27.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:27.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2149301353' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:28.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8689/315081 objects degraded (2.758%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8689/315081 objects degraded (2.758%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:28.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8689/315081 objects degraded (2.758%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:29 smithi086 ceph-mon[97913]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 8689/315081 objects degraded (2.758%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:46:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:29 smithi086 ceph-mon[102754]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 8689/315081 objects degraded (2.758%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:46:29.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:29 smithi137 ceph-mon[90634]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 8689/315081 objects degraded (2.758%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:46:30.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/981021130' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:30.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/981021130' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/981021130' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:31.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:31 smithi086 ceph-mon[102754]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 8466/303000 objects degraded (2.794%); 4.4 MiB/s, 70 objects/s recovering 2024-06-14T22:46:31.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:31 smithi086 ceph-mon[97913]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 8466/303000 objects degraded (2.794%); 4.4 MiB/s, 70 objects/s recovering 2024-06-14T22:46:31.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:31 smithi137 ceph-mon[90634]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 8466/303000 objects degraded (2.794%); 4.4 MiB/s, 70 objects/s recovering 2024-06-14T22:46:32.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:31 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:46:31.992+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:46:32.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:32.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/860930778' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:33.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:33.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/860930778' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:33.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/860930778' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:33.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:33 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:33.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:33 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:46:33.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:33 smithi137 ceph-mon[90634]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 8466/303000 objects degraded (2.794%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T22:46:33.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8466/303000 objects degraded (2.794%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:34.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:33 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:34.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:33 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:46:34.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:33 smithi086 ceph-mon[97913]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 8466/303000 objects degraded (2.794%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T22:46:34.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8466/303000 objects degraded (2.794%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:34.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:33 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:34.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:33 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:46:34.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:33 smithi086 ceph-mon[102754]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 8466/303000 objects degraded (2.794%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T22:46:34.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8466/303000 objects degraded (2.794%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:35.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:46:34] "GET /metrics HTTP/1.1" 200 36442 "" "Prometheus/2.43.0" 2024-06-14T22:46:35.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:35 smithi137 ceph-mon[90634]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 8194/292065 objects degraded (2.806%); 4.3 MiB/s, 69 objects/s recovering 2024-06-14T22:46:35.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1030253829' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:36.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:35 smithi086 ceph-mon[97913]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 8194/292065 objects degraded (2.806%); 4.3 MiB/s, 69 objects/s recovering 2024-06-14T22:46:36.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1030253829' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:36.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:35 smithi086 ceph-mon[102754]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 8194/292065 objects degraded (2.806%); 4.3 MiB/s, 69 objects/s recovering 2024-06-14T22:46:36.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1030253829' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:37.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:37 smithi086 ceph-mon[97913]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 8194/290487 objects degraded (2.821%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:46:37.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/443653420' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:37.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:37 smithi086 ceph-mon[102754]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 8194/290487 objects degraded (2.821%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:46:37.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/443653420' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:37.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:37 smithi137 ceph-mon[90634]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 8194/290487 objects degraded (2.821%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:46:37.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/443653420' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:38.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8194/290487 objects degraded (2.821%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:39.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8194/290487 objects degraded (2.821%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:39.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8194/290487 objects degraded (2.821%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:39.891 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:39 smithi137 ceph-mon[90634]: pgmap v233: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 8194/290487 objects degraded (2.821%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:46:40.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:39 smithi086 ceph-mon[97913]: pgmap v233: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 8194/290487 objects degraded (2.821%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:46:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:39 smithi086 ceph-mon[102754]: pgmap v233: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 8194/290487 objects degraded (2.821%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:46:40.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/517912698' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:41.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/517912698' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:41.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/517912698' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:41.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:41 smithi137 ceph-mon[90634]: pgmap v234: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 7936/278283 objects degraded (2.852%); 3.9 MiB/s, 62 objects/s recovering 2024-06-14T22:46:42.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:41 smithi086 ceph-mon[97913]: pgmap v234: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 7936/278283 objects degraded (2.852%); 3.9 MiB/s, 62 objects/s recovering 2024-06-14T22:46:42.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:41 smithi086 ceph-mon[102754]: pgmap v234: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 7936/278283 objects degraded (2.852%); 3.9 MiB/s, 62 objects/s recovering 2024-06-14T22:46:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:42 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2569722314' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:42 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2569722314' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:43.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:43.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:42 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2569722314' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:43.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:43 smithi137 ceph-mon[90634]: pgmap v235: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 7936/278283 objects degraded (2.852%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:46:43.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 7936/278283 objects degraded (2.852%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:44.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:43 smithi086 ceph-mon[102754]: pgmap v235: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 7936/278283 objects degraded (2.852%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:46:44.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 7936/278283 objects degraded (2.852%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:44.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:43 smithi086 ceph-mon[97913]: pgmap v235: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 7936/278283 objects degraded (2.852%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:46:44.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 7936/278283 objects degraded (2.852%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:44.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:44 smithi137 ceph-mon[90634]: pgmap v236: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 982 op/s; 7665/267633 objects degraded (2.864%); 4.2 MiB/s, 66 objects/s recovering 2024-06-14T22:46:45.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:44 smithi086 ceph-mon[97913]: pgmap v236: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 982 op/s; 7665/267633 objects degraded (2.864%); 4.2 MiB/s, 66 objects/s recovering 2024-06-14T22:46:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:46:44] "GET /metrics HTTP/1.1" 200 36371 "" "Prometheus/2.43.0" 2024-06-14T22:46:45.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:44 smithi086 ceph-mon[102754]: pgmap v236: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 982 op/s; 7665/267633 objects degraded (2.864%); 4.2 MiB/s, 66 objects/s recovering 2024-06-14T22:46:45.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3589771193' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:46.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3589771193' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:46.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3589771193' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:46.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:46 smithi137 ceph-mon[90634]: pgmap v237: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 7665/266100 objects degraded (2.880%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:46:47.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:46 smithi086 ceph-mon[97913]: pgmap v237: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 7665/266100 objects degraded (2.880%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:46:47.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:46 smithi086 ceph-mon[102754]: pgmap v237: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 7665/266100 objects degraded (2.880%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:46:47.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:47 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:46:47.008+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:46:47.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:47 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:47.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:47 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:47.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:47 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:46:47.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/390764449' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:48.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:47 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:48.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:47 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:48.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:47 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:46:48.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/390764449' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:48.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:47 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:48.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:47 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:46:48.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:47 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:46:48.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/390764449' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:48.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 7665/266100 objects degraded (2.880%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:48.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:48 smithi137 ceph-mon[90634]: pgmap v238: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 7665/266100 objects degraded (2.880%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:46:49.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 7665/266100 objects degraded (2.880%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:49.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:48 smithi086 ceph-mon[97913]: pgmap v238: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 7665/266100 objects degraded (2.880%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:46:49.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 7665/266100 objects degraded (2.880%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:49.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:48 smithi086 ceph-mon[102754]: pgmap v238: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 7665/266100 objects degraded (2.880%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:46:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2910404790' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:50.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2910404790' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:50.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2910404790' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:51.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:50 smithi137 ceph-mon[90634]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 7386/253941 objects degraded (2.909%); 3.6 MiB/s, 67 objects/s recovering 2024-06-14T22:46:51.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:50 smithi086 ceph-mon[97913]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 7386/253941 objects degraded (2.909%); 3.6 MiB/s, 67 objects/s recovering 2024-06-14T22:46:51.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:50 smithi086 ceph-mon[102754]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 7386/253941 objects degraded (2.909%); 3.6 MiB/s, 67 objects/s recovering 2024-06-14T22:46:53.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:52 smithi137 ceph-mon[90634]: osdmap e100: 8 total, 8 up, 8 in 2024-06-14T22:46:53.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4036088640' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:53.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:52 smithi137 ceph-mon[90634]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 7386/253941 objects degraded (2.909%); 2.7 MiB/s, 54 objects/s recovering 2024-06-14T22:46:53.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:52 smithi086 ceph-mon[102754]: osdmap e100: 8 total, 8 up, 8 in 2024-06-14T22:46:53.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4036088640' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:53.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:52 smithi086 ceph-mon[102754]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 7386/253941 objects degraded (2.909%); 2.7 MiB/s, 54 objects/s recovering 2024-06-14T22:46:53.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:52 smithi086 ceph-mon[97913]: osdmap e100: 8 total, 8 up, 8 in 2024-06-14T22:46:53.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4036088640' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:53.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:52 smithi086 ceph-mon[97913]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 7386/253941 objects degraded (2.909%); 2.7 MiB/s, 54 objects/s recovering 2024-06-14T22:46:54.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:53 smithi086 ceph-mon[97913]: osdmap e101: 8 total, 8 up, 8 in 2024-06-14T22:46:54.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 7386/253941 objects degraded (2.909%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:54.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:53 smithi086 ceph-mon[102754]: osdmap e101: 8 total, 8 up, 8 in 2024-06-14T22:46:54.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 7386/253941 objects degraded (2.909%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:54.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:53 smithi137 ceph-mon[90634]: osdmap e101: 8 total, 8 up, 8 in 2024-06-14T22:46:54.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 7386/253941 objects degraded (2.909%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:55.232 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:55 smithi086 ceph-mon[102754]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 6978/245310 objects degraded (2.845%); 1.3 MiB/s, 85 objects/s recovering 2024-06-14T22:46:55.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:55 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1529213482' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:55.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:55 smithi086 ceph-mon[97913]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 6978/245310 objects degraded (2.845%); 1.3 MiB/s, 85 objects/s recovering 2024-06-14T22:46:55.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:55 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1529213482' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:55.233 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:46:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:46:54] "GET /metrics HTTP/1.1" 200 36371 "" "Prometheus/2.43.0" 2024-06-14T22:46:55.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:55 smithi137 ceph-mon[90634]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 6978/245310 objects degraded (2.845%); 1.3 MiB/s, 85 objects/s recovering 2024-06-14T22:46:55.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1529213482' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:57 smithi137 ceph-mon[90634]: pgmap v244: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 920 op/s; 6978/243987 objects degraded (2.860%); 1.3 MiB/s, 85 objects/s recovering 2024-06-14T22:46:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2519851008' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:57.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:57 smithi086 ceph-mon[97913]: pgmap v244: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 920 op/s; 6978/243987 objects degraded (2.860%); 1.3 MiB/s, 85 objects/s recovering 2024-06-14T22:46:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2519851008' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:57 smithi086 ceph-mon[102754]: pgmap v244: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 920 op/s; 6978/243987 objects degraded (2.860%); 1.3 MiB/s, 85 objects/s recovering 2024-06-14T22:46:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2519851008' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:46:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:46:58.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6978/243987 objects degraded (2.860%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:58.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6978/243987 objects degraded (2.860%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:58.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6978/243987 objects degraded (2.860%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:46:59.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:46:59 smithi137 ceph-mon[90634]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 414 op/s; 6978/243987 objects degraded (2.860%); 0 B/s, 50 objects/s recovering 2024-06-14T22:46:59.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:46:59 smithi086 ceph-mon[97913]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 414 op/s; 6978/243987 objects degraded (2.860%); 0 B/s, 50 objects/s recovering 2024-06-14T22:46:59.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:46:59 smithi086 ceph-mon[102754]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 414 op/s; 6978/243987 objects degraded (2.860%); 0 B/s, 50 objects/s recovering 2024-06-14T22:47:00.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2308506532' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:00.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2308506532' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:00.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2308506532' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:01.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:01 smithi137 ceph-mon[90634]: pgmap v246: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 867 op/s; 6821/232056 objects degraded (2.939%); 0 B/s, 67 objects/s recovering 2024-06-14T22:47:01.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:01 smithi086 ceph-mon[97913]: pgmap v246: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 867 op/s; 6821/232056 objects degraded (2.939%); 0 B/s, 67 objects/s recovering 2024-06-14T22:47:01.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:01 smithi086 ceph-mon[102754]: pgmap v246: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 867 op/s; 6821/232056 objects degraded (2.939%); 0 B/s, 67 objects/s recovering 2024-06-14T22:47:02.731 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:47:02.731 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2389238875' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:47:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2389238875' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:02.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:47:02.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2389238875' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:03 smithi137 ceph-mon[90634]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 6821/232056 objects degraded (2.939%); 0 B/s, 56 objects/s recovering 2024-06-14T22:47:03.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6821/232056 objects degraded (2.939%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:03.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:03 smithi086 ceph-mon[97913]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 6821/232056 objects degraded (2.939%); 0 B/s, 56 objects/s recovering 2024-06-14T22:47:03.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6821/232056 objects degraded (2.939%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:03.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:03 smithi086 ceph-mon[102754]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 6821/232056 objects degraded (2.939%); 0 B/s, 56 objects/s recovering 2024-06-14T22:47:03.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6821/232056 objects degraded (2.939%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:05.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:47:04] "GET /metrics HTTP/1.1" 200 36432 "" "Prometheus/2.43.0" 2024-06-14T22:47:05.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:05 smithi137 ceph-mon[90634]: pgmap v248: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 6650/222183 objects degraded (2.993%); 0 B/s, 64 objects/s recovering 2024-06-14T22:47:05.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/734927819' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:05 smithi086 ceph-mon[97913]: pgmap v248: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 6650/222183 objects degraded (2.993%); 0 B/s, 64 objects/s recovering 2024-06-14T22:47:05.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/734927819' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:05.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:05 smithi086 ceph-mon[102754]: pgmap v248: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 6650/222183 objects degraded (2.993%); 0 B/s, 64 objects/s recovering 2024-06-14T22:47:05.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/734927819' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:07.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:07 smithi086 ceph-mon[97913]: pgmap v249: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 6650/220824 objects degraded (3.011%); 0 B/s, 27 objects/s recovering 2024-06-14T22:47:07.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:07 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:07.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:07 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:07.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3880870587' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:07.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:07 smithi086 ceph-mon[102754]: pgmap v249: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 6650/220824 objects degraded (3.011%); 0 B/s, 27 objects/s recovering 2024-06-14T22:47:07.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:07 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:07.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:07 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:07.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3880870587' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:07.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:07 smithi137 ceph-mon[90634]: pgmap v249: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 6650/220824 objects degraded (3.011%); 0 B/s, 27 objects/s recovering 2024-06-14T22:47:07.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:07 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:07.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:07 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:07.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3880870587' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:08.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6650/220824 objects degraded (3.011%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:08.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6650/220824 objects degraded (3.011%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:08.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6650/220824 objects degraded (3.011%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:09.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:09 smithi086 ceph-mon[102754]: pgmap v250: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 6650/220824 objects degraded (3.011%); 0 B/s, 27 objects/s recovering 2024-06-14T22:47:09.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:09 smithi086 ceph-mon[97913]: pgmap v250: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 6650/220824 objects degraded (3.011%); 0 B/s, 27 objects/s recovering 2024-06-14T22:47:09.891 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:09 smithi137 ceph-mon[90634]: pgmap v250: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 6650/220824 objects degraded (3.011%); 0 B/s, 27 objects/s recovering 2024-06-14T22:47:10.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1760092301' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:10.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:10 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:10.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:10 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:10.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1760092301' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:10.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:10 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:10.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:10 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:10.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1760092301' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:10.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:10 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:10.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:10 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:11.517 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:47:11.214+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:11 smithi137 ceph-mon[90634]: pgmap v251: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 946 op/s; 6474/209883 objects degraded (3.085%); 0 B/s, 41 objects/s recovering 2024-06-14T22:47:11.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:47:11.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:47:11.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:11.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:47:11.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:47:11.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:47:11.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:47:11.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:11.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[102754]: pgmap v251: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 946 op/s; 6474/209883 objects degraded (3.085%); 0 B/s, 41 objects/s recovering 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[97913]: pgmap v251: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 946 op/s; 6474/209883 objects degraded (3.085%); 0 B/s, 41 objects/s recovering 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:47:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:47:11.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:47:11.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:47:11.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:47:11.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:47:11.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:12.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:12 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:12.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:12 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:12.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:12.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1193837707' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:12 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:12 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:12.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1193837707' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:12.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:12 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:12.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:12 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:12.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:12.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1193837707' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:13.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:13 smithi086 ceph-mon[97913]: pgmap v252: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 6474/209883 objects degraded (3.085%); 0 B/s, 28 objects/s recovering 2024-06-14T22:47:13.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6474/209883 objects degraded (3.085%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:13.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:13 smithi086 ceph-mon[102754]: pgmap v252: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 6474/209883 objects degraded (3.085%); 0 B/s, 28 objects/s recovering 2024-06-14T22:47:13.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6474/209883 objects degraded (3.085%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:13.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:13 smithi137 ceph-mon[90634]: pgmap v252: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 6474/209883 objects degraded (3.085%); 0 B/s, 28 objects/s recovering 2024-06-14T22:47:13.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6474/209883 objects degraded (3.085%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:15.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:47:14] "GET /metrics HTTP/1.1" 200 36426 "" "Prometheus/2.43.0" 2024-06-14T22:47:15.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:15 smithi086 ceph-mon[97913]: pgmap v253: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 6299/199953 objects degraded (3.150%); 0 B/s, 43 objects/s recovering 2024-06-14T22:47:15.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/655924921' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:15.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:15 smithi086 ceph-mon[102754]: pgmap v253: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 6299/199953 objects degraded (3.150%); 0 B/s, 43 objects/s recovering 2024-06-14T22:47:15.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/655924921' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:15.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:15 smithi137 ceph-mon[90634]: pgmap v253: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 6299/199953 objects degraded (3.150%); 0 B/s, 43 objects/s recovering 2024-06-14T22:47:15.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/655924921' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:17 smithi086 ceph-mon[97913]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 656 op/s; 6299/198540 objects degraded (3.173%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:17.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/489517607' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:17.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:17 smithi086 ceph-mon[102754]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 656 op/s; 6299/198540 objects degraded (3.173%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:17.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/489517607' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:17.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:17 smithi137 ceph-mon[90634]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 656 op/s; 6299/198540 objects degraded (3.173%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:17.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/489517607' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:18.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6299/198540 objects degraded (3.173%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:18.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6299/198540 objects degraded (3.173%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:18.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6299/198540 objects degraded (3.173%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:19.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:19 smithi086 ceph-mon[97913]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 6299/198540 objects degraded (3.173%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:19.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:19 smithi086 ceph-mon[102754]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 6299/198540 objects degraded (3.173%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:19.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:19 smithi137 ceph-mon[90634]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 6299/198540 objects degraded (3.173%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:20.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2973551007' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:20.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2973551007' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:20.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2973551007' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:21.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:21 smithi086 ceph-mon[97913]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 6119/186804 objects degraded (3.276%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:21.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:21 smithi086 ceph-mon[102754]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 6119/186804 objects degraded (3.276%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:21.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:21 smithi137 ceph-mon[90634]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 6119/186804 objects degraded (3.276%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:22.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4026936744' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:22.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4026936744' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:22.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4026936744' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:23.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:23 smithi137 ceph-mon[90634]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 640 op/s; 6119/186804 objects degraded (3.276%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:23.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6119/186804 objects degraded (3.276%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:24.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:23 smithi086 ceph-mon[97913]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 640 op/s; 6119/186804 objects degraded (3.276%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:24.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6119/186804 objects degraded (3.276%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:24.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:23 smithi086 ceph-mon[102754]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 640 op/s; 6119/186804 objects degraded (3.276%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:24.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6119/186804 objects degraded (3.276%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:25.331 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:47:24] "GET /metrics HTTP/1.1" 200 36426 "" "Prometheus/2.43.0" 2024-06-14T22:47:25.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:25 smithi137 ceph-mon[90634]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 5934/176484 objects degraded (3.362%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:25.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2379412683' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:26.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:25 smithi086 ceph-mon[97913]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 5934/176484 objects degraded (3.362%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:26.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2379412683' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:26.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:25 smithi086 ceph-mon[102754]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 5934/176484 objects degraded (3.362%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:26.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2379412683' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:26.507 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:26 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:47:26.230+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:26.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:26 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:26.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:26 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:26.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:26 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:27.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:27 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:27.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:27 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:27.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:27 smithi137 ceph-mon[90634]: pgmap v259: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 689 op/s; 5934/175113 objects degraded (3.389%); 0 B/s, 30 objects/s recovering 2024-06-14T22:47:27.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:27.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/718625450' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:27.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:27.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:27.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[97913]: pgmap v259: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 689 op/s; 5934/175113 objects degraded (3.389%); 0 B/s, 30 objects/s recovering 2024-06-14T22:47:27.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:27.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/718625450' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:27.998 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:27.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:27.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[102754]: pgmap v259: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 689 op/s; 5934/175113 objects degraded (3.389%); 0 B/s, 30 objects/s recovering 2024-06-14T22:47:27.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:27.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/718625450' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:28.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5934/175113 objects degraded (3.389%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:29.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5934/175113 objects degraded (3.389%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:29.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5934/175113 objects degraded (3.389%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:29.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:29 smithi137 ceph-mon[90634]: pgmap v260: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 5934/175113 objects degraded (3.389%); 0 B/s, 30 objects/s recovering 2024-06-14T22:47:30.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:29 smithi086 ceph-mon[102754]: pgmap v260: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 5934/175113 objects degraded (3.389%); 0 B/s, 30 objects/s recovering 2024-06-14T22:47:30.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:29 smithi086 ceph-mon[97913]: pgmap v260: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 5934/175113 objects degraded (3.389%); 0 B/s, 30 objects/s recovering 2024-06-14T22:47:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2385067249' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:31.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2385067249' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:31.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2385067249' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:31.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:31 smithi137 ceph-mon[90634]: pgmap v261: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 5762/163968 objects degraded (3.514%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:32.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:31 smithi086 ceph-mon[102754]: pgmap v261: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 5762/163968 objects degraded (3.514%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:32.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:31 smithi086 ceph-mon[97913]: pgmap v261: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 5762/163968 objects degraded (3.514%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:32.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4107229024' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:33.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4107229024' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:33.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4107229024' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:33.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:33 smithi137 ceph-mon[90634]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 5762/163968 objects degraded (3.514%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:33.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5762/163968 objects degraded (3.514%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:34.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:33 smithi086 ceph-mon[102754]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 5762/163968 objects degraded (3.514%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:34.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5762/163968 objects degraded (3.514%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:34.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:33 smithi086 ceph-mon[97913]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 5762/163968 objects degraded (3.514%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:34.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5762/163968 objects degraded (3.514%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:34.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:34 smithi137 ceph-mon[90634]: pgmap v263: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 5580/153822 objects degraded (3.628%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:35.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:34 smithi086 ceph-mon[102754]: pgmap v263: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 5580/153822 objects degraded (3.628%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:35.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:34 smithi086 ceph-mon[97913]: pgmap v263: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 5580/153822 objects degraded (3.628%); 0 B/s, 44 objects/s recovering 2024-06-14T22:47:35.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:47:34] "GET /metrics HTTP/1.1" 200 36422 "" "Prometheus/2.43.0" 2024-06-14T22:47:35.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/955808568' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:36.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/955808568' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:36.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/955808568' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:36.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:36 smithi137 ceph-mon[90634]: pgmap v264: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 5580/152346 objects degraded (3.663%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:37.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:36 smithi086 ceph-mon[97913]: pgmap v264: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 5580/152346 objects degraded (3.663%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:37.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:36 smithi086 ceph-mon[102754]: pgmap v264: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 5580/152346 objects degraded (3.663%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:37.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/266120668' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:38.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/266120668' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:38.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/266120668' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:38.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5580/152346 objects degraded (3.663%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:38.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:38 smithi137 ceph-mon[90634]: pgmap v265: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 5580/152346 objects degraded (3.663%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:39.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5580/152346 objects degraded (3.663%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:39.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:38 smithi086 ceph-mon[97913]: pgmap v265: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 5580/152346 objects degraded (3.663%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:39.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5580/152346 objects degraded (3.663%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:39.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:38 smithi086 ceph-mon[102754]: pgmap v265: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 5580/152346 objects degraded (3.663%); 0 B/s, 29 objects/s recovering 2024-06-14T22:47:39.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:39 smithi137 ceph-mon[90634]: osdmap e102: 8 total, 8 up, 8 in 2024-06-14T22:47:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:39 smithi086 ceph-mon[97913]: osdmap e102: 8 total, 8 up, 8 in 2024-06-14T22:47:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:39 smithi086 ceph-mon[102754]: osdmap e102: 8 total, 8 up, 8 in 2024-06-14T22:47:40.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:40 smithi137 ceph-mon[90634]: osdmap e103: 8 total, 8 up, 8 in 2024-06-14T22:47:40.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4240883042' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:40.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:40 smithi137 ceph-mon[90634]: pgmap v268: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 5414/141069 objects degraded (3.838%); 0 B/s, 43 objects/s recovering 2024-06-14T22:47:41.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:40 smithi086 ceph-mon[97913]: osdmap e103: 8 total, 8 up, 8 in 2024-06-14T22:47:41.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4240883042' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:41.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:40 smithi086 ceph-mon[97913]: pgmap v268: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 5414/141069 objects degraded (3.838%); 0 B/s, 43 objects/s recovering 2024-06-14T22:47:41.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:40 smithi086 ceph-mon[102754]: osdmap e103: 8 total, 8 up, 8 in 2024-06-14T22:47:41.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4240883042' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:41.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:40 smithi086 ceph-mon[102754]: pgmap v268: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 5414/141069 objects degraded (3.838%); 0 B/s, 43 objects/s recovering 2024-06-14T22:47:41.581 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:47:41.246+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:41.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:41.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:41 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:41.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:41 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:42.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:42.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:41 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:42.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:41 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:42.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:42.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:41 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:42.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:41 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:42 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/651246695' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:42.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:42 smithi137 ceph-mon[90634]: pgmap v269: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 531 op/s; 5414/141069 objects degraded (3.838%); 0 B/s, 20 objects/s recovering 2024-06-14T22:47:43.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:42 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/651246695' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:43.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:42 smithi086 ceph-mon[97913]: pgmap v269: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 531 op/s; 5414/141069 objects degraded (3.838%); 0 B/s, 20 objects/s recovering 2024-06-14T22:47:43.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:42 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/651246695' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:43.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:43.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:42 smithi086 ceph-mon[102754]: pgmap v269: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 531 op/s; 5414/141069 objects degraded (3.838%); 0 B/s, 20 objects/s recovering 2024-06-14T22:47:43.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5414/141069 objects degraded (3.838%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:44.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5414/141069 objects degraded (3.838%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:44.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5414/141069 objects degraded (3.838%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:45.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:44 smithi086 ceph-mon[97913]: pgmap v270: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 5071/132156 objects degraded (3.837%); 0 B/s, 63 objects/s recovering 2024-06-14T22:47:45.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:44 smithi086 ceph-mon[102754]: pgmap v270: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 5071/132156 objects degraded (3.837%); 0 B/s, 63 objects/s recovering 2024-06-14T22:47:45.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:47:44] "GET /metrics HTTP/1.1" 200 36421 "" "Prometheus/2.43.0" 2024-06-14T22:47:45.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:44 smithi137 ceph-mon[90634]: pgmap v270: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 5071/132156 objects degraded (3.837%); 0 B/s, 63 objects/s recovering 2024-06-14T22:47:46.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3237807997' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:46.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3237807997' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:46.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3237807997' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:47.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:46 smithi086 ceph-mon[97913]: pgmap v271: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 5071/130566 objects degraded (3.884%); 0 B/s, 63 objects/s recovering 2024-06-14T22:47:47.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:46 smithi086 ceph-mon[102754]: pgmap v271: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 5071/130566 objects degraded (3.884%); 0 B/s, 63 objects/s recovering 2024-06-14T22:47:47.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:46 smithi137 ceph-mon[90634]: pgmap v271: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 5071/130566 objects degraded (3.884%); 0 B/s, 63 objects/s recovering 2024-06-14T22:47:48.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2075660716' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:48.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2075660716' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:48.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2075660716' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:49.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5071/130566 objects degraded (3.884%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:49.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:48 smithi086 ceph-mon[102754]: pgmap v272: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 598 op/s; 5071/130566 objects degraded (3.884%); 0 B/s, 51 objects/s recovering 2024-06-14T22:47:49.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5071/130566 objects degraded (3.884%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:49.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:48 smithi086 ceph-mon[97913]: pgmap v272: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 598 op/s; 5071/130566 objects degraded (3.884%); 0 B/s, 51 objects/s recovering 2024-06-14T22:47:49.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5071/130566 objects degraded (3.884%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:49.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:48 smithi137 ceph-mon[90634]: pgmap v272: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 598 op/s; 5071/130566 objects degraded (3.884%); 0 B/s, 51 objects/s recovering 2024-06-14T22:47:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/717670327' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:50.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/717670327' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:50.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/717670327' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:51.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:50 smithi137 ceph-mon[90634]: pgmap v273: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 4892/120141 objects degraded (4.072%); 0 B/s, 48 objects/s recovering 2024-06-14T22:47:51.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:50 smithi086 ceph-mon[97913]: pgmap v273: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 4892/120141 objects degraded (4.072%); 0 B/s, 48 objects/s recovering 2024-06-14T22:47:51.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:50 smithi086 ceph-mon[102754]: pgmap v273: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 4892/120141 objects degraded (4.072%); 0 B/s, 48 objects/s recovering 2024-06-14T22:47:52.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/820277603' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:52.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/820277603' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:52.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/820277603' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:53.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:53 smithi137 ceph-mon[90634]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 4892/118665 objects degraded (4.123%); 0 B/s, 43 objects/s recovering 2024-06-14T22:47:53.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 4892/120141 objects degraded (4.072%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:53.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:53 smithi086 ceph-mon[97913]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 4892/118665 objects degraded (4.123%); 0 B/s, 43 objects/s recovering 2024-06-14T22:47:53.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 4892/120141 objects degraded (4.072%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:53.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:53 smithi086 ceph-mon[102754]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 4892/118665 objects degraded (4.123%); 0 B/s, 43 objects/s recovering 2024-06-14T22:47:53.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 4892/120141 objects degraded (4.072%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:55.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:47:54] "GET /metrics HTTP/1.1" 200 36421 "" "Prometheus/2.43.0" 2024-06-14T22:47:55.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:55 smithi137 ceph-mon[90634]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 911 op/s; 4698/108252 objects degraded (4.340%); 0 B/s, 59 objects/s recovering 2024-06-14T22:47:55.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2071095204' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:55.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:55 smithi086 ceph-mon[97913]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 911 op/s; 4698/108252 objects degraded (4.340%); 0 B/s, 59 objects/s recovering 2024-06-14T22:47:55.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:55 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2071095204' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:55.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:55 smithi086 ceph-mon[102754]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 911 op/s; 4698/108252 objects degraded (4.340%); 0 B/s, 59 objects/s recovering 2024-06-14T22:47:55.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:55 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2071095204' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:56.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:56.582 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:47:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:47:56.267+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:56.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:56.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:57.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:57 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:57.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:57 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:57.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:57 smithi137 ceph-mon[90634]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 4698/106794 objects degraded (4.399%); 0 B/s, 31 objects/s recovering 2024-06-14T22:47:57.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:57.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3249556238' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[97913]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 4698/106794 objects degraded (4.399%); 0 B/s, 31 objects/s recovering 2024-06-14T22:47:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3249556238' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:47:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:47:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[102754]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 4698/106794 objects degraded (4.399%); 0 B/s, 31 objects/s recovering 2024-06-14T22:47:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:47:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3249556238' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:47:58.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 4698/106794 objects degraded (4.399%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:58.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 4698/106794 objects degraded (4.399%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:58.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 4698/106794 objects degraded (4.399%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:47:59.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:47:59 smithi086 ceph-mon[97913]: pgmap v277: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 4698/106794 objects degraded (4.399%); 0 B/s, 31 objects/s recovering 2024-06-14T22:47:59.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:47:59 smithi086 ceph-mon[102754]: pgmap v277: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 4698/106794 objects degraded (4.399%); 0 B/s, 31 objects/s recovering 2024-06-14T22:47:59.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:47:59 smithi137 ceph-mon[90634]: pgmap v277: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 4698/106794 objects degraded (4.399%); 0 B/s, 31 objects/s recovering 2024-06-14T22:48:00.831 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2818106547' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:00.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2818106547' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:00.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2818106547' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:01.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:01 smithi086 ceph-mon[102754]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 4504/96549 objects degraded (4.665%); 0 B/s, 47 objects/s recovering 2024-06-14T22:48:01.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:01 smithi086 ceph-mon[97913]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 4504/96549 objects degraded (4.665%); 0 B/s, 47 objects/s recovering 2024-06-14T22:48:01.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:01 smithi137 ceph-mon[90634]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 4504/96549 objects degraded (4.665%); 0 B/s, 47 objects/s recovering 2024-06-14T22:48:02.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/608675770' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:02.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/608675770' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:02.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/608675770' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:03.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:03 smithi086 ceph-mon[102754]: pgmap v279: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 4504/95205 objects degraded (4.731%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:03.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 4504/95205 objects degraded (4.731%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:03.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:03 smithi086 ceph-mon[97913]: pgmap v279: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 4504/95205 objects degraded (4.731%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:03.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 4504/95205 objects degraded (4.731%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:03.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:03 smithi137 ceph-mon[90634]: pgmap v279: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 4504/95205 objects degraded (4.731%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:03.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 4504/95205 objects degraded (4.731%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:05.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:48:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:48:04] "GET /metrics HTTP/1.1" 200 36418 "" "Prometheus/2.43.0" 2024-06-14T22:48:05.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:05 smithi086 ceph-mon[102754]: pgmap v280: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 4309/84741 objects degraded (5.085%); 0 B/s, 48 objects/s recovering 2024-06-14T22:48:05.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3170963851' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:05.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:05 smithi086 ceph-mon[97913]: pgmap v280: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 4309/84741 objects degraded (5.085%); 0 B/s, 48 objects/s recovering 2024-06-14T22:48:05.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3170963851' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:05.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:05 smithi137 ceph-mon[90634]: pgmap v280: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 4309/84741 objects degraded (5.085%); 0 B/s, 48 objects/s recovering 2024-06-14T22:48:05.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3170963851' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:07.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:07 smithi086 ceph-mon[97913]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 694 op/s; 4309/83223 objects degraded (5.178%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:07.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2851122958' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:07.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:07 smithi086 ceph-mon[102754]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 694 op/s; 4309/83223 objects degraded (5.178%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:07.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2851122958' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:07.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:07 smithi137 ceph-mon[90634]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 694 op/s; 4309/83223 objects degraded (5.178%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:07.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2851122958' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:08.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 4309/83223 objects degraded (5.178%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:08.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 4309/83223 objects degraded (5.178%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:08.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 4309/83223 objects degraded (5.178%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:09.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:09 smithi086 ceph-mon[97913]: pgmap v282: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 654 op/s; 4309/83223 objects degraded (5.178%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:09.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:09 smithi086 ceph-mon[102754]: pgmap v282: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 654 op/s; 4309/83223 objects degraded (5.178%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:09.892 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:09 smithi137 ceph-mon[90634]: pgmap v282: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 654 op/s; 4309/83223 objects degraded (5.178%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:10.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/398168537' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:10.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/398168537' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:10.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/398168537' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:11 smithi086 ceph-mon[97913]: pgmap v283: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 936 op/s; 4107/73047 objects degraded (5.622%); 0 B/s, 49 objects/s recovering 2024-06-14T22:48:11.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:48:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:11 smithi086 ceph-mon[102754]: pgmap v283: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 936 op/s; 4107/73047 objects degraded (5.622%); 0 B/s, 49 objects/s recovering 2024-06-14T22:48:11.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:48:11.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:11 smithi137 ceph-mon[90634]: pgmap v283: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 936 op/s; 4107/73047 objects degraded (5.622%); 0 B/s, 49 objects/s recovering 2024-06-14T22:48:11.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:48:12.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:12.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1927750726' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:12.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:12.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1927750726' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:12.836 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:12.836 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1927750726' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:13.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:13 smithi086 ceph-mon[102754]: pgmap v284: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 4107/71508 objects degraded (5.743%); 0 B/s, 33 objects/s recovering 2024-06-14T22:48:13.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 4107/71508 objects degraded (5.743%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:13.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:13 smithi086 ceph-mon[97913]: pgmap v284: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 4107/71508 objects degraded (5.743%); 0 B/s, 33 objects/s recovering 2024-06-14T22:48:13.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 4107/71508 objects degraded (5.743%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:13.967 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:13 smithi137 ceph-mon[90634]: pgmap v284: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 4107/71508 objects degraded (5.743%); 0 B/s, 33 objects/s recovering 2024-06-14T22:48:13.967 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 4107/71508 objects degraded (5.743%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:15.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:48:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:48:14] "GET /metrics HTTP/1.1" 200 36419 "" "Prometheus/2.43.0" 2024-06-14T22:48:15.720 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:15 smithi086 ceph-mon[102754]: pgmap v285: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 3882/61875 objects degraded (6.274%); 0 B/s, 51 objects/s recovering 2024-06-14T22:48:15.720 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2240170881' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:15.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:15 smithi086 ceph-mon[97913]: pgmap v285: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 3882/61875 objects degraded (6.274%); 0 B/s, 51 objects/s recovering 2024-06-14T22:48:15.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2240170881' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:15.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:15 smithi137 ceph-mon[90634]: pgmap v285: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 3882/61875 objects degraded (6.274%); 0 B/s, 51 objects/s recovering 2024-06-14T22:48:15.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2240170881' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: pgmap v286: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 3882/60435 objects degraded (6.423%); 0 B/s, 35 objects/s recovering 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:48:17.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:48:17.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:17.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:48:16 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:48:16.876+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:17.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: pgmap v286: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 3882/60435 objects degraded (6.423%); 0 B/s, 35 objects/s recovering 2024-06-14T22:48:17.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:48:17.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:48:17.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:48:17.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:48:17.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:48:17.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:48:17.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:17.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: pgmap v286: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 3882/60435 objects degraded (6.423%); 0 B/s, 35 objects/s recovering 2024-06-14T22:48:17.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:48:17.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:48:17.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:17.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:48:17.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:48:17.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:48:17.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:48:17.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:18.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:18 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:18.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:18 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:18.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/791250841' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:18.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 3882/60435 objects degraded (6.423%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:18.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:18 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:18.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:18 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:18.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/791250841' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:18.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 3882/60435 objects degraded (6.423%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:18.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:18 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:18.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:18 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:18.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/791250841' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:18.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 3882/60435 objects degraded (6.423%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:19.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:19 smithi137 ceph-mon[90634]: pgmap v287: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 3882/60435 objects degraded (6.423%); 0 B/s, 35 objects/s recovering 2024-06-14T22:48:19.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:19 smithi086 ceph-mon[102754]: pgmap v287: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 3882/60435 objects degraded (6.423%); 0 B/s, 35 objects/s recovering 2024-06-14T22:48:19.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:19 smithi086 ceph-mon[97913]: pgmap v287: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 3882/60435 objects degraded (6.423%); 0 B/s, 35 objects/s recovering 2024-06-14T22:48:20.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:20 smithi137 ceph-mon[90634]: osdmap e104: 8 total, 8 up, 8 in 2024-06-14T22:48:20.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3953398838' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:20.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:20 smithi086 ceph-mon[97913]: osdmap e104: 8 total, 8 up, 8 in 2024-06-14T22:48:20.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3953398838' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:20.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:20 smithi086 ceph-mon[102754]: osdmap e104: 8 total, 8 up, 8 in 2024-06-14T22:48:20.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3953398838' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:21.490 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:21 smithi137 ceph-mon[90634]: osdmap e105: 8 total, 8 up, 8 in 2024-06-14T22:48:21.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:21 smithi137 ceph-mon[90634]: pgmap v290: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.0 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 3624/50715 objects degraded (7.146%); 0 B/s, 60 objects/s recovering 2024-06-14T22:48:21.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:21 smithi086 ceph-mon[97913]: osdmap e105: 8 total, 8 up, 8 in 2024-06-14T22:48:21.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:21 smithi086 ceph-mon[97913]: pgmap v290: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.0 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 3624/50715 objects degraded (7.146%); 0 B/s, 60 objects/s recovering 2024-06-14T22:48:21.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:21 smithi086 ceph-mon[102754]: osdmap e105: 8 total, 8 up, 8 in 2024-06-14T22:48:21.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:21 smithi086 ceph-mon[102754]: pgmap v290: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.0 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 3624/50715 objects degraded (7.146%); 0 B/s, 60 objects/s recovering 2024-06-14T22:48:22.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/966348858' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:22.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/966348858' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:22.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/966348858' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:23.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:23 smithi137 ceph-mon[90634]: pgmap v291: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.0 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 519 op/s; 3624/49413 objects degraded (7.334%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 3624/50715 objects degraded (7.146%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:23.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:23 smithi086 ceph-mon[97913]: pgmap v291: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.0 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 519 op/s; 3624/49413 objects degraded (7.334%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:23.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 3624/50715 objects degraded (7.146%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:23.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:23 smithi086 ceph-mon[102754]: pgmap v291: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.0 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 519 op/s; 3624/49413 objects degraded (7.334%); 0 B/s, 32 objects/s recovering 2024-06-14T22:48:23.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 3624/50715 objects degraded (7.146%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:24.241 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:48:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=sqlstore.transactions t=2024-06-14T22:48:23.765553956Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-06-14T22:48:24.241 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:48:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=cleanup t=2024-06-14T22:48:23.767855469Z level=info msg="Completed cleanup jobs" duration=13.579589ms 2024-06-14T22:48:25.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:48:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:48:24] "GET /metrics HTTP/1.1" 200 36419 "" "Prometheus/2.43.0" 2024-06-14T22:48:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:25 smithi137 ceph-mon[90634]: pgmap v292: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 840 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 3333/40365 objects degraded (8.257%); 0 B/s, 68 objects/s recovering 2024-06-14T22:48:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2549719370' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:25.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:25 smithi086 ceph-mon[97913]: pgmap v292: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 840 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 3333/40365 objects degraded (8.257%); 0 B/s, 68 objects/s recovering 2024-06-14T22:48:25.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2549719370' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:25 smithi086 ceph-mon[102754]: pgmap v292: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 840 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 3333/40365 objects degraded (8.257%); 0 B/s, 68 objects/s recovering 2024-06-14T22:48:25.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2549719370' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:27 smithi086 ceph-mon[97913]: pgmap v293: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 811 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 894 op/s; 3333/38955 objects degraded (8.556%); 0 B/s, 68 objects/s recovering 2024-06-14T22:48:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/834025201' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:27.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:27 smithi086 ceph-mon[102754]: pgmap v293: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 811 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 894 op/s; 3333/38955 objects degraded (8.556%); 0 B/s, 68 objects/s recovering 2024-06-14T22:48:27.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:27.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/834025201' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:27.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:27 smithi137 ceph-mon[90634]: pgmap v293: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 811 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 894 op/s; 3333/38955 objects degraded (8.556%); 0 B/s, 68 objects/s recovering 2024-06-14T22:48:27.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:27.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/834025201' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:28.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 3333/38955 objects degraded (8.556%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:28.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 3333/38955 objects degraded (8.556%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:28.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 3333/38955 objects degraded (8.556%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:29.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:29 smithi086 ceph-mon[97913]: pgmap v294: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 811 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 3333/38955 objects degraded (8.556%); 0 B/s, 36 objects/s recovering 2024-06-14T22:48:29.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:29 smithi086 ceph-mon[102754]: pgmap v294: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 811 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 3333/38955 objects degraded (8.556%); 0 B/s, 36 objects/s recovering 2024-06-14T22:48:29.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:29 smithi137 ceph-mon[90634]: pgmap v294: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 811 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 3333/38955 objects degraded (8.556%); 0 B/s, 36 objects/s recovering 2024-06-14T22:48:30.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2890557699' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:30.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2890557699' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:30.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2890557699' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:31.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:31 smithi086 ceph-mon[102754]: pgmap v295: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 603 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 701 op/s; 3171/28983 objects degraded (10.941%); 0 B/s, 43 objects/s recovering 2024-06-14T22:48:31.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:31 smithi086 ceph-mon[97913]: pgmap v295: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 603 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 701 op/s; 3171/28983 objects degraded (10.941%); 0 B/s, 43 objects/s recovering 2024-06-14T22:48:31.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:31 smithi137 ceph-mon[90634]: pgmap v295: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 603 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 701 op/s; 3171/28983 objects degraded (10.941%); 0 B/s, 43 objects/s recovering 2024-06-14T22:48:32.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:48:31 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:48:31.895+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:32.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:32.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1946783908' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:32.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:32.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1946783908' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:32.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:32.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1946783908' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:33 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:33 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:33 smithi086 ceph-mon[97913]: pgmap v296: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 574 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 642 op/s; 3171/27591 objects degraded (11.493%); 0 B/s, 37 objects/s recovering 2024-06-14T22:48:33.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 3171/27591 objects degraded (11.493%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:33 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:33 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:33 smithi086 ceph-mon[102754]: pgmap v296: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 574 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 642 op/s; 3171/27591 objects degraded (11.493%); 0 B/s, 37 objects/s recovering 2024-06-14T22:48:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 3171/27591 objects degraded (11.493%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:33.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:33 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:33.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:33 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:33.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:33 smithi137 ceph-mon[90634]: pgmap v296: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 574 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 642 op/s; 3171/27591 objects degraded (11.493%); 0 B/s, 37 objects/s recovering 2024-06-14T22:48:33.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 3171/27591 objects degraded (11.493%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:35.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:48:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:48:34] "GET /metrics HTTP/1.1" 200 36416 "" "Prometheus/2.43.0" 2024-06-14T22:48:35.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:35 smithi086 ceph-mon[97913]: pgmap v297: 89 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 3171/20679 objects degraded (15.334%); 0 B/s, 37 objects/s recovering 2024-06-14T22:48:35.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1717585963' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:35.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:35 smithi086 ceph-mon[102754]: pgmap v297: 89 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 3171/20679 objects degraded (15.334%); 0 B/s, 37 objects/s recovering 2024-06-14T22:48:35.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1717585963' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:35.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:35 smithi137 ceph-mon[90634]: pgmap v297: 89 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 3171/20679 objects degraded (15.334%); 0 B/s, 37 objects/s recovering 2024-06-14T22:48:35.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1717585963' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:37.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:37 smithi086 ceph-mon[97913]: pgmap v298: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 3004/16293 objects degraded (18.437%); 0 B/s, 27 objects/s recovering 2024-06-14T22:48:37.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/285599436' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:37.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:37 smithi086 ceph-mon[102754]: pgmap v298: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 3004/16293 objects degraded (18.437%); 0 B/s, 27 objects/s recovering 2024-06-14T22:48:37.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/285599436' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:37.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:37 smithi137 ceph-mon[90634]: pgmap v298: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 3004/16293 objects degraded (18.437%); 0 B/s, 27 objects/s recovering 2024-06-14T22:48:37.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/285599436' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:38.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 3004/16293 objects degraded (18.437%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:38.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 3004/16293 objects degraded (18.437%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:38.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 3004/16293 objects degraded (18.437%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:39.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:39 smithi086 ceph-mon[97913]: pgmap v299: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 3004/16293 objects degraded (18.437%); 0 B/s, 27 objects/s recovering 2024-06-14T22:48:39.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:39 smithi086 ceph-mon[102754]: pgmap v299: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 3004/16293 objects degraded (18.437%); 0 B/s, 27 objects/s recovering 2024-06-14T22:48:39.892 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:39 smithi137 ceph-mon[90634]: pgmap v299: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 3004/16293 objects degraded (18.437%); 0 B/s, 27 objects/s recovering 2024-06-14T22:48:40.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4171820752' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:40.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4171820752' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:40.904 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4171820752' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:41.016 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 113512 objects 2024-06-14T22:48:41.016 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :147.75 2024-06-14T22:48:41.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:41 smithi086 ceph-mon[97913]: pgmap v300: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 122 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 2823/5892 objects degraded (47.912%); 0 B/s, 42 objects/s recovering 2024-06-14T22:48:41.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:41 smithi086 ceph-mon[102754]: pgmap v300: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 122 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 2823/5892 objects degraded (47.912%); 0 B/s, 42 objects/s recovering 2024-06-14T22:48:41.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:41 smithi137 ceph-mon[90634]: pgmap v300: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 122 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 2823/5892 objects degraded (47.912%); 0 B/s, 42 objects/s recovering 2024-06-14T22:48:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:42 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2663055923' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:42.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:42 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2663055923' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:42.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:42.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:42 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2663055923' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:42.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:43.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:43 smithi086 ceph-mon[97913]: pgmap v301: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 91 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 2823/4410 objects degraded (64.014%); 0 B/s, 28 objects/s recovering 2024-06-14T22:48:43.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 2823/4410 objects degraded (64.014%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:43.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:43 smithi086 ceph-mon[102754]: pgmap v301: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 91 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 2823/4410 objects degraded (64.014%); 0 B/s, 28 objects/s recovering 2024-06-14T22:48:43.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 2823/4410 objects degraded (64.014%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:43.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:43 smithi137 ceph-mon[90634]: pgmap v301: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 91 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 2823/4410 objects degraded (64.014%); 0 B/s, 28 objects/s recovering 2024-06-14T22:48:43.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 2823/4410 objects degraded (64.014%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:48:44.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:44 smithi086 ceph-mon[97913]: osdmap e106: 8 total, 8 up, 8 in 2024-06-14T22:48:44.866 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:44 smithi086 ceph-mon[102754]: osdmap e106: 8 total, 8 up, 8 in 2024-06-14T22:48:44.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:44 smithi137 ceph-mon[90634]: osdmap e106: 8 total, 8 up, 8 in 2024-06-14T22:48:45.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:48:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:48:44] "GET /metrics HTTP/1.1" 200 36413 "" "Prometheus/2.43.0" 2024-06-14T22:48:45.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:45 smithi137 ceph-mon[90634]: pgmap v303: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 86 active+clean; 24 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 649 op/s; 2690/1185 objects degraded (227.004%); 0 B/s, 48 objects/s recovering 2024-06-14T22:48:45.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2660738790' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:45.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:45 smithi137 ceph-mon[90634]: osdmap e107: 8 total, 8 up, 8 in 2024-06-14T22:48:46.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:45 smithi086 ceph-mon[97913]: pgmap v303: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 86 active+clean; 24 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 649 op/s; 2690/1185 objects degraded (227.004%); 0 B/s, 48 objects/s recovering 2024-06-14T22:48:46.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2660738790' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:46.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:45 smithi086 ceph-mon[97913]: osdmap e107: 8 total, 8 up, 8 in 2024-06-14T22:48:46.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:45 smithi086 ceph-mon[102754]: pgmap v303: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 86 active+clean; 24 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 649 op/s; 2690/1185 objects degraded (227.004%); 0 B/s, 48 objects/s recovering 2024-06-14T22:48:46.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2660738790' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:46.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:45 smithi086 ceph-mon[102754]: osdmap e107: 8 total, 8 up, 8 in 2024-06-14T22:48:46.257 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-06-14T22:48:46.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-06-14T22:48:46.533 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:48:47.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:48:46 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:48:46.912+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:47.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:47 smithi086 ceph-mon[102754]: pgmap v305: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 1727/72 objects degraded (2398.611%); 0 B/s, 159 objects/s recovering 2024-06-14T22:48:47.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/497576804' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:47.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:47 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:47.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:47 smithi086 ceph-mon[97913]: pgmap v305: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 1727/72 objects degraded (2398.611%); 0 B/s, 159 objects/s recovering 2024-06-14T22:48:47.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/497576804' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:47.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:47 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:47.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:47 smithi137 ceph-mon[90634]: pgmap v305: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 1727/72 objects degraded (2398.611%); 0 B/s, 159 objects/s recovering 2024-06-14T22:48:47.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/497576804' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:47.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:47 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:48.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:48 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:48.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:48 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:48.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 1727/72 objects degraded (2398.611%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-14T22:48:49.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:48 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:49.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:48 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:49.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 1727/72 objects degraded (2398.611%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-14T22:48:49.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:48 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:48:49.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:48 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-06-14T22:48:49.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 1727/72 objects degraded (2398.611%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-14T22:48:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_2' removed 2024-06-14T22:48:49.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:49 smithi086 ceph-mon[97913]: pgmap v306: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 242 op/s; 1727/72 objects degraded (2398.611%); 0 B/s, 136 objects/s recovering 2024-06-14T22:48:49.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1622093489' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:48:49.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:49 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:48:49.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2154857967' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:49.935 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:49 smithi086 ceph-mon[102754]: pgmap v306: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 242 op/s; 1727/72 objects degraded (2398.611%); 0 B/s, 136 objects/s recovering 2024-06-14T22:48:49.935 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1622093489' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:48:49.935 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:49 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:48:49.935 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2154857967' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:49.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:49 smithi137 ceph-mon[90634]: pgmap v306: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 242 op/s; 1727/72 objects degraded (2398.611%); 0 B/s, 136 objects/s recovering 2024-06-14T22:48:49.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1622093489' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:48:49.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:49 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:48:49.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2154857967' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:50.232 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T22:48:50.233 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T22:48:50.233 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-06-14T22:48:50.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-06-14T22:48:50.515 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:48:50.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:50 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:48:50.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:50 smithi086 ceph-mon[97913]: osdmap e108: 8 total, 8 up, 8 in 2024-06-14T22:48:50.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:50 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:48:50.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:50 smithi086 ceph-mon[102754]: osdmap e108: 8 total, 8 up, 8 in 2024-06-14T22:48:50.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:50 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:48:50.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:50 smithi137 ceph-mon[90634]: osdmap e108: 8 total, 8 up, 8 in 2024-06-14T22:48:51.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:51 smithi137 ceph-mon[90634]: pgmap v308: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:51.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:51 smithi137 ceph-mon[90634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1727/72 objects degraded (2398.611%), 1 pg degraded, 1 pg undersized) 2024-06-14T22:48:51.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:51 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:48:51.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1720751210' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:51.992 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:51 smithi086 ceph-mon[97913]: pgmap v308: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:51.992 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:51 smithi086 ceph-mon[97913]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1727/72 objects degraded (2398.611%), 1 pg degraded, 1 pg undersized) 2024-06-14T22:48:51.992 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:51 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:48:51.992 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1720751210' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:51.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:51 smithi086 ceph-mon[102754]: pgmap v308: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:51.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:51 smithi086 ceph-mon[102754]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1727/72 objects degraded (2398.611%), 1 pg degraded, 1 pg undersized) 2024-06-14T22:48:51.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:51 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:48:51.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1720751210' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:53.615 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_3' created 2024-06-14T22:48:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:53 smithi137 ceph-mon[90634]: pgmap v309: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/507869908' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-14T22:48:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:53 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-14T22:48:54.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:53 smithi086 ceph-mon[97913]: pgmap v309: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:54.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/507869908' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-14T22:48:54.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:53 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-14T22:48:54.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:53 smithi086 ceph-mon[102754]: pgmap v309: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:54.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/507869908' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-14T22:48:54.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:53 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-14T22:48:54.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-06-14T22:48:54.659 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:48:54.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:54 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-06-14T22:48:54.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:54 smithi086 ceph-mon[97913]: osdmap e109: 8 total, 8 up, 8 in 2024-06-14T22:48:54.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3693095906' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:54.781 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:54 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-06-14T22:48:54.781 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:54 smithi086 ceph-mon[102754]: osdmap e109: 8 total, 8 up, 8 in 2024-06-14T22:48:54.781 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3693095906' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:54 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-06-14T22:48:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:54 smithi137 ceph-mon[90634]: osdmap e109: 8 total, 8 up, 8 in 2024-06-14T22:48:54.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:54 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3693095906' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:55.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:48:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:48:54] "GET /metrics HTTP/1.1" 200 36413 "" "Prometheus/2.43.0" 2024-06-14T22:48:55.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:55 smithi137 ceph-mon[90634]: pgmap v311: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:55.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:55 smithi137 ceph-mon[90634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:48:55.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:55 smithi137 ceph-mon[90634]: osdmap e110: 8 total, 8 up, 8 in 2024-06-14T22:48:56.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:55 smithi086 ceph-mon[97913]: pgmap v311: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:56.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:55 smithi086 ceph-mon[97913]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:48:56.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:55 smithi086 ceph-mon[97913]: osdmap e110: 8 total, 8 up, 8 in 2024-06-14T22:48:56.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:55 smithi086 ceph-mon[102754]: pgmap v311: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:56.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:55 smithi086 ceph-mon[102754]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:48:56.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:55 smithi086 ceph-mon[102754]: osdmap e110: 8 total, 8 up, 8 in 2024-06-14T22:48:56.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:56 smithi086 ceph-mon[97913]: osdmap e111: 8 total, 8 up, 8 in 2024-06-14T22:48:56.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2309557173' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:56.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:56 smithi086 ceph-mon[102754]: osdmap e111: 8 total, 8 up, 8 in 2024-06-14T22:48:56.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2309557173' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:56.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:56 smithi137 ceph-mon[90634]: osdmap e111: 8 total, 8 up, 8 in 2024-06-14T22:48:56.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2309557173' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:57.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:57 smithi086 ceph-mon[102754]: pgmap v314: 89 pgs: 7 creating+peering, 6 unknown, 76 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:57 smithi086 ceph-mon[97913]: pgmap v314: 89 pgs: 7 creating+peering, 6 unknown, 76 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:57 smithi137 ceph-mon[90634]: pgmap v314: 89 pgs: 7 creating+peering, 6 unknown, 76 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:48:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:48:58.646 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-06-14T22:48:58.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3433148885' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:58.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1830562845' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:48:58.927 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3433148885' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:58.927 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1830562845' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:48:58.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3433148885' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:48:58.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1830562845' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:48:59.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:59 smithi137 ceph-mon[90634]: pgmap v315: 89 pgs: 7 creating+peering, 6 unknown, 76 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:48:59.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:59 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1830562845' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:48:59.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:48:59 smithi137 ceph-mon[90634]: osdmap e112: 8 total, 8 up, 8 in 2024-06-14T22:49:00.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:59 smithi086 ceph-mon[97913]: pgmap v315: 89 pgs: 7 creating+peering, 6 unknown, 76 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:49:00.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:59 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1830562845' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:49:00.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:48:59 smithi086 ceph-mon[97913]: osdmap e112: 8 total, 8 up, 8 in 2024-06-14T22:49:00.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:59 smithi086 ceph-mon[102754]: pgmap v315: 89 pgs: 7 creating+peering, 6 unknown, 76 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:49:00.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:59 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1830562845' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:49:00.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:48:59 smithi086 ceph-mon[102754]: osdmap e112: 8 total, 8 up, 8 in 2024-06-14T22:49:00.707 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/614233257' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:00.707 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/614233257' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:00.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/614233257' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:01.732 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-06-14T22:49:01.735 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T22:49:01.736 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T22:49:01.788 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T22:49:01.788 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T22:49:01.788 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_122122 2024-06-14T22:49:01.788 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:49:01.788 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T22:49:01.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:01 smithi137 ceph-mon[90634]: pgmap v317: 89 pgs: 89 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:49:01.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:01 smithi137 ceph-mon[90634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:49:01.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:01 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:49:02.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:01 smithi086 ceph-mon[97913]: pgmap v317: 89 pgs: 89 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:49:02.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:01 smithi086 ceph-mon[97913]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:49:02.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:01 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:49:02.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:01 smithi086 ceph-mon[102754]: pgmap v317: 89 pgs: 89 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:49:02.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:01 smithi086 ceph-mon[102754]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:49:02.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:01 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:49:02.789 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 3897 3881 242.549 242.562 0.00442345 0.00410821 2024-06-14T22:49:02.810 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:49:02.811 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:49:02.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:49:03.789 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 16 7527 7511 234.701 226.875 0.00630071 0.00424999 2024-06-14T22:49:03.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:03 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:49:03.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:03 smithi137 ceph-mon[90634]: Upgrade: osd.1 is safe to restart 2024-06-14T22:49:03.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:03 smithi137 ceph-mon[90634]: pgmap v318: 89 pgs: 89 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:49:03.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1951681395' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:04.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:03 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:49:04.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:03 smithi086 ceph-mon[97913]: Upgrade: osd.1 is safe to restart 2024-06-14T22:49:04.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:03 smithi086 ceph-mon[97913]: pgmap v318: 89 pgs: 89 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:49:04.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1951681395' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:04.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:03 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-14T22:49:04.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:03 smithi086 ceph-mon[102754]: Upgrade: osd.1 is safe to restart 2024-06-14T22:49:04.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:03 smithi086 ceph-mon[102754]: pgmap v318: 89 pgs: 89 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-06-14T22:49:04.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1951681395' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:04.789 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 16 10790 10774 224.441 203.938 0.00592205 0.00444795 2024-06-14T22:49:04.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:04 smithi086 ceph-mon[97913]: Upgrade: Updating osd.1 2024-06-14T22:49:04.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:04 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:04.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:04 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-14T22:49:04.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:04 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:49:04.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:04 smithi086 ceph-mon[102754]: Upgrade: Updating osd.1 2024-06-14T22:49:04.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:04 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:04.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:04 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-14T22:49:04.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:04 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:49:04.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:04 smithi137 ceph-mon[90634]: Upgrade: Updating osd.1 2024-06-14T22:49:04.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:04 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:04.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:04 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-14T22:49:04.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:04 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:49:05.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:49:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:49:04] "GET /metrics HTTP/1.1" 200 36350 "" "Prometheus/2.43.0" 2024-06-14T22:49:05.789 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 16 13938 13922 217.508 196.75 0.0109849 0.00458923 2024-06-14T22:49:05.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:05 smithi086 ceph-mon[97913]: Deploying daemon osd.1 on smithi086 2024-06-14T22:49:05.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:05 smithi086 ceph-mon[97913]: pgmap v319: 89 pgs: 89 active+clean; 281 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 32 MiB/s wr, 509 op/s 2024-06-14T22:49:05.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2630352766' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:05.977 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:05 smithi086 ceph-mon[102754]: Deploying daemon osd.1 on smithi086 2024-06-14T22:49:05.977 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:05 smithi086 ceph-mon[102754]: pgmap v319: 89 pgs: 89 active+clean; 281 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 32 MiB/s wr, 509 op/s 2024-06-14T22:49:05.977 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2630352766' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:05.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:05 smithi137 ceph-mon[90634]: Deploying daemon osd.1 on smithi086 2024-06-14T22:49:05.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:05 smithi137 ceph-mon[90634]: pgmap v319: 89 pgs: 89 active+clean; 281 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 32 MiB/s wr, 509 op/s 2024-06-14T22:49:05.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2630352766' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:06.789 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 16 16984 16968 212.078 190.375 0.00296702 0.00470862 2024-06-14T22:49:06.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:06 smithi137 ceph-mon[90634]: pgmap v320: 89 pgs: 89 active+clean; 548 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s 2024-06-14T22:49:07.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:06 smithi086 ceph-mon[97913]: pgmap v320: 89 pgs: 89 active+clean; 548 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s 2024-06-14T22:49:07.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:06 smithi086 ceph-mon[102754]: pgmap v320: 89 pgs: 89 active+clean; 548 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s 2024-06-14T22:49:07.789 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 19774 19758 205.791 174.375 0.00847776 0.00485193 2024-06-14T22:49:07.832 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:07 smithi086 systemd[1]: Stopping Ceph osd.1 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:49:08.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:08 smithi086 ceph-mon[97913]: osd.1 marked itself down and dead 2024-06-14T22:49:08.332 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:08 smithi086 ceph-mon[102754]: osd.1 marked itself down and dead 2024-06-14T22:49:08.333 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[49514]: 2024-06-14T22:49:08.057+0000 7fd6b48ab640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T22:49:08.333 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[49514]: 2024-06-14T22:49:08.057+0000 7fd6b48ab640 -1 osd.1 112 *** Got signal Terminated *** 2024-06-14T22:49:08.333 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:08 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[49514]: 2024-06-14T22:49:08.057+0000 7fd6b48ab640 -1 osd.1 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T22:49:08.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:08 smithi137 ceph-mon[90634]: osd.1 marked itself down and dead 2024-06-14T22:49:08.790 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 16 20587 20571 183.649 50.8125 0.00160669 0.00492436 2024-06-14T22:49:09.037 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:08 smithi086 podman[122658]: 2024-06-14 22:49:08.723590057 +0000 UTC m=+0.826272749 container died 88baee12d7cb8b05d107238bac5777186f239e239feb400f86f362fe53e400db (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:49:09.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[97913]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:49:09.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[97913]: osdmap e113: 8 total, 7 up, 8 in 2024-06-14T22:49:09.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/854501461' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:09.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[97913]: pgmap v322: 89 pgs: 14 stale+active+clean, 75 active+clean; 548 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 56 MiB/s wr, 894 op/s 2024-06-14T22:49:09.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[97913]: osdmap e114: 8 total, 7 up, 8 in 2024-06-14T22:49:09.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[102754]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:49:09.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[102754]: osdmap e113: 8 total, 7 up, 8 in 2024-06-14T22:49:09.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/854501461' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:09.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[102754]: pgmap v322: 89 pgs: 14 stale+active+clean, 75 active+clean; 548 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 56 MiB/s wr, 894 op/s 2024-06-14T22:49:09.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:09 smithi086 ceph-mon[102754]: osdmap e114: 8 total, 7 up, 8 in 2024-06-14T22:49:09.333 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:09 smithi086 podman[122658]: 2024-06-14 22:49:09.130868498 +0000 UTC m=+1.233551196 container cleanup 88baee12d7cb8b05d107238bac5777186f239e239feb400f86f362fe53e400db (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-06-14T22:49:09.334 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:09 smithi086 bash[122658]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1 2024-06-14T22:49:09.392 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:09 smithi137 ceph-mon[90634]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:49:09.392 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:09 smithi137 ceph-mon[90634]: osdmap e113: 8 total, 7 up, 8 in 2024-06-14T22:49:09.392 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/854501461' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:09.392 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:09 smithi137 ceph-mon[90634]: pgmap v322: 89 pgs: 14 stale+active+clean, 75 active+clean; 548 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 56 MiB/s wr, 894 op/s 2024-06-14T22:49:09.392 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:09 smithi137 ceph-mon[90634]: osdmap e114: 8 total, 7 up, 8 in 2024-06-14T22:49:09.790 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 22659 22643 176.87 129.5 0.00472546 0.00564809 2024-06-14T22:49:09.832 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:09 smithi086 podman[122679]: 2024-06-14 22:49:09.399475537 +0000 UTC m=+0.673390963 container remove 88baee12d7cb8b05d107238bac5777186f239e239feb400f86f362fe53e400db (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, ceph=True, io.buildah.version=1.36.0) 2024-06-14T22:49:10.259 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:09 smithi086 podman[122761]: 2024-06-14 22:49:09.832339032 +0000 UTC m=+0.035191983 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:49:10.260 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:09 smithi086 podman[122761]: 2024-06-14 22:49:09.983776164 +0000 UTC m=+0.186629105 container create 95d267d0ea8604caf24e1ea987760226a58c0768e42b5c1cebf27483ca2bbdce (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True) 2024-06-14T22:49:10.546 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:10 smithi086 podman[122761]: 2024-06-14 22:49:10.261798275 +0000 UTC m=+0.464651230 container init 95d267d0ea8604caf24e1ea987760226a58c0768e42b5c1cebf27483ca2bbdce (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:49:10.546 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:10 smithi086 podman[122761]: 2024-06-14 22:49:10.27243349 +0000 UTC m=+0.475286433 container start 95d267d0ea8604caf24e1ea987760226a58c0768e42b5c1cebf27483ca2bbdce (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-14T22:49:10.546 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:10 smithi086 podman[122761]: 2024-06-14 22:49:10.386357778 +0000 UTC m=+0.589210724 container attach 95d267d0ea8604caf24e1ea987760226a58c0768e42b5c1cebf27483ca2bbdce (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, ceph=True, org.label-schema.schema-version=1.0) 2024-06-14T22:49:10.790 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 25545 25529 177.257 180.375 0.00475196 0.00563631 2024-06-14T22:49:10.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:10 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-14T22:49:10.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:10 smithi086 ceph-mon[102754]: Health check failed: Degraded data redundancy: 6658/56094 objects degraded (11.869%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:10.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:10 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-14T22:49:10.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:10 smithi086 ceph-mon[97913]: Health check failed: Degraded data redundancy: 6658/56094 objects degraded (11.869%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:10.833 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:10 smithi086 podman[122761]: 2024-06-14 22:49:10.722417622 +0000 UTC m=+0.925270563 container died 95d267d0ea8604caf24e1ea987760226a58c0768e42b5c1cebf27483ca2bbdce (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-14T22:49:10.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:10 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-14T22:49:10.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:10 smithi137 ceph-mon[90634]: Health check failed: Degraded data redundancy: 6658/56094 objects degraded (11.869%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:11.772 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:11 smithi086 ceph-mon[97913]: pgmap v324: 89 pgs: 11 active+undersized, 19 peering, 2 stale+active+clean, 5 active+undersized+degraded, 52 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s; 6658/56094 objects degraded (11.869%); 16 KiB/s, 0 objects/s recovering 2024-06-14T22:49:11.772 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1415934452' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:11.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:11 smithi086 ceph-mon[102754]: pgmap v324: 89 pgs: 11 active+undersized, 19 peering, 2 stale+active+clean, 5 active+undersized+degraded, 52 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s; 6658/56094 objects degraded (11.869%); 16 KiB/s, 0 objects/s recovering 2024-06-14T22:49:11.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1415934452' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:11.773 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:11 smithi086 podman[122827]: 2024-06-14 22:49:11.440752518 +0000 UTC m=+0.709717307 container remove 95d267d0ea8604caf24e1ea987760226a58c0768e42b5c1cebf27483ca2bbdce (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:49:11.773 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:11 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.1.service: Deactivated successfully. 2024-06-14T22:49:11.773 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:11 smithi086 systemd[1]: Stopped Ceph osd.1 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:49:11.773 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:11 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.1.service: Consumed 16min 28.175s CPU time. 2024-06-14T22:49:11.773 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:11 smithi086 systemd[1]: Starting Ceph osd.1 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:49:11.790 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 16 28548 28532 178.298 187.688 0.00253863 0.00560339 2024-06-14T22:49:11.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:11 smithi137 ceph-mon[90634]: pgmap v324: 89 pgs: 11 active+undersized, 19 peering, 2 stale+active+clean, 5 active+undersized+degraded, 52 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s; 6658/56094 objects degraded (11.869%); 16 KiB/s, 0 objects/s recovering 2024-06-14T22:49:11.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1415934452' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:12.332 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:12 smithi086 podman[122924]: 2024-06-14 22:49:11.975052008 +0000 UTC m=+0.028034570 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:49:12.332 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:12 smithi086 podman[122924]: 2024-06-14 22:49:12.28405993 +0000 UTC m=+0.337042480 container create 733b4879d5116b60521b4c55b7764734140f0bde5cdc3b3ba0e32879603887c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T22:49:12.790 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 31372 31356 178.133 176.5 0.00264694 0.00560805 2024-06-14T22:49:13.332 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:12 smithi086 podman[122924]: 2024-06-14 22:49:12.837107344 +0000 UTC m=+0.890089903 container init 733b4879d5116b60521b4c55b7764734140f0bde5cdc3b3ba0e32879603887c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-14T22:49:13.332 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:12 smithi086 podman[122924]: 2024-06-14 22:49:12.848239877 +0000 UTC m=+0.901222434 container start 733b4879d5116b60521b4c55b7764734140f0bde5cdc3b3ba0e32879603887c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-14T22:49:13.333 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:12 smithi086 podman[122924]: 2024-06-14 22:49:12.936717538 +0000 UTC m=+0.989700091 container attach 733b4879d5116b60521b4c55b7764734140f0bde5cdc3b3ba0e32879603887c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-14T22:49:13.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:13.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:13.599 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:13.599 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[97913]: pgmap v325: 89 pgs: 13 active+undersized, 19 peering, 7 active+undersized+degraded, 50 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s; 8245/66057 objects degraded (12.482%); 24 KiB/s, 0 objects/s recovering 2024-06-14T22:49:13.599 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3921824913' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:13.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:13.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:13.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:13.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[102754]: pgmap v325: 89 pgs: 13 active+undersized, 19 peering, 7 active+undersized+degraded, 50 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s; 8245/66057 objects degraded (12.482%); 24 KiB/s, 0 objects/s recovering 2024-06-14T22:49:13.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3921824913' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate[122938]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 bash[122924]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate[122938]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 bash[122924]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate[122938]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 bash[122924]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate[122938]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 bash[122924]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate[122938]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 bash[122924]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate[122938]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 bash[122924]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-14T22:49:13.600 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate[122938]: --> ceph-volume raw activate successful for osd ID: 1 2024-06-14T22:49:13.601 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 bash[122924]: --> ceph-volume raw activate successful for osd ID: 1 2024-06-14T22:49:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:13 smithi137 ceph-mon[90634]: pgmap v325: 89 pgs: 13 active+undersized, 19 peering, 7 active+undersized+degraded, 50 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s; 8245/66057 objects degraded (12.482%); 24 KiB/s, 0 objects/s recovering 2024-06-14T22:49:13.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3921824913' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:13.790 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 16 33953 33937 176.731 161.312 0.00370691 0.00565375 2024-06-14T22:49:13.947 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:13 smithi086 podman[122924]: 2024-06-14 22:49:13.602536442 +0000 UTC m=+1.655518998 container died 733b4879d5116b60521b4c55b7764734140f0bde5cdc3b3ba0e32879603887c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-06-14T22:49:14.646 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:14 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-06-14T22:49:14.646 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:14 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-06-14T22:49:14.646 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:14 smithi086 podman[123099]: 2024-06-14 22:49:14.349604088 +0000 UTC m=+0.738677710 container remove 733b4879d5116b60521b4c55b7764734140f0bde5cdc3b3ba0e32879603887c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-14T22:49:14.790 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 36656 36640 176.13 168.938 0.00398387 0.00567289 2024-06-14T22:49:14.923 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:49:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:49:14] "GET /metrics HTTP/1.1" 200 36493 "" "Prometheus/2.43.0" 2024-06-14T22:49:14.923 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:14 smithi086 podman[123154]: 2024-06-14 22:49:14.554905925 +0000 UTC m=+0.034322335 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:49:14.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:14 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-06-14T22:49:15.184 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:14 smithi086 podman[123154]: 2024-06-14 22:49:14.925896872 +0000 UTC m=+0.405313260 container create 50c27a83b12af885134b5ddae3a9dd317998a06a7c1c3565f5096b9d3eecc5d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-14T22:49:15.550 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:15 smithi086 podman[123154]: 2024-06-14 22:49:15.186691838 +0000 UTC m=+0.666108246 container init 50c27a83b12af885134b5ddae3a9dd317998a06a7c1c3565f5096b9d3eecc5d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-14T22:49:15.551 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:15 smithi086 podman[123154]: 2024-06-14 22:49:15.195541434 +0000 UTC m=+0.674957829 container start 50c27a83b12af885134b5ddae3a9dd317998a06a7c1c3565f5096b9d3eecc5d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1, ceph=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T22:49:15.551 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-osd[123189]: -- 172.21.15.86:0/1169379289 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55d783c39600 con 0x55d783bc5c00 2024-06-14T22:49:15.551 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-osd[123189]: -- 172.21.15.86:0/1169379289 <== mon.2 v2:172.21.15.86:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d783c271e0 con 0x55d783bc5c00 2024-06-14T22:49:15.551 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:15 smithi086 bash[123154]: 50c27a83b12af885134b5ddae3a9dd317998a06a7c1c3565f5096b9d3eecc5d9 2024-06-14T22:49:15.551 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:15 smithi086 systemd[1]: Started Ceph osd.1 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:49:15.790 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 39009 38993 174.053 147.062 0.00512998 0.00574037 2024-06-14T22:49:15.832 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[123167]: 2024-06-14T22:49:15.753+0000 7f25c9806640 -1 Falling back to public interface 2024-06-14T22:49:15.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-mon[97913]: pgmap v326: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 1.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s; 11155/84963 objects degraded (13.129%); 40 KiB/s, 0 objects/s recovering 2024-06-14T22:49:15.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:15.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:15.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:49:15.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-mon[102754]: pgmap v326: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 1.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s; 11155/84963 objects degraded (13.129%); 40 KiB/s, 0 objects/s recovering 2024-06-14T22:49:15.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:15.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:15.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:49:15.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:15 smithi137 ceph-mon[90634]: pgmap v326: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 1.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s; 11155/84963 objects degraded (13.129%); 40 KiB/s, 0 objects/s recovering 2024-06-14T22:49:15.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:15.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:15.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:49:16.790 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 41403 41387 172.424 149.625 0.00785387 0.00579508 2024-06-14T22:49:16.813 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:16 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4128873867' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:16.813 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:16 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:16.813 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:16 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:16.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4128873867' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:16.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:16 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:16.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:16 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:16.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:16 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4128873867' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:16.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:16 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:16.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:16 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:17.792 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 43688 43672 170.557 142.812 0.00649774 0.00585828 2024-06-14T22:49:17.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:17 smithi086 ceph-mon[97913]: pgmap v327: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.2 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 200 MiB/s wr, 3.20k op/s; 15600/106227 objects degraded (14.686%); 38 KiB/s, 0 objects/s recovering 2024-06-14T22:49:17.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:17 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 15600/106227 objects degraded (14.686%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:17.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:17 smithi086 ceph-mon[102754]: pgmap v327: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.2 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 200 MiB/s wr, 3.20k op/s; 15600/106227 objects degraded (14.686%); 38 KiB/s, 0 objects/s recovering 2024-06-14T22:49:17.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:17 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 15600/106227 objects degraded (14.686%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:17.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:17 smithi137 ceph-mon[90634]: pgmap v327: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.2 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 200 MiB/s wr, 3.20k op/s; 15600/106227 objects degraded (14.686%); 38 KiB/s, 0 objects/s recovering 2024-06-14T22:49:17.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:17 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 15600/106227 objects degraded (14.686%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:18.792 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 16 45586 45570 167.502 118.625 0.00396667 0.0059654 2024-06-14T22:49:18.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3997117313' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:18.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3997117313' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:18.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3997117313' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:19.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:19 smithi086 ceph-mon[97913]: pgmap v328: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.2 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 166 MiB/s wr, 2.66k op/s; 15600/106227 objects degraded (14.686%); 32 KiB/s, 0 objects/s recovering 2024-06-14T22:49:19.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:19 smithi086 ceph-mon[102754]: pgmap v328: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.2 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 166 MiB/s wr, 2.66k op/s; 15600/106227 objects degraded (14.686%); 32 KiB/s, 0 objects/s recovering 2024-06-14T22:49:19.792 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 47795 47779 165.866 138.062 0.00441138 0.00602457 2024-06-14T22:49:19.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:19 smithi137 ceph-mon[90634]: pgmap v328: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.2 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 166 MiB/s wr, 2.66k op/s; 15600/106227 objects degraded (14.686%); 32 KiB/s, 0 objects/s recovering 2024-06-14T22:49:20.792 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 49705 49689 163.419 119.375 0.00964491 0.00611425 2024-06-14T22:49:21.792 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:49:21.795535+0000 min lat: 0.00117459 max lat: 1.02781 avg lat: 0.00624364 2024-06-14T22:49:21.792 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:49:21.792 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 51237 51221 160.035 95.75 0.00388612 0.00624364 2024-06-14T22:49:21.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:21 smithi086 ceph-mon[97913]: pgmap v329: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 161 MiB/s wr, 2.57k op/s; 19818/132027 objects degraded (15.011%); 28 KiB/s, 0 objects/s recovering 2024-06-14T22:49:21.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/179281808' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:21.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:21 smithi086 ceph-mon[102754]: pgmap v329: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 161 MiB/s wr, 2.57k op/s; 19818/132027 objects degraded (15.011%); 28 KiB/s, 0 objects/s recovering 2024-06-14T22:49:21.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/179281808' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:21.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:21 smithi137 ceph-mon[90634]: pgmap v329: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 161 MiB/s wr, 2.57k op/s; 19818/132027 objects degraded (15.011%); 28 KiB/s, 0 objects/s recovering 2024-06-14T22:49:21.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/179281808' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:22.792 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 16 53050 53034 157.81 113.312 0.00580912 0.00632973 2024-06-14T22:49:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:23 smithi137 ceph-mon[90634]: pgmap v330: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s; 20538/140196 objects degraded (14.649%); 16 KiB/s, 0 objects/s recovering 2024-06-14T22:49:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 19818/132027 objects degraded (15.011%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2419032898' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:23.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:23 smithi137 ceph-mon[90634]: from='osd.1 [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-14T22:49:23.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:23 smithi137 ceph-mon[90634]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-14T22:49:23.793 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 54668 54652 155.233 101.125 0.00743347 0.0064367 2024-06-14T22:49:23.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[102754]: pgmap v330: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s; 20538/140196 objects degraded (14.649%); 16 KiB/s, 0 objects/s recovering 2024-06-14T22:49:23.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:23.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 19818/132027 objects degraded (15.011%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:23.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2419032898' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:23.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[102754]: from='osd.1 [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-14T22:49:23.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[102754]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-14T22:49:23.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[97913]: pgmap v330: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s; 20538/140196 objects degraded (14.649%); 16 KiB/s, 0 objects/s recovering 2024-06-14T22:49:23.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:23.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 19818/132027 objects degraded (15.011%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:23.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2419032898' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:23.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[97913]: from='osd.1 [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-14T22:49:23.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-mon[97913]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-14T22:49:23.835 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[123167]: 2024-06-14T22:49:23.360+0000 7f25c9806640 -1 osd.1 112 log_to_monitors true 2024-06-14T22:49:23.835 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 22:49:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[123167]: 2024-06-14T22:49:23.499+0000 7f25c15b0640 -1 osd.1 112 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:49:24.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:24 smithi137 ceph-mon[90634]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-14T22:49:24.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:24 smithi137 ceph-mon[90634]: osdmap e115: 8 total, 7 up, 8 in 2024-06-14T22:49:24.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:24 smithi137 ceph-mon[90634]: from='osd.1 [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:49:24.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:24 smithi137 ceph-mon[90634]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:49:24.793 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 55952 55936 151.973 80.25 0.00260307 0.00648751 2024-06-14T22:49:24.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:24 smithi086 ceph-mon[97913]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-14T22:49:24.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:24 smithi086 ceph-mon[97913]: osdmap e115: 8 total, 7 up, 8 in 2024-06-14T22:49:24.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:24 smithi086 ceph-mon[97913]: from='osd.1 [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:49:24.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:24 smithi086 ceph-mon[97913]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:49:24.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:24 smithi086 ceph-mon[102754]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-14T22:49:24.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:24 smithi086 ceph-mon[102754]: osdmap e115: 8 total, 7 up, 8 in 2024-06-14T22:49:24.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:24 smithi086 ceph-mon[102754]: from='osd.1 [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:49:24.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:24 smithi086 ceph-mon[102754]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:49:25.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:49:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:49:24] "GET /metrics HTTP/1.1" 200 36493 "" "Prometheus/2.43.0" 2024-06-14T22:49:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:25 smithi137 ceph-mon[90634]: pgmap v332: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s; 21732/154086 objects degraded (14.104%) 2024-06-14T22:49:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:25 smithi137 ceph-mon[90634]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:49:25.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:25 smithi137 ceph-mon[90634]: osd.1 [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517] boot 2024-06-14T22:49:25.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:25 smithi137 ceph-mon[90634]: osdmap e116: 8 total, 8 up, 8 in 2024-06-14T22:49:25.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:49:25.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:25 smithi137 ceph-mon[90634]: osdmap e117: 8 total, 8 up, 8 in 2024-06-14T22:49:25.757 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[102754]: pgmap v332: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s; 21732/154086 objects degraded (14.104%) 2024-06-14T22:49:25.757 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[102754]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:49:25.757 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[102754]: osd.1 [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517] boot 2024-06-14T22:49:25.757 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[102754]: osdmap e116: 8 total, 8 up, 8 in 2024-06-14T22:49:25.757 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:49:25.757 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[102754]: osdmap e117: 8 total, 8 up, 8 in 2024-06-14T22:49:25.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[97913]: pgmap v332: 89 pgs: 29 active+undersized, 10 active+undersized+degraded, 50 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s; 21732/154086 objects degraded (14.104%) 2024-06-14T22:49:25.758 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[97913]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:49:25.758 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[97913]: osd.1 [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517] boot 2024-06-14T22:49:25.758 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[97913]: osdmap e116: 8 total, 8 up, 8 in 2024-06-14T22:49:25.758 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-14T22:49:25.758 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:25 smithi086 ceph-mon[97913]: osdmap e117: 8 total, 8 up, 8 in 2024-06-14T22:49:25.793 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 55952 55936 145.642 0 - 0.00648751 2024-06-14T22:49:26.793 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 56163 56147 140.344 6.59375 0.00888494 0.00711908 2024-06-14T22:49:26.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1595316648' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:26.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:26 smithi086 ceph-mon[102754]: osdmap e118: 8 total, 8 up, 8 in 2024-06-14T22:49:26.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1595316648' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:26.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:26 smithi086 ceph-mon[97913]: osdmap e118: 8 total, 8 up, 8 in 2024-06-14T22:49:26.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1595316648' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:26.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:26 smithi137 ceph-mon[90634]: osdmap e118: 8 total, 8 up, 8 in 2024-06-14T22:49:27.793 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 57295 57279 137.666 70.75 0.00695328 0.00725489 2024-06-14T22:49:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:27 smithi086 ceph-mon[97913]: pgmap v335: 89 pgs: 1 remapped+peering, 2 unknown, 5 peering, 24 active+undersized, 5 active+undersized+degraded, 52 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s; 9765/163974 objects degraded (5.955%) 2024-06-14T22:49:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:27.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:27.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:27 smithi086 ceph-mon[102754]: pgmap v335: 89 pgs: 1 remapped+peering, 2 unknown, 5 peering, 24 active+undersized, 5 active+undersized+degraded, 52 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s; 9765/163974 objects degraded (5.955%) 2024-06-14T22:49:27.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:27.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:27.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:27 smithi137 ceph-mon[90634]: pgmap v335: 89 pgs: 1 remapped+peering, 2 unknown, 5 peering, 24 active+undersized, 5 active+undersized+degraded, 52 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s; 9765/163974 objects degraded (5.955%) 2024-06-14T22:49:27.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:27.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:28.793 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 58226 58210 134.721 58.1875 0.010256 0.00741421 2024-06-14T22:49:28.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:28.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:28.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 9765/163974 objects degraded (5.955%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:28.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3200885866' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:29.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:29.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:29.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 9765/163974 objects degraded (5.955%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:29.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3200885866' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:29.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:29.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:29.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 9765/163974 objects degraded (5.955%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:29.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3200885866' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:29.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 59241 59225 132.176 63.4375 0.0128093 0.00756 2024-06-14T22:49:29.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:29 smithi137 ceph-mon[90634]: pgmap v337: 89 pgs: 1 remapped+peering, 2 unknown, 5 peering, 24 active+undersized, 5 active+undersized+degraded, 52 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 9765/163974 objects degraded (5.955%) 2024-06-14T22:49:29.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:29 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:29.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:29 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:30.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:29 smithi086 ceph-mon[97913]: pgmap v337: 89 pgs: 1 remapped+peering, 2 unknown, 5 peering, 24 active+undersized, 5 active+undersized+degraded, 52 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 9765/163974 objects degraded (5.955%) 2024-06-14T22:49:30.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:29 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:30.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:29 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:30.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:29 smithi086 ceph-mon[102754]: pgmap v337: 89 pgs: 1 remapped+peering, 2 unknown, 5 peering, 24 active+undersized, 5 active+undersized+degraded, 52 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 9765/163974 objects degraded (5.955%) 2024-06-14T22:49:30.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:29 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:30.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:29 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:30.793 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 60386 60370 130.085 71.5625 0.011205 0.0076819 2024-06-14T22:49:30.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3332165991' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:30.964 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3332165991' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:30.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3332165991' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:31.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 61551 61535 128.176 72.8125 0.00767529 0.00779562 2024-06-14T22:49:31.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:31 smithi137 ceph-mon[90634]: pgmap v338: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 82 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 13063/173406 objects degraded (7.533%); 1.1 MiB/s, 17 objects/s recovering 2024-06-14T22:49:32.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:31 smithi086 ceph-mon[97913]: pgmap v338: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 82 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 13063/173406 objects degraded (7.533%); 1.1 MiB/s, 17 objects/s recovering 2024-06-14T22:49:32.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:31 smithi086 ceph-mon[102754]: pgmap v338: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 82 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 13063/173406 objects degraded (7.533%); 1.1 MiB/s, 17 objects/s recovering 2024-06-14T22:49:32.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 62689 62673 126.336 71.125 0.0275298 0.00790809 2024-06-14T22:49:32.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:32 smithi137 ceph-mon[90634]: pgmap v339: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 58 MiB/s wr, 935 op/s; 15361/176487 objects degraded (8.704%); 858 KiB/s, 13 objects/s recovering 2024-06-14T22:49:33.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:32 smithi086 ceph-mon[97913]: pgmap v339: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 58 MiB/s wr, 935 op/s; 15361/176487 objects degraded (8.704%); 858 KiB/s, 13 objects/s recovering 2024-06-14T22:49:33.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:32 smithi086 ceph-mon[102754]: pgmap v339: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 58 MiB/s wr, 935 op/s; 15361/176487 objects degraded (8.704%); 858 KiB/s, 13 objects/s recovering 2024-06-14T22:49:33.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 63813 63797 124.583 70.25 0.0133212 0.00802114 2024-06-14T22:49:33.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 15361/176487 objects degraded (8.704%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:33.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2925431908' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:34.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 15361/176487 objects degraded (8.704%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:34.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2925431908' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:34.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 15361/176487 objects degraded (8.704%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:34.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2925431908' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:34.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 16 64994 64978 123.044 73.8125 0.0148616 0.00812144 2024-06-14T22:49:34.832 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:49:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:49:34.433+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:49:35.194 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:49:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:49:34] "GET /metrics HTTP/1.1" 200 36335 "" "Prometheus/2.43.0" 2024-06-14T22:49:35.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:35.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:35.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:49:35.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:49:35.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:35.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:49:35.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:49:35.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:49:35.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:49:35.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:35.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:35.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:49:35.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:35 smithi137 ceph-mon[90634]: pgmap v340: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s; 15152/184950 objects degraded (8.192%); 2.2 MiB/s, 35 objects/s recovering 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:49:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[97913]: pgmap v340: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s; 15152/184950 objects degraded (8.192%); 2.2 MiB/s, 35 objects/s recovering 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:49:35.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:49:35.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:35.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:35.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:49:35.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:35 smithi086 ceph-mon[102754]: pgmap v340: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s; 15152/184950 objects degraded (8.192%); 2.2 MiB/s, 35 objects/s recovering 2024-06-14T22:49:35.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 66115 66099 121.486 70.0625 0.0220248 0.00822401 2024-06-14T22:49:36.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1735939325' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:36.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1735939325' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:36.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1735939325' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:36.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 67295 67279 120.122 73.75 0.011205 0.00831859 2024-06-14T22:49:37.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:37 smithi137 ceph-mon[90634]: pgmap v341: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s; 15152/193290 objects degraded (7.839%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T22:49:37.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:37 smithi086 ceph-mon[97913]: pgmap v341: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s; 15152/193290 objects degraded (7.839%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T22:49:37.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:37 smithi086 ceph-mon[102754]: pgmap v341: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s; 15152/193290 objects degraded (7.839%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T22:49:37.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 68453 68437 118.796 72.375 0.0113315 0.00841118 2024-06-14T22:49:38.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 15152/193290 objects degraded (7.839%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:38.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:38 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1597683537' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 15152/193290 objects degraded (7.839%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:38.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:38 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1597683537' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:38.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 15152/193290 objects degraded (7.839%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:38.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:38 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1597683537' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:38.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 69535 69519 117.413 67.625 0.0095087 0.00851139 2024-06-14T22:49:39.582 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:39 smithi086 ceph-mon[102754]: pgmap v342: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 51 MiB/s wr, 815 op/s; 15152/193290 objects degraded (7.839%); 1.6 MiB/s, 26 objects/s recovering 2024-06-14T22:49:39.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:39 smithi086 ceph-mon[97913]: pgmap v342: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 51 MiB/s wr, 815 op/s; 15152/193290 objects degraded (7.839%); 1.6 MiB/s, 26 objects/s recovering 2024-06-14T22:49:39.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:39 smithi137 ceph-mon[90634]: pgmap v342: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 51 MiB/s wr, 815 op/s; 15152/193290 objects degraded (7.839%); 1.6 MiB/s, 26 objects/s recovering 2024-06-14T22:49:39.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 70686 70670 116.216 71.9375 0.011831 0.00859947 2024-06-14T22:49:40.794 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 16 71780 71764 114.989 68.375 0.0144618 0.00869149 2024-06-14T22:49:41.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:49:41.797751+0000 min lat: 0.00117459 max lat: 2.1576 avg lat: 0.00877492 2024-06-14T22:49:41.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:49:41.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 72918 72902 113.892 71.125 0.0104547 0.00877492 2024-06-14T22:49:41.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:41 smithi086 ceph-mon[97913]: pgmap v343: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 14920/206235 objects degraded (7.234%); 2.9 MiB/s, 45 objects/s recovering 2024-06-14T22:49:41.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4107318745' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:41.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:41 smithi086 ceph-mon[102754]: pgmap v343: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 14920/206235 objects degraded (7.234%); 2.9 MiB/s, 45 objects/s recovering 2024-06-14T22:49:41.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4107318745' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:41.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:41 smithi137 ceph-mon[90634]: pgmap v343: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 14920/206235 objects degraded (7.234%); 2.9 MiB/s, 45 objects/s recovering 2024-06-14T22:49:41.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4107318745' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:42.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 74033 74017 112.814 69.6875 0.00550548 0.00885915 2024-06-14T22:49:42.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:42.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:42.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:43.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 16 75062 75046 111.659 64.3125 0.00818112 0.00895005 2024-06-14T22:49:43.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:43 smithi086 ceph-mon[102754]: pgmap v344: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 14920/210570 objects degraded (7.086%); 2.3 MiB/s, 36 objects/s recovering 2024-06-14T22:49:43.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 14920/210570 objects degraded (7.086%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:43.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4007082099' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:43.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:43 smithi086 ceph-mon[97913]: pgmap v344: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 14920/210570 objects degraded (7.086%); 2.3 MiB/s, 36 objects/s recovering 2024-06-14T22:49:43.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 14920/210570 objects degraded (7.086%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:43.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4007082099' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:43.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:43 smithi137 ceph-mon[90634]: pgmap v344: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 14920/210570 objects degraded (7.086%); 2.3 MiB/s, 36 objects/s recovering 2024-06-14T22:49:43.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 14920/210570 objects degraded (7.086%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:43.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4007082099' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:44.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 76175 76159 110.68 69.5625 0.00587485 0.0090293 2024-06-14T22:49:45.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:49:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:49:44] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.43.0" 2024-06-14T22:49:45.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 16 77224 77208 109.655 65.5625 0.00467683 0.00911351 2024-06-14T22:49:45.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:45 smithi086 ceph-mon[102754]: pgmap v345: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 14656/218700 objects degraded (6.701%); 3.7 MiB/s, 58 objects/s recovering 2024-06-14T22:49:45.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:45 smithi086 ceph-mon[97913]: pgmap v345: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 14656/218700 objects degraded (6.701%); 3.7 MiB/s, 58 objects/s recovering 2024-06-14T22:49:45.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:45 smithi137 ceph-mon[90634]: pgmap v345: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 14656/218700 objects degraded (6.701%); 3.7 MiB/s, 58 objects/s recovering 2024-06-14T22:49:46.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 78296 78280 108.707 67 0.00779133 0.00919409 2024-06-14T22:49:46.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1154897949' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:46.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1154897949' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:46.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1154897949' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:47.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 16 79375 79359 107.809 67.4375 0.00747989 0.00927004 2024-06-14T22:49:47.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:47 smithi086 ceph-mon[97913]: pgmap v346: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 14656/226980 objects degraded (6.457%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:49:47.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:47 smithi086 ceph-mon[102754]: pgmap v346: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 14656/226980 objects degraded (6.457%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:49:47.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:47 smithi137 ceph-mon[90634]: pgmap v346: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 14656/226980 objects degraded (6.457%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:49:48.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 80398 80382 106.876 63.9375 0.0051596 0.00935105 2024-06-14T22:49:48.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 14656/226980 objects degraded (6.457%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:48.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2485151187' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:48.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 14656/226980 objects degraded (6.457%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:48.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2485151187' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:48.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 14656/226980 objects degraded (6.457%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:48.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2485151187' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:49.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 81483 81467 106.062 67.8125 0.015455 0.00942286 2024-06-14T22:49:49.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:49 smithi086 ceph-mon[97913]: pgmap v347: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 58 MiB/s wr, 935 op/s; 14656/226980 objects degraded (6.457%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:49:49.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:49 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:49.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:49 smithi086 ceph-mon[102754]: pgmap v347: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 58 MiB/s wr, 935 op/s; 14656/226980 objects degraded (6.457%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:49:49.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:49 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:49.833 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:49:49 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:49:49.449+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:49:49.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:49 smithi137 ceph-mon[90634]: pgmap v347: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 58 MiB/s wr, 935 op/s; 14656/226980 objects degraded (6.457%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:49:49.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:49 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:50.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 82512 82496 105.21 64.3125 0.0130595 0.0094987 2024-06-14T22:49:50.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:50 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:50.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:50 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:49:50.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:50 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:50.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:50 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:49:50.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:50 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:49:50.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:50 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:49:51.795 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 83567 83551 104.424 65.9375 0.00342689 0.0095705 2024-06-14T22:49:51.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:51 smithi086 ceph-mon[97913]: pgmap v348: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 14386/238974 objects degraded (6.020%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:49:51.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1570789683' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:51.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:51 smithi086 ceph-mon[102754]: pgmap v348: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 14386/238974 objects degraded (6.020%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:49:51.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1570789683' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:51.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:51 smithi137 ceph-mon[90634]: pgmap v348: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 14386/238974 objects degraded (6.020%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:49:51.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1570789683' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:52.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 84653 84637 103.708 67.875 0.00905218 0.00963632 2024-06-14T22:49:53.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 16 85610 85594 102.863 59.8125 0.0396823 0.00971564 2024-06-14T22:49:53.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:53 smithi086 ceph-mon[97913]: pgmap v349: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 14386/243033 objects degraded (5.919%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:49:53.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 14386/243033 objects degraded (5.919%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:53.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1240644239' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:53.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:53 smithi086 ceph-mon[102754]: pgmap v349: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 14386/243033 objects degraded (5.919%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:49:53.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 14386/243033 objects degraded (5.919%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:53.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1240644239' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:53 smithi137 ceph-mon[90634]: pgmap v349: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 14386/243033 objects degraded (5.919%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:49:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 14386/243033 objects degraded (5.919%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1240644239' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:54.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 16 86685 86669 102.19 67.1875 0.00574747 0.00977895 2024-06-14T22:49:55.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:49:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:49:54] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.43.0" 2024-06-14T22:49:55.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 87641 87625 101.404 59.75 0.0113098 0.00985546 2024-06-14T22:49:55.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:55 smithi086 ceph-mon[97913]: pgmap v350: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 14077/250941 objects degraded (5.610%); 4.4 MiB/s, 70 objects/s recovering 2024-06-14T22:49:55.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:55 smithi086 ceph-mon[102754]: pgmap v350: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 14077/250941 objects degraded (5.610%); 4.4 MiB/s, 70 objects/s recovering 2024-06-14T22:49:55.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:55 smithi137 ceph-mon[90634]: pgmap v350: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 14077/250941 objects degraded (5.610%); 4.4 MiB/s, 70 objects/s recovering 2024-06-14T22:49:56.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 88673 88657 100.733 64.5 0.00496594 0.00992188 2024-06-14T22:49:56.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3775971203' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:57.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3775971203' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:57.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3775971203' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:57.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 89691 89675 100.07 63.625 0.0304536 0.00998701 2024-06-14T22:49:57.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:57 smithi086 ceph-mon[97913]: pgmap v351: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 14077/258597 objects degraded (5.444%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:49:57.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:57.984 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:57 smithi086 ceph-mon[102754]: pgmap v351: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 14077/258597 objects degraded (5.444%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:49:57.984 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:57 smithi137 ceph-mon[90634]: pgmap v351: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 14077/258597 objects degraded (5.444%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:49:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:49:58.717 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 14077/258597 objects degraded (5.444%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:58.717 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1788989272' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:58.717 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 14077/258597 objects degraded (5.444%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:58.717 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1788989272' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:58.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 90537 90521 99.2424 52.875 0.0388388 0.010071 2024-06-14T22:49:58.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 14077/258597 objects degraded (5.444%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:49:58.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1788989272' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:49:59.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 16 91418 91402 98.4806 55.0625 0.0226298 0.0101482 2024-06-14T22:49:59.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:49:59 smithi137 ceph-mon[90634]: pgmap v352: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 55 MiB/s wr, 877 op/s; 14077/258597 objects degraded (5.444%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:50:00.035 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:49:59 smithi086 ceph-mon[102754]: pgmap v352: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 55 MiB/s wr, 877 op/s; 14077/258597 objects degraded (5.444%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:50:00.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:49:59 smithi086 ceph-mon[97913]: pgmap v352: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 55 MiB/s wr, 877 op/s; 14077/258597 objects degraded (5.444%); 3.0 MiB/s, 48 objects/s recovering 2024-06-14T22:50:00.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 92351 92335 97.7997 58.3125 0.00602848 0.0102198 2024-06-14T22:50:00.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:00 smithi137 ceph-mon[90634]: Health detail: HEALTH_WARN Degraded data redundancy: 14077/258597 objects degraded (5.444%), 7 pgs degraded 2024-06-14T22:50:00.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:00 smithi137 ceph-mon[90634]: [WRN] PG_DEGRADED: Degraded data redundancy: 14077/258597 objects degraded (5.444%), 7 pgs degraded 2024-06-14T22:50:00.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:00 smithi137 ceph-mon[90634]: pg 8.0 is active+recovery_wait+undersized+degraded+remapped, acting [3,5] 2024-06-14T22:50:00.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:00 smithi137 ceph-mon[90634]: pg 8.1 is active+recovery_wait+undersized+degraded+remapped, acting [0,2] 2024-06-14T22:50:00.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:00 smithi137 ceph-mon[90634]: pg 8.3 is active+recovery_wait+undersized+degraded+remapped, acting [5,6] 2024-06-14T22:50:00.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:00 smithi137 ceph-mon[90634]: pg 8.7 is active+recovery_wait+undersized+degraded+remapped, acting [3,6] 2024-06-14T22:50:00.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:00 smithi137 ceph-mon[90634]: pg 8.a is active+recovering+undersized+degraded+remapped, acting [2,5] 2024-06-14T22:50:00.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:00 smithi137 ceph-mon[90634]: pg 8.b is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-06-14T22:50:00.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:00 smithi137 ceph-mon[90634]: pg 8.f is active+recovery_wait+undersized+degraded+remapped, acting [3,2] 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[102754]: Health detail: HEALTH_WARN Degraded data redundancy: 14077/258597 objects degraded (5.444%), 7 pgs degraded 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[102754]: [WRN] PG_DEGRADED: Degraded data redundancy: 14077/258597 objects degraded (5.444%), 7 pgs degraded 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[102754]: pg 8.0 is active+recovery_wait+undersized+degraded+remapped, acting [3,5] 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[102754]: pg 8.1 is active+recovery_wait+undersized+degraded+remapped, acting [0,2] 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[102754]: pg 8.3 is active+recovery_wait+undersized+degraded+remapped, acting [5,6] 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[102754]: pg 8.7 is active+recovery_wait+undersized+degraded+remapped, acting [3,6] 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[102754]: pg 8.a is active+recovering+undersized+degraded+remapped, acting [2,5] 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[102754]: pg 8.b is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[102754]: pg 8.f is active+recovery_wait+undersized+degraded+remapped, acting [3,2] 2024-06-14T22:50:01.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[97913]: Health detail: HEALTH_WARN Degraded data redundancy: 14077/258597 objects degraded (5.444%), 7 pgs degraded 2024-06-14T22:50:01.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[97913]: [WRN] PG_DEGRADED: Degraded data redundancy: 14077/258597 objects degraded (5.444%), 7 pgs degraded 2024-06-14T22:50:01.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[97913]: pg 8.0 is active+recovery_wait+undersized+degraded+remapped, acting [3,5] 2024-06-14T22:50:01.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[97913]: pg 8.1 is active+recovery_wait+undersized+degraded+remapped, acting [0,2] 2024-06-14T22:50:01.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[97913]: pg 8.3 is active+recovery_wait+undersized+degraded+remapped, acting [5,6] 2024-06-14T22:50:01.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[97913]: pg 8.7 is active+recovery_wait+undersized+degraded+remapped, acting [3,6] 2024-06-14T22:50:01.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[97913]: pg 8.a is active+recovering+undersized+degraded+remapped, acting [2,5] 2024-06-14T22:50:01.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[97913]: pg 8.b is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-06-14T22:50:01.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:00 smithi086 ceph-mon[97913]: pg 8.f is active+recovery_wait+undersized+degraded+remapped, acting [3,2] 2024-06-14T22:50:01.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:50:01.799558+0000 min lat: 0.00117459 max lat: 2.1576 avg lat: 0.0102792 2024-06-14T22:50:01.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:50:01.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 93372 93356 97.2332 63.8125 0.029597 0.0102792 2024-06-14T22:50:01.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:01 smithi137 ceph-mon[90634]: pgmap v353: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 13782/269736 objects degraded (5.109%); 4.6 MiB/s, 72 objects/s recovering 2024-06-14T22:50:01.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2026607694' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:02.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:01 smithi086 ceph-mon[97913]: pgmap v353: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 13782/269736 objects degraded (5.109%); 4.6 MiB/s, 72 objects/s recovering 2024-06-14T22:50:02.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2026607694' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:02.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:01 smithi086 ceph-mon[102754]: pgmap v353: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 13782/269736 objects degraded (5.109%); 4.6 MiB/s, 72 objects/s recovering 2024-06-14T22:50:02.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2026607694' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:02.796 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 94383 94367 96.675 63.1875 0.0218546 0.0103384 2024-06-14T22:50:03.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 95327 95311 96.0673 59 0.00623381 0.0104035 2024-06-14T22:50:03.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:03 smithi137 ceph-mon[90634]: pgmap v354: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 60 MiB/s wr, 953 op/s; 13782/273318 objects degraded (5.042%); 3.1 MiB/s, 50 objects/s recovering 2024-06-14T22:50:03.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 13782/273318 objects degraded (5.042%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:03.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1505522044' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:04.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:03 smithi086 ceph-mon[102754]: pgmap v354: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 60 MiB/s wr, 953 op/s; 13782/273318 objects degraded (5.042%); 3.1 MiB/s, 50 objects/s recovering 2024-06-14T22:50:04.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 13782/273318 objects degraded (5.042%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:04.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1505522044' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:04.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:03 smithi086 ceph-mon[97913]: pgmap v354: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 60 MiB/s wr, 953 op/s; 13782/273318 objects degraded (5.042%); 3.1 MiB/s, 50 objects/s recovering 2024-06-14T22:50:04.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 13782/273318 objects degraded (5.042%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:04.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1505522044' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:04.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 96314 96298 95.5215 61.6875 0.0098814 0.0104632 2024-06-14T22:50:04.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:04 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:04.833 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:50:04.467+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:50:04.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:04 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:04.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:04 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:05.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:50:04] "GET /metrics HTTP/1.1" 200 36497 "" "Prometheus/2.43.0" 2024-06-14T22:50:05.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 97222 97206 94.9157 56.75 0.0108507 0.0105299 2024-06-14T22:50:05.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:05 smithi137 ceph-mon[90634]: pgmap v355: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 13408/280437 objects degraded (4.781%); 5.1 MiB/s, 81 objects/s recovering 2024-06-14T22:50:05.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:05 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:05.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:05 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:50:06.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:05 smithi086 ceph-mon[97913]: pgmap v355: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 13408/280437 objects degraded (4.781%); 5.1 MiB/s, 81 objects/s recovering 2024-06-14T22:50:06.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:05 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:06.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:05 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:50:06.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:05 smithi086 ceph-mon[102754]: pgmap v355: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 13408/280437 objects degraded (4.781%); 5.1 MiB/s, 81 objects/s recovering 2024-06-14T22:50:06.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:05 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:06.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:05 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-06-14T22:50:06.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 98241 98225 94.4351 63.6875 0.0100659 0.0105777 2024-06-14T22:50:06.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3968395637' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:07.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3968395637' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:07.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3968395637' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:07.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 98458 98442 93.2098 13.5625 0.0164348 0.0107222 2024-06-14T22:50:07.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:07 smithi137 ceph-mon[90634]: pgmap v356: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 13408/287640 objects degraded (4.661%); 3.5 MiB/s, 55 objects/s recovering 2024-06-14T22:50:07.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:07 smithi137 ceph-mon[90634]: osdmap e119: 8 total, 8 up, 8 in 2024-06-14T22:50:08.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:07 smithi086 ceph-mon[97913]: pgmap v356: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 13408/287640 objects degraded (4.661%); 3.5 MiB/s, 55 objects/s recovering 2024-06-14T22:50:08.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:07 smithi086 ceph-mon[97913]: osdmap e119: 8 total, 8 up, 8 in 2024-06-14T22:50:08.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:07 smithi086 ceph-mon[102754]: pgmap v356: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 13408/287640 objects degraded (4.661%); 3.5 MiB/s, 55 objects/s recovering 2024-06-14T22:50:08.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:07 smithi086 ceph-mon[102754]: osdmap e119: 8 total, 8 up, 8 in 2024-06-14T22:50:08.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 99403 99387 92.7001 59.0625 0.0087284 0.0107814 2024-06-14T22:50:08.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:08 smithi137 ceph-mon[90634]: osdmap e120: 8 total, 8 up, 8 in 2024-06-14T22:50:08.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 13408/287640 objects degraded (4.661%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:08.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/957157823' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:09.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:08 smithi086 ceph-mon[97913]: osdmap e120: 8 total, 8 up, 8 in 2024-06-14T22:50:09.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 13408/287640 objects degraded (4.661%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:09.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/957157823' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:09.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:08 smithi086 ceph-mon[102754]: osdmap e120: 8 total, 8 up, 8 in 2024-06-14T22:50:09.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 13408/287640 objects degraded (4.661%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:09.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/957157823' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:09.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 100407 100391 92.2596 62.75 0.00725103 0.0108326 2024-06-14T22:50:09.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:09 smithi137 ceph-mon[90634]: pgmap v359: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 745 op/s; 13408/287640 objects degraded (4.661%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T22:50:10.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:09 smithi086 ceph-mon[97913]: pgmap v359: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 745 op/s; 13408/287640 objects degraded (4.661%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T22:50:10.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:09 smithi086 ceph-mon[102754]: pgmap v359: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 745 op/s; 13408/287640 objects degraded (4.661%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T22:50:10.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 101351 101335 91.7775 59 0.0092628 0.0108901 2024-06-14T22:50:11.797 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 102352 102336 91.3601 62.5625 0.00785653 0.0109404 2024-06-14T22:50:11.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:11 smithi137 ceph-mon[90634]: pgmap v360: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s; 13042/296715 objects degraded (4.395%); 5.8 MiB/s, 92 objects/s recovering 2024-06-14T22:50:11.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2062208983' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:12.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:11 smithi086 ceph-mon[97913]: pgmap v360: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s; 13042/296715 objects degraded (4.395%); 5.8 MiB/s, 92 objects/s recovering 2024-06-14T22:50:12.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2062208983' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:12.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:11 smithi086 ceph-mon[102754]: pgmap v360: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s; 13042/296715 objects degraded (4.395%); 5.8 MiB/s, 92 objects/s recovering 2024-06-14T22:50:12.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2062208983' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:12.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 103328 103312 90.9324 61 0.0334947 0.0109917 2024-06-14T22:50:12.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:13.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:13.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:13.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 104261 104245 90.4793 58.3125 0.0151877 0.0110466 2024-06-14T22:50:13.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:13 smithi137 ceph-mon[90634]: pgmap v361: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 50 MiB/s wr, 805 op/s; 13042/299775 objects degraded (4.351%); 2.9 MiB/s, 45 objects/s recovering 2024-06-14T22:50:13.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 13042/299775 objects degraded (4.351%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:13.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/448708707' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:14.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:13 smithi086 ceph-mon[97913]: pgmap v361: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 50 MiB/s wr, 805 op/s; 13042/299775 objects degraded (4.351%); 2.9 MiB/s, 45 objects/s recovering 2024-06-14T22:50:14.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 13042/299775 objects degraded (4.351%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:14.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/448708707' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:14.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:13 smithi086 ceph-mon[102754]: pgmap v361: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 50 MiB/s wr, 805 op/s; 13042/299775 objects degraded (4.351%); 2.9 MiB/s, 45 objects/s recovering 2024-06-14T22:50:14.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 13042/299775 objects degraded (4.351%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:14.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/448708707' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:14.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 105235 105219 90.0737 60.875 0.00609955 0.0110964 2024-06-14T22:50:15.332 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:50:14] "GET /metrics HTTP/1.1" 200 36501 "" "Prometheus/2.43.0" 2024-06-14T22:50:15.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 16 106150 106134 89.6292 57.1875 0.0135692 0.0111512 2024-06-14T22:50:15.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:15 smithi137 ceph-mon[90634]: pgmap v362: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 51 MiB/s wr, 812 op/s; 12823/307137 objects degraded (4.175%); 4.6 MiB/s, 73 objects/s recovering 2024-06-14T22:50:16.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:15 smithi086 ceph-mon[102754]: pgmap v362: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 51 MiB/s wr, 812 op/s; 12823/307137 objects degraded (4.175%); 4.6 MiB/s, 73 objects/s recovering 2024-06-14T22:50:16.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:15 smithi086 ceph-mon[97913]: pgmap v362: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 51 MiB/s wr, 812 op/s; 12823/307137 objects degraded (4.175%); 4.6 MiB/s, 73 objects/s recovering 2024-06-14T22:50:16.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 107113 107097 89.2366 60.1875 0.0123655 0.0112003 2024-06-14T22:50:16.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:16 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3391136051' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:17.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:16 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3391136051' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:17.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3391136051' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:17.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 108069 108053 88.8486 59.75 0.0206599 0.0112497 2024-06-14T22:50:17.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:17 smithi137 ceph-mon[90634]: pgmap v363: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s; 12823/314421 objects degraded (4.078%); 3.7 MiB/s, 59 objects/s recovering 2024-06-14T22:50:18.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:17 smithi086 ceph-mon[97913]: pgmap v363: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s; 12823/314421 objects degraded (4.078%); 3.7 MiB/s, 59 objects/s recovering 2024-06-14T22:50:18.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:17 smithi086 ceph-mon[102754]: pgmap v363: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s; 12823/314421 objects degraded (4.078%); 3.7 MiB/s, 59 objects/s recovering 2024-06-14T22:50:18.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 16 108953 108937 88.4122 55.25 0.00504509 0.0113051 2024-06-14T22:50:18.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 12823/314421 objects degraded (4.078%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:18.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1157518282' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:19.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 12823/314421 objects degraded (4.078%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:19.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1157518282' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:19.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 12823/314421 objects degraded (4.078%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:19.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1157518282' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:19.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 109897 109881 88.035 59 0.0225903 0.011354 2024-06-14T22:50:19.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:19 smithi086 ceph-mon[102754]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 52 MiB/s wr, 825 op/s; 12823/314421 objects degraded (4.078%); 3.4 MiB/s, 54 objects/s recovering 2024-06-14T22:50:19.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:19 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:19.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:19 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:19.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:19 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:19.833 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:19 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:50:19.486+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:19.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:19 smithi086 ceph-mon[97913]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 52 MiB/s wr, 825 op/s; 12823/314421 objects degraded (4.078%); 3.4 MiB/s, 54 objects/s recovering 2024-06-14T22:50:19.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:19 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:19.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:19 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:19.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:19 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:19.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:19 smithi137 ceph-mon[90634]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 52 MiB/s wr, 825 op/s; 12823/314421 objects degraded (4.078%); 3.4 MiB/s, 54 objects/s recovering 2024-06-14T22:50:19.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:19 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:19.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:19 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:19.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:19 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:20.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 110781 110765 87.62 55.25 0.0196472 0.0114075 2024-06-14T22:50:20.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:20 smithi137 ceph-mon[90634]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 12588/325059 objects degraded (3.873%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:50:21.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:20 smithi086 ceph-mon[97913]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 12588/325059 objects degraded (3.873%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:50:21.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:20 smithi086 ceph-mon[102754]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 12588/325059 objects degraded (3.873%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:50:21.798 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:50:21.801402+0000 min lat: 0.00117459 max lat: 2.1576 avg lat: 0.0114546 2024-06-14T22:50:21.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:50:21.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 111716 111700 87.2551 58.4375 0.0436595 0.0114546 2024-06-14T22:50:21.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1408240003' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:22.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1408240003' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:22.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1408240003' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:22.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 112636 112620 86.8877 57.5 0.00805832 0.0115022 2024-06-14T22:50:23.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:22 smithi086 ceph-mon[97913]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 55 MiB/s wr, 882 op/s; 12588/328485 objects degraded (3.832%); 2.4 MiB/s, 37 objects/s recovering 2024-06-14T22:50:23.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:22 smithi086 ceph-mon[102754]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 55 MiB/s wr, 882 op/s; 12588/328485 objects degraded (3.832%); 2.4 MiB/s, 37 objects/s recovering 2024-06-14T22:50:23.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:22 smithi137 ceph-mon[90634]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 55 MiB/s wr, 882 op/s; 12588/328485 objects degraded (3.832%); 2.4 MiB/s, 37 objects/s recovering 2024-06-14T22:50:23.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 113505 113489 86.4904 54.3125 0.00687125 0.0115562 2024-06-14T22:50:24.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 12588/328485 objects degraded (3.832%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:24.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3320627084' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:24.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 12588/328485 objects degraded (3.832%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:24.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3320627084' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:24.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 12588/328485 objects degraded (3.832%), 6 pgs degraded (PG_DEGRADED) 2024-06-14T22:50:24.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3320627084' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:24.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 16 114440 114424 86.1524 58.4375 0.0154459 0.0116015 2024-06-14T22:50:25.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:24 smithi086 ceph-mon[97913]: pgmap v367: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s; 12339/335283 objects degraded (3.680%); 3.7 MiB/s, 58 objects/s recovering 2024-06-14T22:50:25.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:24 smithi086 ceph-mon[102754]: pgmap v367: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s; 12339/335283 objects degraded (3.680%); 3.7 MiB/s, 58 objects/s recovering 2024-06-14T22:50:25.083 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:50:24] "GET /metrics HTTP/1.1" 200 36501 "" "Prometheus/2.43.0" 2024-06-14T22:50:25.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:24 smithi137 ceph-mon[90634]: pgmap v367: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s; 12339/335283 objects degraded (3.680%); 3.7 MiB/s, 58 objects/s recovering 2024-06-14T22:50:25.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 115318 115302 85.78 54.875 0.00909691 0.0116522 2024-06-14T22:50:26.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3839514772' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:26.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3839514772' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:26.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3839514772' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:26.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 116231 116215 85.4421 57.0625 0.00917152 0.0116987 2024-06-14T22:50:27.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:26 smithi086 ceph-mon[97913]: pgmap v368: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 970 op/s; 12339/342081 objects degraded (3.607%); 2.5 MiB/s, 40 objects/s recovering 2024-06-14T22:50:27.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:26 smithi086 ceph-mon[102754]: pgmap v368: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 970 op/s; 12339/342081 objects degraded (3.607%); 2.5 MiB/s, 40 objects/s recovering 2024-06-14T22:50:27.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:26 smithi137 ceph-mon[90634]: pgmap v368: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 970 op/s; 12339/342081 objects degraded (3.607%); 2.5 MiB/s, 40 objects/s recovering 2024-06-14T22:50:27.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 117164 117148 85.1266 58.3125 0.00733853 0.0117418 2024-06-14T22:50:28.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:28.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:28.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:28.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 118002 117986 84.7501 52.375 0.0448963 0.0117939 2024-06-14T22:50:29.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:29 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 12339/342081 objects degraded (3.607%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:29.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:29 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2540115435' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:29.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:29 smithi086 ceph-mon[97913]: pgmap v369: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 48 MiB/s wr, 768 op/s; 12339/342081 objects degraded (3.607%); 2.5 MiB/s, 40 objects/s recovering 2024-06-14T22:50:29.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:29 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 12339/342081 objects degraded (3.607%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:29.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:29 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2540115435' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:29.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:29 smithi086 ceph-mon[102754]: pgmap v369: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 48 MiB/s wr, 768 op/s; 12339/342081 objects degraded (3.607%); 2.5 MiB/s, 40 objects/s recovering 2024-06-14T22:50:29.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:29 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 12339/342081 objects degraded (3.607%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:29.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:29 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2540115435' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:29.491 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:29 smithi137 ceph-mon[90634]: pgmap v369: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 48 MiB/s wr, 768 op/s; 12339/342081 objects degraded (3.607%); 2.5 MiB/s, 40 objects/s recovering 2024-06-14T22:50:29.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 118908 118892 84.4304 56.625 0.0155911 0.011839 2024-06-14T22:50:30.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 119751 119735 84.0737 52.6875 0.0227929 0.0118875 2024-06-14T22:50:31.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 90 5 120586 120581 83.727 52.875 0.0128327 0.0119382 2024-06-14T22:50:31.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:31 smithi086 ceph-mon[97913]: pgmap v370: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 12082/352341 objects degraded (3.429%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T22:50:31.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/752261875' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:31.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:31 smithi086 ceph-mon[102754]: pgmap v370: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 12082/352341 objects degraded (3.429%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T22:50:31.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/752261875' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:31.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:31 smithi137 ceph-mon[90634]: pgmap v370: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 12082/352341 objects degraded (3.429%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T22:50:31.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/752261875' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:32.799 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0131 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 120586 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 83.7281 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 47.9712 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 242.562 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 0 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1339 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 767.549 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 3881 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 0 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0119385 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.0302984 2024-06-14T22:50:32.800 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 2.1576 2024-06-14T22:50:32.801 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00117459 2024-06-14T22:50:32.807 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T22:50:33.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:33 smithi086 ceph-mon[97913]: pgmap v371: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s; 12082/355587 objects degraded (3.398%); 2.6 MiB/s, 42 objects/s recovering 2024-06-14T22:50:33.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 12082/355587 objects degraded (3.398%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:33.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2097691104' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:33 smithi086 ceph-mon[102754]: pgmap v371: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s; 12082/355587 objects degraded (3.398%); 2.6 MiB/s, 42 objects/s recovering 2024-06-14T22:50:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 12082/355587 objects degraded (3.398%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:33.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2097691104' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:33.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:33 smithi137 ceph-mon[90634]: pgmap v371: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s; 12082/355587 objects degraded (3.398%); 2.6 MiB/s, 42 objects/s recovering 2024-06-14T22:50:33.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 12082/355587 objects degraded (3.398%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:33.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2097691104' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:34.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:34 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:50:34.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:34 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:50:34.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:34 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:50:35.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:50:34] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.43.0" 2024-06-14T22:50:35.797 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:35 smithi137 ceph-mon[90634]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 54 MiB/s wr, 897 op/s; 11601/358764 objects degraded (3.234%); 5.1 MiB/s, 82 objects/s recovering 2024-06-14T22:50:35.797 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:35.797 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:35.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:35 smithi086 ceph-mon[97913]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 54 MiB/s wr, 897 op/s; 11601/358764 objects degraded (3.234%); 5.1 MiB/s, 82 objects/s recovering 2024-06-14T22:50:35.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:35.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:35.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:35 smithi086 ceph-mon[102754]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 54 MiB/s wr, 897 op/s; 11601/358764 objects degraded (3.234%); 5.1 MiB/s, 82 objects/s recovering 2024-06-14T22:50:35.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:35.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:36.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/126380017' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:36.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/126380017' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:36.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/126380017' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:37.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:37 smithi086 ceph-mon[97913]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 46 MiB/s wr, 849 op/s; 11601/357789 objects degraded (3.242%); 3.8 MiB/s, 61 objects/s recovering 2024-06-14T22:50:37.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:37 smithi086 ceph-mon[102754]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 46 MiB/s wr, 849 op/s; 11601/357789 objects degraded (3.242%); 3.8 MiB/s, 61 objects/s recovering 2024-06-14T22:50:37.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:37 smithi137 ceph-mon[90634]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 46 MiB/s wr, 849 op/s; 11601/357789 objects degraded (3.242%); 3.8 MiB/s, 61 objects/s recovering 2024-06-14T22:50:38.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 11601/357789 objects degraded (3.242%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:38.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:38 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3157667974' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:38.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 11601/357789 objects degraded (3.242%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:38.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:38 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3157667974' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:38.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 11601/357789 objects degraded (3.242%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:38.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:38 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3157667974' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:39.643 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:39 smithi137 ceph-mon[90634]: pgmap v374: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 660 op/s; 11601/357789 objects degraded (3.242%); 3.8 MiB/s, 61 objects/s recovering 2024-06-14T22:50:39.752 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:39 smithi086 ceph-mon[102754]: pgmap v374: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 660 op/s; 11601/357789 objects degraded (3.242%); 3.8 MiB/s, 61 objects/s recovering 2024-06-14T22:50:39.752 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:39 smithi086 ceph-mon[97913]: pgmap v374: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 660 op/s; 11601/357789 objects degraded (3.242%); 3.8 MiB/s, 61 objects/s recovering 2024-06-14T22:50:41.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:50:41.057+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:41.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: pgmap v375: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 917 op/s; 11316/348528 objects degraded (3.247%); 5.3 MiB/s, 85 objects/s recovering 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1225999782' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: pgmap v375: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 917 op/s; 11316/348528 objects degraded (3.247%); 5.3 MiB/s, 85 objects/s recovering 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1225999782' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:50:41.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: pgmap v375: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 34 MiB/s wr, 917 op/s; 11316/348528 objects degraded (3.247%); 5.3 MiB/s, 85 objects/s recovering 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1225999782' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:50:41.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:50:41.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:50:41.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:42.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:42 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:42.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:42 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:42.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:42 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:42.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:42 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:42.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:42 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:42.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:42 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:43 smithi086 ceph-mon[97913]: pgmap v376: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 16 MiB/s wr, 718 op/s; 11316/345447 objects degraded (3.276%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:50:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 11316/348528 objects degraded (3.247%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:43.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:43 smithi086 ceph-mon[97913]: osdmap e121: 8 total, 8 up, 8 in 2024-06-14T22:50:43.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:43.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:43 smithi086 ceph-mon[102754]: pgmap v376: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 16 MiB/s wr, 718 op/s; 11316/345447 objects degraded (3.276%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:50:43.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 11316/348528 objects degraded (3.247%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:43.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:43 smithi086 ceph-mon[102754]: osdmap e121: 8 total, 8 up, 8 in 2024-06-14T22:50:43.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:43.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:43 smithi137 ceph-mon[90634]: pgmap v376: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 16 MiB/s wr, 718 op/s; 11316/345447 objects degraded (3.276%); 4.0 MiB/s, 63 objects/s recovering 2024-06-14T22:50:43.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 11316/348528 objects degraded (3.247%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:43.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:43 smithi137 ceph-mon[90634]: osdmap e121: 8 total, 8 up, 8 in 2024-06-14T22:50:44.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1052556619' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:44.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:44 smithi086 ceph-mon[97913]: osdmap e122: 8 total, 8 up, 8 in 2024-06-14T22:50:44.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1052556619' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:44.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:44 smithi086 ceph-mon[102754]: osdmap e122: 8 total, 8 up, 8 in 2024-06-14T22:50:44.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1052556619' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:44.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:44 smithi137 ceph-mon[90634]: osdmap e122: 8 total, 8 up, 8 in 2024-06-14T22:50:45.313 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:50:44] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.43.0" 2024-06-14T22:50:45.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:45 smithi086 ceph-mon[97913]: pgmap v379: 89 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 767 op/s; 11073/339366 objects degraded (3.263%); 4.1 MiB/s, 65 objects/s recovering 2024-06-14T22:50:45.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:45 smithi086 ceph-mon[102754]: pgmap v379: 89 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 767 op/s; 11073/339366 objects degraded (3.263%); 4.1 MiB/s, 65 objects/s recovering 2024-06-14T22:50:45.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:45 smithi137 ceph-mon[90634]: pgmap v379: 89 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 767 op/s; 11073/339366 objects degraded (3.263%); 4.1 MiB/s, 65 objects/s recovering 2024-06-14T22:50:46.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2760584028' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:46.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2760584028' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:46.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2760584028' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:47.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:47 smithi086 ceph-mon[102754]: pgmap v380: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 10932/333954 objects degraded (3.274%); 5.2 MiB/s, 83 objects/s recovering 2024-06-14T22:50:47.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:47 smithi086 ceph-mon[97913]: pgmap v380: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 10932/333954 objects degraded (3.274%); 5.2 MiB/s, 83 objects/s recovering 2024-06-14T22:50:47.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:47 smithi137 ceph-mon[90634]: pgmap v380: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 10932/333954 objects degraded (3.274%); 5.2 MiB/s, 83 objects/s recovering 2024-06-14T22:50:48.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10932/333954 objects degraded (3.274%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:48.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10932/333954 objects degraded (3.274%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:48.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10932/333954 objects degraded (3.274%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:49.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1163055290' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:49.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:49 smithi137 ceph-mon[90634]: pgmap v381: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 10932/333954 objects degraded (3.274%); 3.0 MiB/s, 47 objects/s recovering 2024-06-14T22:50:49.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1163055290' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:49.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:49 smithi086 ceph-mon[97913]: pgmap v381: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 10932/333954 objects degraded (3.274%); 3.0 MiB/s, 47 objects/s recovering 2024-06-14T22:50:49.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1163055290' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:49.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:49 smithi086 ceph-mon[102754]: pgmap v381: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 10932/333954 objects degraded (3.274%); 3.0 MiB/s, 47 objects/s recovering 2024-06-14T22:50:51.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:51 smithi086 ceph-mon[97913]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 10771/324873 objects degraded (3.315%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:50:51.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3258663158' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:51.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:51 smithi086 ceph-mon[102754]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 10771/324873 objects degraded (3.315%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:50:51.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3258663158' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:51.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:51 smithi137 ceph-mon[90634]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 10771/324873 objects degraded (3.315%); 4.3 MiB/s, 68 objects/s recovering 2024-06-14T22:50:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3258663158' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:53.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:53 smithi086 ceph-mon[97913]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 10771/321795 objects degraded (3.347%); 3.6 MiB/s, 58 objects/s recovering 2024-06-14T22:50:53.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10771/321795 objects degraded (3.347%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:53.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3632654502' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:53.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:53 smithi086 ceph-mon[102754]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 10771/321795 objects degraded (3.347%); 3.6 MiB/s, 58 objects/s recovering 2024-06-14T22:50:53.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10771/321795 objects degraded (3.347%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:53.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3632654502' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:53 smithi137 ceph-mon[90634]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 10771/321795 objects degraded (3.347%); 3.6 MiB/s, 58 objects/s recovering 2024-06-14T22:50:53.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10771/321795 objects degraded (3.347%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:53.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3632654502' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:55.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:50:54] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.43.0" 2024-06-14T22:50:55.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:55 smithi086 ceph-mon[97913]: pgmap v384: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 764 op/s; 10771/315543 objects degraded (3.413%); 1.8 MiB/s, 29 objects/s recovering 2024-06-14T22:50:55.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:55 smithi086 ceph-mon[102754]: pgmap v384: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 764 op/s; 10771/315543 objects degraded (3.413%); 1.8 MiB/s, 29 objects/s recovering 2024-06-14T22:50:55.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:55 smithi137 ceph-mon[90634]: pgmap v384: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 764 op/s; 10771/315543 objects degraded (3.413%); 1.8 MiB/s, 29 objects/s recovering 2024-06-14T22:50:56.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:50:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:50:56.073+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:56.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1862174276' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:56.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1862174276' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:56.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:56.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1862174276' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:56.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:57 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:57 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:57 smithi086 ceph-mon[97913]: pgmap v385: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 839 op/s; 10601/309153 objects degraded (3.429%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T22:50:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:57.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:57 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:57.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:57 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:57.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:57 smithi086 ceph-mon[102754]: pgmap v385: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 839 op/s; 10601/309153 objects degraded (3.429%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T22:50:57.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:57 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:50:57.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:57 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:50:57.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:57 smithi137 ceph-mon[90634]: pgmap v385: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 839 op/s; 10601/309153 objects degraded (3.429%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T22:50:57.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:50:58.754 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10601/309153 objects degraded (3.429%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:58.754 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3417895653' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:58.754 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10601/309153 objects degraded (3.429%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:58.754 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3417895653' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:58.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10601/309153 objects degraded (3.429%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:50:58.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3417895653' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:50:59.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:50:59 smithi086 ceph-mon[102754]: pgmap v386: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 10601/309153 objects degraded (3.429%); 1.7 MiB/s, 27 objects/s recovering 2024-06-14T22:50:59.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:50:59 smithi086 ceph-mon[97913]: pgmap v386: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 10601/309153 objects degraded (3.429%); 1.7 MiB/s, 27 objects/s recovering 2024-06-14T22:50:59.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:50:59 smithi137 ceph-mon[90634]: pgmap v386: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 10601/309153 objects degraded (3.429%); 1.7 MiB/s, 27 objects/s recovering 2024-06-14T22:51:01.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:01 smithi086 ceph-mon[97913]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 937 op/s; 10425/300210 objects degraded (3.473%); 2.6 MiB/s, 42 objects/s recovering 2024-06-14T22:51:01.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4235143343' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:01.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:01 smithi086 ceph-mon[102754]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 937 op/s; 10425/300210 objects degraded (3.473%); 2.6 MiB/s, 42 objects/s recovering 2024-06-14T22:51:01.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4235143343' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:01.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:01 smithi137 ceph-mon[90634]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 937 op/s; 10425/300210 objects degraded (3.473%); 2.6 MiB/s, 42 objects/s recovering 2024-06-14T22:51:01.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4235143343' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:03.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:03 smithi086 ceph-mon[102754]: pgmap v388: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 10425/297309 objects degraded (3.506%); 1.8 MiB/s, 28 objects/s recovering 2024-06-14T22:51:03.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10425/297309 objects degraded (3.506%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:03.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3358534864' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:03.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:03 smithi086 ceph-mon[97913]: pgmap v388: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 10425/297309 objects degraded (3.506%); 1.8 MiB/s, 28 objects/s recovering 2024-06-14T22:51:03.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10425/297309 objects degraded (3.506%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:03.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3358534864' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:03.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:03 smithi137 ceph-mon[90634]: pgmap v388: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 10425/297309 objects degraded (3.506%); 1.8 MiB/s, 28 objects/s recovering 2024-06-14T22:51:03.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10425/297309 objects degraded (3.506%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:03.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3358534864' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:05.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:51:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:51:04] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.43.0" 2024-06-14T22:51:05.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:05 smithi137 ceph-mon[90634]: pgmap v389: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 10425/291660 objects degraded (3.574%); 1.8 MiB/s, 28 objects/s recovering 2024-06-14T22:51:06.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:05 smithi086 ceph-mon[97913]: pgmap v389: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 10425/291660 objects degraded (3.574%); 1.8 MiB/s, 28 objects/s recovering 2024-06-14T22:51:06.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:05 smithi086 ceph-mon[102754]: pgmap v389: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 10425/291660 objects degraded (3.574%); 1.8 MiB/s, 28 objects/s recovering 2024-06-14T22:51:06.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/232470558' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:07.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/232470558' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:07.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/232470558' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:07.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:07 smithi137 ceph-mon[90634]: pgmap v390: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 10235/285471 objects degraded (3.585%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:51:08.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:07 smithi086 ceph-mon[102754]: pgmap v390: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 10235/285471 objects degraded (3.585%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:51:08.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:07 smithi086 ceph-mon[97913]: pgmap v390: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 10235/285471 objects degraded (3.585%); 2.8 MiB/s, 44 objects/s recovering 2024-06-14T22:51:08.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10235/285471 objects degraded (3.585%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:08.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2342011368' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:09.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10235/285471 objects degraded (3.585%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:09.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2342011368' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:09.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10235/285471 objects degraded (3.585%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:09.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2342011368' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:09.893 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:09 smithi137 ceph-mon[90634]: pgmap v391: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 10235/285471 objects degraded (3.585%); 1.9 MiB/s, 30 objects/s recovering 2024-06-14T22:51:10.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:09 smithi086 ceph-mon[102754]: pgmap v391: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 10235/285471 objects degraded (3.585%); 1.9 MiB/s, 30 objects/s recovering 2024-06-14T22:51:10.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:09 smithi086 ceph-mon[97913]: pgmap v391: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 10235/285471 objects degraded (3.585%); 1.9 MiB/s, 30 objects/s recovering 2024-06-14T22:51:11.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:51:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:51:11.090+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:11.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:11 smithi137 ceph-mon[90634]: pgmap v392: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 909 op/s; 10040/276396 objects degraded (3.632%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T22:51:11.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/346755544' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:11.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:12.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:11 smithi086 ceph-mon[97913]: pgmap v392: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 909 op/s; 10040/276396 objects degraded (3.632%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T22:51:12.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/346755544' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:12.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:12.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:11 smithi086 ceph-mon[102754]: pgmap v392: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 909 op/s; 10040/276396 objects degraded (3.632%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T22:51:12.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/346755544' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:12.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:12.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:12 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:12.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:12 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:12.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:13.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:12 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:13.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:12 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:13.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:13.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:12 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:13.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:12 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:13.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:13.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:13 smithi137 ceph-mon[90634]: pgmap v393: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 10040/273306 objects degraded (3.674%); 2.0 MiB/s, 32 objects/s recovering 2024-06-14T22:51:13.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10040/273306 objects degraded (3.674%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:13.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2452904106' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:14.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:13 smithi086 ceph-mon[97913]: pgmap v393: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 10040/273306 objects degraded (3.674%); 2.0 MiB/s, 32 objects/s recovering 2024-06-14T22:51:14.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10040/273306 objects degraded (3.674%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:14.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2452904106' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:14.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:13 smithi086 ceph-mon[102754]: pgmap v393: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 10040/273306 objects degraded (3.674%); 2.0 MiB/s, 32 objects/s recovering 2024-06-14T22:51:14.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10040/273306 objects degraded (3.674%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:14.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2452904106' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:15.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:51:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:51:14] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.43.0" 2024-06-14T22:51:15.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:15 smithi137 ceph-mon[90634]: pgmap v394: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 10040/267348 objects degraded (3.755%); 2.0 MiB/s, 32 objects/s recovering 2024-06-14T22:51:16.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:15 smithi086 ceph-mon[97913]: pgmap v394: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 10040/267348 objects degraded (3.755%); 2.0 MiB/s, 32 objects/s recovering 2024-06-14T22:51:16.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:15 smithi086 ceph-mon[102754]: pgmap v394: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 10040/267348 objects degraded (3.755%); 2.0 MiB/s, 32 objects/s recovering 2024-06-14T22:51:16.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:16 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3920673786' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:17.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3920673786' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:17.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:16 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3920673786' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:17.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:17 smithi137 ceph-mon[90634]: pgmap v395: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 9851/261255 objects degraded (3.771%); 3.0 MiB/s, 47 objects/s recovering 2024-06-14T22:51:18.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:17 smithi086 ceph-mon[97913]: pgmap v395: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 9851/261255 objects degraded (3.771%); 3.0 MiB/s, 47 objects/s recovering 2024-06-14T22:51:18.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:17 smithi086 ceph-mon[102754]: pgmap v395: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 9851/261255 objects degraded (3.771%); 3.0 MiB/s, 47 objects/s recovering 2024-06-14T22:51:18.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 9851/261255 objects degraded (3.771%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:18.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/382190911' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:19.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 9851/261255 objects degraded (3.771%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:19.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/382190911' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:19.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 9851/261255 objects degraded (3.771%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:19.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/382190911' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:19.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:19 smithi137 ceph-mon[90634]: pgmap v396: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 9851/261255 objects degraded (3.771%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T22:51:20.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:19 smithi086 ceph-mon[97913]: pgmap v396: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 9851/261255 objects degraded (3.771%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T22:51:20.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:19 smithi086 ceph-mon[102754]: pgmap v396: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 9851/261255 objects degraded (3.771%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T22:51:21.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:21 smithi137 ceph-mon[90634]: pgmap v397: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 9628/252531 objects degraded (3.813%); 3.2 MiB/s, 50 objects/s recovering 2024-06-14T22:51:21.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4045448435' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:22.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:21 smithi086 ceph-mon[97913]: pgmap v397: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 9628/252531 objects degraded (3.813%); 3.2 MiB/s, 50 objects/s recovering 2024-06-14T22:51:22.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4045448435' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:22.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:21 smithi086 ceph-mon[102754]: pgmap v397: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 9628/252531 objects degraded (3.813%); 3.2 MiB/s, 50 objects/s recovering 2024-06-14T22:51:22.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4045448435' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:23.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:23 smithi137 ceph-mon[90634]: pgmap v398: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 9628/249306 objects degraded (3.862%); 2.1 MiB/s, 34 objects/s recovering 2024-06-14T22:51:23.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 9628/249306 objects degraded (3.862%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:23.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1366219318' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:24.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:23 smithi086 ceph-mon[97913]: pgmap v398: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 9628/249306 objects degraded (3.862%); 2.1 MiB/s, 34 objects/s recovering 2024-06-14T22:51:24.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 9628/249306 objects degraded (3.862%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:24.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1366219318' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:24.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:23 smithi086 ceph-mon[102754]: pgmap v398: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 9628/249306 objects degraded (3.862%); 2.1 MiB/s, 34 objects/s recovering 2024-06-14T22:51:24.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 9628/249306 objects degraded (3.862%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:24.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1366219318' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:25.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:51:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:51:24] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.43.0" 2024-06-14T22:51:25.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:25 smithi137 ceph-mon[90634]: pgmap v399: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 9628/243279 objects degraded (3.958%); 2.1 MiB/s, 34 objects/s recovering 2024-06-14T22:51:26.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:25 smithi086 ceph-mon[97913]: pgmap v399: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 9628/243279 objects degraded (3.958%); 2.1 MiB/s, 34 objects/s recovering 2024-06-14T22:51:26.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:25 smithi086 ceph-mon[102754]: pgmap v399: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 9628/243279 objects degraded (3.958%); 2.1 MiB/s, 34 objects/s recovering 2024-06-14T22:51:26.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:51:26 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:51:26.107+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:26.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4130386574' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:26.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:26 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:27.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4130386574' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:27.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:26 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:27.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4130386574' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:27.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:26 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:27 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:27 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:27 smithi137 ceph-mon[90634]: pgmap v400: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 9390/237261 objects degraded (3.958%); 3.4 MiB/s, 54 objects/s recovering 2024-06-14T22:51:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:28.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:27 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:28.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:27 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:28.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:27 smithi086 ceph-mon[102754]: pgmap v400: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 9390/237261 objects degraded (3.958%); 3.4 MiB/s, 54 objects/s recovering 2024-06-14T22:51:28.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:28.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:27 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:28.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:27 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:28.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:27 smithi086 ceph-mon[97913]: pgmap v400: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 9390/237261 objects degraded (3.958%); 3.4 MiB/s, 54 objects/s recovering 2024-06-14T22:51:28.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:28.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 9390/237261 objects degraded (3.958%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:28.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2943547206' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:29.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 9390/237261 objects degraded (3.958%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:29.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2943547206' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:29.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 9390/237261 objects degraded (3.958%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:29.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2943547206' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:29.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:29 smithi137 ceph-mon[90634]: pgmap v401: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 9390/237261 objects degraded (3.958%); 2.4 MiB/s, 38 objects/s recovering 2024-06-14T22:51:30.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:29 smithi086 ceph-mon[97913]: pgmap v401: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 9390/237261 objects degraded (3.958%); 2.4 MiB/s, 38 objects/s recovering 2024-06-14T22:51:30.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:29 smithi086 ceph-mon[102754]: pgmap v401: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 9390/237261 objects degraded (3.958%); 2.4 MiB/s, 38 objects/s recovering 2024-06-14T22:51:31.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:31 smithi137 ceph-mon[90634]: pgmap v402: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 916 op/s; 9131/228234 objects degraded (4.001%); 3.7 MiB/s, 59 objects/s recovering 2024-06-14T22:51:31.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2033177081' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:32.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:31 smithi086 ceph-mon[97913]: pgmap v402: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 916 op/s; 9131/228234 objects degraded (4.001%); 3.7 MiB/s, 59 objects/s recovering 2024-06-14T22:51:32.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2033177081' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:32.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:31 smithi086 ceph-mon[102754]: pgmap v402: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 916 op/s; 9131/228234 objects degraded (4.001%); 3.7 MiB/s, 59 objects/s recovering 2024-06-14T22:51:32.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2033177081' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:33.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:33 smithi137 ceph-mon[90634]: pgmap v403: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 763 op/s; 9131/225024 objects degraded (4.058%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:51:33.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 9131/225024 objects degraded (4.058%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:33.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/672621360' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:34.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:33 smithi086 ceph-mon[97913]: pgmap v403: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 763 op/s; 9131/225024 objects degraded (4.058%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:51:34.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 9131/225024 objects degraded (4.058%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:34.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/672621360' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:34.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:33 smithi086 ceph-mon[102754]: pgmap v403: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 763 op/s; 9131/225024 objects degraded (4.058%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:51:34.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 9131/225024 objects degraded (4.058%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:34.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/672621360' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:35.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:51:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:51:34] "GET /metrics HTTP/1.1" 200 36504 "" "Prometheus/2.43.0" 2024-06-14T22:51:35.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:35 smithi137 ceph-mon[90634]: pgmap v404: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 9131/218814 objects degraded (4.173%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:51:35.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:35 smithi137 ceph-mon[90634]: osdmap e123: 8 total, 8 up, 8 in 2024-06-14T22:51:36.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:35 smithi086 ceph-mon[97913]: pgmap v404: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 9131/218814 objects degraded (4.173%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:51:36.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:35 smithi086 ceph-mon[97913]: osdmap e123: 8 total, 8 up, 8 in 2024-06-14T22:51:36.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:35 smithi086 ceph-mon[102754]: pgmap v404: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 9131/218814 objects degraded (4.173%); 2.6 MiB/s, 41 objects/s recovering 2024-06-14T22:51:36.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:35 smithi086 ceph-mon[102754]: osdmap e123: 8 total, 8 up, 8 in 2024-06-14T22:51:37.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:36 smithi086 ceph-mon[97913]: osdmap e124: 8 total, 8 up, 8 in 2024-06-14T22:51:37.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2419940247' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:37.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:36 smithi086 ceph-mon[97913]: pgmap v407: 89 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1k op/s; 8740/213243 objects degraded (4.099%); 5.1 MiB/s, 81 objects/s recovering 2024-06-14T22:51:37.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:36 smithi086 ceph-mon[102754]: osdmap e124: 8 total, 8 up, 8 in 2024-06-14T22:51:37.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2419940247' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:37.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:36 smithi086 ceph-mon[102754]: pgmap v407: 89 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1k op/s; 8740/213243 objects degraded (4.099%); 5.1 MiB/s, 81 objects/s recovering 2024-06-14T22:51:37.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:36 smithi137 ceph-mon[90634]: osdmap e124: 8 total, 8 up, 8 in 2024-06-14T22:51:37.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2419940247' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:37.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:36 smithi137 ceph-mon[90634]: pgmap v407: 89 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1k op/s; 8740/213243 objects degraded (4.099%); 5.1 MiB/s, 81 objects/s recovering 2024-06-14T22:51:38.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8740/213243 objects degraded (4.099%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:38.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8740/213243 objects degraded (4.099%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:38.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8740/213243 objects degraded (4.099%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:39.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:39 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1044233754' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:39.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:39 smithi137 ceph-mon[90634]: pgmap v408: 89 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 8740/213243 objects degraded (4.099%); 3.1 MiB/s, 48 objects/s recovering 2024-06-14T22:51:39.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:39 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1044233754' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:39.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:39 smithi086 ceph-mon[97913]: pgmap v408: 89 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 8740/213243 objects degraded (4.099%); 3.1 MiB/s, 48 objects/s recovering 2024-06-14T22:51:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:39 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1044233754' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:39 smithi086 ceph-mon[102754]: pgmap v408: 89 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 8740/213243 objects degraded (4.099%); 3.1 MiB/s, 48 objects/s recovering 2024-06-14T22:51:41.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:41 smithi086 ceph-mon[97913]: pgmap v409: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 8615/205425 objects degraded (4.194%); 4.0 MiB/s, 64 objects/s recovering 2024-06-14T22:51:41.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1051821981' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:41.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:41 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:51:41.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:41 smithi086 ceph-mon[102754]: pgmap v409: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 8615/205425 objects degraded (4.194%); 4.0 MiB/s, 64 objects/s recovering 2024-06-14T22:51:41.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1051821981' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:41.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:41 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:51:41.862 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:41 smithi137 ceph-mon[90634]: pgmap v409: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 816 op/s; 8615/205425 objects degraded (4.194%); 4.0 MiB/s, 64 objects/s recovering 2024-06-14T22:51:41.862 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1051821981' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:41.862 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:41 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:51:42.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:42.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:42.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:43.618 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:43 smithi086 ceph-mon[97913]: pgmap v410: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 680 op/s; 8615/202482 objects degraded (4.255%); 4.0 MiB/s, 64 objects/s recovering 2024-06-14T22:51:43.618 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8615/202482 objects degraded (4.255%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:43.618 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1644848126' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:43.618 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:43 smithi086 ceph-mon[102754]: pgmap v410: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 680 op/s; 8615/202482 objects degraded (4.255%); 4.0 MiB/s, 64 objects/s recovering 2024-06-14T22:51:43.619 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8615/202482 objects degraded (4.255%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:43.619 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1644848126' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:43.882 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:43 smithi137 ceph-mon[90634]: pgmap v410: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 680 op/s; 8615/202482 objects degraded (4.255%); 4.0 MiB/s, 64 objects/s recovering 2024-06-14T22:51:43.883 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8615/202482 objects degraded (4.255%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:43.883 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1644848126' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:44.871 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:51:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:51:44] "GET /metrics HTTP/1.1" 200 36504 "" "Prometheus/2.43.0" 2024-06-14T22:51:45.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:45 smithi086 ceph-mon[102754]: pgmap v411: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 8615/196836 objects degraded (4.377%); 3.3 MiB/s, 52 objects/s recovering 2024-06-14T22:51:45.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:45 smithi086 ceph-mon[97913]: pgmap v411: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 8615/196836 objects degraded (4.377%); 3.3 MiB/s, 52 objects/s recovering 2024-06-14T22:51:45.881 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:45 smithi137 ceph-mon[90634]: pgmap v411: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 8615/196836 objects degraded (4.377%); 3.3 MiB/s, 52 objects/s recovering 2024-06-14T22:51:46.896 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/890049631' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:46.896 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/890049631' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:46.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/890049631' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:47.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:47 smithi137 ceph-mon[90634]: pgmap v412: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 8438/191250 objects degraded (4.412%); 2.2 MiB/s, 42 objects/s recovering 2024-06-14T22:51:47.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:47 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:47.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:47 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:48.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:47 smithi086 ceph-mon[102754]: pgmap v412: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 8438/191250 objects degraded (4.412%); 2.2 MiB/s, 42 objects/s recovering 2024-06-14T22:51:48.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:47 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:48.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:47 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:48.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:47 smithi086 ceph-mon[97913]: pgmap v412: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 8438/191250 objects degraded (4.412%); 2.2 MiB/s, 42 objects/s recovering 2024-06-14T22:51:48.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:47 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:48.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:47 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:48.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8438/191250 objects degraded (4.412%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:48.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2190529453' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:48.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8438/191250 objects degraded (4.412%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:48.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2190529453' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:48.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8438/191250 objects degraded (4.412%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:48.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2190529453' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:49.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:49 smithi137 ceph-mon[90634]: pgmap v413: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 8438/191250 objects degraded (4.412%); 1.2 MiB/s, 25 objects/s recovering 2024-06-14T22:51:50.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:49 smithi086 ceph-mon[97913]: pgmap v413: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 8438/191250 objects degraded (4.412%); 1.2 MiB/s, 25 objects/s recovering 2024-06-14T22:51:50.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:49 smithi086 ceph-mon[102754]: pgmap v413: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 8438/191250 objects degraded (4.412%); 1.2 MiB/s, 25 objects/s recovering 2024-06-14T22:51:51.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:51:51.329+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:51.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: pgmap v414: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 8265/182931 objects degraded (4.518%); 1.2 MiB/s, 39 objects/s recovering 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1503150837' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: pgmap v414: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 8265/182931 objects degraded (4.518%); 1.2 MiB/s, 39 objects/s recovering 2024-06-14T22:51:51.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1503150837' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:51:51.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: pgmap v414: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 8265/182931 objects degraded (4.518%); 1.2 MiB/s, 39 objects/s recovering 2024-06-14T22:51:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1503150837' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:51:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:51:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:51:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:51:51.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:51:51.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:51:51.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:51:51.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:52.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:52 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:52.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:52 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:52.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:52 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:52.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:52 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:52.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:52 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:51:52.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:52 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:51:53.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:53 smithi086 ceph-mon[97913]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 8265/180186 objects degraded (4.587%); 586 KiB/s, 29 objects/s recovering 2024-06-14T22:51:53.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8265/180186 objects degraded (4.587%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:53.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2178750409' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:53.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:53 smithi086 ceph-mon[102754]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 8265/180186 objects degraded (4.587%); 586 KiB/s, 29 objects/s recovering 2024-06-14T22:51:53.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8265/180186 objects degraded (4.587%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:53.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2178750409' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:53.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:53 smithi137 ceph-mon[90634]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 8265/180186 objects degraded (4.587%); 586 KiB/s, 29 objects/s recovering 2024-06-14T22:51:53.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8265/180186 objects degraded (4.587%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:53.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2178750409' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:55.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:51:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:51:54] "GET /metrics HTTP/1.1" 200 36504 "" "Prometheus/2.43.0" 2024-06-14T22:51:55.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:55 smithi086 ceph-mon[97913]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 8265/174114 objects degraded (4.747%); 586 KiB/s, 29 objects/s recovering 2024-06-14T22:51:55.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:55 smithi086 ceph-mon[102754]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 8265/174114 objects degraded (4.747%); 586 KiB/s, 29 objects/s recovering 2024-06-14T22:51:55.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:55 smithi137 ceph-mon[90634]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 8265/174114 objects degraded (4.747%); 586 KiB/s, 29 objects/s recovering 2024-06-14T22:51:56.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/111854715' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:56.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/111854715' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:56.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/111854715' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:57 smithi086 ceph-mon[97913]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 8093/168135 objects degraded (4.813%); 586 KiB/s, 43 objects/s recovering 2024-06-14T22:51:57.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:57 smithi086 ceph-mon[102754]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 8093/168135 objects degraded (4.813%); 586 KiB/s, 43 objects/s recovering 2024-06-14T22:51:57.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:57.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:57 smithi137 ceph-mon[90634]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 8093/168135 objects degraded (4.813%); 586 KiB/s, 43 objects/s recovering 2024-06-14T22:51:57.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:51:58.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8093/168135 objects degraded (4.813%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:58.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3969904154' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:58.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8093/168135 objects degraded (4.813%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:58.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3969904154' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:58.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8093/168135 objects degraded (4.813%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:51:58.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3969904154' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:51:59.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:51:59 smithi137 ceph-mon[90634]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 8093/168135 objects degraded (4.813%); 0 B/s, 28 objects/s recovering 2024-06-14T22:52:00.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:51:59 smithi086 ceph-mon[102754]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 8093/168135 objects degraded (4.813%); 0 B/s, 28 objects/s recovering 2024-06-14T22:52:00.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:51:59 smithi086 ceph-mon[97913]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 8093/168135 objects degraded (4.813%); 0 B/s, 28 objects/s recovering 2024-06-14T22:52:01.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:01 smithi137 ceph-mon[90634]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 7912/159198 objects degraded (4.970%); 0 B/s, 43 objects/s recovering 2024-06-14T22:52:01.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2423568404' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:02.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:01 smithi086 ceph-mon[102754]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 7912/159198 objects degraded (4.970%); 0 B/s, 43 objects/s recovering 2024-06-14T22:52:02.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2423568404' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:02.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:01 smithi086 ceph-mon[97913]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 7912/159198 objects degraded (4.970%); 0 B/s, 43 objects/s recovering 2024-06-14T22:52:02.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2423568404' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:03.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:03 smithi137 ceph-mon[90634]: pgmap v420: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 7912/156276 objects degraded (5.063%); 0 B/s, 29 objects/s recovering 2024-06-14T22:52:03.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 7912/156276 objects degraded (5.063%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:03.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2108044820' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:04.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:03 smithi086 ceph-mon[97913]: pgmap v420: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 7912/156276 objects degraded (5.063%); 0 B/s, 29 objects/s recovering 2024-06-14T22:52:04.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 7912/156276 objects degraded (5.063%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:04.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2108044820' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:04.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:03 smithi086 ceph-mon[102754]: pgmap v420: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 7912/156276 objects degraded (5.063%); 0 B/s, 29 objects/s recovering 2024-06-14T22:52:04.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 7912/156276 objects degraded (5.063%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:04.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2108044820' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:05.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:52:04] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.43.0" 2024-06-14T22:52:05.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:05 smithi137 ceph-mon[90634]: pgmap v421: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 7912/150480 objects degraded (5.258%); 0 B/s, 29 objects/s recovering 2024-06-14T22:52:06.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:05 smithi086 ceph-mon[97913]: pgmap v421: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 7912/150480 objects degraded (5.258%); 0 B/s, 29 objects/s recovering 2024-06-14T22:52:06.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:05 smithi086 ceph-mon[102754]: pgmap v421: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 7912/150480 objects degraded (5.258%); 0 B/s, 29 objects/s recovering 2024-06-14T22:52:06.606 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:52:06.346+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:06.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2424426022' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:06.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:07.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2424426022' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:07.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:07.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2424426022' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:07.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:07.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:07 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:07.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:07 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:07.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:07 smithi137 ceph-mon[90634]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 7716/144123 objects degraded (5.354%); 0 B/s, 45 objects/s recovering 2024-06-14T22:52:08.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:07 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:08.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:07 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:08.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:07 smithi086 ceph-mon[102754]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 7716/144123 objects degraded (5.354%); 0 B/s, 45 objects/s recovering 2024-06-14T22:52:08.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:07 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:08.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:07 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:08.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:07 smithi086 ceph-mon[97913]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 7716/144123 objects degraded (5.354%); 0 B/s, 45 objects/s recovering 2024-06-14T22:52:08.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 7716/144123 objects degraded (5.354%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:08.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1856267090' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:09.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 7716/144123 objects degraded (5.354%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:09.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1856267090' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:09.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 7716/144123 objects degraded (5.354%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:09.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1856267090' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:09.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:09 smithi086 ceph-mon[97913]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 7716/144123 objects degraded (5.354%); 0 B/s, 31 objects/s recovering 2024-06-14T22:52:09.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:09 smithi086 ceph-mon[102754]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 7716/144123 objects degraded (5.354%); 0 B/s, 31 objects/s recovering 2024-06-14T22:52:09.893 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:09 smithi137 ceph-mon[90634]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 7716/144123 objects degraded (5.354%); 0 B/s, 31 objects/s recovering 2024-06-14T22:52:11.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:11 smithi137 ceph-mon[90634]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 7501/134955 objects degraded (5.558%); 0 B/s, 49 objects/s recovering 2024-06-14T22:52:11.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3949712901' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:12.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:11 smithi086 ceph-mon[102754]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 7501/134955 objects degraded (5.558%); 0 B/s, 49 objects/s recovering 2024-06-14T22:52:12.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3949712901' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:12.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:11 smithi086 ceph-mon[97913]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 7501/134955 objects degraded (5.558%); 0 B/s, 49 objects/s recovering 2024-06-14T22:52:12.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3949712901' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:12.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:13.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:13.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:13.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:13 smithi137 ceph-mon[90634]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 7501/131925 objects degraded (5.686%); 0 B/s, 34 objects/s recovering 2024-06-14T22:52:13.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 7501/131925 objects degraded (5.686%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:13.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/880983605' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:14.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:13 smithi086 ceph-mon[97913]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 7501/131925 objects degraded (5.686%); 0 B/s, 34 objects/s recovering 2024-06-14T22:52:14.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 7501/131925 objects degraded (5.686%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:14.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/880983605' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:14.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:13 smithi086 ceph-mon[102754]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 757 op/s; 7501/131925 objects degraded (5.686%); 0 B/s, 34 objects/s recovering 2024-06-14T22:52:14.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 7501/131925 objects degraded (5.686%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:14.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/880983605' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:15.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:52:14] "GET /metrics HTTP/1.1" 200 36488 "" "Prometheus/2.43.0" 2024-06-14T22:52:15.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:15 smithi137 ceph-mon[90634]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 7501/126000 objects degraded (5.953%); 0 B/s, 34 objects/s recovering 2024-06-14T22:52:16.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:15 smithi086 ceph-mon[97913]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 7501/126000 objects degraded (5.953%); 0 B/s, 34 objects/s recovering 2024-06-14T22:52:16.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:15 smithi086 ceph-mon[102754]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 7501/126000 objects degraded (5.953%); 0 B/s, 34 objects/s recovering 2024-06-14T22:52:16.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:16 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3184932479' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:17.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3184932479' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:17.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:16 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3184932479' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:17.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:17 smithi137 ceph-mon[90634]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 7262/119691 objects degraded (6.067%); 0 B/s, 54 objects/s recovering 2024-06-14T22:52:18.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:17 smithi086 ceph-mon[97913]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 7262/119691 objects degraded (6.067%); 0 B/s, 54 objects/s recovering 2024-06-14T22:52:18.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:17 smithi086 ceph-mon[102754]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 7262/119691 objects degraded (6.067%); 0 B/s, 54 objects/s recovering 2024-06-14T22:52:18.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 7262/119691 objects degraded (6.067%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:18.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1800039353' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:19.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 7262/119691 objects degraded (6.067%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:19.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1800039353' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:19.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 7262/119691 objects degraded (6.067%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:19.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1800039353' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:19.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:19 smithi137 ceph-mon[90634]: pgmap v428: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 678 op/s; 7262/119691 objects degraded (6.067%); 0 B/s, 37 objects/s recovering 2024-06-14T22:52:20.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:19 smithi086 ceph-mon[97913]: pgmap v428: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 678 op/s; 7262/119691 objects degraded (6.067%); 0 B/s, 37 objects/s recovering 2024-06-14T22:52:20.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:19 smithi086 ceph-mon[102754]: pgmap v428: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 678 op/s; 7262/119691 objects degraded (6.067%); 0 B/s, 37 objects/s recovering 2024-06-14T22:52:21.686 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:52:21.367+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:21.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:21 smithi137 ceph-mon[90634]: pgmap v429: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 7030/110256 objects degraded (6.376%); 0 B/s, 57 objects/s recovering 2024-06-14T22:52:21.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1687998594' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:21.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:22.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:21 smithi086 ceph-mon[97913]: pgmap v429: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 7030/110256 objects degraded (6.376%); 0 B/s, 57 objects/s recovering 2024-06-14T22:52:22.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1687998594' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:22.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:22.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:21 smithi086 ceph-mon[102754]: pgmap v429: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 7030/110256 objects degraded (6.376%); 0 B/s, 57 objects/s recovering 2024-06-14T22:52:22.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1687998594' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:22.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:22.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:22 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:22.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:22 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:23.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:22 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:23.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:22 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:23.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:22 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:23.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:22 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:23.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:23 smithi137 ceph-mon[90634]: pgmap v430: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 771 op/s; 7030/107166 objects degraded (6.560%); 0 B/s, 39 objects/s recovering 2024-06-14T22:52:23.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 7030/107166 objects degraded (6.560%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:23.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1820974270' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:24.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:23 smithi086 ceph-mon[102754]: pgmap v430: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 771 op/s; 7030/107166 objects degraded (6.560%); 0 B/s, 39 objects/s recovering 2024-06-14T22:52:24.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 7030/107166 objects degraded (6.560%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:24.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1820974270' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:24.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:23 smithi086 ceph-mon[97913]: pgmap v430: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 771 op/s; 7030/107166 objects degraded (6.560%); 0 B/s, 39 objects/s recovering 2024-06-14T22:52:24.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 7030/107166 objects degraded (6.560%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:24.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1820974270' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:25.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:52:24] "GET /metrics HTTP/1.1" 200 36488 "" "Prometheus/2.43.0" 2024-06-14T22:52:25.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:25 smithi137 ceph-mon[90634]: pgmap v431: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 7030/100890 objects degraded (6.968%); 0 B/s, 39 objects/s recovering 2024-06-14T22:52:26.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:25 smithi086 ceph-mon[97913]: pgmap v431: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 7030/100890 objects degraded (6.968%); 0 B/s, 39 objects/s recovering 2024-06-14T22:52:26.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:25 smithi086 ceph-mon[102754]: pgmap v431: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 861 op/s; 7030/100890 objects degraded (6.968%); 0 B/s, 39 objects/s recovering 2024-06-14T22:52:26.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/500494596' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:27.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/500494596' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:27.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/500494596' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:27.984 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:27 smithi086 ceph-mon[102754]: pgmap v432: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 6769/94740 objects degraded (7.145%); 0 B/s, 60 objects/s recovering 2024-06-14T22:52:27.984 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:27.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:27 smithi086 ceph-mon[97913]: pgmap v432: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 6769/94740 objects degraded (7.145%); 0 B/s, 60 objects/s recovering 2024-06-14T22:52:27.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:27 smithi137 ceph-mon[90634]: pgmap v432: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 6769/94740 objects degraded (7.145%); 0 B/s, 60 objects/s recovering 2024-06-14T22:52:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:28.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6769/94740 objects degraded (7.145%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:28.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3759607907' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:29.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6769/94740 objects degraded (7.145%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:29.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3759607907' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:29.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6769/94740 objects degraded (7.145%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:29.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3759607907' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:29.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:29 smithi137 ceph-mon[90634]: pgmap v433: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 6769/94740 objects degraded (7.145%); 0 B/s, 41 objects/s recovering 2024-06-14T22:52:30.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:29 smithi086 ceph-mon[97913]: pgmap v433: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 6769/94740 objects degraded (7.145%); 0 B/s, 41 objects/s recovering 2024-06-14T22:52:30.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:29 smithi086 ceph-mon[102754]: pgmap v433: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 6769/94740 objects degraded (7.145%); 0 B/s, 41 objects/s recovering 2024-06-14T22:52:31.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:30 smithi086 ceph-mon[97913]: osdmap e125: 8 total, 8 up, 8 in 2024-06-14T22:52:31.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:30 smithi086 ceph-mon[102754]: osdmap e125: 8 total, 8 up, 8 in 2024-06-14T22:52:31.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:30 smithi137 ceph-mon[90634]: osdmap e125: 8 total, 8 up, 8 in 2024-06-14T22:52:32.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:31 smithi086 ceph-mon[97913]: pgmap v435: 89 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 821 op/s; 6547/85602 objects degraded (7.648%); 0 B/s, 48 objects/s recovering 2024-06-14T22:52:32.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:31 smithi086 ceph-mon[97913]: osdmap e126: 8 total, 8 up, 8 in 2024-06-14T22:52:32.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1885041464' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:32.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:31 smithi086 ceph-mon[102754]: pgmap v435: 89 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 821 op/s; 6547/85602 objects degraded (7.648%); 0 B/s, 48 objects/s recovering 2024-06-14T22:52:32.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:31 smithi086 ceph-mon[102754]: osdmap e126: 8 total, 8 up, 8 in 2024-06-14T22:52:32.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1885041464' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:32.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:31 smithi137 ceph-mon[90634]: pgmap v435: 89 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 821 op/s; 6547/85602 objects degraded (7.648%); 0 B/s, 48 objects/s recovering 2024-06-14T22:52:32.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:31 smithi137 ceph-mon[90634]: osdmap e126: 8 total, 8 up, 8 in 2024-06-14T22:52:32.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1885041464' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:33.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:32 smithi086 ceph-mon[97913]: pgmap v437: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 6349/82914 objects degraded (7.657%); 0 B/s, 85 objects/s recovering 2024-06-14T22:52:33.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:32 smithi086 ceph-mon[102754]: pgmap v437: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 6349/82914 objects degraded (7.657%); 0 B/s, 85 objects/s recovering 2024-06-14T22:52:33.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:32 smithi137 ceph-mon[90634]: pgmap v437: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 6349/82914 objects degraded (7.657%); 0 B/s, 85 objects/s recovering 2024-06-14T22:52:34.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6349/82914 objects degraded (7.657%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:34.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3196051870' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:34.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6349/82914 objects degraded (7.657%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:34.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3196051870' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:34.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6349/82914 objects degraded (7.657%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:34.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3196051870' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:35.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:34 smithi086 ceph-mon[97913]: pgmap v438: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.6 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 6349/77547 objects degraded (8.187%); 0 B/s, 52 objects/s recovering 2024-06-14T22:52:35.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:34 smithi086 ceph-mon[102754]: pgmap v438: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.6 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 6349/77547 objects degraded (8.187%); 0 B/s, 52 objects/s recovering 2024-06-14T22:52:35.084 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:52:34] "GET /metrics HTTP/1.1" 200 36493 "" "Prometheus/2.43.0" 2024-06-14T22:52:35.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:34 smithi137 ceph-mon[90634]: pgmap v438: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.6 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 6349/77547 objects degraded (8.187%); 0 B/s, 52 objects/s recovering 2024-06-14T22:52:36.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1982199219' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:36.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1982199219' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:36.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1982199219' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:36.833 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:36 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:52:36.383+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:37.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:37 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:37.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:37 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:37.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:37 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:37.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:37 smithi086 ceph-mon[97913]: pgmap v439: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 6170/71880 objects degraded (8.584%); 0 B/s, 74 objects/s recovering 2024-06-14T22:52:37.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:37 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:37.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:37 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:37.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:37 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:37.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:37 smithi086 ceph-mon[102754]: pgmap v439: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 6170/71880 objects degraded (8.584%); 0 B/s, 74 objects/s recovering 2024-06-14T22:52:37.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:37 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:37.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:37 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:37.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:37 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:37.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:37 smithi137 ceph-mon[90634]: pgmap v439: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 6170/71880 objects degraded (8.584%); 0 B/s, 74 objects/s recovering 2024-06-14T22:52:38.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6170/71880 objects degraded (8.584%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:38.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6170/71880 objects degraded (8.584%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:38.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6170/71880 objects degraded (8.584%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:39.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:39 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/376845684' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:39.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:39 smithi137 ceph-mon[90634]: pgmap v440: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 6170/71880 objects degraded (8.584%); 0 B/s, 68 objects/s recovering 2024-06-14T22:52:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:39 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/376845684' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:39.583 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:39 smithi086 ceph-mon[102754]: pgmap v440: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 6170/71880 objects degraded (8.584%); 0 B/s, 68 objects/s recovering 2024-06-14T22:52:39.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:39 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/376845684' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:39.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:39 smithi086 ceph-mon[97913]: pgmap v440: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 6170/71880 objects degraded (8.584%); 0 B/s, 68 objects/s recovering 2024-06-14T22:52:41.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:41 smithi086 ceph-mon[102754]: pgmap v441: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 764 op/s; 6170/62676 objects degraded (9.844%); 0 B/s, 37 objects/s recovering 2024-06-14T22:52:41.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1064210380' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:41.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:41 smithi086 ceph-mon[97913]: pgmap v441: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 764 op/s; 6170/62676 objects degraded (9.844%); 0 B/s, 37 objects/s recovering 2024-06-14T22:52:41.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1064210380' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:41.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:41 smithi137 ceph-mon[90634]: pgmap v441: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 764 op/s; 6170/62676 objects degraded (9.844%); 0 B/s, 37 objects/s recovering 2024-06-14T22:52:41.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1064210380' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:42.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:43.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:43.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:43.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:43 smithi137 ceph-mon[90634]: pgmap v442: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 738 op/s; 5973/59541 objects degraded (10.032%); 0 B/s, 48 objects/s recovering 2024-06-14T22:52:43.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5973/59541 objects degraded (10.032%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:43.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/401458034' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:44.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:43 smithi086 ceph-mon[102754]: pgmap v442: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 738 op/s; 5973/59541 objects degraded (10.032%); 0 B/s, 48 objects/s recovering 2024-06-14T22:52:44.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5973/59541 objects degraded (10.032%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:44.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/401458034' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:44.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:43 smithi086 ceph-mon[97913]: pgmap v442: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 738 op/s; 5973/59541 objects degraded (10.032%); 0 B/s, 48 objects/s recovering 2024-06-14T22:52:44.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5973/59541 objects degraded (10.032%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:44.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/401458034' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:45.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:52:44] "GET /metrics HTTP/1.1" 200 36487 "" "Prometheus/2.43.0" 2024-06-14T22:52:45.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:45 smithi137 ceph-mon[90634]: pgmap v443: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.1 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 5973/53241 objects degraded (11.219%); 0 B/s, 31 objects/s recovering 2024-06-14T22:52:46.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:45 smithi086 ceph-mon[102754]: pgmap v443: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.1 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 5973/53241 objects degraded (11.219%); 0 B/s, 31 objects/s recovering 2024-06-14T22:52:46.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:45 smithi086 ceph-mon[97913]: pgmap v443: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.1 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 5973/53241 objects degraded (11.219%); 0 B/s, 31 objects/s recovering 2024-06-14T22:52:46.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1383679587' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:47.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1383679587' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:47.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1383679587' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:47.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:47 smithi137 ceph-mon[90634]: pgmap v444: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 979 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 5774/47031 objects degraded (12.277%); 0 B/s, 47 objects/s recovering 2024-06-14T22:52:48.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:47 smithi086 ceph-mon[102754]: pgmap v444: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 979 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 5774/47031 objects degraded (12.277%); 0 B/s, 47 objects/s recovering 2024-06-14T22:52:48.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:47 smithi086 ceph-mon[97913]: pgmap v444: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 979 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 5774/47031 objects degraded (12.277%); 0 B/s, 47 objects/s recovering 2024-06-14T22:52:48.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5774/47031 objects degraded (12.277%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:48.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/24917161' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:49.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5774/47031 objects degraded (12.277%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:49.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/24917161' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:49.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5774/47031 objects degraded (12.277%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:49.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/24917161' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:49.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:49 smithi137 ceph-mon[90634]: pgmap v445: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 979 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 689 op/s; 5774/47031 objects degraded (12.277%); 0 B/s, 32 objects/s recovering 2024-06-14T22:52:50.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:49 smithi086 ceph-mon[97913]: pgmap v445: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 979 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 689 op/s; 5774/47031 objects degraded (12.277%); 0 B/s, 32 objects/s recovering 2024-06-14T22:52:50.095 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:49 smithi086 ceph-mon[102754]: pgmap v445: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 979 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 689 op/s; 5774/47031 objects degraded (12.277%); 0 B/s, 32 objects/s recovering 2024-06-14T22:52:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:51 smithi137 ceph-mon[90634]: pgmap v446: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 782 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 5774/37566 objects degraded (15.370%); 0 B/s, 32 objects/s recovering 2024-06-14T22:52:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3306931830' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:52:52.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:51 smithi086 ceph-mon[97913]: pgmap v446: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 782 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 5774/37566 objects degraded (15.370%); 0 B/s, 32 objects/s recovering 2024-06-14T22:52:52.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3306931830' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:52.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:52:52.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:51 smithi086 ceph-mon[102754]: pgmap v446: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 782 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 5774/37566 objects degraded (15.370%); 0 B/s, 32 objects/s recovering 2024-06-14T22:52:52.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3306931830' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:52.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:52:53.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:53 smithi137 ceph-mon[90634]: pgmap v447: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 717 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 5571/34476 objects degraded (16.159%); 0 B/s, 49 objects/s recovering 2024-06-14T22:52:53.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5571/34476 objects degraded (16.159%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:53.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/519039541' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:54.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:53 smithi086 ceph-mon[97913]: pgmap v447: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 717 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 5571/34476 objects degraded (16.159%); 0 B/s, 49 objects/s recovering 2024-06-14T22:52:54.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5571/34476 objects degraded (16.159%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:54.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/519039541' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:54.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:53 smithi086 ceph-mon[102754]: pgmap v447: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 717 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 5571/34476 objects degraded (16.159%); 0 B/s, 49 objects/s recovering 2024-06-14T22:52:54.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5571/34476 objects degraded (16.159%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:54.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/519039541' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:54.890 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:52:54] "GET /metrics HTTP/1.1" 200 36487 "" "Prometheus/2.43.0" 2024-06-14T22:52:55.793 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:55 smithi086 ceph-mon[102754]: pgmap v448: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 595 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 858 op/s; 5571/28617 objects degraded (19.467%); 0 B/s, 33 objects/s recovering 2024-06-14T22:52:55.793 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:55 smithi086 ceph-mon[97913]: pgmap v448: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 595 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 858 op/s; 5571/28617 objects degraded (19.467%); 0 B/s, 33 objects/s recovering 2024-06-14T22:52:55.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:55 smithi137 ceph-mon[90634]: pgmap v448: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 595 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 858 op/s; 5571/28617 objects degraded (19.467%); 0 B/s, 33 objects/s recovering 2024-06-14T22:52:56.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/817632938' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:56.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:56.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/817632938' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:57.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/817632938' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:57.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:57 smithi086 ceph-mon[97913]: pgmap v449: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 477 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 5366/22959 objects degraded (23.372%); 0 B/s, 50 objects/s recovering 2024-06-14T22:52:57.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:57.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:57 smithi086 ceph-mon[102754]: pgmap v449: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 477 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 5366/22959 objects degraded (23.372%); 0 B/s, 50 objects/s recovering 2024-06-14T22:52:57.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:57.986 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.986 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:57 smithi137 ceph-mon[90634]: pgmap v449: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 477 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 840 op/s; 5366/22959 objects degraded (23.372%); 0 B/s, 50 objects/s recovering 2024-06-14T22:52:57.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:52:57.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:57.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:58.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:52:58.140+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:58.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5366/22959 objects degraded (23.372%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3704352511' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:58.956 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:52:58.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:52:58.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:58.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:52:58.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:52:58.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:52:58.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5366/22959 objects degraded (23.372%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:58.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:52:58.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:58.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3704352511' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:58.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:52:58.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:52:58.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:52:58.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:52:58.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:52:58.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:52:58.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5366/22959 objects degraded (23.372%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:52:58.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:52:58.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:58.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3704352511' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:52:59.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:59 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:52:59.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:59 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:52:59.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:52:59 smithi137 ceph-mon[90634]: pgmap v450: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 477 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 5366/22959 objects degraded (23.372%); 0 B/s, 33 objects/s recovering 2024-06-14T22:53:00.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:59 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:53:00.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:59 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:53:00.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:52:59 smithi086 ceph-mon[97913]: pgmap v450: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 477 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 5366/22959 objects degraded (23.372%); 0 B/s, 33 objects/s recovering 2024-06-14T22:53:00.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:59 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:53:00.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:59 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-06-14T22:53:00.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:52:59 smithi086 ceph-mon[102754]: pgmap v450: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 477 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 5366/22959 objects degraded (23.372%); 0 B/s, 33 objects/s recovering 2024-06-14T22:53:01.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:01 smithi137 ceph-mon[90634]: pgmap v451: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 296 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 5366/14244 objects degraded (37.672%); 0 B/s, 33 objects/s recovering 2024-06-14T22:53:01.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/731935606' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:02.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:01 smithi086 ceph-mon[102754]: pgmap v451: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 296 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 5366/14244 objects degraded (37.672%); 0 B/s, 33 objects/s recovering 2024-06-14T22:53:02.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/731935606' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:02.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:01 smithi086 ceph-mon[97913]: pgmap v451: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 296 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 5366/14244 objects degraded (37.672%); 0 B/s, 33 objects/s recovering 2024-06-14T22:53:02.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/731935606' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:03.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:03 smithi137 ceph-mon[90634]: pgmap v452: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 236 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 5173/11394 objects degraded (45.401%); 0 B/s, 50 objects/s recovering 2024-06-14T22:53:03.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5173/11394 objects degraded (45.401%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:53:03.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1478106720' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:04.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:03 smithi086 ceph-mon[102754]: pgmap v452: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 236 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 5173/11394 objects degraded (45.401%); 0 B/s, 50 objects/s recovering 2024-06-14T22:53:04.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5173/11394 objects degraded (45.401%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:53:04.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1478106720' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:04.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:03 smithi086 ceph-mon[97913]: pgmap v452: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 236 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 5173/11394 objects degraded (45.401%); 0 B/s, 50 objects/s recovering 2024-06-14T22:53:04.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5173/11394 objects degraded (45.401%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:53:04.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1478106720' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:04.342 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 120586 objects 2024-06-14T22:53:04.342 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :151.534 2024-06-14T22:53:05.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:53:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:53:04] "GET /metrics HTTP/1.1" 200 36487 "" "Prometheus/2.43.0" 2024-06-14T22:53:05.794 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-06-14T22:53:05.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-06-14T22:53:05.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:05 smithi137 ceph-mon[90634]: pgmap v453: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 108 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 5173/5238 objects degraded (98.759%); 0 B/s, 33 objects/s recovering 2024-06-14T22:53:06.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:05 smithi086 ceph-mon[97913]: pgmap v453: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 108 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 5173/5238 objects degraded (98.759%); 0 B/s, 33 objects/s recovering 2024-06-14T22:53:06.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:05 smithi086 ceph-mon[102754]: pgmap v453: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 108 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 5173/5238 objects degraded (98.759%); 0 B/s, 33 objects/s recovering 2024-06-14T22:53:06.107 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:53:06.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3930968305' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:07.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3930968305' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:07.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3930968305' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:07.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:07 smithi086 ceph-mon[97913]: pgmap v454: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 13 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 4442/675 objects degraded (658.074%); 0 B/s, 94 objects/s recovering 2024-06-14T22:53:07.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:07 smithi086 ceph-mon[97913]: osdmap e127: 8 total, 8 up, 8 in 2024-06-14T22:53:07.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:07 smithi086 ceph-mon[102754]: pgmap v454: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 13 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 4442/675 objects degraded (658.074%); 0 B/s, 94 objects/s recovering 2024-06-14T22:53:07.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:07 smithi086 ceph-mon[102754]: osdmap e127: 8 total, 8 up, 8 in 2024-06-14T22:53:07.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:07 smithi137 ceph-mon[90634]: pgmap v454: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 13 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 4442/675 objects degraded (658.074%); 0 B/s, 94 objects/s recovering 2024-06-14T22:53:07.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:07 smithi137 ceph-mon[90634]: osdmap e127: 8 total, 8 up, 8 in 2024-06-14T22:53:08.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:08 smithi137 ceph-mon[90634]: osdmap e128: 8 total, 8 up, 8 in 2024-06-14T22:53:08.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 4442/675 objects degraded (658.074%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:53:08.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2090185723' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:09.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:08 smithi086 ceph-mon[97913]: osdmap e128: 8 total, 8 up, 8 in 2024-06-14T22:53:09.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 4442/675 objects degraded (658.074%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:53:09.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2090185723' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:09.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:08 smithi086 ceph-mon[102754]: osdmap e128: 8 total, 8 up, 8 in 2024-06-14T22:53:09.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 4442/675 objects degraded (658.074%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T22:53:09.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2090185723' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:09.689 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_3' removed 2024-06-14T22:53:09.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:09 smithi137 ceph-mon[90634]: pgmap v457: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 13 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 4442/675 objects degraded (658.074%); 0 B/s, 115 objects/s recovering 2024-06-14T22:53:09.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4039899861' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:53:10.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:09 smithi086 ceph-mon[97913]: pgmap v457: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 13 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 4442/675 objects degraded (658.074%); 0 B/s, 115 objects/s recovering 2024-06-14T22:53:10.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4039899861' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:53:10.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:09 smithi086 ceph-mon[102754]: pgmap v457: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 13 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 4442/675 objects degraded (658.074%); 0 B/s, 115 objects/s recovering 2024-06-14T22:53:10.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4039899861' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T22:53:10.355 INFO:teuthology.task.sequential:In sequential, running task print... 2024-06-14T22:53:10.355 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-06-14T22:53:10.356 DEBUG:teuthology.parallel:result is None 2024-06-14T22:53:10.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4039899861' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:53:10.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:10 smithi137 ceph-mon[90634]: osdmap e129: 8 total, 8 up, 8 in 2024-06-14T22:53:10.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/993238545' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:11.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4039899861' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:53:11.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:10 smithi086 ceph-mon[97913]: osdmap e129: 8 total, 8 up, 8 in 2024-06-14T22:53:11.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/993238545' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:11.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4039899861' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T22:53:11.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:10 smithi086 ceph-mon[102754]: osdmap e129: 8 total, 8 up, 8 in 2024-06-14T22:53:11.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/993238545' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:11.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:11 smithi137 ceph-mon[90634]: pgmap v459: 73 pgs: 73 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:11.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:11 smithi137 ceph-mon[90634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4442/675 objects degraded (658.074%), 3 pgs degraded, 3 pgs undersized) 2024-06-14T22:53:11.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:11 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:53:12.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:11 smithi086 ceph-mon[97913]: pgmap v459: 73 pgs: 73 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:12.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:11 smithi086 ceph-mon[97913]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4442/675 objects degraded (658.074%), 3 pgs degraded, 3 pgs undersized) 2024-06-14T22:53:12.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:11 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:53:12.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:11 smithi086 ceph-mon[102754]: pgmap v459: 73 pgs: 73 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:12.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:11 smithi086 ceph-mon[102754]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4442/675 objects degraded (658.074%), 3 pgs degraded, 3 pgs undersized) 2024-06-14T22:53:12.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:11 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:53:13.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:13 smithi137 ceph-mon[90634]: pgmap v460: 73 pgs: 73 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/389861860' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:53:13.828 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:13.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:13.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[102754]: pgmap v460: 73 pgs: 73 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:13.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/389861860' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:13.829 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:53:13.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:13.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:13.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[97913]: pgmap v460: 73 pgs: 73 active+clean; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:13.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/389861860' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:13.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:53:14.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:14 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:53:14.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:14 smithi137 ceph-mon[90634]: Upgrade: osd.2 is safe to restart 2024-06-14T22:53:14.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:14 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:53:14.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:14 smithi086 ceph-mon[97913]: Upgrade: osd.2 is safe to restart 2024-06-14T22:53:14.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:14 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-14T22:53:14.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:14 smithi086 ceph-mon[102754]: Upgrade: osd.2 is safe to restart 2024-06-14T22:53:15.083 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:53:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:53:14] "GET /metrics HTTP/1.1" 200 34128 "" "Prometheus/2.43.0" 2024-06-14T22:53:15.479 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[97913]: pgmap v461: 73 pgs: 73 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:15.479 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:15.479 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-14T22:53:15.479 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:53:15.479 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3294095203' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:15.479 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[102754]: pgmap v461: 73 pgs: 73 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:15.479 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:15.479 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-14T22:53:15.479 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:53:15.480 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3294095203' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:15.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:15 smithi137 ceph-mon[90634]: pgmap v461: 73 pgs: 73 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:15.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:15.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-14T22:53:15.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:53:15.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3294095203' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:16.565 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:16 smithi086 ceph-mon[97913]: Upgrade: Updating osd.2 2024-06-14T22:53:16.566 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:16 smithi086 ceph-mon[97913]: Deploying daemon osd.2 on smithi086 2024-06-14T22:53:16.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:16 smithi086 ceph-mon[102754]: Upgrade: Updating osd.2 2024-06-14T22:53:16.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:16 smithi086 ceph-mon[102754]: Deploying daemon osd.2 on smithi086 2024-06-14T22:53:16.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:16 smithi137 ceph-mon[90634]: Upgrade: Updating osd.2 2024-06-14T22:53:16.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:16 smithi137 ceph-mon[90634]: Deploying daemon osd.2 on smithi086 2024-06-14T22:53:17.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:17 smithi086 ceph-mon[97913]: pgmap v462: 73 pgs: 73 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:17.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3512146854' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:17.532 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:17 smithi086 ceph-mon[102754]: pgmap v462: 73 pgs: 73 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:17.532 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3512146854' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:17.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:17 smithi137 ceph-mon[90634]: pgmap v462: 73 pgs: 73 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:17.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3512146854' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:18.156 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:17 smithi086 systemd[1]: Stopping Ceph osd.2 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:53:18.499 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:18 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[55497]: 2024-06-14T22:53:18.156+0000 7f5996b75640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T22:53:18.499 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:18 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[55497]: 2024-06-14T22:53:18.156+0000 7f5996b75640 -1 osd.2 129 *** Got signal Terminated *** 2024-06-14T22:53:18.499 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:18 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[55497]: 2024-06-14T22:53:18.156+0000 7f5996b75640 -1 osd.2 129 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T22:53:18.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:18 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:18.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:18 smithi086 ceph-mon[97913]: osd.2 marked itself down and dead 2024-06-14T22:53:18.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:18 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:18.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:18 smithi086 ceph-mon[102754]: osd.2 marked itself down and dead 2024-06-14T22:53:18.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:18 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:18.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:18 smithi137 ceph-mon[90634]: osd.2 marked itself down and dead 2024-06-14T22:53:19.329 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:19 smithi086 podman[131850]: 2024-06-14 22:53:19.045894081 +0000 UTC m=+1.001602786 container died fbf5ce68d6eb3d54932031feaeb319f4f8fd02f03f7842afb41886a8d3ba2cab (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-06-14T22:53:19.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:19 smithi086 ceph-mon[97913]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:53:19.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:19 smithi086 ceph-mon[97913]: osdmap e130: 8 total, 7 up, 8 in 2024-06-14T22:53:19.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:19 smithi086 ceph-mon[97913]: pgmap v464: 73 pgs: 2 stale+active+clean, 71 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:19.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:19 smithi086 ceph-mon[102754]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:53:19.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:19 smithi086 ceph-mon[102754]: osdmap e130: 8 total, 7 up, 8 in 2024-06-14T22:53:19.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:19 smithi086 ceph-mon[102754]: pgmap v464: 73 pgs: 2 stale+active+clean, 71 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:19.584 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:19 smithi086 podman[131850]: 2024-06-14 22:53:19.428589889 +0000 UTC m=+1.384298610 container cleanup fbf5ce68d6eb3d54932031feaeb319f4f8fd02f03f7842afb41886a8d3ba2cab (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-14T22:53:19.584 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:19 smithi086 bash[131850]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2 2024-06-14T22:53:19.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:19 smithi137 ceph-mon[90634]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:53:19.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:19 smithi137 ceph-mon[90634]: osdmap e130: 8 total, 7 up, 8 in 2024-06-14T22:53:19.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:19 smithi137 ceph-mon[90634]: pgmap v464: 73 pgs: 2 stale+active+clean, 71 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:20.084 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:19 smithi086 podman[131870]: 2024-06-14 22:53:19.746814119 +0000 UTC m=+0.700130923 container remove fbf5ce68d6eb3d54932031feaeb319f4f8fd02f03f7842afb41886a8d3ba2cab (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-06-14T22:53:20.358 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:20 smithi086 podman[131987]: 2024-06-14 22:53:20.007230625 +0000 UTC m=+0.017046805 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:53:20.358 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:20 smithi086 podman[131987]: 2024-06-14 22:53:20.106076653 +0000 UTC m=+0.115892875 container create 493e2feb350b57e4539c5d319da79986edad985e013c79ad7a97c64b76ea3bc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-14T22:53:20.358 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:20 smithi086 podman[131987]: 2024-06-14 22:53:20.359357751 +0000 UTC m=+0.369173940 container init 493e2feb350b57e4539c5d319da79986edad985e013c79ad7a97c64b76ea3bc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:53:20.670 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:20 smithi086 ceph-mon[97913]: osdmap e131: 8 total, 7 up, 8 in 2024-06-14T22:53:20.670 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1685613023' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:20.670 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:20 smithi086 ceph-mon[102754]: osdmap e131: 8 total, 7 up, 8 in 2024-06-14T22:53:20.670 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1685613023' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:20.670 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:20 smithi086 podman[131987]: 2024-06-14 22:53:20.365773636 +0000 UTC m=+0.375589816 container start 493e2feb350b57e4539c5d319da79986edad985e013c79ad7a97c64b76ea3bc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:53:20.671 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:20 smithi086 podman[131987]: 2024-06-14 22:53:20.450448369 +0000 UTC m=+0.460264555 container attach 493e2feb350b57e4539c5d319da79986edad985e013c79ad7a97c64b76ea3bc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:53:20.961 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:20 smithi086 podman[131987]: 2024-06-14 22:53:20.671520747 +0000 UTC m=+0.681336928 container died 493e2feb350b57e4539c5d319da79986edad985e013c79ad7a97c64b76ea3bc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:53:20.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:20 smithi137 ceph-mon[90634]: osdmap e131: 8 total, 7 up, 8 in 2024-06-14T22:53:20.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1685613023' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:21.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:21 smithi086 podman[132031]: 2024-06-14 22:53:21.245523392 +0000 UTC m=+0.568337909 container remove 493e2feb350b57e4539c5d319da79986edad985e013c79ad7a97c64b76ea3bc9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-14T22:53:21.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:21 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.2.service: Deactivated successfully. 2024-06-14T22:53:21.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:21 smithi086 systemd[1]: Stopped Ceph osd.2 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:53:21.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:21 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.2.service: Consumed 26min 19.381s CPU time. 2024-06-14T22:53:21.517 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:21 smithi086 systemd[1]: Starting Ceph osd.2 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:53:21.780 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:21 smithi086 ceph-mon[97913]: pgmap v466: 73 pgs: 4 active+undersized, 12 peering, 57 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:21.781 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:21 smithi086 ceph-mon[102754]: pgmap v466: 73 pgs: 4 active+undersized, 12 peering, 57 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:21.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:21 smithi086 podman[132134]: 2024-06-14 22:53:21.586412787 +0000 UTC m=+0.017203858 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:53:21.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:21 smithi086 podman[132134]: 2024-06-14 22:53:21.696500424 +0000 UTC m=+0.127291496 container create 67cd40ca658690b147d301513fb494587ee55f64a8c2e2510b1230229c24794e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T22:53:21.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:21 smithi137 ceph-mon[90634]: pgmap v466: 73 pgs: 4 active+undersized, 12 peering, 57 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:22.031 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:21 smithi086 podman[132134]: 2024-06-14 22:53:21.941255482 +0000 UTC m=+0.372046568 container init 67cd40ca658690b147d301513fb494587ee55f64a8c2e2510b1230229c24794e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-14T22:53:22.031 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:21 smithi086 podman[132134]: 2024-06-14 22:53:21.947400202 +0000 UTC m=+0.378191266 container start 67cd40ca658690b147d301513fb494587ee55f64a8c2e2510b1230229c24794e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240528) 2024-06-14T22:53:22.303 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 podman[132134]: 2024-06-14 22:53:22.032507295 +0000 UTC m=+0.463298361 container attach 67cd40ca658690b147d301513fb494587ee55f64a8c2e2510b1230229c24794e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3) 2024-06-14T22:53:22.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-14T22:53:22.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2280735411' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:22.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-14T22:53:22.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2280735411' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:22.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[132170]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-14T22:53:22.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 bash[132134]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-14T22:53:22.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[132170]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-14T22:53:22.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 bash[132134]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-14T22:53:22.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[132170]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-14T22:53:22.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 bash[132134]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-14T22:53:22.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[132170]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-14T22:53:22.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 bash[132134]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-14T22:53:22.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[132170]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-14T22:53:22.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 bash[132134]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-14T22:53:22.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[132170]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-14T22:53:22.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 bash[132134]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-14T22:53:22.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[132170]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-14T22:53:22.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 bash[132134]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-14T22:53:22.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:22 smithi086 podman[132134]: 2024-06-14 22:53:22.357926243 +0000 UTC m=+0.788717306 container died 67cd40ca658690b147d301513fb494587ee55f64a8c2e2510b1230229c24794e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS) 2024-06-14T22:53:22.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:22 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-14T22:53:22.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2280735411' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:22.743 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:53:22 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=sqlstore.transactions t=2024-06-14T22:53:22.315380182Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-06-14T22:53:23.334 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:23 smithi086 podman[132307]: 2024-06-14 22:53:23.019562537 +0000 UTC m=+0.655936617 container remove 67cd40ca658690b147d301513fb494587ee55f64a8c2e2510b1230229c24794e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-14T22:53:23.334 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:23 smithi086 podman[132371]: 2024-06-14 22:53:23.132587844 +0000 UTC m=+0.017229550 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:53:23.334 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:23 smithi086 podman[132371]: 2024-06-14 22:53:23.245599784 +0000 UTC m=+0.130241527 container create 866667701cf9b5515eb213854504f9204c5a578c5d786ef9c6bb3686f7c771be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:53:23.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:23 smithi086 ceph-mon[102754]: pgmap v467: 73 pgs: 5 active+undersized, 12 peering, 1 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 2/72 objects degraded (2.778%) 2024-06-14T22:53:23.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:23 smithi086 ceph-mon[97913]: pgmap v467: 73 pgs: 5 active+undersized, 12 peering, 1 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 2/72 objects degraded (2.778%) 2024-06-14T22:53:23.834 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:23 smithi086 podman[132371]: 2024-06-14 22:53:23.498250159 +0000 UTC m=+0.382891862 container init 866667701cf9b5515eb213854504f9204c5a578c5d786ef9c6bb3686f7c771be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_CLEAN=True) 2024-06-14T22:53:23.834 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:23 smithi086 podman[132371]: 2024-06-14 22:53:23.504134284 +0000 UTC m=+0.388775982 container start 866667701cf9b5515eb213854504f9204c5a578c5d786ef9c6bb3686f7c771be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:53:23.834 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:23 smithi086 ceph-osd[132409]: -- 172.21.15.86:0/3537907519 <== mon.2 v2:172.21.15.86:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55614cfa31e0 con 0x55614dd22000 2024-06-14T22:53:23.834 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:23 smithi086 bash[132371]: 866667701cf9b5515eb213854504f9204c5a578c5d786ef9c6bb3686f7c771be 2024-06-14T22:53:23.834 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:23 smithi086 systemd[1]: Started Ceph osd.2 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:53:23.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:23 smithi137 ceph-mon[90634]: pgmap v467: 73 pgs: 5 active+undersized, 12 peering, 1 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 2/72 objects degraded (2.778%) 2024-06-14T22:53:24.334 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[132386]: 2024-06-14T22:53:24.044+0000 7fb4bc8f4640 -1 Falling back to public interface 2024-06-14T22:53:24.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[97913]: Health check failed: Degraded data redundancy: 2/72 objects degraded (2.778%), 1 pg degraded (PG_DEGRADED) 2024-06-14T22:53:24.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:24.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:24.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:53:24.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1466241886' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:24.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[102754]: Health check failed: Degraded data redundancy: 2/72 objects degraded (2.778%), 1 pg degraded (PG_DEGRADED) 2024-06-14T22:53:24.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:24.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:24.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:53:24.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1466241886' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:24.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:24 smithi137 ceph-mon[90634]: Health check failed: Degraded data redundancy: 2/72 objects degraded (2.778%), 1 pg degraded (PG_DEGRADED) 2024-06-14T22:53:24.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:24 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:24.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:24 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:24.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:24 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:53:24.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:24 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1466241886' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:25.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:53:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:53:24] "GET /metrics HTTP/1.1" 200 34128 "" "Prometheus/2.43.0" 2024-06-14T22:53:25.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:25 smithi086 ceph-mon[97913]: pgmap v468: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:25.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:25 smithi086 ceph-mon[102754]: pgmap v468: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:25.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:25 smithi137 ceph-mon[90634]: pgmap v468: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:26.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:26 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T22:53:26.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4087278399' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:26.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:26 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T22:53:26.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4087278399' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:26.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:26 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T22:53:26.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4087278399' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:27.657 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[102754]: pgmap v469: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:27.657 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:27.657 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:27.657 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:27.657 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:27.657 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[97913]: pgmap v469: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:27.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:27.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:27.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:27.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:27 smithi137 ceph-mon[90634]: pgmap v469: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:29.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:29.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:29.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:28 smithi137 ceph-mon[90634]: pgmap v470: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:29.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3110197830' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:29.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:29.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:29.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:28 smithi086 ceph-mon[97913]: pgmap v470: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:29.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3110197830' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:29.271 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:29.271 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:29.271 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:28 smithi086 ceph-mon[102754]: pgmap v470: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:29.271 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3110197830' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:29.583 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[132386]: 2024-06-14T22:53:29.270+0000 7fb4bc8f4640 -1 osd.2 129 log_to_monitors true 2024-06-14T22:53:30.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:30 smithi086 ceph-mon[97913]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T22:53:30.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:30 smithi086 ceph-mon[102754]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T22:53:30.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:30 smithi137 ceph-mon[90634]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T22:53:30.833 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 22:53:30 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[132386]: 2024-06-14T22:53:30.487+0000 7fb4b0696640 -1 osd.2 129 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:53:31.318 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[97913]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-14T22:53:31.319 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[97913]: osdmap e132: 8 total, 7 up, 8 in 2024-06-14T22:53:31.319 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[97913]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:53:31.319 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[97913]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' 2024-06-14T22:53:31.319 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[97913]: pgmap v472: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:31.319 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[102754]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-14T22:53:31.319 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[102754]: osdmap e132: 8 total, 7 up, 8 in 2024-06-14T22:53:31.319 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[102754]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:53:31.319 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[102754]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' 2024-06-14T22:53:31.320 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-mon[102754]: pgmap v472: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:31.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:31 smithi137 ceph-mon[90634]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-14T22:53:31.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:31 smithi137 ceph-mon[90634]: osdmap e132: 8 total, 7 up, 8 in 2024-06-14T22:53:31.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:31 smithi137 ceph-mon[90634]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:53:31.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:31 smithi137 ceph-mon[90634]: from='osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022]' entity='osd.2' 2024-06-14T22:53:31.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:31 smithi137 ceph-mon[90634]: pgmap v472: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:31.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:53:31 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:53:31.473+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-06-14T22:53:32.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2431001203' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:32.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:32.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:32.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022] boot 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: osdmap e133: 8 total, 8 up, 8 in 2024-06-14T22:53:32.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2431001203' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-06-14T22:53:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:53:32.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022] boot 2024-06-14T22:53:32.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: osdmap e133: 8 total, 8 up, 8 in 2024-06-14T22:53:32.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:32 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:53:32.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2431001203' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-06-14T22:53:32.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:53:32.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: osd.2 [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022] boot 2024-06-14T22:53:32.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: osdmap e133: 8 total, 8 up, 8 in 2024-06-14T22:53:32.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:32 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T22:53:33.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:33 smithi137 ceph-mon[90634]: osdmap e134: 8 total, 8 up, 8 in 2024-06-14T22:53:33.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:33 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:33.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:33 smithi137 ceph-mon[90634]: pgmap v475: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:33.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-06-14T22:53:33.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2909008335' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:33.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[97913]: osdmap e134: 8 total, 8 up, 8 in 2024-06-14T22:53:33.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:33.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[97913]: pgmap v475: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:33.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-06-14T22:53:33.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2909008335' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:33.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[102754]: osdmap e134: 8 total, 8 up, 8 in 2024-06-14T22:53:33.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:33.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[102754]: pgmap v475: 73 pgs: 16 active+undersized, 2 active+undersized+degraded, 55 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:33.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-06-14T22:53:33.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2909008335' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:35.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:53:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:53:34] "GET /metrics HTTP/1.1" 200 34124 "" "Prometheus/2.43.0" 2024-06-14T22:53:35.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:35 smithi137 ceph-mon[90634]: pgmap v476: 73 pgs: 6 active+undersized, 1 active+undersized+degraded, 66 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2/72 objects degraded (2.778%) 2024-06-14T22:53:36.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:35 smithi086 ceph-mon[97913]: pgmap v476: 73 pgs: 6 active+undersized, 1 active+undersized+degraded, 66 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2/72 objects degraded (2.778%) 2024-06-14T22:53:36.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:35 smithi086 ceph-mon[102754]: pgmap v476: 73 pgs: 6 active+undersized, 1 active+undersized+degraded, 66 active+clean; 582 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 2/72 objects degraded (2.778%) 2024-06-14T22:53:36.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/808112231' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:37.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/808112231' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:37.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/808112231' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:37.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:37 smithi137 ceph-mon[90634]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:37.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:37 smithi137 ceph-mon[90634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/72 objects degraded (2.778%), 1 pg degraded) 2024-06-14T22:53:37.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:37 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:53:38.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:37 smithi086 ceph-mon[97913]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:38.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:37 smithi086 ceph-mon[97913]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/72 objects degraded (2.778%), 1 pg degraded) 2024-06-14T22:53:38.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:37 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:53:38.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:37 smithi086 ceph-mon[102754]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:38.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:37 smithi086 ceph-mon[102754]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/72 objects degraded (2.778%), 1 pg degraded) 2024-06-14T22:53:38.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:37 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:53:38.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:38 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2924602497' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:39.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:38 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2924602497' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:39.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:38 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2924602497' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:39.893 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:39 smithi137 ceph-mon[90634]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:40.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:39 smithi086 ceph-mon[97913]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:40.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:39 smithi086 ceph-mon[102754]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:40.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3396763178' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:41.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3396763178' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:41.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3396763178' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:41.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:41 smithi137 ceph-mon[90634]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:42.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:41 smithi086 ceph-mon[97913]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:42.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:41 smithi086 ceph-mon[102754]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:43.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:43.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/980683019' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:43 smithi137 ceph-mon[90634]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:43.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:43.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:43.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/980683019' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:43.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:43.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[97913]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:43.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:43.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:43.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/980683019' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:43.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:43.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:43 smithi086 ceph-mon[102754]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:45.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:53:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:53:44] "GET /metrics HTTP/1.1" 200 34122 "" "Prometheus/2.43.0" 2024-06-14T22:53:45.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:45 smithi137 ceph-mon[90634]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:45.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4093110481' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:46.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:45 smithi086 ceph-mon[97913]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:46.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4093110481' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:46.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:45 smithi086 ceph-mon[102754]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:46.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4093110481' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:46.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:46 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:46.870 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:46 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:46.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:46 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:47.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:47 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:47.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:47 smithi137 ceph-mon[90634]: Upgrade: osd.3 is safe to restart 2024-06-14T22:53:47.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:47 smithi137 ceph-mon[90634]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:47.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/649918300' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:48.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:47 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:48.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:47 smithi086 ceph-mon[97913]: Upgrade: osd.3 is safe to restart 2024-06-14T22:53:48.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:47 smithi086 ceph-mon[97913]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:48.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/649918300' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:48.075 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:47 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-14T22:53:48.075 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:47 smithi086 ceph-mon[102754]: Upgrade: osd.3 is safe to restart 2024-06-14T22:53:48.075 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:47 smithi086 ceph-mon[102754]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:48.076 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/649918300' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:49.490 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[97913]: Upgrade: Updating osd.3 2024-06-14T22:53:49.491 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:49.491 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-14T22:53:49.491 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:53:49.491 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[97913]: Deploying daemon osd.3 on smithi086 2024-06-14T22:53:49.491 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[97913]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:49.491 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[102754]: Upgrade: Updating osd.3 2024-06-14T22:53:49.491 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:49.491 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-14T22:53:49.491 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:53:49.492 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[102754]: Deploying daemon osd.3 on smithi086 2024-06-14T22:53:49.492 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:49 smithi086 ceph-mon[102754]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:49.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:49 smithi137 ceph-mon[90634]: Upgrade: Updating osd.3 2024-06-14T22:53:49.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:49 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:49.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:49 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-14T22:53:49.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:49 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:53:49.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:49 smithi137 ceph-mon[90634]: Deploying daemon osd.3 on smithi086 2024-06-14T22:53:49.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:49 smithi137 ceph-mon[90634]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:50.483 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:50 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1408075156' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:50.484 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:50 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1408075156' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:50.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:50 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1408075156' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:51.424 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:51 smithi086 ceph-mon[97913]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:51.424 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:51 smithi086 ceph-mon[102754]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:51.425 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:51 smithi086 systemd[1]: Stopping Ceph osd.3 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:53:51.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:51 smithi137 ceph-mon[90634]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:51.834 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:51 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[61825]: 2024-06-14T22:53:51.424+0000 7f49dcf1f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T22:53:51.834 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:51 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[61825]: 2024-06-14T22:53:51.424+0000 7f49dcf1f640 -1 osd.3 134 *** Got signal Terminated *** 2024-06-14T22:53:51.834 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:51 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[61825]: 2024-06-14T22:53:51.424+0000 7f49dcf1f640 -1 osd.3 134 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T22:53:52.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:52 smithi137 ceph-mon[90634]: osd.3 marked itself down and dead 2024-06-14T22:53:52.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2414373141' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:52.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:52 smithi086 ceph-mon[97913]: osd.3 marked itself down and dead 2024-06-14T22:53:52.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2414373141' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:52.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:52 smithi086 ceph-mon[102754]: osd.3 marked itself down and dead 2024-06-14T22:53:52.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2414373141' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:53.011 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:52 smithi086 podman[137331]: 2024-06-14 22:53:52.683261049 +0000 UTC m=+1.377120678 container died bee260f44ae79f02f042ff2ffe5905b0e2397375b5812d19ed187b273a425537 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0) 2024-06-14T22:53:53.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:53 smithi086 ceph-mon[102754]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:53:53.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:53 smithi086 ceph-mon[102754]: osdmap e135: 8 total, 7 up, 8 in 2024-06-14T22:53:53.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:53 smithi086 ceph-mon[102754]: pgmap v486: 73 pgs: 13 stale+active+clean, 60 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:53.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:53 smithi086 ceph-mon[97913]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:53:53.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:53 smithi086 ceph-mon[97913]: osdmap e135: 8 total, 7 up, 8 in 2024-06-14T22:53:53.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:53 smithi086 ceph-mon[97913]: pgmap v486: 73 pgs: 13 stale+active+clean, 60 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:53.335 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:53 smithi086 podman[137331]: 2024-06-14 22:53:53.222190169 +0000 UTC m=+1.916049787 container cleanup bee260f44ae79f02f042ff2ffe5905b0e2397375b5812d19ed187b273a425537 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:53:53.335 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:53 smithi086 bash[137331]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3 2024-06-14T22:53:53.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:53 smithi137 ceph-mon[90634]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T22:53:53.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:53 smithi137 ceph-mon[90634]: osdmap e135: 8 total, 7 up, 8 in 2024-06-14T22:53:53.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:53 smithi137 ceph-mon[90634]: pgmap v486: 73 pgs: 13 stale+active+clean, 60 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail 2024-06-14T22:53:53.834 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:53 smithi086 podman[137372]: 2024-06-14 22:53:53.474164519 +0000 UTC m=+0.791181655 container remove bee260f44ae79f02f042ff2ffe5905b0e2397375b5812d19ed187b273a425537 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T22:53:54.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:54 smithi086 ceph-mon[97913]: osdmap e136: 8 total, 7 up, 8 in 2024-06-14T22:53:54.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1547033655' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:54.168 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:54 smithi086 ceph-mon[102754]: osdmap e136: 8 total, 7 up, 8 in 2024-06-14T22:53:54.168 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1547033655' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:54.169 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:53 smithi086 podman[137490]: 2024-06-14 22:53:53.81361063 +0000 UTC m=+0.016858039 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:53:54.169 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:53 smithi086 podman[137490]: 2024-06-14 22:53:53.941779118 +0000 UTC m=+0.145026509 container create 44fb6f14d3a2de1d5f0ada20533e0e06d9147c3f145ea6223f6d7a37746e4e49 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3) 2024-06-14T22:53:54.425 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:54 smithi086 podman[137490]: 2024-06-14 22:53:54.169368766 +0000 UTC m=+0.372616165 container init 44fb6f14d3a2de1d5f0ada20533e0e06d9147c3f145ea6223f6d7a37746e4e49 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-14T22:53:54.425 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:54 smithi086 podman[137490]: 2024-06-14 22:53:54.175011887 +0000 UTC m=+0.378259285 container start 44fb6f14d3a2de1d5f0ada20533e0e06d9147c3f145ea6223f6d7a37746e4e49 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-14T22:53:54.425 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:54 smithi086 podman[137490]: 2024-06-14 22:53:54.252388881 +0000 UTC m=+0.455636286 container attach 44fb6f14d3a2de1d5f0ada20533e0e06d9147c3f145ea6223f6d7a37746e4e49 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T22:53:54.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:54 smithi137 ceph-mon[90634]: osdmap e136: 8 total, 7 up, 8 in 2024-06-14T22:53:54.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:54 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1547033655' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:54.762 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:54 smithi086 podman[137490]: 2024-06-14 22:53:54.426665711 +0000 UTC m=+0.629913105 container died 44fb6f14d3a2de1d5f0ada20533e0e06d9147c3f145ea6223f6d7a37746e4e49 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240528, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T22:53:55.055 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:53:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:53:54] "GET /metrics HTTP/1.1" 200 34122 "" "Prometheus/2.43.0" 2024-06-14T22:53:55.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:55 smithi086 ceph-mon[97913]: pgmap v488: 73 pgs: 18 active+undersized, 4 stale+active+clean, 2 active+undersized+degraded, 49 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3/72 objects degraded (4.167%) 2024-06-14T22:53:55.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:55 smithi086 ceph-mon[102754]: pgmap v488: 73 pgs: 18 active+undersized, 4 stale+active+clean, 2 active+undersized+degraded, 49 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3/72 objects degraded (4.167%) 2024-06-14T22:53:55.334 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 podman[137536]: 2024-06-14 22:53:55.056196911 +0000 UTC m=+0.620703589 container remove 44fb6f14d3a2de1d5f0ada20533e0e06d9147c3f145ea6223f6d7a37746e4e49 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=) 2024-06-14T22:53:55.334 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.3.service: Deactivated successfully. 2024-06-14T22:53:55.334 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 systemd[1]: Stopped Ceph osd.3 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:53:55.335 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.3.service: Consumed 28min 55.480s CPU time. 2024-06-14T22:53:55.335 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 systemd[1]: Starting Ceph osd.3 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T22:53:55.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:55 smithi137 ceph-mon[90634]: pgmap v488: 73 pgs: 18 active+undersized, 4 stale+active+clean, 2 active+undersized+degraded, 49 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3/72 objects degraded (4.167%) 2024-06-14T22:53:55.776 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 podman[137640]: 2024-06-14 22:53:55.401497723 +0000 UTC m=+0.017185186 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:53:55.776 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 podman[137640]: 2024-06-14 22:53:55.532757024 +0000 UTC m=+0.148444542 container create 0fb53ec5346d06f0af19417c3d20a6abd84eabcc8cbd284b2149f0ecbbcf59a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=) 2024-06-14T22:53:55.776 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 podman[137640]: 2024-06-14 22:53:55.777267089 +0000 UTC m=+0.392954549 container init 0fb53ec5346d06f0af19417c3d20a6abd84eabcc8cbd284b2149f0ecbbcf59a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-14T22:53:56.084 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 podman[137640]: 2024-06-14 22:53:55.783277533 +0000 UTC m=+0.398964984 container start 0fb53ec5346d06f0af19417c3d20a6abd84eabcc8cbd284b2149f0ecbbcf59a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-14T22:53:56.084 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:55 smithi086 podman[137640]: 2024-06-14 22:53:55.868409652 +0000 UTC m=+0.484097104 container attach 0fb53ec5346d06f0af19417c3d20a6abd84eabcc8cbd284b2149f0ecbbcf59a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-06-14T22:53:56.487 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-mon[97913]: Health check failed: Degraded data redundancy: 3/72 objects degraded (4.167%), 2 pgs degraded (PG_DEGRADED) 2024-06-14T22:53:56.487 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1363596697' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:56.488 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-mon[102754]: Health check failed: Degraded data redundancy: 3/72 objects degraded (4.167%), 2 pgs degraded (PG_DEGRADED) 2024-06-14T22:53:56.488 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1363596697' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:56.488 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate[137655]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-14T22:53:56.488 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 bash[137640]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-14T22:53:56.488 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate[137655]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-14T22:53:56.488 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 bash[137640]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-14T22:53:56.488 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate[137655]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-14T22:53:56.488 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 bash[137640]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-14T22:53:56.488 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate[137655]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-14T22:53:56.488 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 bash[137640]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-14T22:53:56.489 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate[137655]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-06-14T22:53:56.489 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 bash[137640]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-06-14T22:53:56.489 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate[137655]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-14T22:53:56.489 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 bash[137640]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-14T22:53:56.489 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate[137655]: --> ceph-volume raw activate successful for osd ID: 3 2024-06-14T22:53:56.489 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 bash[137640]: --> ceph-volume raw activate successful for osd ID: 3 2024-06-14T22:53:56.489 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 podman[137640]: 2024-06-14 22:53:56.208906484 +0000 UTC m=+0.824593934 container died 0fb53ec5346d06f0af19417c3d20a6abd84eabcc8cbd284b2149f0ecbbcf59a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-14T22:53:56.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:56 smithi137 ceph-mon[90634]: Health check failed: Degraded data redundancy: 3/72 objects degraded (4.167%), 2 pgs degraded (PG_DEGRADED) 2024-06-14T22:53:56.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1363596697' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:57.136 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:56 smithi086 podman[137829]: 2024-06-14 22:53:56.872396417 +0000 UTC m=+0.657785953 container remove 0fb53ec5346d06f0af19417c3d20a6abd84eabcc8cbd284b2149f0ecbbcf59a4 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-14T22:53:57.137 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:57 smithi086 podman[137892]: 2024-06-14 22:53:56.986335059 +0000 UTC m=+0.017101273 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:53:57.137 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:57 smithi086 podman[137892]: 2024-06-14 22:53:57.098469832 +0000 UTC m=+0.129236038 container create 47b006e23b2a60d7adea74948de7e51ac45bd6657fc24c6adb76e41138487488 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.35.3, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.vendor=CentOS) 2024-06-14T22:53:57.424 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:57 smithi086 ceph-mon[97913]: pgmap v489: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:57.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:57 smithi086 ceph-mon[102754]: pgmap v489: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:57.425 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:57 smithi086 podman[137892]: 2024-06-14 22:53:57.334368738 +0000 UTC m=+0.365134949 container init 47b006e23b2a60d7adea74948de7e51ac45bd6657fc24c6adb76e41138487488 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, maintainer=Guillaume Abrioux ) 2024-06-14T22:53:57.425 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:57 smithi086 podman[137892]: 2024-06-14 22:53:57.340178685 +0000 UTC m=+0.370944891 container start 47b006e23b2a60d7adea74948de7e51ac45bd6657fc24c6adb76e41138487488 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T22:53:57.425 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:57 smithi086 ceph-osd[137929]: -- 172.21.15.86:0/3002174751 <== mon.2 v2:172.21.15.86:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56076fa311e0 con 0x56076f9cfc00 2024-06-14T22:53:57.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:57 smithi137 ceph-mon[90634]: pgmap v489: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:57.833 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:57 smithi086 bash[137892]: 47b006e23b2a60d7adea74948de7e51ac45bd6657fc24c6adb76e41138487488 2024-06-14T22:53:57.834 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:57 smithi086 systemd[1]: Started Ceph osd.3 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T22:53:58.164 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:53:57 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T22:53:57.877+0000 7f61b5a68640 -1 Falling back to public interface 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:58.536 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:53:58.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:58.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:53:58.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:58.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:53:58.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:53:59.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:59 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3032334577' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:59.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:53:59 smithi086 ceph-mon[97913]: pgmap v490: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:59.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:59 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3032334577' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:59.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:53:59 smithi086 ceph-mon[102754]: pgmap v490: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:53:59.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:59 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3032334577' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:53:59.672 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:53:59 smithi137 ceph-mon[90634]: pgmap v490: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:01.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:01 smithi137 ceph-mon[90634]: pgmap v491: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/119269487' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:02.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[97913]: pgmap v491: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:02.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/119269487' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:02.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:02.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:02.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:02.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:02.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[102754]: pgmap v491: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:02.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/119269487' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:02.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:02.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:02.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:02.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:03.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:03.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:03.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:02 smithi086 ceph-mon[97913]: pgmap v492: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:03.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:03.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:03.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:02 smithi086 ceph-mon[102754]: pgmap v492: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:03.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:03.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:03.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:02 smithi137 ceph-mon[90634]: pgmap v492: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:03.617 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:54:03 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T22:54:03.381+0000 7f61b5a68640 -1 osd.3 134 log_to_monitors true 2024-06-14T22:54:04.018 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3035582589' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:04.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded (PG_DEGRADED) 2024-06-14T22:54:04.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:03 smithi086 ceph-mon[102754]: from='osd.3 [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:54:04.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:03 smithi086 ceph-mon[102754]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:54:04.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3035582589' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:04.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded (PG_DEGRADED) 2024-06-14T22:54:04.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:03 smithi086 ceph-mon[97913]: from='osd.3 [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:54:04.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:03 smithi086 ceph-mon[97913]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:54:04.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3035582589' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:04.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded (PG_DEGRADED) 2024-06-14T22:54:04.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:03 smithi137 ceph-mon[90634]: from='osd.3 [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:54:04.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:03 smithi137 ceph-mon[90634]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-14T22:54:04.334 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T22:54:04.267+0000 7f61a980a640 -1 osd.3 134 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T22:54:05.083 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:54:04] "GET /metrics HTTP/1.1" 200 34124 "" "Prometheus/2.43.0" 2024-06-14T22:54:05.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[97913]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-14T22:54:05.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[97913]: osdmap e137: 8 total, 7 up, 8 in 2024-06-14T22:54:05.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[97913]: from='osd.3 [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:54:05.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[97913]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:54:05.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[97913]: from='osd.3 ' entity='osd.3' 2024-06-14T22:54:05.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[97913]: pgmap v494: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:05.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[102754]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-14T22:54:05.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[102754]: osdmap e137: 8 total, 7 up, 8 in 2024-06-14T22:54:05.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[102754]: from='osd.3 [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:54:05.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[102754]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:54:05.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[102754]: from='osd.3 ' entity='osd.3' 2024-06-14T22:54:05.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:04 smithi086 ceph-mon[102754]: pgmap v494: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:05.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:04 smithi137 ceph-mon[90634]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-14T22:54:05.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:04 smithi137 ceph-mon[90634]: osdmap e137: 8 total, 7 up, 8 in 2024-06-14T22:54:05.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:04 smithi137 ceph-mon[90634]: from='osd.3 [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:54:05.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:04 smithi137 ceph-mon[90634]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T22:54:05.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:04 smithi137 ceph-mon[90634]: from='osd.3 ' entity='osd.3' 2024-06-14T22:54:05.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:04 smithi137 ceph-mon[90634]: pgmap v494: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:05.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T22:54:05.447+0000 7f48e8f21640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-06-14T22:54:06.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2732431298' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:06.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:54:06.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: osd.3 [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285] boot 2024-06-14T22:54:06.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: osdmap e138: 8 total, 8 up, 8 in 2024-06-14T22:54:06.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:54:06.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:06.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:06.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:54:06.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[97913]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2732431298' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:54:06.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: osd.3 [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285] boot 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: osdmap e138: 8 total, 8 up, 8 in 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-14T22:54:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-14T22:54:06.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:05 smithi086 ceph-mon[102754]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2732431298' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: osd.3 [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285] boot 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: osdmap e138: 8 total, 8 up, 8 in 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:54:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:06.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:06.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:06.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-14T22:54:06.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-14T22:54:06.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:05 smithi137 ceph-mon[90634]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-06-14T22:54:07.523 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:07 smithi086 ceph-mon[97913]: osdmap e139: 8 total, 8 up, 8 in 2024-06-14T22:54:07.523 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:07 smithi086 ceph-mon[97913]: pgmap v497: 73 pgs: 1 activating, 23 active+undersized, 3 active+undersized+degraded, 46 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:07.523 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:07 smithi086 ceph-mon[102754]: osdmap e139: 8 total, 8 up, 8 in 2024-06-14T22:54:07.523 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:07 smithi086 ceph-mon[102754]: pgmap v497: 73 pgs: 1 activating, 23 active+undersized, 3 active+undersized+degraded, 46 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:07.524 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable)": 4 2024-06-14T22:54:07.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:07 smithi137 ceph-mon[90634]: osdmap e139: 8 total, 8 up, 8 in 2024-06-14T22:54:07.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:07 smithi137 ceph-mon[90634]: pgmap v497: 73 pgs: 1 activating, 23 active+undersized, 3 active+undersized+degraded, 46 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:08.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch upgrade pause' 2024-06-14T22:54:08.521 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:08.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1723208926' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:08.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1723208926' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:08.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1723208926' entity='client.admin' cmd=[{"prefix": "osd versions"}]: dispatch 2024-06-14T22:54:09.542 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:09 smithi086 ceph-mon[97913]: pgmap v498: 73 pgs: 1 activating, 23 active+undersized, 3 active+undersized+degraded, 46 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:09.542 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:09 smithi086 ceph-mon[102754]: pgmap v498: 73 pgs: 1 activating, 23 active+undersized, 3 active+undersized+degraded, 46 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:09.643 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:09 smithi137 ceph-mon[90634]: pgmap v498: 73 pgs: 1 activating, 23 active+undersized, 3 active+undersized+degraded, 46 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4/72 objects degraded (5.556%) 2024-06-14T22:54:10.658 INFO:teuthology.orchestra.run.smithi086.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:54:10.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:10 smithi137 ceph-mon[90634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded) 2024-06-14T22:54:10.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:10 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:54:11.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:10 smithi086 ceph-mon[97913]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded) 2024-06-14T22:54:11.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:10 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:54:11.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:10 smithi086 ceph-mon[102754]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded) 2024-06-14T22:54:11.025 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:10 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:54:11.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch ps' 2024-06-14T22:54:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:11.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:11 smithi086 ceph-mon[97913]: pgmap v499: 73 pgs: 1 activating, 72 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:11.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:11 smithi086 ceph-mon[97913]: from='client.35367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:54:11.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:11 smithi086 ceph-mon[97913]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:54:11.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:11.972 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:11 smithi086 ceph-mon[102754]: pgmap v499: 73 pgs: 1 activating, 72 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:11.972 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:11 smithi086 ceph-mon[102754]: from='client.35367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:54:11.972 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:11 smithi086 ceph-mon[102754]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:54:11.972 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:11.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:11 smithi137 ceph-mon[90634]: pgmap v499: 73 pgs: 1 activating, 72 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:11.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:11 smithi137 ceph-mon[90634]: from='client.35367 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:54:11.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:11 smithi137 ceph-mon[90634]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T22:54:11.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:13.368 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:13.369 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:13.369 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:13 smithi086 ceph-mon[102754]: pgmap v500: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:13.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:13.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:13.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:13 smithi086 ceph-mon[97913]: pgmap v500: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:13.541 INFO:teuthology.orchestra.run.smithi086.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-14T22:54:13.541 INFO:teuthology.orchestra.run.smithi086.stdout:alertmanager.a smithi086 *:9093,9094 running (16m) 12s ago 20m 25.8M - 0.25.0 c8568f914cd2 1c9917382faa 2024-06-14T22:54:13.541 INFO:teuthology.orchestra.run.smithi086.stdout:grafana.a smithi137 *:3000 running (15m) 11s ago 20m 84.4M - 9.4.7 954c08fa6188 12b0617a16e5 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi086.nhdjdb smithi086 running (19m) 12s ago 19m 20.7M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 2bd94d6f47de 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi137.yjevpg smithi137 running (19m) 11s ago 19m 16.0M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4151acede02a 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.x smithi137 *:8443,9283,8765 running (15m) 11s ago 24m 445M - 18.2.2-1443-g6f4b0956 e7282a3bffba 086e4706077c 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.y smithi086 *:8443,9283,8765 running (15m) 12s ago 26m 497M - 18.2.2-1443-g6f4b0956 e7282a3bffba edafb7b2b182 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:mon.a smithi086 running (14m) 12s ago 26m 65.3M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 0fe78616d3a2 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:mon.b smithi137 running (11m) 11s ago 24m 47.8M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 74ce1e795ab2 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:mon.c smithi086 running (12m) 12s ago 24m 53.2M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 1c4e71d265f0 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.a smithi086 *:9100 running (16m) 12s ago 20m 15.6M - 1.5.0 0da6a335fe13 d1de50e60879 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.b smithi137 *:9100 running (15m) 11s ago 20m 16.4M - 1.5.0 0da6a335fe13 b8c1542255f2 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0 smithi086 running (9m) 12s ago 24m 762M 2480M 18.2.2-1443-g6f4b0956 e7282a3bffba 62732bd1f802 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1 smithi086 running (4m) 12s ago 23m 263M 2480M 18.2.2-1443-g6f4b0956 e7282a3bffba 50c27a83b12a 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2 smithi086 running (50s) 12s ago 23m 88.7M 2480M 18.2.2-1443-g6f4b0956 e7282a3bffba 866667701cf9 2024-06-14T22:54:13.542 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3 smithi086 running (16s) 12s ago 22m 14.3M 2480M 18.2.2-1443-g6f4b0956 e7282a3bffba 47b006e23b2a 2024-06-14T22:54:13.543 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4 smithi137 running (22m) 11s ago 22m 550M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4b2729030303 2024-06-14T22:54:13.543 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5 smithi137 running (22m) 11s ago 22m 665M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 e01c2182297f 2024-06-14T22:54:13.543 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6 smithi137 running (21m) 11s ago 21m 842M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 7210ba0a183d 2024-06-14T22:54:13.543 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7 smithi137 running (21m) 11s ago 21m 704M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 a9ca11c000f2 2024-06-14T22:54:13.543 INFO:teuthology.orchestra.run.smithi086.stdout:prometheus.a smithi137 *:9095 running (15m) 11s ago 20m 69.8M - 2.43.0 a07b618ecd1d 1604035f49df 2024-06-14T22:54:13.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:13 smithi137 ceph-mon[90634]: pgmap v500: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:14.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph orch ps' 2024-06-14T22:54:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:14.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:14 smithi137 ceph-mon[90634]: from='client.45296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:54:14.753 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:14 smithi086 ceph-mon[97913]: from='client.45296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:54:14.753 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:14 smithi086 ceph-mon[102754]: from='client.45296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:54:15.084 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:54:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:54:14] "GET /metrics HTTP/1.1" 200 34121 "" "Prometheus/2.43.0" 2024-06-14T22:54:15.742 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:15 smithi137 ceph-mon[90634]: pgmap v501: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:15.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:15 smithi086 ceph-mon[97913]: pgmap v501: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:15.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:15 smithi086 ceph-mon[102754]: pgmap v501: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:16.613 INFO:teuthology.orchestra.run.smithi086.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-14T22:54:16.613 INFO:teuthology.orchestra.run.smithi086.stdout:alertmanager.a smithi086 *:9093,9094 running (16m) 15s ago 20m 25.8M - 0.25.0 c8568f914cd2 1c9917382faa 2024-06-14T22:54:16.613 INFO:teuthology.orchestra.run.smithi086.stdout:grafana.a smithi137 *:3000 running (15m) 15s ago 20m 84.4M - 9.4.7 954c08fa6188 12b0617a16e5 2024-06-14T22:54:16.613 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi086.nhdjdb smithi086 running (19m) 15s ago 19m 20.7M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 2bd94d6f47de 2024-06-14T22:54:16.613 INFO:teuthology.orchestra.run.smithi086.stdout:mds.foo.smithi137.yjevpg smithi137 running (19m) 15s ago 19m 16.0M - 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4151acede02a 2024-06-14T22:54:16.613 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.x smithi137 *:8443,9283,8765 running (15m) 15s ago 24m 445M - 18.2.2-1443-g6f4b0956 e7282a3bffba 086e4706077c 2024-06-14T22:54:16.613 INFO:teuthology.orchestra.run.smithi086.stdout:mgr.y smithi086 *:8443,9283,8765 running (15m) 15s ago 26m 497M - 18.2.2-1443-g6f4b0956 e7282a3bffba edafb7b2b182 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:mon.a smithi086 running (14m) 15s ago 26m 65.3M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 0fe78616d3a2 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:mon.b smithi137 running (11m) 15s ago 24m 47.8M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 74ce1e795ab2 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:mon.c smithi086 running (12m) 15s ago 24m 53.2M 2048M 18.2.2-1443-g6f4b0956 e7282a3bffba 1c4e71d265f0 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.a smithi086 *:9100 running (16m) 15s ago 20m 15.6M - 1.5.0 0da6a335fe13 d1de50e60879 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:node-exporter.b smithi137 *:9100 running (15m) 15s ago 20m 16.4M - 1.5.0 0da6a335fe13 b8c1542255f2 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0 smithi086 running (9m) 15s ago 24m 762M 2480M 18.2.2-1443-g6f4b0956 e7282a3bffba 62732bd1f802 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1 smithi086 running (5m) 15s ago 23m 263M 2480M 18.2.2-1443-g6f4b0956 e7282a3bffba 50c27a83b12a 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2 smithi086 running (53s) 15s ago 23m 88.7M 2480M 18.2.2-1443-g6f4b0956 e7282a3bffba 866667701cf9 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3 smithi086 running (19s) 15s ago 22m 14.3M 2480M 18.2.2-1443-g6f4b0956 e7282a3bffba 47b006e23b2a 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4 smithi137 running (22m) 15s ago 22m 550M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 4b2729030303 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5 smithi137 running (22m) 15s ago 22m 665M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 e01c2182297f 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6 smithi137 running (21m) 15s ago 21m 842M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 7210ba0a183d 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7 smithi137 running (21m) 15s ago 21m 704M 2480M 17.2.7-1008-ge47ff4b8 37f9eedc23c3 a9ca11c000f2 2024-06-14T22:54:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:prometheus.a smithi137 *:9095 running (15m) 15s ago 20m 69.8M - 2.43.0 a07b618ecd1d 1604035f49df 2024-06-14T22:54:17.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -e sha1=6f4b09560ed399c3f139402a02f389c2fc8bb2cc -- bash -c 'ceph versions' 2024-06-14T22:54:17.485 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:17.890 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:17 smithi086 ceph-mon[97913]: pgmap v502: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:17.891 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:17 smithi086 ceph-mon[97913]: from='client.45302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:54:17.891 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:17 smithi086 ceph-mon[102754]: pgmap v502: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:17.891 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:17 smithi086 ceph-mon[102754]: from='client.45302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:54:17.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:17 smithi137 ceph-mon[90634]: pgmap v502: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:17.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:17 smithi137 ceph-mon[90634]: from='client.45302 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T22:54:19.423 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:54:19.423 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": { 2024-06-14T22:54:19.423 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable)": 3 2024-06-14T22:54:19.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:54:19.423 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": { 2024-06-14T22:54:19.423 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable)": 2 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": { 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)": 4, 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable)": 4 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: "mds": { 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)": 2 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: "overall": { 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)": 6, 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph version 18.2.2-1443-g6f4b0956 (6f4b09560ed399c3f139402a02f389c2fc8bb2cc) reef (stable)": 9 2024-06-14T22:54:19.424 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:54:19.425 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:54:19.776 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:19 smithi086 ceph-mon[97913]: pgmap v503: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:19.776 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:19 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2564969515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:19.777 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:19 smithi086 ceph-mon[102754]: pgmap v503: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:19.777 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:19 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2564969515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:19.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:19 smithi137 ceph-mon[90634]: pgmap v503: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:19.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:19 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2564969515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-14T22:54:20.078 DEBUG:teuthology.parallel:result is None 2024-06-14T22:54:20.078 INFO:teuthology.run_tasks:Running task print... 2024-06-14T22:54:20.086 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-06-14T22:54:20.087 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-06-14T22:54:20.136 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-14T22:54:20.137 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-14T22:54:20.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph mgr dump --format=json 2024-06-14T22:54:20.391 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:20.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:54:20.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:54:20.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:54:20.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:20.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:54:20.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:54:20.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:54:20.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:20.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:54:20.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:54:20.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:54:20.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:20.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:54:20.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:54:20.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:54:20.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:20.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:54:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:54:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:54:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:54:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:54:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:54:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:21.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:21 smithi086 ceph-mon[97913]: pgmap v504: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:21.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:21 smithi086 ceph-mon[102754]: pgmap v504: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:21.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:21 smithi137 ceph-mon[90634]: pgmap v504: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:22.341 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:54:22.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1849182411' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-14T22:54:22.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:22 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:23.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1849182411' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-14T22:54:23.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:22 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:23.057 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1849182411' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-14T22:54:23.058 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:22 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:54:23.083 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":39,"flags":0,"active_gid":25130,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":293345125},{"type":"v1","addr":"172.21.15.86:6801","nonce":293345125}]},"active_addr":"172.21.15.86:6801/293345125","active_change":"2024-06-14T22:39:42.196070+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24871,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.86:8443/","prometheus":"http://172.21.15.86:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":77,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2276181328}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2226602393}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":1801540798}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3545952777}]}]} 2024-06-14T22:54:23.087 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-14T22:54:23.087 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-14T22:54:23.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-14T22:54:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:23.744 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:23 smithi086 ceph-mon[97913]: pgmap v505: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:23.744 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:23 smithi086 ceph-mon[102754]: pgmap v505: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:23.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:23 smithi137 ceph-mon[90634]: pgmap v505: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:25.084 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:54:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:54:24] "GET /metrics HTTP/1.1" 200 34121 "" "Prometheus/2.43.0" 2024-06-14T22:54:25.182 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:54:25.183 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":139,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T22:54:06.263307+0000","last_up_change":"2024-06-14T22:54:05.259648+0000","last_in_change":"2024-06-14T22:32:42.063674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-14T22:33:31.126149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-06-14T22:34:44.677867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-06-14T22:34:45.034624+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":83,"up_from":91,"up_thru":138,"down_at":84,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6803","nonce":2971024428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6805","nonce":2971024428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6809","nonce":2971024428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6807","nonce":2971024428}]},"public_addr":"172.21.15.86:6803/2971024428","cluster_addr":"172.21.15.86:6805/2971024428","heartbeat_back_addr":"172.21.15.86:6809/2971024428","heartbeat_front_addr":"172.21.15.86:6807/2971024428","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":112,"up_from":116,"up_thru":138,"down_at":113,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6811","nonce":1358489517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6813","nonce":1358489517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6817","nonce":1358489517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6815","nonce":1358489517}]},"public_addr":"172.21.15.86:6811/1358489517","cluster_addr":"172.21.15.86:6813/1358489517","heartbeat_back_addr":"172.21.15.86:6817/1358489517","heartbeat_front_addr":"172.21.15.86:6815/1358489517","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":129,"up_from":133,"up_thru":138,"down_at":130,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3223348022},{"type":"v1","addr":"172.21.15.86:6819","nonce":3223348022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3223348022},{"type":"v1","addr":"172.21.15.86:6821","nonce":3223348022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3223348022},{"type":"v1","addr":"172.21.15.86:6825","nonce":3223348022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3223348022},{"type":"v1","addr":"172.21.15.86:6823","nonce":3223348022}]},"public_addr":"172.21.15.86:6819/3223348022","cluster_addr":"172.21.15.86:6821/3223348022","heartbeat_back_addr":"172.21.15.86:6825/3223348022","heartbeat_front_addr":"172.21.15.86:6823/3223348022","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":134,"up_from":138,"up_thru":138,"down_at":135,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6827","nonce":1034049285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6829","nonce":1034049285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6833","nonce":1034049285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6831","nonce":1034049285}]},"public_addr":"172.21.15.86:6827/1034049285","cluster_addr":"172.21.15.86:6829/1034049285","heartbeat_back_addr":"172.21.15.86:6833/1034049285","heartbeat_front_addr":"172.21.15.86:6831/1034049285","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6809","nonce":2122959324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6811","nonce":2122959324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6815","nonce":2122959324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6813","nonce":2122959324}]},"public_addr":"172.21.15.137:6809/2122959324","cluster_addr":"172.21.15.137:6811/2122959324","heartbeat_back_addr":"172.21.15.137:6815/2122959324","heartbeat_front_addr":"172.21.15.137:6813/2122959324","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6817","nonce":2398798362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6819","nonce":2398798362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6823","nonce":2398798362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6821","nonce":2398798362}]},"public_addr":"172.21.15.137:6817/2398798362","cluster_addr":"172.21.15.137:6819/2398798362","heartbeat_back_addr":"172.21.15.137:6823/2398798362","heartbeat_front_addr":"172.21.15.137:6821/2398798362","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":83},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":112},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":129},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":134},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:0/3024645077":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/2298288322":"2024-06-15T22:39:42.195791+0000","172.21.15.86:6800/1921939101":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/1328684620":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/648695835":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/358698000":"2024-06-15T22:39:42.195791+0000","172.21.15.137:0/3481534205":"2024-06-15T22:38:44.349280+0000","172.21.15.86:6801/2254669699":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/2741142752":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/418538598":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/1759721543":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2254669699":"2024-06-15T22:37:41.915228+0000","172.21.15.137:6834/3389028913":"2024-06-15T22:38:44.349280+0000","172.21.15.86:6801/1921939101":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3724463773":"2024-06-15T22:33:27.518416+0000","172.21.15.137:6835/3389028913":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/2407056636":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/2914169147":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/3094754408":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/2665974068":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/3471286695":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/3167284010":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1470436259":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3107951276":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/3941467574":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/1167860215":"2024-06-15T22:37:41.915228+0000","172.21.15.86:6801/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T22:54:25.805 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:25 smithi086 ceph-mon[97913]: pgmap v506: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:25.805 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2501684958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:54:25.805 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:25 smithi086 ceph-mon[102754]: pgmap v506: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:25.805 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2501684958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:54:25.830 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-14T22:54:25.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-14T22:54:25.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:25 smithi137 ceph-mon[90634]: pgmap v506: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:25.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2501684958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:54:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:27.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:27 smithi086 ceph-mon[97913]: pgmap v507: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:27.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:27.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:27 smithi086 ceph-mon[102754]: pgmap v507: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:27.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:27.893 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:54:27.894 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":139,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T22:54:06.263307+0000","last_up_change":"2024-06-14T22:54:05.259648+0000","last_in_change":"2024-06-14T22:32:42.063674+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-14T22:33:31.126149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-06-14T22:34:44.677867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-06-14T22:34:45.034624+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":83,"up_from":91,"up_thru":138,"down_at":84,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6803","nonce":2971024428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6805","nonce":2971024428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6809","nonce":2971024428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6807","nonce":2971024428}]},"public_addr":"172.21.15.86:6803/2971024428","cluster_addr":"172.21.15.86:6805/2971024428","heartbeat_back_addr":"172.21.15.86:6809/2971024428","heartbeat_front_addr":"172.21.15.86:6807/2971024428","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":112,"up_from":116,"up_thru":138,"down_at":113,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6811","nonce":1358489517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6813","nonce":1358489517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6817","nonce":1358489517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6815","nonce":1358489517}]},"public_addr":"172.21.15.86:6811/1358489517","cluster_addr":"172.21.15.86:6813/1358489517","heartbeat_back_addr":"172.21.15.86:6817/1358489517","heartbeat_front_addr":"172.21.15.86:6815/1358489517","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":129,"up_from":133,"up_thru":138,"down_at":130,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3223348022},{"type":"v1","addr":"172.21.15.86:6819","nonce":3223348022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3223348022},{"type":"v1","addr":"172.21.15.86:6821","nonce":3223348022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3223348022},{"type":"v1","addr":"172.21.15.86:6825","nonce":3223348022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3223348022},{"type":"v1","addr":"172.21.15.86:6823","nonce":3223348022}]},"public_addr":"172.21.15.86:6819/3223348022","cluster_addr":"172.21.15.86:6821/3223348022","heartbeat_back_addr":"172.21.15.86:6825/3223348022","heartbeat_front_addr":"172.21.15.86:6823/3223348022","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":134,"up_from":138,"up_thru":138,"down_at":135,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6827","nonce":1034049285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6829","nonce":1034049285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6833","nonce":1034049285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6831","nonce":1034049285}]},"public_addr":"172.21.15.86:6827/1034049285","cluster_addr":"172.21.15.86:6829/1034049285","heartbeat_back_addr":"172.21.15.86:6833/1034049285","heartbeat_front_addr":"172.21.15.86:6831/1034049285","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6809","nonce":2122959324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6811","nonce":2122959324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6815","nonce":2122959324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2122959324},{"type":"v1","addr":"172.21.15.137:6813","nonce":2122959324}]},"public_addr":"172.21.15.137:6809/2122959324","cluster_addr":"172.21.15.137:6811/2122959324","heartbeat_back_addr":"172.21.15.137:6815/2122959324","heartbeat_front_addr":"172.21.15.137:6813/2122959324","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6817","nonce":2398798362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6819","nonce":2398798362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6823","nonce":2398798362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2398798362},{"type":"v1","addr":"172.21.15.137:6821","nonce":2398798362}]},"public_addr":"172.21.15.137:6817/2398798362","cluster_addr":"172.21.15.137:6819/2398798362","heartbeat_back_addr":"172.21.15.137:6823/2398798362","heartbeat_front_addr":"172.21.15.137:6821/2398798362","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":83},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":112},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":129},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":134},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:0/3481534205":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/1167860215":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/3024645077":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/2298288322":"2024-06-15T22:39:42.195791+0000","172.21.15.86:6801/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/418538598":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/358698000":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/1759721543":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2254669699":"2024-06-15T22:37:41.915228+0000","172.21.15.137:6834/3389028913":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/1328684620":"2024-06-15T22:39:42.195791+0000","172.21.15.137:0/2407056636":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/2914169147":"2024-06-15T22:38:44.349280+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3724463773":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/648695835":"2024-06-15T22:39:42.195791+0000","172.21.15.137:6835/3389028913":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/3471286695":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/2741142752":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6801/2254669699":"2024-06-15T22:37:41.915228+0000","172.21.15.137:0/2665974068":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/3094754408":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/3167284010":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/1921939101":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6801/1921939101":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/3941467574":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/1470436259":"2024-06-15T22:39:42.195791+0000","172.21.15.86:6800/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3107951276":"2024-06-15T22:37:41.915228+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T22:54:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:27 smithi137 ceph-mon[90634]: pgmap v507: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:27.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:28.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-14T22:54:28.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-14T22:54:28.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-14T22:54:28.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-14T22:54:28.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-14T22:54:28.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-14T22:54:28.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-14T22:54:28.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-14T22:54:28.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2647540760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:54:28.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2647540760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:54:28.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2647540760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T22:54:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:29.151 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:29.159 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:29.159 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:29.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:29 smithi086 ceph-mon[97913]: pgmap v508: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:29.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:29 smithi086 ceph-mon[102754]: pgmap v508: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:29.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:29 smithi137 ceph-mon[90634]: pgmap v508: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:31.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:31 smithi137 ceph-mon[90634]: pgmap v509: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:32.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:31 smithi086 ceph-mon[97913]: pgmap v509: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:32.047 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:31 smithi086 ceph-mon[102754]: pgmap v509: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:33.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:33 smithi086 ceph-mon[97913]: pgmap v510: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:33.950 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:33 smithi086 ceph-mon[102754]: pgmap v510: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:33.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:33 smithi137 ceph-mon[90634]: pgmap v510: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:34.998 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:54:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:54:34] "GET /metrics HTTP/1.1" 200 34124 "" "Prometheus/2.43.0" 2024-06-14T22:54:35.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:35 smithi086 ceph-mon[97913]: pgmap v511: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:35.830 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:35 smithi086 ceph-mon[102754]: pgmap v511: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:35.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:35 smithi137 ceph-mon[90634]: pgmap v511: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:37.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:37 smithi086 ceph-mon[97913]: pgmap v512: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:37.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:37 smithi086 ceph-mon[102754]: pgmap v512: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:37.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:37 smithi137 ceph-mon[90634]: pgmap v512: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:39 smithi137 ceph-mon[90634]: pgmap v513: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:40.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:39 smithi086 ceph-mon[97913]: pgmap v513: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:39 smithi086 ceph-mon[102754]: pgmap v513: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:40.544 INFO:teuthology.orchestra.run.smithi086.stdout:128849019157 2024-06-14T22:54:40.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-14T22:54:41.301 INFO:teuthology.orchestra.run.smithi086.stdout:390842024055 2024-06-14T22:54:41.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-14T22:54:41.888 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:41 smithi086 ceph-mon[97913]: pgmap v514: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:41.888 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:41 smithi086 ceph-mon[102754]: pgmap v514: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:41.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:41 smithi137 ceph-mon[90634]: pgmap v514: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:42.205 INFO:teuthology.orchestra.run.smithi086.stdout:571230650383 2024-06-14T22:54:42.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-14T22:54:42.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:42.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:42.892 INFO:teuthology.orchestra.run.smithi086.stdout:197568495882 2024-06-14T22:54:42.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-14T22:54:42.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:43.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:43 smithi086 ceph-mon[97913]: pgmap v515: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:43.729 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:43 smithi086 ceph-mon[102754]: pgmap v515: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:43.752 INFO:teuthology.orchestra.run.smithi086.stdout:498216206401 2024-06-14T22:54:43.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-14T22:54:43.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:43 smithi137 ceph-mon[90634]: pgmap v515: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:45.204 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:54:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:54:44] "GET /metrics HTTP/1.1" 200 34121 "" "Prometheus/2.43.0" 2024-06-14T22:54:45.228 INFO:teuthology.orchestra.run.smithi086.stdout:163208757523 2024-06-14T22:54:45.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-14T22:54:45.698 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:45.698 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:45.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:45 smithi086 ceph-mon[97913]: pgmap v516: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:45.822 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:45 smithi086 ceph-mon[102754]: pgmap v516: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:45.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:45 smithi137 ceph-mon[90634]: pgmap v516: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:46.184 INFO:teuthology.orchestra.run.smithi086.stdout:231928234248 2024-06-14T22:54:46.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-14T22:54:46.392 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:46.848 INFO:teuthology.orchestra.run.smithi086.stdout:592705486858 2024-06-14T22:54:46.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-14T22:54:47.973 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:47 smithi086 ceph-mon[97913]: pgmap v517: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:47.973 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:47 smithi086 ceph-mon[102754]: pgmap v517: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:47.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:47 smithi137 ceph-mon[90634]: pgmap v517: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:49.492 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:49 smithi137 ceph-mon[90634]: pgmap v518: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:49.514 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:49.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:49 smithi086 ceph-mon[97913]: pgmap v518: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:49.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:49 smithi086 ceph-mon[102754]: pgmap v518: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:50.799 INFO:teuthology.orchestra.run.smithi086.stdout:390842024057 2024-06-14T22:54:51.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 390842024055 got 390842024057 for osd.0 2024-06-14T22:54:51.505 DEBUG:teuthology.parallel:result is None 2024-06-14T22:54:51.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:51 smithi086 ceph-mon[97913]: pgmap v519: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:51.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/550493195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T22:54:51.777 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:51 smithi086 ceph-mon[102754]: pgmap v519: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:51.777 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/550493195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T22:54:51.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:51 smithi137 ceph-mon[90634]: pgmap v519: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:51.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/550493195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T22:54:52.155 INFO:teuthology.orchestra.run.smithi086.stdout:571230650386 2024-06-14T22:54:52.828 INFO:teuthology.orchestra.run.smithi086.stdout:128849019160 2024-06-14T22:54:52.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/934018384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T22:54:52.828 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/934018384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T22:54:52.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/934018384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T22:54:53.491 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019157 got 128849019160 for osd.4 2024-06-14T22:54:53.492 DEBUG:teuthology.parallel:result is None 2024-06-14T22:54:53.509 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:53.509 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:53.509 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:53.510 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:54:53.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:53 smithi086 ceph-mon[97913]: pgmap v520: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:53.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3430488626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T22:54:53.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:53 smithi086 ceph-mon[102754]: pgmap v520: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:53.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3430488626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T22:54:53.910 INFO:tasks.cephadm.ceph_manager.ceph:need seq 571230650383 got 571230650386 for osd.2 2024-06-14T22:54:53.910 DEBUG:teuthology.parallel:result is None 2024-06-14T22:54:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:53 smithi137 ceph-mon[90634]: pgmap v520: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3430488626' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T22:54:55.080 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:54:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:54:54] "GET /metrics HTTP/1.1" 200 34121 "" "Prometheus/2.43.0" 2024-06-14T22:54:55.816 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:55 smithi086 ceph-mon[102754]: pgmap v521: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:55.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:55 smithi086 ceph-mon[97913]: pgmap v521: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:55.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:55 smithi137 ceph-mon[90634]: pgmap v521: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:57.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:57 smithi086 ceph-mon[102754]: pgmap v522: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:57.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:57.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:57 smithi086 ceph-mon[97913]: pgmap v522: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:57.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:57 smithi137 ceph-mon[90634]: pgmap v522: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:54:58.033 INFO:teuthology.orchestra.run.smithi086.stdout:197568495886 2024-06-14T22:54:58.919 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1399464275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T22:54:58.920 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1399464275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T22:54:58.946 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495882 got 197568495886 for osd.6 2024-06-14T22:54:58.946 DEBUG:teuthology.parallel:result is None 2024-06-14T22:54:58.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1399464275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T22:54:59.759 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:54:59 smithi086 ceph-mon[97913]: pgmap v523: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:59.759 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:54:59 smithi086 ceph-mon[102754]: pgmap v523: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:54:59.831 INFO:teuthology.orchestra.run.smithi086.stdout:231928234250 2024-06-14T22:54:59.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:54:59 smithi137 ceph-mon[90634]: pgmap v523: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:00.892 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3089867331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T22:55:00.892 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3089867331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T22:55:00.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3089867331' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T22:55:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:163208757526 2024-06-14T22:55:01.574 INFO:teuthology.orchestra.run.smithi086.stdout:498216206405 2024-06-14T22:55:01.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:01 smithi086 ceph-mon[97913]: pgmap v524: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:01.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1188209081' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T22:55:01.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1260242977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T22:55:01.815 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:01 smithi086 ceph-mon[102754]: pgmap v524: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:01.815 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1188209081' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T22:55:01.815 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1260242977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T22:55:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:01 smithi137 ceph-mon[90634]: pgmap v524: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1188209081' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T22:55:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1260242977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T22:55:02.402 INFO:tasks.cephadm.ceph_manager.ceph:need seq 498216206401 got 498216206405 for osd.1 2024-06-14T22:55:02.403 DEBUG:teuthology.parallel:result is None 2024-06-14T22:55:03.006 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757523 got 163208757526 for osd.5 2024-06-14T22:55:03.006 DEBUG:teuthology.parallel:result is None 2024-06-14T22:55:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:592705486861 2024-06-14T22:55:03.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234248 got 231928234250 for osd.7 2024-06-14T22:55:03.667 DEBUG:teuthology.parallel:result is None 2024-06-14T22:55:03.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:03 smithi137 ceph-mon[90634]: pgmap v525: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:03.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/802066431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T22:55:04.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:03 smithi086 ceph-mon[97913]: pgmap v525: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:04.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/802066431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T22:55:04.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:03 smithi086 ceph-mon[102754]: pgmap v525: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:04.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/802066431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T22:55:04.364 INFO:tasks.cephadm.ceph_manager.ceph:need seq 592705486858 got 592705486861 for osd.3 2024-06-14T22:55:04.364 DEBUG:teuthology.parallel:result is None 2024-06-14T22:55:04.364 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-14T22:55:04.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:55:04.618 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:05.025 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:55:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:55:04] "GET /metrics HTTP/1.1" 200 34120 "" "Prometheus/2.43.0" 2024-06-14T22:55:05.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:05 smithi137 ceph-mon[90634]: pgmap v526: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:06.008 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:05 smithi086 ceph-mon[97913]: pgmap v526: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:06.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:05 smithi086 ceph-mon[102754]: pgmap v526: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:06.423 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:55:06.423 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:55:07.063 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-06-14T22:55:04.560275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":300,"num_write_kb":2445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3203148,"kb_used_data":14612,"kb_used_omap":29,"kb_used_meta":3188386,"kb_avail":746758068,"statfs":{"total":767960285184,"available":764680261632,"internally_reserved":0,"allocated":14962688,"data_stored":8968614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":3264907941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002918"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279751+0000","last_change":"2024-06-14T22:54:06.279751+0000","last_active":"2024-06-14T22:54:06.279751+0000","last_peered":"2024-06-14T22:54:06.279751+0000","last_clean":"2024-06-14T22:54:06.279751+0000","last_became_active":"2024-06-14T22:54:06.279443+0000","last_became_peered":"2024-06-14T22:54:06.279443+0000","last_unstale":"2024-06-14T22:54:06.279751+0000","last_undegraded":"2024-06-14T22:54:06.279751+0000","last_fullsized":"2024-06-14T22:54:06.279751+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:13:26.144924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278747+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T22:54:05.278747+0000","last_peered":"2024-06-14T22:54:05.278747+0000","last_clean":"2024-06-14T22:54:05.278747+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T22:54:05.278747+0000","last_undegraded":"2024-06-14T22:54:05.278747+0000","last_fullsized":"2024-06-14T22:54:05.278747+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":99,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.274802+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T22:54:06.274802+0000","last_peered":"2024-06-14T22:54:06.274802+0000","last_clean":"2024-06-14T22:54:06.274802+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T22:54:06.274802+0000","last_undegraded":"2024-06-14T22:54:06.274802+0000","last_fullsized":"2024-06-14T22:54:06.274802+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277962+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T22:54:06.277962+0000","last_peered":"2024-06-14T22:54:06.277962+0000","last_clean":"2024-06-14T22:54:06.277962+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T22:54:06.277962+0000","last_undegraded":"2024-06-14T22:54:06.277962+0000","last_fullsized":"2024-06-14T22:54:06.277962+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":89,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.278179+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T22:54:06.278179+0000","last_peered":"2024-06-14T22:54:06.278179+0000","last_clean":"2024-06-14T22:54:06.278179+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T22:54:06.278179+0000","last_undegraded":"2024-06-14T22:54:06.278179+0000","last_fullsized":"2024-06-14T22:54:06.278179+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":83,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278562+0000","last_change":"2024-06-14T22:34:45.813267+0000","last_active":"2024-06-14T22:54:05.278562+0000","last_peered":"2024-06-14T22:54:05.278562+0000","last_clean":"2024-06-14T22:54:05.278562+0000","last_became_active":"2024-06-14T22:34:45.813003+0000","last_became_peered":"2024-06-14T22:34:45.813003+0000","last_unstale":"2024-06-14T22:54:05.278562+0000","last_undegraded":"2024-06-14T22:54:05.278562+0000","last_fullsized":"2024-06-14T22:54:05.278562+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:46.446483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280927+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T22:54:06.280927+0000","last_peered":"2024-06-14T22:54:06.280927+0000","last_clean":"2024-06-14T22:54:06.280927+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T22:54:06.280927+0000","last_undegraded":"2024-06-14T22:54:06.280927+0000","last_fullsized":"2024-06-14T22:54:06.280927+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":108,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280520+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T22:54:05.280520+0000","last_peered":"2024-06-14T22:54:05.280520+0000","last_clean":"2024-06-14T22:54:05.280520+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T22:54:05.280520+0000","last_undegraded":"2024-06-14T22:54:05.280520+0000","last_fullsized":"2024-06-14T22:54:05.280520+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280442+0000","last_change":"2024-06-14T22:44:51.839766+0000","last_active":"2024-06-14T22:54:05.280442+0000","last_peered":"2024-06-14T22:54:05.280442+0000","last_clean":"2024-06-14T22:54:05.280442+0000","last_became_active":"2024-06-14T22:44:51.838970+0000","last_became_peered":"2024-06-14T22:44:51.838970+0000","last_unstale":"2024-06-14T22:54:05.280442+0000","last_undegraded":"2024-06-14T22:54:05.280442+0000","last_fullsized":"2024-06-14T22:54:05.280442+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:31:59.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280352+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:54:05.280352+0000","last_peered":"2024-06-14T22:54:05.280352+0000","last_clean":"2024-06-14T22:54:05.280352+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:54:05.280352+0000","last_undegraded":"2024-06-14T22:54:05.280352+0000","last_fullsized":"2024-06-14T22:54:05.280352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280342+0000","last_change":"2024-06-14T22:49:25.474899+0000","last_active":"2024-06-14T22:54:05.280342+0000","last_peered":"2024-06-14T22:54:05.280342+0000","last_clean":"2024-06-14T22:54:05.280342+0000","last_became_active":"2024-06-14T22:49:25.474251+0000","last_became_peered":"2024-06-14T22:49:25.474251+0000","last_unstale":"2024-06-14T22:54:05.280342+0000","last_undegraded":"2024-06-14T22:54:05.280342+0000","last_fullsized":"2024-06-14T22:54:05.280342+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:22:05.971278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281348+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T22:54:06.281348+0000","last_peered":"2024-06-14T22:54:06.281348+0000","last_clean":"2024-06-14T22:54:06.281348+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T22:54:06.281348+0000","last_undegraded":"2024-06-14T22:54:06.281348+0000","last_fullsized":"2024-06-14T22:54:06.281348+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":84,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279141+0000","last_change":"2024-06-14T22:34:45.810186+0000","last_active":"2024-06-14T22:54:05.279141+0000","last_peered":"2024-06-14T22:54:05.279141+0000","last_clean":"2024-06-14T22:54:05.279141+0000","last_became_active":"2024-06-14T22:34:45.809869+0000","last_became_peered":"2024-06-14T22:34:45.809869+0000","last_unstale":"2024-06-14T22:54:05.279141+0000","last_undegraded":"2024-06-14T22:54:05.279141+0000","last_fullsized":"2024-06-14T22:54:05.279141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:03:45.652442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278987+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T22:54:05.278987+0000","last_peered":"2024-06-14T22:54:05.278987+0000","last_clean":"2024-06-14T22:54:05.278987+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T22:54:05.278987+0000","last_undegraded":"2024-06-14T22:54:05.278987+0000","last_fullsized":"2024-06-14T22:54:05.278987+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280349+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T22:54:06.280349+0000","last_peered":"2024-06-14T22:54:06.280349+0000","last_clean":"2024-06-14T22:54:06.280349+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T22:54:06.280349+0000","last_undegraded":"2024-06-14T22:54:06.280349+0000","last_fullsized":"2024-06-14T22:54:06.280349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.276516+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T22:54:05.276516+0000","last_peered":"2024-06-14T22:54:05.276516+0000","last_clean":"2024-06-14T22:54:05.276516+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T22:54:05.276516+0000","last_undegraded":"2024-06-14T22:54:05.276516+0000","last_fullsized":"2024-06-14T22:54:05.276516+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280994+0000","last_change":"2024-06-14T22:54:06.280994+0000","last_active":"2024-06-14T22:54:06.280994+0000","last_peered":"2024-06-14T22:54:06.280994+0000","last_clean":"2024-06-14T22:54:06.280994+0000","last_became_active":"2024-06-14T22:54:06.279883+0000","last_became_peered":"2024-06-14T22:54:06.279883+0000","last_unstale":"2024-06-14T22:54:06.280994+0000","last_undegraded":"2024-06-14T22:54:06.280994+0000","last_fullsized":"2024-06-14T22:54:06.280994+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:51.156630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":99,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280497+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T22:54:06.280497+0000","last_peered":"2024-06-14T22:54:06.280497+0000","last_clean":"2024-06-14T22:54:06.280497+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T22:54:06.280497+0000","last_undegraded":"2024-06-14T22:54:06.280497+0000","last_fullsized":"2024-06-14T22:54:06.280497+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279157+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T22:54:05.279157+0000","last_peered":"2024-06-14T22:54:05.279157+0000","last_clean":"2024-06-14T22:54:05.279157+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T22:54:05.279157+0000","last_undegraded":"2024-06-14T22:54:05.279157+0000","last_fullsized":"2024-06-14T22:54:05.279157+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":109,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275417+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T22:54:05.275417+0000","last_peered":"2024-06-14T22:54:05.275417+0000","last_clean":"2024-06-14T22:54:05.275417+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T22:54:05.275417+0000","last_undegraded":"2024-06-14T22:54:05.275417+0000","last_fullsized":"2024-06-14T22:54:05.275417+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275776+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T22:54:05.275776+0000","last_peered":"2024-06-14T22:54:05.275776+0000","last_clean":"2024-06-14T22:54:05.275776+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T22:54:05.275776+0000","last_undegraded":"2024-06-14T22:54:05.275776+0000","last_fullsized":"2024-06-14T22:54:05.275776+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275483+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T22:54:05.275483+0000","last_peered":"2024-06-14T22:54:05.275483+0000","last_clean":"2024-06-14T22:54:05.275483+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T22:54:05.275483+0000","last_undegraded":"2024-06-14T22:54:05.275483+0000","last_fullsized":"2024-06-14T22:54:05.275483+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":91,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281907+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T22:54:06.281907+0000","last_peered":"2024-06-14T22:54:06.281907+0000","last_clean":"2024-06-14T22:54:06.281907+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T22:54:06.281907+0000","last_undegraded":"2024-06-14T22:54:06.281907+0000","last_fullsized":"2024-06-14T22:54:06.281907+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282328+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T22:54:06.282328+0000","last_peered":"2024-06-14T22:54:06.282328+0000","last_clean":"2024-06-14T22:54:06.282328+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T22:54:06.282328+0000","last_undegraded":"2024-06-14T22:54:06.282328+0000","last_fullsized":"2024-06-14T22:54:06.282328+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":480,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279291+0000","last_change":"2024-06-14T22:44:51.840491+0000","last_active":"2024-06-14T22:54:05.279291+0000","last_peered":"2024-06-14T22:54:05.279291+0000","last_clean":"2024-06-14T22:54:05.279291+0000","last_became_active":"2024-06-14T22:44:51.839990+0000","last_became_peered":"2024-06-14T22:44:51.839990+0000","last_unstale":"2024-06-14T22:54:05.279291+0000","last_undegraded":"2024-06-14T22:54:05.279291+0000","last_fullsized":"2024-06-14T22:54:05.279291+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:51:24.139036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281690+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T22:54:05.281690+0000","last_peered":"2024-06-14T22:54:05.281690+0000","last_clean":"2024-06-14T22:54:05.281690+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T22:54:05.281690+0000","last_undegraded":"2024-06-14T22:54:05.281690+0000","last_fullsized":"2024-06-14T22:54:05.281690+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":64,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279205+0000","last_change":"2024-06-14T22:54:06.279205+0000","last_active":"2024-06-14T22:54:06.279205+0000","last_peered":"2024-06-14T22:54:06.279205+0000","last_clean":"2024-06-14T22:54:06.279205+0000","last_became_active":"2024-06-14T22:54:06.278792+0000","last_became_peered":"2024-06-14T22:54:06.278792+0000","last_unstale":"2024-06-14T22:54:06.279205+0000","last_undegraded":"2024-06-14T22:54:06.279205+0000","last_fullsized":"2024-06-14T22:54:06.279205+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:55:02.899941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279434+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:54:05.279434+0000","last_peered":"2024-06-14T22:54:05.279434+0000","last_clean":"2024-06-14T22:54:05.279434+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:54:05.279434+0000","last_undegraded":"2024-06-14T22:54:05.279434+0000","last_fullsized":"2024-06-14T22:54:05.279434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":121,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279201+0000","last_change":"2024-06-14T22:49:25.476578+0000","last_active":"2024-06-14T22:54:05.279201+0000","last_peered":"2024-06-14T22:54:05.279201+0000","last_clean":"2024-06-14T22:54:05.279201+0000","last_became_active":"2024-06-14T22:49:25.476183+0000","last_became_peered":"2024-06-14T22:49:25.476183+0000","last_unstale":"2024-06-14T22:54:05.279201+0000","last_undegraded":"2024-06-14T22:54:05.279201+0000","last_fullsized":"2024-06-14T22:54:05.279201+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:01:44.585320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277248+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T22:54:06.277248+0000","last_peered":"2024-06-14T22:54:06.277248+0000","last_clean":"2024-06-14T22:54:06.277248+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T22:54:06.277248+0000","last_undegraded":"2024-06-14T22:54:06.277248+0000","last_fullsized":"2024-06-14T22:54:06.277248+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278487+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T22:54:05.278487+0000","last_peered":"2024-06-14T22:54:05.278487+0000","last_clean":"2024-06-14T22:54:05.278487+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T22:54:05.278487+0000","last_undegraded":"2024-06-14T22:54:05.278487+0000","last_fullsized":"2024-06-14T22:54:05.278487+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.275775+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T22:54:06.275775+0000","last_peered":"2024-06-14T22:54:06.275775+0000","last_clean":"2024-06-14T22:54:06.275775+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T22:54:06.275775+0000","last_undegraded":"2024-06-14T22:54:06.275775+0000","last_fullsized":"2024-06-14T22:54:06.275775+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270774+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T22:54:05.270774+0000","last_peered":"2024-06-14T22:54:05.270774+0000","last_clean":"2024-06-14T22:54:05.270774+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T22:54:05.270774+0000","last_undegraded":"2024-06-14T22:54:05.270774+0000","last_fullsized":"2024-06-14T22:54:05.270774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270053+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T22:54:05.270053+0000","last_peered":"2024-06-14T22:54:05.270053+0000","last_clean":"2024-06-14T22:54:05.270053+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T22:54:05.270053+0000","last_undegraded":"2024-06-14T22:54:05.270053+0000","last_fullsized":"2024-06-14T22:54:05.270053+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281419+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T22:54:05.281419+0000","last_peered":"2024-06-14T22:54:05.281419+0000","last_clean":"2024-06-14T22:54:05.281419+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T22:54:05.281419+0000","last_undegraded":"2024-06-14T22:54:05.281419+0000","last_fullsized":"2024-06-14T22:54:05.281419+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275641+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T22:54:05.275641+0000","last_peered":"2024-06-14T22:54:05.275641+0000","last_clean":"2024-06-14T22:54:05.275641+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T22:54:05.275641+0000","last_undegraded":"2024-06-14T22:54:05.275641+0000","last_fullsized":"2024-06-14T22:54:05.275641+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":119,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280880+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T22:54:06.280880+0000","last_peered":"2024-06-14T22:54:06.280880+0000","last_clean":"2024-06-14T22:54:06.280880+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T22:54:06.280880+0000","last_undegraded":"2024-06-14T22:54:06.280880+0000","last_fullsized":"2024-06-14T22:54:06.280880+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":117,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280346+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T22:54:06.280346+0000","last_peered":"2024-06-14T22:54:06.280346+0000","last_clean":"2024-06-14T22:54:06.280346+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T22:54:06.280346+0000","last_undegraded":"2024-06-14T22:54:06.280346+0000","last_fullsized":"2024-06-14T22:54:06.280346+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270215+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T22:54:05.270215+0000","last_peered":"2024-06-14T22:54:05.270215+0000","last_clean":"2024-06-14T22:54:05.270215+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T22:54:05.270215+0000","last_undegraded":"2024-06-14T22:54:05.270215+0000","last_fullsized":"2024-06-14T22:54:05.270215+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270541+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T22:54:05.270541+0000","last_peered":"2024-06-14T22:54:05.270541+0000","last_clean":"2024-06-14T22:54:05.270541+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T22:54:05.270541+0000","last_undegraded":"2024-06-14T22:54:05.270541+0000","last_fullsized":"2024-06-14T22:54:05.270541+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270139+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T22:54:05.270139+0000","last_peered":"2024-06-14T22:54:05.270139+0000","last_clean":"2024-06-14T22:54:05.270139+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T22:54:05.270139+0000","last_undegraded":"2024-06-14T22:54:05.270139+0000","last_fullsized":"2024-06-14T22:54:05.270139+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279227+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T22:54:05.279227+0000","last_peered":"2024-06-14T22:54:05.279227+0000","last_clean":"2024-06-14T22:54:05.279227+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T22:54:05.279227+0000","last_undegraded":"2024-06-14T22:54:05.279227+0000","last_fullsized":"2024-06-14T22:54:05.279227+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":74,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.276447+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T22:54:05.276447+0000","last_peered":"2024-06-14T22:54:05.276447+0000","last_clean":"2024-06-14T22:54:05.276447+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T22:54:05.276447+0000","last_undegraded":"2024-06-14T22:54:05.276447+0000","last_fullsized":"2024-06-14T22:54:05.276447+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.284651+0000","last_change":"2024-06-14T22:54:06.284651+0000","last_active":"2024-06-14T22:54:06.284651+0000","last_peered":"2024-06-14T22:54:06.284651+0000","last_clean":"2024-06-14T22:54:06.284651+0000","last_became_active":"2024-06-14T22:54:06.283564+0000","last_became_peered":"2024-06-14T22:54:06.283564+0000","last_unstale":"2024-06-14T22:54:06.284651+0000","last_undegraded":"2024-06-14T22:54:06.284651+0000","last_fullsized":"2024-06-14T22:54:06.284651+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:34.916272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270118+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T22:54:05.270118+0000","last_peered":"2024-06-14T22:54:05.270118+0000","last_clean":"2024-06-14T22:54:05.270118+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T22:54:05.270118+0000","last_undegraded":"2024-06-14T22:54:05.270118+0000","last_fullsized":"2024-06-14T22:54:05.270118+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281419+0000","last_change":"2024-06-14T22:54:06.281419+0000","last_active":"2024-06-14T22:54:06.281419+0000","last_peered":"2024-06-14T22:54:06.281419+0000","last_clean":"2024-06-14T22:54:06.281419+0000","last_became_active":"2024-06-14T22:54:06.279482+0000","last_became_peered":"2024-06-14T22:54:06.279482+0000","last_unstale":"2024-06-14T22:54:06.281419+0000","last_undegraded":"2024-06-14T22:54:06.281419+0000","last_fullsized":"2024-06-14T22:54:06.281419+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:06:37.039006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.277756+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T22:54:05.277756+0000","last_peered":"2024-06-14T22:54:05.277756+0000","last_clean":"2024-06-14T22:54:05.277756+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T22:54:05.277756+0000","last_undegraded":"2024-06-14T22:54:05.277756+0000","last_fullsized":"2024-06-14T22:54:05.277756+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280283+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T22:54:05.280283+0000","last_peered":"2024-06-14T22:54:05.280283+0000","last_clean":"2024-06-14T22:54:05.280283+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T22:54:05.280283+0000","last_undegraded":"2024-06-14T22:54:05.280283+0000","last_fullsized":"2024-06-14T22:54:05.280283+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275645+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:54:05.275645+0000","last_peered":"2024-06-14T22:54:05.275645+0000","last_clean":"2024-06-14T22:54:05.275645+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:54:05.275645+0000","last_undegraded":"2024-06-14T22:54:05.275645+0000","last_fullsized":"2024-06-14T22:54:05.275645+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275589+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T22:54:05.275589+0000","last_peered":"2024-06-14T22:54:05.275589+0000","last_clean":"2024-06-14T22:54:05.275589+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T22:54:05.275589+0000","last_undegraded":"2024-06-14T22:54:05.275589+0000","last_fullsized":"2024-06-14T22:54:05.275589+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":105,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275170+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:54:05.275170+0000","last_peered":"2024-06-14T22:54:05.275170+0000","last_clean":"2024-06-14T22:54:05.275170+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:54:05.275170+0000","last_undegraded":"2024-06-14T22:54:05.275170+0000","last_fullsized":"2024-06-14T22:54:05.275170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270411+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T22:54:05.270411+0000","last_peered":"2024-06-14T22:54:05.270411+0000","last_clean":"2024-06-14T22:54:05.270411+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T22:54:05.270411+0000","last_undegraded":"2024-06-14T22:54:05.270411+0000","last_fullsized":"2024-06-14T22:54:05.270411+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.275561+0000","last_change":"2024-06-14T22:54:06.275561+0000","last_active":"2024-06-14T22:54:06.275561+0000","last_peered":"2024-06-14T22:54:06.275561+0000","last_clean":"2024-06-14T22:54:06.275561+0000","last_became_active":"2024-06-14T22:54:06.274930+0000","last_became_peered":"2024-06-14T22:54:06.274930+0000","last_unstale":"2024-06-14T22:54:06.275561+0000","last_undegraded":"2024-06-14T22:54:06.275561+0000","last_fullsized":"2024-06-14T22:54:06.275561+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:11.022230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281532+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T22:54:05.281532+0000","last_peered":"2024-06-14T22:54:05.281532+0000","last_clean":"2024-06-14T22:54:05.281532+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T22:54:05.281532+0000","last_undegraded":"2024-06-14T22:54:05.281532+0000","last_fullsized":"2024-06-14T22:54:05.281532+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":64,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279332+0000","last_change":"2024-06-14T22:54:06.279332+0000","last_active":"2024-06-14T22:54:06.279332+0000","last_peered":"2024-06-14T22:54:06.279332+0000","last_clean":"2024-06-14T22:54:06.279332+0000","last_became_active":"2024-06-14T22:54:06.279020+0000","last_became_peered":"2024-06-14T22:54:06.279020+0000","last_unstale":"2024-06-14T22:54:06.279332+0000","last_undegraded":"2024-06-14T22:54:06.279332+0000","last_fullsized":"2024-06-14T22:54:06.279332+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:52:57.777668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.285313+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T22:54:06.285313+0000","last_peered":"2024-06-14T22:54:06.285313+0000","last_clean":"2024-06-14T22:54:06.285313+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T22:54:06.285313+0000","last_undegraded":"2024-06-14T22:54:06.285313+0000","last_fullsized":"2024-06-14T22:54:06.285313+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281615+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T22:54:06.281615+0000","last_peered":"2024-06-14T22:54:06.281615+0000","last_clean":"2024-06-14T22:54:06.281615+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T22:54:06.281615+0000","last_undegraded":"2024-06-14T22:54:06.281615+0000","last_fullsized":"2024-06-14T22:54:06.281615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280203+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T22:54:05.280203+0000","last_peered":"2024-06-14T22:54:05.280203+0000","last_clean":"2024-06-14T22:54:05.280203+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T22:54:05.280203+0000","last_undegraded":"2024-06-14T22:54:05.280203+0000","last_fullsized":"2024-06-14T22:54:05.280203+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":108,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282120+0000","last_change":"2024-06-14T22:54:06.282120+0000","last_active":"2024-06-14T22:54:06.282120+0000","last_peered":"2024-06-14T22:54:06.282120+0000","last_clean":"2024-06-14T22:54:06.282120+0000","last_became_active":"2024-06-14T22:54:06.280898+0000","last_became_peered":"2024-06-14T22:54:06.280898+0000","last_unstale":"2024-06-14T22:54:06.282120+0000","last_undegraded":"2024-06-14T22:54:06.282120+0000","last_fullsized":"2024-06-14T22:54:06.282120+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:30:45.544457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.278300+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T22:54:06.278300+0000","last_peered":"2024-06-14T22:54:06.278300+0000","last_clean":"2024-06-14T22:54:06.278300+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T22:54:06.278300+0000","last_undegraded":"2024-06-14T22:54:06.278300+0000","last_fullsized":"2024-06-14T22:54:06.278300+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.284728+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T22:54:06.284728+0000","last_peered":"2024-06-14T22:54:06.284728+0000","last_clean":"2024-06-14T22:54:06.284728+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T22:54:06.284728+0000","last_undegraded":"2024-06-14T22:54:06.284728+0000","last_fullsized":"2024-06-14T22:54:06.284728+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280483+0000","last_change":"2024-06-14T22:54:06.280483+0000","last_active":"2024-06-14T22:54:06.280483+0000","last_peered":"2024-06-14T22:54:06.280483+0000","last_clean":"2024-06-14T22:54:06.280483+0000","last_became_active":"2024-06-14T22:54:06.278985+0000","last_became_peered":"2024-06-14T22:54:06.278985+0000","last_unstale":"2024-06-14T22:54:06.280483+0000","last_undegraded":"2024-06-14T22:54:06.280483+0000","last_fullsized":"2024-06-14T22:54:06.280483+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:43:40.478580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":118,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.274772+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T22:54:05.274772+0000","last_peered":"2024-06-14T22:54:05.274772+0000","last_clean":"2024-06-14T22:54:05.274772+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T22:54:05.274772+0000","last_undegraded":"2024-06-14T22:54:05.274772+0000","last_fullsized":"2024-06-14T22:54:05.274772+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":109,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277110+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T22:54:06.277110+0000","last_peered":"2024-06-14T22:54:06.277110+0000","last_clean":"2024-06-14T22:54:06.277110+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T22:54:06.277110+0000","last_undegraded":"2024-06-14T22:54:06.277110+0000","last_fullsized":"2024-06-14T22:54:06.277110+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":77,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.270191+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T22:54:06.270191+0000","last_peered":"2024-06-14T22:54:06.270191+0000","last_clean":"2024-06-14T22:54:06.270191+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T22:54:06.270191+0000","last_undegraded":"2024-06-14T22:54:06.270191+0000","last_fullsized":"2024-06-14T22:54:06.270191+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281117+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T22:54:06.281117+0000","last_peered":"2024-06-14T22:54:06.281117+0000","last_clean":"2024-06-14T22:54:06.281117+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T22:54:06.281117+0000","last_undegraded":"2024-06-14T22:54:06.281117+0000","last_fullsized":"2024-06-14T22:54:06.281117+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280162+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T22:54:05.280162+0000","last_peered":"2024-06-14T22:54:05.280162+0000","last_clean":"2024-06-14T22:54:05.280162+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T22:54:05.280162+0000","last_undegraded":"2024-06-14T22:54:05.280162+0000","last_fullsized":"2024-06-14T22:54:05.280162+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280576+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T22:54:06.280576+0000","last_peered":"2024-06-14T22:54:06.280576+0000","last_clean":"2024-06-14T22:54:06.280576+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T22:54:06.280576+0000","last_undegraded":"2024-06-14T22:54:06.280576+0000","last_fullsized":"2024-06-14T22:54:06.280576+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":85,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.282162+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:54:05.282162+0000","last_peered":"2024-06-14T22:54:05.282162+0000","last_clean":"2024-06-14T22:54:05.282162+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:54:05.282162+0000","last_undegraded":"2024-06-14T22:54:05.282162+0000","last_fullsized":"2024-06-14T22:54:05.282162+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":123,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282972+0000","last_change":"2024-06-14T22:54:06.282972+0000","last_active":"2024-06-14T22:54:06.282972+0000","last_peered":"2024-06-14T22:54:06.282972+0000","last_clean":"2024-06-14T22:54:06.282972+0000","last_became_active":"2024-06-14T22:54:06.282622+0000","last_became_peered":"2024-06-14T22:54:06.282622+0000","last_unstale":"2024-06-14T22:54:06.282972+0000","last_undegraded":"2024-06-14T22:54:06.282972+0000","last_fullsized":"2024-06-14T22:54:06.282972+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:31:22.051955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":108,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279284+0000","last_change":"2024-06-14T22:54:06.279284+0000","last_active":"2024-06-14T22:54:06.279284+0000","last_peered":"2024-06-14T22:54:06.279284+0000","last_clean":"2024-06-14T22:54:06.279284+0000","last_became_active":"2024-06-14T22:54:06.278861+0000","last_became_peered":"2024-06-14T22:54:06.278861+0000","last_unstale":"2024-06-14T22:54:06.279284+0000","last_undegraded":"2024-06-14T22:54:06.279284+0000","last_fullsized":"2024-06-14T22:54:06.279284+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:17:47.260769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.273550+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T22:54:05.273550+0000","last_peered":"2024-06-14T22:54:05.273550+0000","last_clean":"2024-06-14T22:54:05.273550+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T22:54:05.273550+0000","last_undegraded":"2024-06-14T22:54:05.273550+0000","last_fullsized":"2024-06-14T22:54:05.273550+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275021+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T22:54:05.275021+0000","last_peered":"2024-06-14T22:54:05.275021+0000","last_clean":"2024-06-14T22:54:05.275021+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T22:54:05.275021+0000","last_undegraded":"2024-06-14T22:54:05.275021+0000","last_fullsized":"2024-06-14T22:54:05.275021+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234252,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600240,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":93144912,"statfs":{"total":95995035648,"available":95380389888,"internally_reserved":0,"allocated":2260992,"data_stored":1493426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.0780000000000001,"15min":1.167},"min":{"1min":0.28399999999999997,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.69199999999999995,"5min":9.3919999999999995,"15min":9.3919999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":1.353,"15min":1.2989999999999999},"min":{"1min":0.29999999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.74199999999999999,"5min":10.416,"15min":10.416},"last":0.60999999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.212,"15min":1.3700000000000001},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80500000000000005,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":1.3560000000000001,"15min":1.464},"min":{"1min":0.38100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.59799999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.5509999999999999,"15min":1.4099999999999999},"min":{"1min":0.39500000000000002,"5min":0.30399999999999999,"15min":0.26100000000000001},"max":{"1min":3.129,"5min":7.4249999999999998,"15min":10.199},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.254,"15min":1.4450000000000001},"min":{"1min":0.41199999999999998,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":2.8490000000000002,"5min":7.032,"15min":7.4610000000000003},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.67000000000000004,"15min":0.80400000000000005},"min":{"1min":0.27300000000000002,"5min":0.248,"15min":0.214},"max":{"1min":0.67900000000000005,"5min":5.8449999999999998,"15min":9.7430000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.33900000000000002,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":0.64500000000000002,"5min":4.2709999999999999,"15min":6.6660000000000004},"last":0.38}]},{"osd":5,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.752,"15min":0.77200000000000002},"min":{"1min":0.34999999999999998,"5min":0.26100000000000001,"15min":0.22},"max":{"1min":0.70599999999999996,"5min":3.9860000000000002,"15min":7.9870000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.88,"15min":0.88600000000000001},"min":{"1min":0.33500000000000002,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.76200000000000001,"5min":5.3659999999999997,"15min":9.5120000000000005},"last":0.40999999999999998}]},{"osd":6,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.75800000000000001,"15min":0.81999999999999995},"min":{"1min":0.39400000000000002,"5min":0.308,"15min":0.28100000000000003},"max":{"1min":0.72499999999999998,"5min":5.806,"15min":5.806},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.76700000000000002,"15min":0.71999999999999997},"min":{"1min":0.39000000000000001,"5min":0.311,"15min":0.29899999999999999},"max":{"1min":0.70299999999999996,"5min":4.3090000000000002,"15min":5.7430000000000003},"last":0.42999999999999999}]}]},{"osd":6,"up_from":46,"seq":197568495888,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311836,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":93433316,"statfs":{"total":95995035648,"available":95675715584,"internally_reserved":0,"allocated":2244608,"data_stored":1492957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":1.5880000000000001,"15min":1.2050000000000001},"min":{"1min":0.39800000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.72099999999999997,"5min":31.358000000000001,"15min":31.358000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.417,"15min":1.458},"min":{"1min":0.36199999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.69899999999999995,"5min":29.399000000000001,"15min":29.399000000000001},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.224,"15min":1.304},"min":{"1min":0.39400000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73399999999999999,"5min":9.9559999999999995,"15min":9.9559999999999995},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":1.2769999999999999,"15min":1.276},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.67400000000000004,"5min":9.5510000000000002,"15min":9.5510000000000002},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:53:14 2024","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":1.7669999999999999,"15min":1.607},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":6.9219999999999997,"5min":28.693999999999999,"15min":28.693999999999999},"last":0.505},{"interface":"front","average":{"1min":1.649,"5min":1.6839999999999999,"15min":1.5960000000000001},"min":{"1min":0.35099999999999998,"5min":0.27900000000000003,"15min":0.25700000000000001},"max":{"1min":8.7200000000000006,"5min":28.300000000000001,"15min":28.300000000000001},"last":0.52900000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":1.2110000000000001,"15min":0.88800000000000001},"min":{"1min":0.35299999999999998,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":0.71999999999999997,"5min":47.698,"15min":47.698},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.496,"5min":1.181,"15min":0.82599999999999996},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.23899999999999999},"max":{"1min":0.72699999999999998,"5min":48.197000000000003,"15min":48.197000000000003},"last":0.41699999999999998}]},{"osd":5,"last update":"Fri Jun 14 22:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.88600000000000001,"15min":0.79100000000000004},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.22600000000000001},"max":{"1min":0.75800000000000001,"5min":8.6500000000000004,"15min":8.6500000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.95399999999999996,"15min":0.86199999999999999},"min":{"1min":0.41199999999999998,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":8.6349999999999998,"15min":9.5609999999999999},"last":0.39600000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:54:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.84199999999999997,"15min":0.82099999999999995},"min":{"1min":0.38100000000000001,"5min":0.31,"15min":0.28799999999999998},"max":{"1min":0.82299999999999995,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.90500000000000003,"15min":0.83299999999999996},"min":{"1min":0.44500000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78100000000000003,"5min":8.673,"15min":8.673},"last":0.57899999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757527,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896852,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":92848300,"statfs":{"total":95995035648,"available":95076659200,"internally_reserved":0,"allocated":1642496,"data_stored":898403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":1.1299999999999999,"15min":1.2270000000000001},"min":{"1min":0.41099999999999998,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.69899999999999995,"5min":6.0369999999999999,"15min":17.766999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":1.046,"15min":1.2689999999999999},"min":{"1min":0.38500000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70999999999999996,"5min":5.2439999999999998,"15min":17.806000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.2809999999999999,"15min":1.357},"min":{"1min":0.36099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66400000000000003,"5min":7.694,"15min":7.694},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.501,"5min":1.129,"15min":1.169},"min":{"1min":0.33700000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.72599999999999998,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:52:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8660000000000001,"5min":1.393,"15min":1.5449999999999999},"min":{"1min":0.379,"5min":0.25700000000000001,"15min":0.23999999999999999},"max":{"1min":6.0229999999999997,"5min":7.7599999999999998,"15min":10.128},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.484,"15min":1.5549999999999999},"min":{"1min":0.36699999999999999,"5min":0.28299999999999997,"15min":0.27300000000000002},"max":{"1min":5.8550000000000004,"5min":5.8550000000000004,"15min":9.048},"last":0.61699999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.60599999999999998,"15min":0.73599999999999999},"min":{"1min":0.313,"5min":0.23300000000000001,"15min":0.214},"max":{"1min":0.64400000000000002,"5min":4.9790000000000001,"15min":11.771000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.69699999999999995,"15min":0.74299999999999999},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.23200000000000001},"max":{"1min":0.66100000000000003,"5min":5.8109999999999999,"15min":10.032},"last":0.33100000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.84699999999999998,"15min":0.86599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.23000000000000001},"max":{"1min":0.67900000000000005,"5min":9.6020000000000003,"15min":9.8369999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.85699999999999998,"15min":0.88300000000000001},"min":{"1min":0.33300000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.74399999999999999,"5min":12.339,"15min":12.339},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.75800000000000001,"15min":0.82299999999999995},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":5.2629999999999999,"15min":10.462999999999999},"last":0.501},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.72999999999999998,"15min":0.83899999999999997},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69299999999999995,"5min":10.638999999999999,"15min":10.638999999999999},"last":0.51100000000000001}]}]},{"osd":4,"up_from":30,"seq":128849019163,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599652,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":93145500,"statfs":{"total":95995035648,"available":95380992000,"internally_reserved":0,"allocated":1658880,"data_stored":902723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.0640000000000001,"15min":1.0609999999999999},"min":{"1min":0.39500000000000002,"5min":0.219,"15min":0.218},"max":{"1min":0.70199999999999996,"5min":4.641,"15min":7.5650000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":1.1499999999999999,"15min":1.365},"min":{"1min":0.39800000000000002,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":0.65400000000000003,"5min":5.7569999999999997,"15min":9.1470000000000002},"last":0.47099999999999997}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":1.1779999999999999,"15min":1.1499999999999999},"min":{"1min":0.34899999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65300000000000002,"5min":5.141,"15min":5.141},"last":0.441},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.2030000000000001,"15min":1.3360000000000001},"min":{"1min":0.35299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.70799999999999996,"5min":6.2709999999999999,"15min":6.2709999999999999},"last":0.55200000000000005}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Jun 14 22:53:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.2150000000000001,"15min":1.4339999999999999},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.24299999999999999},"max":{"1min":5.2830000000000004,"5min":6.2460000000000004,"15min":9.2750000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.764,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.317,"5min":0.27800000000000002,"15min":0.27300000000000002},"max":{"1min":5.6859999999999999,"5min":10.249000000000001,"15min":10.249000000000001},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:54:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.72599999999999998,"15min":0.749},"min":{"1min":0.31,"5min":0.24199999999999999,"15min":0.21199999999999999},"max":{"1min":0.67900000000000005,"5min":3.964,"15min":5.774},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":0.78200000000000003,"5min":4.1269999999999998,"15min":4.7480000000000002},"last":0.378}]},{"osd":6,"last update":"Fri Jun 14 22:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.65200000000000002,"15min":0.78100000000000003},"min":{"1min":0.34399999999999997,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":3.8929999999999998,"15min":10.904999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61499999999999999,"15min":0.77200000000000002},"min":{"1min":0.371,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":0.64600000000000002,"5min":3.7450000000000001,"15min":9.2390000000000008},"last":0.48999999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:54:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.63,"15min":0.752},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.246},"max":{"1min":0.72399999999999998,"5min":3.7029999999999998,"15min":6.298},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.74299999999999999,"15min":0.76400000000000001},"min":{"1min":0.38300000000000001,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.73499999999999999,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":0.56799999999999995}]}]},{"osd":3,"up_from":138,"seq":592705486862,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77328,"kb_used_data":1604,"kb_used_omap":4,"kb_used_meta":75707,"kb_avail":93667824,"statfs":{"total":95995035648,"available":95915851776,"internally_reserved":0,"allocated":1642496,"data_stored":898176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":77524446},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":2,"up_from":133,"seq":571230650388,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62544,"kb_used_data":1604,"kb_used_omap":6,"kb_used_meta":60921,"kb_avail":93682608,"statfs":{"total":95995035648,"available":95930990592,"internally_reserved":0,"allocated":1642496,"data_stored":898390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":62383605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.34899999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.59099999999999997}]},{"osd":6,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.505}]},{"osd":7,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.55000000000000004}]}]},{"osd":1,"up_from":116,"seq":498216206406,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":213204,"kb_used_data":1600,"kb_used_omap":5,"kb_used_meta":211578,"kb_avail":93531948,"statfs":{"total":95995035648,"available":95776714752,"internally_reserved":0,"allocated":1638400,"data_stored":898086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":216656152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":1.8759999999999999,"15min":2.3620000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41299999999999998,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.4159999999999999,"15min":1.746},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.433,"5min":10.426,"15min":10.426},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Jun 14 22:53:32 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":2.073,"15min":2.2200000000000002},"min":{"1min":0.317,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.3860000000000001,"5min":20.297999999999998,"15min":20.297999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.5409999999999999,"5min":2.427,"15min":2.4910000000000001},"min":{"1min":0.32100000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.8630000000000004,"5min":20.260000000000002,"15min":20.260000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.5229999999999999,"15min":2.0209999999999999},"min":{"1min":0.41799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.95399999999999996,"5min":8.8599999999999994,"15min":8.8599999999999994},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.625,"15min":1.9490000000000001},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88600000000000001,"5min":11.161,"15min":11.161},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.6719999999999999,"15min":1.8420000000000001},"min":{"1min":0.40799999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":1.52,"15min":2.0760000000000001},"min":{"1min":0.438,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.97899999999999998,"5min":8.125,"15min":8.125},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":1.7749999999999999,"15min":1.929},"min":{"1min":0.39700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89700000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":1.4870000000000001,"15min":2.0750000000000002},"min":{"1min":0.44,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.92300000000000004,"5min":8.1479999999999997,"15min":8.1479999999999997},"last":0.51300000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.589,"15min":2.089},"min":{"1min":0.45900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79900000000000004,"5min":9.0960000000000001,"15min":9.0960000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.621,"5min":1.5600000000000001,"15min":1.702},"min":{"1min":0.48199999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96499999999999997,"5min":9.5449999999999999,"15min":9.5449999999999999},"last":0.55800000000000005}]}]},{"osd":0,"up_from":91,"seq":390842024060,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":441492,"kb_used_data":2180,"kb_used_omap":5,"kb_used_meta":439290,"kb_avail":93303660,"statfs":{"total":95995035648,"available":95542947840,"internally_reserved":0,"allocated":2232320,"data_stored":1486453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":449833050},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:54:27 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.7989999999999999,"15min":1.609},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.44600000000000001,"5min":9.6999999999999993,"15min":9.6999999999999993},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":1.895,"15min":2.2869999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.58199999999999996,"5min":10.613,"15min":10.613},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:54:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.44},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.49299999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:53:08 2024","interfaces":[{"interface":"back","average":{"1min":2.7589999999999999,"5min":2.4350000000000001,"15min":2.411},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":8.5199999999999996,"5min":13.749000000000001,"15min":13.749000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":2.4199999999999999,"5min":2.3079999999999998,"15min":2.1549999999999998},"min":{"1min":0.32300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.0339999999999998,"5min":9.8109999999999999,"15min":12.768000000000001},"last":0.47399999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.427,"15min":1.6850000000000001},"min":{"1min":0.40000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.83999999999999997,"5min":8.7889999999999997,"15min":8.9510000000000005},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.2929999999999999,"15min":1.3460000000000001},"min":{"1min":0.436,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.87,"5min":13.574,"15min":13.574},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.3999999999999999,"15min":1.391},"min":{"1min":0.41399999999999998,"5min":0.317,"15min":0.29299999999999998},"max":{"1min":0.84099999999999997,"5min":13.616,"15min":13.616},"last":0.748},{"interface":"front","average":{"1min":0.64700000000000002,"5min":1.3620000000000001,"15min":1.4850000000000001},"min":{"1min":0.45300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.0409999999999999,"5min":7.6509999999999998,"15min":9.6620000000000008},"last":0.68100000000000005}]},{"osd":6,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":1.4039999999999999,"15min":1.421},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81699999999999995,"5min":13.645,"15min":13.645},"last":0.627},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.248,"15min":1.5780000000000001},"min":{"1min":0.44800000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.022,"5min":7.0119999999999996,"15min":9.8599999999999994},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.2509999999999999,"15min":1.617},"min":{"1min":0.42399999999999999,"5min":0.38700000000000001,"15min":0.33100000000000002},"max":{"1min":1.0009999999999999,"5min":6.9870000000000001,"15min":9.6370000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":1.4339999999999999,"15min":1.7989999999999999},"min":{"1min":0.44400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.84799999999999998,"5min":8.8290000000000006,"15min":8.9900000000000002},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:55:07.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:55:07.320 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:07.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:07 smithi086 ceph-mon[97913]: from='client.45428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:07.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:07 smithi086 ceph-mon[97913]: pgmap v527: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:07.740 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:07 smithi086 ceph-mon[102754]: from='client.45428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:07.740 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:07 smithi086 ceph-mon[102754]: pgmap v527: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:07 smithi137 ceph-mon[90634]: from='client.45428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:07 smithi137 ceph-mon[90634]: pgmap v527: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:09.245 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:55:09.246 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:55:09.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:09 smithi086 ceph-mon[97913]: pgmap v528: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:09.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:09 smithi086 ceph-mon[102754]: pgmap v528: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:09.885 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-06-14T22:55:08.561160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":300,"num_write_kb":2445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3203148,"kb_used_data":14612,"kb_used_omap":29,"kb_used_meta":3188386,"kb_avail":746758068,"statfs":{"total":767960285184,"available":764680261632,"internally_reserved":0,"allocated":14962688,"data_stored":8968614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":3264907941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002660"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279751+0000","last_change":"2024-06-14T22:54:06.279751+0000","last_active":"2024-06-14T22:54:06.279751+0000","last_peered":"2024-06-14T22:54:06.279751+0000","last_clean":"2024-06-14T22:54:06.279751+0000","last_became_active":"2024-06-14T22:54:06.279443+0000","last_became_peered":"2024-06-14T22:54:06.279443+0000","last_unstale":"2024-06-14T22:54:06.279751+0000","last_undegraded":"2024-06-14T22:54:06.279751+0000","last_fullsized":"2024-06-14T22:54:06.279751+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:13:26.144924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278747+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T22:54:05.278747+0000","last_peered":"2024-06-14T22:54:05.278747+0000","last_clean":"2024-06-14T22:54:05.278747+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T22:54:05.278747+0000","last_undegraded":"2024-06-14T22:54:05.278747+0000","last_fullsized":"2024-06-14T22:54:05.278747+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":99,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.274802+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T22:54:06.274802+0000","last_peered":"2024-06-14T22:54:06.274802+0000","last_clean":"2024-06-14T22:54:06.274802+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T22:54:06.274802+0000","last_undegraded":"2024-06-14T22:54:06.274802+0000","last_fullsized":"2024-06-14T22:54:06.274802+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277962+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T22:54:06.277962+0000","last_peered":"2024-06-14T22:54:06.277962+0000","last_clean":"2024-06-14T22:54:06.277962+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T22:54:06.277962+0000","last_undegraded":"2024-06-14T22:54:06.277962+0000","last_fullsized":"2024-06-14T22:54:06.277962+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":89,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.278179+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T22:54:06.278179+0000","last_peered":"2024-06-14T22:54:06.278179+0000","last_clean":"2024-06-14T22:54:06.278179+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T22:54:06.278179+0000","last_undegraded":"2024-06-14T22:54:06.278179+0000","last_fullsized":"2024-06-14T22:54:06.278179+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":83,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278562+0000","last_change":"2024-06-14T22:34:45.813267+0000","last_active":"2024-06-14T22:54:05.278562+0000","last_peered":"2024-06-14T22:54:05.278562+0000","last_clean":"2024-06-14T22:54:05.278562+0000","last_became_active":"2024-06-14T22:34:45.813003+0000","last_became_peered":"2024-06-14T22:34:45.813003+0000","last_unstale":"2024-06-14T22:54:05.278562+0000","last_undegraded":"2024-06-14T22:54:05.278562+0000","last_fullsized":"2024-06-14T22:54:05.278562+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:46.446483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280927+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T22:54:06.280927+0000","last_peered":"2024-06-14T22:54:06.280927+0000","last_clean":"2024-06-14T22:54:06.280927+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T22:54:06.280927+0000","last_undegraded":"2024-06-14T22:54:06.280927+0000","last_fullsized":"2024-06-14T22:54:06.280927+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":108,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280520+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T22:54:05.280520+0000","last_peered":"2024-06-14T22:54:05.280520+0000","last_clean":"2024-06-14T22:54:05.280520+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T22:54:05.280520+0000","last_undegraded":"2024-06-14T22:54:05.280520+0000","last_fullsized":"2024-06-14T22:54:05.280520+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280442+0000","last_change":"2024-06-14T22:44:51.839766+0000","last_active":"2024-06-14T22:54:05.280442+0000","last_peered":"2024-06-14T22:54:05.280442+0000","last_clean":"2024-06-14T22:54:05.280442+0000","last_became_active":"2024-06-14T22:44:51.838970+0000","last_became_peered":"2024-06-14T22:44:51.838970+0000","last_unstale":"2024-06-14T22:54:05.280442+0000","last_undegraded":"2024-06-14T22:54:05.280442+0000","last_fullsized":"2024-06-14T22:54:05.280442+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:31:59.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280352+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:54:05.280352+0000","last_peered":"2024-06-14T22:54:05.280352+0000","last_clean":"2024-06-14T22:54:05.280352+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:54:05.280352+0000","last_undegraded":"2024-06-14T22:54:05.280352+0000","last_fullsized":"2024-06-14T22:54:05.280352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280342+0000","last_change":"2024-06-14T22:49:25.474899+0000","last_active":"2024-06-14T22:54:05.280342+0000","last_peered":"2024-06-14T22:54:05.280342+0000","last_clean":"2024-06-14T22:54:05.280342+0000","last_became_active":"2024-06-14T22:49:25.474251+0000","last_became_peered":"2024-06-14T22:49:25.474251+0000","last_unstale":"2024-06-14T22:54:05.280342+0000","last_undegraded":"2024-06-14T22:54:05.280342+0000","last_fullsized":"2024-06-14T22:54:05.280342+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:22:05.971278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281348+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T22:54:06.281348+0000","last_peered":"2024-06-14T22:54:06.281348+0000","last_clean":"2024-06-14T22:54:06.281348+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T22:54:06.281348+0000","last_undegraded":"2024-06-14T22:54:06.281348+0000","last_fullsized":"2024-06-14T22:54:06.281348+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":84,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279141+0000","last_change":"2024-06-14T22:34:45.810186+0000","last_active":"2024-06-14T22:54:05.279141+0000","last_peered":"2024-06-14T22:54:05.279141+0000","last_clean":"2024-06-14T22:54:05.279141+0000","last_became_active":"2024-06-14T22:34:45.809869+0000","last_became_peered":"2024-06-14T22:34:45.809869+0000","last_unstale":"2024-06-14T22:54:05.279141+0000","last_undegraded":"2024-06-14T22:54:05.279141+0000","last_fullsized":"2024-06-14T22:54:05.279141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:03:45.652442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278987+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T22:54:05.278987+0000","last_peered":"2024-06-14T22:54:05.278987+0000","last_clean":"2024-06-14T22:54:05.278987+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T22:54:05.278987+0000","last_undegraded":"2024-06-14T22:54:05.278987+0000","last_fullsized":"2024-06-14T22:54:05.278987+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280349+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T22:54:06.280349+0000","last_peered":"2024-06-14T22:54:06.280349+0000","last_clean":"2024-06-14T22:54:06.280349+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T22:54:06.280349+0000","last_undegraded":"2024-06-14T22:54:06.280349+0000","last_fullsized":"2024-06-14T22:54:06.280349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.276516+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T22:54:05.276516+0000","last_peered":"2024-06-14T22:54:05.276516+0000","last_clean":"2024-06-14T22:54:05.276516+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T22:54:05.276516+0000","last_undegraded":"2024-06-14T22:54:05.276516+0000","last_fullsized":"2024-06-14T22:54:05.276516+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280994+0000","last_change":"2024-06-14T22:54:06.280994+0000","last_active":"2024-06-14T22:54:06.280994+0000","last_peered":"2024-06-14T22:54:06.280994+0000","last_clean":"2024-06-14T22:54:06.280994+0000","last_became_active":"2024-06-14T22:54:06.279883+0000","last_became_peered":"2024-06-14T22:54:06.279883+0000","last_unstale":"2024-06-14T22:54:06.280994+0000","last_undegraded":"2024-06-14T22:54:06.280994+0000","last_fullsized":"2024-06-14T22:54:06.280994+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:51.156630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":99,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280497+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T22:54:06.280497+0000","last_peered":"2024-06-14T22:54:06.280497+0000","last_clean":"2024-06-14T22:54:06.280497+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T22:54:06.280497+0000","last_undegraded":"2024-06-14T22:54:06.280497+0000","last_fullsized":"2024-06-14T22:54:06.280497+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279157+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T22:54:05.279157+0000","last_peered":"2024-06-14T22:54:05.279157+0000","last_clean":"2024-06-14T22:54:05.279157+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T22:54:05.279157+0000","last_undegraded":"2024-06-14T22:54:05.279157+0000","last_fullsized":"2024-06-14T22:54:05.279157+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":109,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275417+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T22:54:05.275417+0000","last_peered":"2024-06-14T22:54:05.275417+0000","last_clean":"2024-06-14T22:54:05.275417+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T22:54:05.275417+0000","last_undegraded":"2024-06-14T22:54:05.275417+0000","last_fullsized":"2024-06-14T22:54:05.275417+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275776+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T22:54:05.275776+0000","last_peered":"2024-06-14T22:54:05.275776+0000","last_clean":"2024-06-14T22:54:05.275776+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T22:54:05.275776+0000","last_undegraded":"2024-06-14T22:54:05.275776+0000","last_fullsized":"2024-06-14T22:54:05.275776+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275483+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T22:54:05.275483+0000","last_peered":"2024-06-14T22:54:05.275483+0000","last_clean":"2024-06-14T22:54:05.275483+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T22:54:05.275483+0000","last_undegraded":"2024-06-14T22:54:05.275483+0000","last_fullsized":"2024-06-14T22:54:05.275483+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":91,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281907+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T22:54:06.281907+0000","last_peered":"2024-06-14T22:54:06.281907+0000","last_clean":"2024-06-14T22:54:06.281907+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T22:54:06.281907+0000","last_undegraded":"2024-06-14T22:54:06.281907+0000","last_fullsized":"2024-06-14T22:54:06.281907+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282328+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T22:54:06.282328+0000","last_peered":"2024-06-14T22:54:06.282328+0000","last_clean":"2024-06-14T22:54:06.282328+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T22:54:06.282328+0000","last_undegraded":"2024-06-14T22:54:06.282328+0000","last_fullsized":"2024-06-14T22:54:06.282328+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":480,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279291+0000","last_change":"2024-06-14T22:44:51.840491+0000","last_active":"2024-06-14T22:54:05.279291+0000","last_peered":"2024-06-14T22:54:05.279291+0000","last_clean":"2024-06-14T22:54:05.279291+0000","last_became_active":"2024-06-14T22:44:51.839990+0000","last_became_peered":"2024-06-14T22:44:51.839990+0000","last_unstale":"2024-06-14T22:54:05.279291+0000","last_undegraded":"2024-06-14T22:54:05.279291+0000","last_fullsized":"2024-06-14T22:54:05.279291+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:51:24.139036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281690+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T22:54:05.281690+0000","last_peered":"2024-06-14T22:54:05.281690+0000","last_clean":"2024-06-14T22:54:05.281690+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T22:54:05.281690+0000","last_undegraded":"2024-06-14T22:54:05.281690+0000","last_fullsized":"2024-06-14T22:54:05.281690+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":64,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279205+0000","last_change":"2024-06-14T22:54:06.279205+0000","last_active":"2024-06-14T22:54:06.279205+0000","last_peered":"2024-06-14T22:54:06.279205+0000","last_clean":"2024-06-14T22:54:06.279205+0000","last_became_active":"2024-06-14T22:54:06.278792+0000","last_became_peered":"2024-06-14T22:54:06.278792+0000","last_unstale":"2024-06-14T22:54:06.279205+0000","last_undegraded":"2024-06-14T22:54:06.279205+0000","last_fullsized":"2024-06-14T22:54:06.279205+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:55:02.899941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279434+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:54:05.279434+0000","last_peered":"2024-06-14T22:54:05.279434+0000","last_clean":"2024-06-14T22:54:05.279434+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:54:05.279434+0000","last_undegraded":"2024-06-14T22:54:05.279434+0000","last_fullsized":"2024-06-14T22:54:05.279434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":121,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279201+0000","last_change":"2024-06-14T22:49:25.476578+0000","last_active":"2024-06-14T22:54:05.279201+0000","last_peered":"2024-06-14T22:54:05.279201+0000","last_clean":"2024-06-14T22:54:05.279201+0000","last_became_active":"2024-06-14T22:49:25.476183+0000","last_became_peered":"2024-06-14T22:49:25.476183+0000","last_unstale":"2024-06-14T22:54:05.279201+0000","last_undegraded":"2024-06-14T22:54:05.279201+0000","last_fullsized":"2024-06-14T22:54:05.279201+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:01:44.585320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277248+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T22:54:06.277248+0000","last_peered":"2024-06-14T22:54:06.277248+0000","last_clean":"2024-06-14T22:54:06.277248+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T22:54:06.277248+0000","last_undegraded":"2024-06-14T22:54:06.277248+0000","last_fullsized":"2024-06-14T22:54:06.277248+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278487+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T22:54:05.278487+0000","last_peered":"2024-06-14T22:54:05.278487+0000","last_clean":"2024-06-14T22:54:05.278487+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T22:54:05.278487+0000","last_undegraded":"2024-06-14T22:54:05.278487+0000","last_fullsized":"2024-06-14T22:54:05.278487+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.275775+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T22:54:06.275775+0000","last_peered":"2024-06-14T22:54:06.275775+0000","last_clean":"2024-06-14T22:54:06.275775+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T22:54:06.275775+0000","last_undegraded":"2024-06-14T22:54:06.275775+0000","last_fullsized":"2024-06-14T22:54:06.275775+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270774+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T22:54:05.270774+0000","last_peered":"2024-06-14T22:54:05.270774+0000","last_clean":"2024-06-14T22:54:05.270774+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T22:54:05.270774+0000","last_undegraded":"2024-06-14T22:54:05.270774+0000","last_fullsized":"2024-06-14T22:54:05.270774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270053+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T22:54:05.270053+0000","last_peered":"2024-06-14T22:54:05.270053+0000","last_clean":"2024-06-14T22:54:05.270053+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T22:54:05.270053+0000","last_undegraded":"2024-06-14T22:54:05.270053+0000","last_fullsized":"2024-06-14T22:54:05.270053+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281419+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T22:54:05.281419+0000","last_peered":"2024-06-14T22:54:05.281419+0000","last_clean":"2024-06-14T22:54:05.281419+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T22:54:05.281419+0000","last_undegraded":"2024-06-14T22:54:05.281419+0000","last_fullsized":"2024-06-14T22:54:05.281419+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275641+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T22:54:05.275641+0000","last_peered":"2024-06-14T22:54:05.275641+0000","last_clean":"2024-06-14T22:54:05.275641+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T22:54:05.275641+0000","last_undegraded":"2024-06-14T22:54:05.275641+0000","last_fullsized":"2024-06-14T22:54:05.275641+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":119,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280880+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T22:54:06.280880+0000","last_peered":"2024-06-14T22:54:06.280880+0000","last_clean":"2024-06-14T22:54:06.280880+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T22:54:06.280880+0000","last_undegraded":"2024-06-14T22:54:06.280880+0000","last_fullsized":"2024-06-14T22:54:06.280880+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":117,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280346+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T22:54:06.280346+0000","last_peered":"2024-06-14T22:54:06.280346+0000","last_clean":"2024-06-14T22:54:06.280346+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T22:54:06.280346+0000","last_undegraded":"2024-06-14T22:54:06.280346+0000","last_fullsized":"2024-06-14T22:54:06.280346+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270215+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T22:54:05.270215+0000","last_peered":"2024-06-14T22:54:05.270215+0000","last_clean":"2024-06-14T22:54:05.270215+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T22:54:05.270215+0000","last_undegraded":"2024-06-14T22:54:05.270215+0000","last_fullsized":"2024-06-14T22:54:05.270215+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270541+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T22:54:05.270541+0000","last_peered":"2024-06-14T22:54:05.270541+0000","last_clean":"2024-06-14T22:54:05.270541+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T22:54:05.270541+0000","last_undegraded":"2024-06-14T22:54:05.270541+0000","last_fullsized":"2024-06-14T22:54:05.270541+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270139+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T22:54:05.270139+0000","last_peered":"2024-06-14T22:54:05.270139+0000","last_clean":"2024-06-14T22:54:05.270139+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T22:54:05.270139+0000","last_undegraded":"2024-06-14T22:54:05.270139+0000","last_fullsized":"2024-06-14T22:54:05.270139+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279227+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T22:54:05.279227+0000","last_peered":"2024-06-14T22:54:05.279227+0000","last_clean":"2024-06-14T22:54:05.279227+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T22:54:05.279227+0000","last_undegraded":"2024-06-14T22:54:05.279227+0000","last_fullsized":"2024-06-14T22:54:05.279227+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":74,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.276447+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T22:54:05.276447+0000","last_peered":"2024-06-14T22:54:05.276447+0000","last_clean":"2024-06-14T22:54:05.276447+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T22:54:05.276447+0000","last_undegraded":"2024-06-14T22:54:05.276447+0000","last_fullsized":"2024-06-14T22:54:05.276447+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.284651+0000","last_change":"2024-06-14T22:54:06.284651+0000","last_active":"2024-06-14T22:54:06.284651+0000","last_peered":"2024-06-14T22:54:06.284651+0000","last_clean":"2024-06-14T22:54:06.284651+0000","last_became_active":"2024-06-14T22:54:06.283564+0000","last_became_peered":"2024-06-14T22:54:06.283564+0000","last_unstale":"2024-06-14T22:54:06.284651+0000","last_undegraded":"2024-06-14T22:54:06.284651+0000","last_fullsized":"2024-06-14T22:54:06.284651+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:34.916272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270118+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T22:54:05.270118+0000","last_peered":"2024-06-14T22:54:05.270118+0000","last_clean":"2024-06-14T22:54:05.270118+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T22:54:05.270118+0000","last_undegraded":"2024-06-14T22:54:05.270118+0000","last_fullsized":"2024-06-14T22:54:05.270118+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281419+0000","last_change":"2024-06-14T22:54:06.281419+0000","last_active":"2024-06-14T22:54:06.281419+0000","last_peered":"2024-06-14T22:54:06.281419+0000","last_clean":"2024-06-14T22:54:06.281419+0000","last_became_active":"2024-06-14T22:54:06.279482+0000","last_became_peered":"2024-06-14T22:54:06.279482+0000","last_unstale":"2024-06-14T22:54:06.281419+0000","last_undegraded":"2024-06-14T22:54:06.281419+0000","last_fullsized":"2024-06-14T22:54:06.281419+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:06:37.039006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.277756+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T22:54:05.277756+0000","last_peered":"2024-06-14T22:54:05.277756+0000","last_clean":"2024-06-14T22:54:05.277756+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T22:54:05.277756+0000","last_undegraded":"2024-06-14T22:54:05.277756+0000","last_fullsized":"2024-06-14T22:54:05.277756+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280283+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T22:54:05.280283+0000","last_peered":"2024-06-14T22:54:05.280283+0000","last_clean":"2024-06-14T22:54:05.280283+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T22:54:05.280283+0000","last_undegraded":"2024-06-14T22:54:05.280283+0000","last_fullsized":"2024-06-14T22:54:05.280283+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275645+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:54:05.275645+0000","last_peered":"2024-06-14T22:54:05.275645+0000","last_clean":"2024-06-14T22:54:05.275645+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:54:05.275645+0000","last_undegraded":"2024-06-14T22:54:05.275645+0000","last_fullsized":"2024-06-14T22:54:05.275645+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275589+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T22:54:05.275589+0000","last_peered":"2024-06-14T22:54:05.275589+0000","last_clean":"2024-06-14T22:54:05.275589+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T22:54:05.275589+0000","last_undegraded":"2024-06-14T22:54:05.275589+0000","last_fullsized":"2024-06-14T22:54:05.275589+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":105,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275170+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:54:05.275170+0000","last_peered":"2024-06-14T22:54:05.275170+0000","last_clean":"2024-06-14T22:54:05.275170+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:54:05.275170+0000","last_undegraded":"2024-06-14T22:54:05.275170+0000","last_fullsized":"2024-06-14T22:54:05.275170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270411+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T22:54:05.270411+0000","last_peered":"2024-06-14T22:54:05.270411+0000","last_clean":"2024-06-14T22:54:05.270411+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T22:54:05.270411+0000","last_undegraded":"2024-06-14T22:54:05.270411+0000","last_fullsized":"2024-06-14T22:54:05.270411+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.275561+0000","last_change":"2024-06-14T22:54:06.275561+0000","last_active":"2024-06-14T22:54:06.275561+0000","last_peered":"2024-06-14T22:54:06.275561+0000","last_clean":"2024-06-14T22:54:06.275561+0000","last_became_active":"2024-06-14T22:54:06.274930+0000","last_became_peered":"2024-06-14T22:54:06.274930+0000","last_unstale":"2024-06-14T22:54:06.275561+0000","last_undegraded":"2024-06-14T22:54:06.275561+0000","last_fullsized":"2024-06-14T22:54:06.275561+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:11.022230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281532+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T22:54:05.281532+0000","last_peered":"2024-06-14T22:54:05.281532+0000","last_clean":"2024-06-14T22:54:05.281532+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T22:54:05.281532+0000","last_undegraded":"2024-06-14T22:54:05.281532+0000","last_fullsized":"2024-06-14T22:54:05.281532+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":64,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279332+0000","last_change":"2024-06-14T22:54:06.279332+0000","last_active":"2024-06-14T22:54:06.279332+0000","last_peered":"2024-06-14T22:54:06.279332+0000","last_clean":"2024-06-14T22:54:06.279332+0000","last_became_active":"2024-06-14T22:54:06.279020+0000","last_became_peered":"2024-06-14T22:54:06.279020+0000","last_unstale":"2024-06-14T22:54:06.279332+0000","last_undegraded":"2024-06-14T22:54:06.279332+0000","last_fullsized":"2024-06-14T22:54:06.279332+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:52:57.777668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.285313+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T22:54:06.285313+0000","last_peered":"2024-06-14T22:54:06.285313+0000","last_clean":"2024-06-14T22:54:06.285313+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T22:54:06.285313+0000","last_undegraded":"2024-06-14T22:54:06.285313+0000","last_fullsized":"2024-06-14T22:54:06.285313+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281615+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T22:54:06.281615+0000","last_peered":"2024-06-14T22:54:06.281615+0000","last_clean":"2024-06-14T22:54:06.281615+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T22:54:06.281615+0000","last_undegraded":"2024-06-14T22:54:06.281615+0000","last_fullsized":"2024-06-14T22:54:06.281615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280203+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T22:54:05.280203+0000","last_peered":"2024-06-14T22:54:05.280203+0000","last_clean":"2024-06-14T22:54:05.280203+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T22:54:05.280203+0000","last_undegraded":"2024-06-14T22:54:05.280203+0000","last_fullsized":"2024-06-14T22:54:05.280203+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":108,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282120+0000","last_change":"2024-06-14T22:54:06.282120+0000","last_active":"2024-06-14T22:54:06.282120+0000","last_peered":"2024-06-14T22:54:06.282120+0000","last_clean":"2024-06-14T22:54:06.282120+0000","last_became_active":"2024-06-14T22:54:06.280898+0000","last_became_peered":"2024-06-14T22:54:06.280898+0000","last_unstale":"2024-06-14T22:54:06.282120+0000","last_undegraded":"2024-06-14T22:54:06.282120+0000","last_fullsized":"2024-06-14T22:54:06.282120+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:30:45.544457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.278300+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T22:54:06.278300+0000","last_peered":"2024-06-14T22:54:06.278300+0000","last_clean":"2024-06-14T22:54:06.278300+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T22:54:06.278300+0000","last_undegraded":"2024-06-14T22:54:06.278300+0000","last_fullsized":"2024-06-14T22:54:06.278300+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.284728+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T22:54:06.284728+0000","last_peered":"2024-06-14T22:54:06.284728+0000","last_clean":"2024-06-14T22:54:06.284728+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T22:54:06.284728+0000","last_undegraded":"2024-06-14T22:54:06.284728+0000","last_fullsized":"2024-06-14T22:54:06.284728+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280483+0000","last_change":"2024-06-14T22:54:06.280483+0000","last_active":"2024-06-14T22:54:06.280483+0000","last_peered":"2024-06-14T22:54:06.280483+0000","last_clean":"2024-06-14T22:54:06.280483+0000","last_became_active":"2024-06-14T22:54:06.278985+0000","last_became_peered":"2024-06-14T22:54:06.278985+0000","last_unstale":"2024-06-14T22:54:06.280483+0000","last_undegraded":"2024-06-14T22:54:06.280483+0000","last_fullsized":"2024-06-14T22:54:06.280483+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:43:40.478580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":118,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.274772+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T22:54:05.274772+0000","last_peered":"2024-06-14T22:54:05.274772+0000","last_clean":"2024-06-14T22:54:05.274772+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T22:54:05.274772+0000","last_undegraded":"2024-06-14T22:54:05.274772+0000","last_fullsized":"2024-06-14T22:54:05.274772+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":109,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277110+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T22:54:06.277110+0000","last_peered":"2024-06-14T22:54:06.277110+0000","last_clean":"2024-06-14T22:54:06.277110+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T22:54:06.277110+0000","last_undegraded":"2024-06-14T22:54:06.277110+0000","last_fullsized":"2024-06-14T22:54:06.277110+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":77,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.270191+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T22:54:06.270191+0000","last_peered":"2024-06-14T22:54:06.270191+0000","last_clean":"2024-06-14T22:54:06.270191+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T22:54:06.270191+0000","last_undegraded":"2024-06-14T22:54:06.270191+0000","last_fullsized":"2024-06-14T22:54:06.270191+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281117+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T22:54:06.281117+0000","last_peered":"2024-06-14T22:54:06.281117+0000","last_clean":"2024-06-14T22:54:06.281117+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T22:54:06.281117+0000","last_undegraded":"2024-06-14T22:54:06.281117+0000","last_fullsized":"2024-06-14T22:54:06.281117+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280162+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T22:54:05.280162+0000","last_peered":"2024-06-14T22:54:05.280162+0000","last_clean":"2024-06-14T22:54:05.280162+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T22:54:05.280162+0000","last_undegraded":"2024-06-14T22:54:05.280162+0000","last_fullsized":"2024-06-14T22:54:05.280162+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280576+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T22:54:06.280576+0000","last_peered":"2024-06-14T22:54:06.280576+0000","last_clean":"2024-06-14T22:54:06.280576+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T22:54:06.280576+0000","last_undegraded":"2024-06-14T22:54:06.280576+0000","last_fullsized":"2024-06-14T22:54:06.280576+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":85,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.282162+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:54:05.282162+0000","last_peered":"2024-06-14T22:54:05.282162+0000","last_clean":"2024-06-14T22:54:05.282162+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:54:05.282162+0000","last_undegraded":"2024-06-14T22:54:05.282162+0000","last_fullsized":"2024-06-14T22:54:05.282162+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":123,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282972+0000","last_change":"2024-06-14T22:54:06.282972+0000","last_active":"2024-06-14T22:54:06.282972+0000","last_peered":"2024-06-14T22:54:06.282972+0000","last_clean":"2024-06-14T22:54:06.282972+0000","last_became_active":"2024-06-14T22:54:06.282622+0000","last_became_peered":"2024-06-14T22:54:06.282622+0000","last_unstale":"2024-06-14T22:54:06.282972+0000","last_undegraded":"2024-06-14T22:54:06.282972+0000","last_fullsized":"2024-06-14T22:54:06.282972+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:31:22.051955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":108,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279284+0000","last_change":"2024-06-14T22:54:06.279284+0000","last_active":"2024-06-14T22:54:06.279284+0000","last_peered":"2024-06-14T22:54:06.279284+0000","last_clean":"2024-06-14T22:54:06.279284+0000","last_became_active":"2024-06-14T22:54:06.278861+0000","last_became_peered":"2024-06-14T22:54:06.278861+0000","last_unstale":"2024-06-14T22:54:06.279284+0000","last_undegraded":"2024-06-14T22:54:06.279284+0000","last_fullsized":"2024-06-14T22:54:06.279284+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:17:47.260769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.273550+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T22:54:05.273550+0000","last_peered":"2024-06-14T22:54:05.273550+0000","last_clean":"2024-06-14T22:54:05.273550+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T22:54:05.273550+0000","last_undegraded":"2024-06-14T22:54:05.273550+0000","last_fullsized":"2024-06-14T22:54:05.273550+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275021+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T22:54:05.275021+0000","last_peered":"2024-06-14T22:54:05.275021+0000","last_clean":"2024-06-14T22:54:05.275021+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T22:54:05.275021+0000","last_undegraded":"2024-06-14T22:54:05.275021+0000","last_fullsized":"2024-06-14T22:54:05.275021+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234252,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600240,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":93144912,"statfs":{"total":95995035648,"available":95380389888,"internally_reserved":0,"allocated":2260992,"data_stored":1493426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.0780000000000001,"15min":1.167},"min":{"1min":0.28399999999999997,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.69199999999999995,"5min":9.3919999999999995,"15min":9.3919999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":1.353,"15min":1.2989999999999999},"min":{"1min":0.29999999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.74199999999999999,"5min":10.416,"15min":10.416},"last":0.60999999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.212,"15min":1.3700000000000001},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80500000000000005,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":1.3560000000000001,"15min":1.464},"min":{"1min":0.38100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.59799999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.5509999999999999,"15min":1.4099999999999999},"min":{"1min":0.39500000000000002,"5min":0.30399999999999999,"15min":0.26100000000000001},"max":{"1min":3.129,"5min":7.4249999999999998,"15min":10.199},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.254,"15min":1.4450000000000001},"min":{"1min":0.41199999999999998,"5min":0.26100000000000001,"15min":0.254},"max":{"1min":2.8490000000000002,"5min":7.032,"15min":7.4610000000000003},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.67000000000000004,"15min":0.80400000000000005},"min":{"1min":0.27300000000000002,"5min":0.248,"15min":0.214},"max":{"1min":0.67900000000000005,"5min":5.8449999999999998,"15min":9.7430000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.33900000000000002,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":0.64500000000000002,"5min":4.2709999999999999,"15min":6.6660000000000004},"last":0.38}]},{"osd":5,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.752,"15min":0.77200000000000002},"min":{"1min":0.34999999999999998,"5min":0.26100000000000001,"15min":0.22},"max":{"1min":0.70599999999999996,"5min":3.9860000000000002,"15min":7.9870000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.88,"15min":0.88600000000000001},"min":{"1min":0.33500000000000002,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.76200000000000001,"5min":5.3659999999999997,"15min":9.5120000000000005},"last":0.40999999999999998}]},{"osd":6,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.75800000000000001,"15min":0.81999999999999995},"min":{"1min":0.39400000000000002,"5min":0.308,"15min":0.28100000000000003},"max":{"1min":0.72499999999999998,"5min":5.806,"15min":5.806},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.76700000000000002,"15min":0.71999999999999997},"min":{"1min":0.39000000000000001,"5min":0.311,"15min":0.29899999999999999},"max":{"1min":0.70299999999999996,"5min":4.3090000000000002,"15min":5.7430000000000003},"last":0.42999999999999999}]}]},{"osd":6,"up_from":46,"seq":197568495888,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311836,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":93433316,"statfs":{"total":95995035648,"available":95675715584,"internally_reserved":0,"allocated":2244608,"data_stored":1492957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":1.5880000000000001,"15min":1.2050000000000001},"min":{"1min":0.39800000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.72099999999999997,"5min":31.358000000000001,"15min":31.358000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.417,"15min":1.458},"min":{"1min":0.36199999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.69899999999999995,"5min":29.399000000000001,"15min":29.399000000000001},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.224,"15min":1.304},"min":{"1min":0.39400000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73399999999999999,"5min":9.9559999999999995,"15min":9.9559999999999995},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":1.2769999999999999,"15min":1.276},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.67400000000000004,"5min":9.5510000000000002,"15min":9.5510000000000002},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:53:14 2024","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":1.7669999999999999,"15min":1.607},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":6.9219999999999997,"5min":28.693999999999999,"15min":28.693999999999999},"last":0.505},{"interface":"front","average":{"1min":1.649,"5min":1.6839999999999999,"15min":1.5960000000000001},"min":{"1min":0.35099999999999998,"5min":0.27900000000000003,"15min":0.25700000000000001},"max":{"1min":8.7200000000000006,"5min":28.300000000000001,"15min":28.300000000000001},"last":0.52900000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":1.2110000000000001,"15min":0.88800000000000001},"min":{"1min":0.35299999999999998,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":0.71999999999999997,"5min":47.698,"15min":47.698},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.496,"5min":1.181,"15min":0.82599999999999996},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.23899999999999999},"max":{"1min":0.72699999999999998,"5min":48.197000000000003,"15min":48.197000000000003},"last":0.41699999999999998}]},{"osd":5,"last update":"Fri Jun 14 22:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.88600000000000001,"15min":0.79100000000000004},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.22600000000000001},"max":{"1min":0.75800000000000001,"5min":8.6500000000000004,"15min":8.6500000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.95399999999999996,"15min":0.86199999999999999},"min":{"1min":0.41199999999999998,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":8.6349999999999998,"15min":9.5609999999999999},"last":0.39600000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:54:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.84199999999999997,"15min":0.82099999999999995},"min":{"1min":0.38100000000000001,"5min":0.31,"15min":0.28799999999999998},"max":{"1min":0.82299999999999995,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.90500000000000003,"15min":0.83299999999999996},"min":{"1min":0.44500000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78100000000000003,"5min":8.673,"15min":8.673},"last":0.57899999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757527,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896852,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":92848300,"statfs":{"total":95995035648,"available":95076659200,"internally_reserved":0,"allocated":1642496,"data_stored":898403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":1.1299999999999999,"15min":1.2270000000000001},"min":{"1min":0.41099999999999998,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.69899999999999995,"5min":6.0369999999999999,"15min":17.766999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":1.046,"15min":1.2689999999999999},"min":{"1min":0.38500000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70999999999999996,"5min":5.2439999999999998,"15min":17.806000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.2809999999999999,"15min":1.357},"min":{"1min":0.36099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66400000000000003,"5min":7.694,"15min":7.694},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.501,"5min":1.129,"15min":1.169},"min":{"1min":0.33700000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.72599999999999998,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:52:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8660000000000001,"5min":1.393,"15min":1.5449999999999999},"min":{"1min":0.379,"5min":0.25700000000000001,"15min":0.23999999999999999},"max":{"1min":6.0229999999999997,"5min":7.7599999999999998,"15min":10.128},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.484,"15min":1.5549999999999999},"min":{"1min":0.36699999999999999,"5min":0.28299999999999997,"15min":0.27300000000000002},"max":{"1min":5.8550000000000004,"5min":5.8550000000000004,"15min":9.048},"last":0.61699999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.60599999999999998,"15min":0.73599999999999999},"min":{"1min":0.313,"5min":0.23300000000000001,"15min":0.214},"max":{"1min":0.64400000000000002,"5min":4.9790000000000001,"15min":11.771000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.69699999999999995,"15min":0.74299999999999999},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.23200000000000001},"max":{"1min":0.66100000000000003,"5min":5.8109999999999999,"15min":10.032},"last":0.33100000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.84699999999999998,"15min":0.86599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.23000000000000001},"max":{"1min":0.67900000000000005,"5min":9.6020000000000003,"15min":9.8369999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.85699999999999998,"15min":0.88300000000000001},"min":{"1min":0.33300000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.74399999999999999,"5min":12.339,"15min":12.339},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.75800000000000001,"15min":0.82299999999999995},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":5.2629999999999999,"15min":10.462999999999999},"last":0.501},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.72999999999999998,"15min":0.83899999999999997},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69299999999999995,"5min":10.638999999999999,"15min":10.638999999999999},"last":0.51100000000000001}]}]},{"osd":4,"up_from":30,"seq":128849019163,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599652,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":93145500,"statfs":{"total":95995035648,"available":95380992000,"internally_reserved":0,"allocated":1658880,"data_stored":902723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.0640000000000001,"15min":1.0609999999999999},"min":{"1min":0.39500000000000002,"5min":0.219,"15min":0.218},"max":{"1min":0.70199999999999996,"5min":4.641,"15min":7.5650000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":1.1499999999999999,"15min":1.365},"min":{"1min":0.39800000000000002,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":0.65400000000000003,"5min":5.7569999999999997,"15min":9.1470000000000002},"last":0.47099999999999997}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":1.1779999999999999,"15min":1.1499999999999999},"min":{"1min":0.34899999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65300000000000002,"5min":5.141,"15min":5.141},"last":0.441},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.2030000000000001,"15min":1.3360000000000001},"min":{"1min":0.35299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.70799999999999996,"5min":6.2709999999999999,"15min":6.2709999999999999},"last":0.55200000000000005}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Jun 14 22:53:21 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.2150000000000001,"15min":1.4339999999999999},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.24299999999999999},"max":{"1min":5.2830000000000004,"5min":6.2460000000000004,"15min":9.2750000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.764,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.317,"5min":0.27800000000000002,"15min":0.27300000000000002},"max":{"1min":5.6859999999999999,"5min":10.249000000000001,"15min":10.249000000000001},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:54:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.72599999999999998,"15min":0.749},"min":{"1min":0.31,"5min":0.24199999999999999,"15min":0.21199999999999999},"max":{"1min":0.67900000000000005,"5min":3.964,"15min":5.774},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":0.78200000000000003,"5min":4.1269999999999998,"15min":4.7480000000000002},"last":0.378}]},{"osd":6,"last update":"Fri Jun 14 22:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.65200000000000002,"15min":0.78100000000000003},"min":{"1min":0.34399999999999997,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":3.8929999999999998,"15min":10.904999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61499999999999999,"15min":0.77200000000000002},"min":{"1min":0.371,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":0.64600000000000002,"5min":3.7450000000000001,"15min":9.2390000000000008},"last":0.48999999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:54:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.63,"15min":0.752},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.246},"max":{"1min":0.72399999999999998,"5min":3.7029999999999998,"15min":6.298},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.74299999999999999,"15min":0.76400000000000001},"min":{"1min":0.38300000000000001,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.73499999999999999,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":0.56799999999999995}]}]},{"osd":3,"up_from":138,"seq":592705486862,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77328,"kb_used_data":1604,"kb_used_omap":4,"kb_used_meta":75707,"kb_avail":93667824,"statfs":{"total":95995035648,"available":95915851776,"internally_reserved":0,"allocated":1642496,"data_stored":898176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":77524446},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]}]},{"osd":2,"up_from":133,"seq":571230650389,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62544,"kb_used_data":1604,"kb_used_omap":6,"kb_used_meta":60921,"kb_avail":93682608,"statfs":{"total":95995035648,"available":95930990592,"internally_reserved":0,"allocated":1642496,"data_stored":898390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":62383605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.252},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.32600000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.30499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":4,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.49399999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.53600000000000003}]},{"osd":7,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.50700000000000001}]}]},{"osd":1,"up_from":116,"seq":498216206406,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":213204,"kb_used_data":1600,"kb_used_omap":5,"kb_used_meta":211578,"kb_avail":93531948,"statfs":{"total":95995035648,"available":95776714752,"internally_reserved":0,"allocated":1638400,"data_stored":898086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":216656152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":1.8759999999999999,"15min":2.3620000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41299999999999998,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.4159999999999999,"15min":1.746},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.433,"5min":10.426,"15min":10.426},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Jun 14 22:53:32 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":2.073,"15min":2.2200000000000002},"min":{"1min":0.317,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.3860000000000001,"5min":20.297999999999998,"15min":20.297999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.5409999999999999,"5min":2.427,"15min":2.4910000000000001},"min":{"1min":0.32100000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.8630000000000004,"5min":20.260000000000002,"15min":20.260000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.5229999999999999,"15min":2.0209999999999999},"min":{"1min":0.41799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.95399999999999996,"5min":8.8599999999999994,"15min":8.8599999999999994},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.625,"15min":1.9490000000000001},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88600000000000001,"5min":11.161,"15min":11.161},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.6719999999999999,"15min":1.8420000000000001},"min":{"1min":0.40799999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":1.52,"15min":2.0760000000000001},"min":{"1min":0.438,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.97899999999999998,"5min":8.125,"15min":8.125},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":1.7749999999999999,"15min":1.929},"min":{"1min":0.39700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89700000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":1.4870000000000001,"15min":2.0750000000000002},"min":{"1min":0.44,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.92300000000000004,"5min":8.1479999999999997,"15min":8.1479999999999997},"last":0.51300000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.589,"15min":2.089},"min":{"1min":0.45900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79900000000000004,"5min":9.0960000000000001,"15min":9.0960000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.621,"5min":1.5600000000000001,"15min":1.702},"min":{"1min":0.48199999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96499999999999997,"5min":9.5449999999999999,"15min":9.5449999999999999},"last":0.55800000000000005}]}]},{"osd":0,"up_from":91,"seq":390842024061,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":441492,"kb_used_data":2180,"kb_used_omap":5,"kb_used_meta":439290,"kb_avail":93303660,"statfs":{"total":95995035648,"available":95542947840,"internally_reserved":0,"allocated":2232320,"data_stored":1486453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":449833050},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:54:27 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.7989999999999999,"15min":1.609},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.44600000000000001,"5min":9.6999999999999993,"15min":9.6999999999999993},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.34999999999999998,"5min":1.895,"15min":2.2869999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.58199999999999996,"5min":10.613,"15min":10.613},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:54:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.33600000000000002}]},{"osd":3,"last update":"Fri Jun 14 22:53:08 2024","interfaces":[{"interface":"back","average":{"1min":2.7589999999999999,"5min":2.4350000000000001,"15min":2.411},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":8.5199999999999996,"5min":13.749000000000001,"15min":13.749000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":2.4199999999999999,"5min":2.3079999999999998,"15min":2.1549999999999998},"min":{"1min":0.32300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.0339999999999998,"5min":9.8109999999999999,"15min":12.768000000000001},"last":0.316}]},{"osd":4,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.427,"15min":1.6850000000000001},"min":{"1min":0.40000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.83999999999999997,"5min":8.7889999999999997,"15min":8.9510000000000005},"last":0.505},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.2929999999999999,"15min":1.3460000000000001},"min":{"1min":0.436,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.87,"5min":13.574,"15min":13.574},"last":0.434}]},{"osd":5,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.3999999999999999,"15min":1.391},"min":{"1min":0.41399999999999998,"5min":0.317,"15min":0.29299999999999998},"max":{"1min":0.84099999999999997,"5min":13.616,"15min":13.616},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":1.3620000000000001,"15min":1.4850000000000001},"min":{"1min":0.45300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.0409999999999999,"5min":7.6509999999999998,"15min":9.6620000000000008},"last":0.46899999999999997}]},{"osd":6,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":1.4039999999999999,"15min":1.421},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81699999999999995,"5min":13.645,"15min":13.645},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.248,"15min":1.5780000000000001},"min":{"1min":0.44800000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.022,"5min":7.0119999999999996,"15min":9.8599999999999994},"last":0.443}]},{"osd":7,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.2509999999999999,"15min":1.617},"min":{"1min":0.42399999999999999,"5min":0.38700000000000001,"15min":0.33100000000000002},"max":{"1min":1.0009999999999999,"5min":6.9870000000000001,"15min":9.6370000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":1.4339999999999999,"15min":1.7989999999999999},"min":{"1min":0.44400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.84799999999999998,"5min":8.8290000000000006,"15min":8.9900000000000002},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:55:09.888 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-14T22:55:09.888 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-14T22:55:09.888 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-14T22:55:09.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-14T22:55:09.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:09 smithi137 ceph-mon[90634]: pgmap v528: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:10.138 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:10.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:10 smithi086 ceph-mon[97913]: from='client.45434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:10.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:10 smithi086 ceph-mon[102754]: from='client.45434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:10.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:10 smithi137 ceph-mon[90634]: from='client.45434 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:11.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:11 smithi137 ceph-mon[90634]: pgmap v529: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:12.007 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:55:12.007 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-14T22:55:12.008 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:11 smithi086 ceph-mon[97913]: pgmap v529: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:12.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:11 smithi086 ceph-mon[102754]: pgmap v529: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:12.678 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-14T22:55:12.678 INFO:teuthology.run_tasks:Running task print... 2024-06-14T22:55:12.691 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-06-14T22:55:12.691 INFO:teuthology.run_tasks:Running task parallel... 2024-06-14T22:55:12.772 INFO:teuthology.task.parallel:starting parallel... 2024-06-14T22:55:12.772 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-06-14T22:55:12.773 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-06-14T22:55:12.774 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-06-14T22:55:12.775 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri5-testing-2024-06-08-0902-reef'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-reef-drop-centos8', 'sha1': '906f73e382c3916fe5507b8227cc56e63a20794f'}} 2024-06-14T22:55:12.775 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-06-14T22:55:12.775 INFO:tasks.thrashosds:Beginning thrashosds... 2024-06-14T22:55:12.775 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-14T22:55:12.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:55:12.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/310727373' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-14T22:55:12.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:13.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/310727373' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-14T22:55:13.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:13.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/310727373' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-14T22:55:13.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:13.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:13 smithi137 ceph-mon[90634]: pgmap v530: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:14.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:13 smithi086 ceph-mon[97913]: pgmap v530: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:14.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:13 smithi086 ceph-mon[102754]: pgmap v530: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:14.833 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:55:14.833 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:55:15.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:55:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:55:14] "GET /metrics HTTP/1.1" 200 34119 "" "Prometheus/2.43.0" 2024-06-14T22:55:15.498 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-06-14T22:55:14.562508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":300,"num_write_kb":2445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3203148,"kb_used_data":14612,"kb_used_omap":29,"kb_used_meta":3188386,"kb_avail":746758068,"statfs":{"total":767960285184,"available":764680261632,"internally_reserved":0,"allocated":14962688,"data_stored":8968614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":3264907941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002555"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279751+0000","last_change":"2024-06-14T22:54:06.279751+0000","last_active":"2024-06-14T22:54:06.279751+0000","last_peered":"2024-06-14T22:54:06.279751+0000","last_clean":"2024-06-14T22:54:06.279751+0000","last_became_active":"2024-06-14T22:54:06.279443+0000","last_became_peered":"2024-06-14T22:54:06.279443+0000","last_unstale":"2024-06-14T22:54:06.279751+0000","last_undegraded":"2024-06-14T22:54:06.279751+0000","last_fullsized":"2024-06-14T22:54:06.279751+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:13:26.144924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278747+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T22:54:05.278747+0000","last_peered":"2024-06-14T22:54:05.278747+0000","last_clean":"2024-06-14T22:54:05.278747+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T22:54:05.278747+0000","last_undegraded":"2024-06-14T22:54:05.278747+0000","last_fullsized":"2024-06-14T22:54:05.278747+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":99,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.274802+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T22:54:06.274802+0000","last_peered":"2024-06-14T22:54:06.274802+0000","last_clean":"2024-06-14T22:54:06.274802+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T22:54:06.274802+0000","last_undegraded":"2024-06-14T22:54:06.274802+0000","last_fullsized":"2024-06-14T22:54:06.274802+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277962+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T22:54:06.277962+0000","last_peered":"2024-06-14T22:54:06.277962+0000","last_clean":"2024-06-14T22:54:06.277962+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T22:54:06.277962+0000","last_undegraded":"2024-06-14T22:54:06.277962+0000","last_fullsized":"2024-06-14T22:54:06.277962+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":89,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.278179+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T22:54:06.278179+0000","last_peered":"2024-06-14T22:54:06.278179+0000","last_clean":"2024-06-14T22:54:06.278179+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T22:54:06.278179+0000","last_undegraded":"2024-06-14T22:54:06.278179+0000","last_fullsized":"2024-06-14T22:54:06.278179+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":83,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278562+0000","last_change":"2024-06-14T22:34:45.813267+0000","last_active":"2024-06-14T22:54:05.278562+0000","last_peered":"2024-06-14T22:54:05.278562+0000","last_clean":"2024-06-14T22:54:05.278562+0000","last_became_active":"2024-06-14T22:34:45.813003+0000","last_became_peered":"2024-06-14T22:34:45.813003+0000","last_unstale":"2024-06-14T22:54:05.278562+0000","last_undegraded":"2024-06-14T22:54:05.278562+0000","last_fullsized":"2024-06-14T22:54:05.278562+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:46.446483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280927+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T22:54:06.280927+0000","last_peered":"2024-06-14T22:54:06.280927+0000","last_clean":"2024-06-14T22:54:06.280927+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T22:54:06.280927+0000","last_undegraded":"2024-06-14T22:54:06.280927+0000","last_fullsized":"2024-06-14T22:54:06.280927+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":108,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280520+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T22:54:05.280520+0000","last_peered":"2024-06-14T22:54:05.280520+0000","last_clean":"2024-06-14T22:54:05.280520+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T22:54:05.280520+0000","last_undegraded":"2024-06-14T22:54:05.280520+0000","last_fullsized":"2024-06-14T22:54:05.280520+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280442+0000","last_change":"2024-06-14T22:44:51.839766+0000","last_active":"2024-06-14T22:54:05.280442+0000","last_peered":"2024-06-14T22:54:05.280442+0000","last_clean":"2024-06-14T22:54:05.280442+0000","last_became_active":"2024-06-14T22:44:51.838970+0000","last_became_peered":"2024-06-14T22:44:51.838970+0000","last_unstale":"2024-06-14T22:54:05.280442+0000","last_undegraded":"2024-06-14T22:54:05.280442+0000","last_fullsized":"2024-06-14T22:54:05.280442+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:31:59.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280352+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:54:05.280352+0000","last_peered":"2024-06-14T22:54:05.280352+0000","last_clean":"2024-06-14T22:54:05.280352+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:54:05.280352+0000","last_undegraded":"2024-06-14T22:54:05.280352+0000","last_fullsized":"2024-06-14T22:54:05.280352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280342+0000","last_change":"2024-06-14T22:49:25.474899+0000","last_active":"2024-06-14T22:54:05.280342+0000","last_peered":"2024-06-14T22:54:05.280342+0000","last_clean":"2024-06-14T22:54:05.280342+0000","last_became_active":"2024-06-14T22:49:25.474251+0000","last_became_peered":"2024-06-14T22:49:25.474251+0000","last_unstale":"2024-06-14T22:54:05.280342+0000","last_undegraded":"2024-06-14T22:54:05.280342+0000","last_fullsized":"2024-06-14T22:54:05.280342+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:22:05.971278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281348+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T22:54:06.281348+0000","last_peered":"2024-06-14T22:54:06.281348+0000","last_clean":"2024-06-14T22:54:06.281348+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T22:54:06.281348+0000","last_undegraded":"2024-06-14T22:54:06.281348+0000","last_fullsized":"2024-06-14T22:54:06.281348+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":84,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279141+0000","last_change":"2024-06-14T22:34:45.810186+0000","last_active":"2024-06-14T22:54:05.279141+0000","last_peered":"2024-06-14T22:54:05.279141+0000","last_clean":"2024-06-14T22:54:05.279141+0000","last_became_active":"2024-06-14T22:34:45.809869+0000","last_became_peered":"2024-06-14T22:34:45.809869+0000","last_unstale":"2024-06-14T22:54:05.279141+0000","last_undegraded":"2024-06-14T22:54:05.279141+0000","last_fullsized":"2024-06-14T22:54:05.279141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:03:45.652442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278987+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T22:54:05.278987+0000","last_peered":"2024-06-14T22:54:05.278987+0000","last_clean":"2024-06-14T22:54:05.278987+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T22:54:05.278987+0000","last_undegraded":"2024-06-14T22:54:05.278987+0000","last_fullsized":"2024-06-14T22:54:05.278987+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280349+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T22:54:06.280349+0000","last_peered":"2024-06-14T22:54:06.280349+0000","last_clean":"2024-06-14T22:54:06.280349+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T22:54:06.280349+0000","last_undegraded":"2024-06-14T22:54:06.280349+0000","last_fullsized":"2024-06-14T22:54:06.280349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.276516+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T22:54:05.276516+0000","last_peered":"2024-06-14T22:54:05.276516+0000","last_clean":"2024-06-14T22:54:05.276516+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T22:54:05.276516+0000","last_undegraded":"2024-06-14T22:54:05.276516+0000","last_fullsized":"2024-06-14T22:54:05.276516+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280994+0000","last_change":"2024-06-14T22:54:06.280994+0000","last_active":"2024-06-14T22:54:06.280994+0000","last_peered":"2024-06-14T22:54:06.280994+0000","last_clean":"2024-06-14T22:54:06.280994+0000","last_became_active":"2024-06-14T22:54:06.279883+0000","last_became_peered":"2024-06-14T22:54:06.279883+0000","last_unstale":"2024-06-14T22:54:06.280994+0000","last_undegraded":"2024-06-14T22:54:06.280994+0000","last_fullsized":"2024-06-14T22:54:06.280994+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:51.156630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":99,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280497+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T22:54:06.280497+0000","last_peered":"2024-06-14T22:54:06.280497+0000","last_clean":"2024-06-14T22:54:06.280497+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T22:54:06.280497+0000","last_undegraded":"2024-06-14T22:54:06.280497+0000","last_fullsized":"2024-06-14T22:54:06.280497+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279157+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T22:54:05.279157+0000","last_peered":"2024-06-14T22:54:05.279157+0000","last_clean":"2024-06-14T22:54:05.279157+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T22:54:05.279157+0000","last_undegraded":"2024-06-14T22:54:05.279157+0000","last_fullsized":"2024-06-14T22:54:05.279157+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":109,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275417+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T22:54:05.275417+0000","last_peered":"2024-06-14T22:54:05.275417+0000","last_clean":"2024-06-14T22:54:05.275417+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T22:54:05.275417+0000","last_undegraded":"2024-06-14T22:54:05.275417+0000","last_fullsized":"2024-06-14T22:54:05.275417+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275776+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T22:54:05.275776+0000","last_peered":"2024-06-14T22:54:05.275776+0000","last_clean":"2024-06-14T22:54:05.275776+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T22:54:05.275776+0000","last_undegraded":"2024-06-14T22:54:05.275776+0000","last_fullsized":"2024-06-14T22:54:05.275776+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275483+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T22:54:05.275483+0000","last_peered":"2024-06-14T22:54:05.275483+0000","last_clean":"2024-06-14T22:54:05.275483+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T22:54:05.275483+0000","last_undegraded":"2024-06-14T22:54:05.275483+0000","last_fullsized":"2024-06-14T22:54:05.275483+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":91,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281907+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T22:54:06.281907+0000","last_peered":"2024-06-14T22:54:06.281907+0000","last_clean":"2024-06-14T22:54:06.281907+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T22:54:06.281907+0000","last_undegraded":"2024-06-14T22:54:06.281907+0000","last_fullsized":"2024-06-14T22:54:06.281907+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282328+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T22:54:06.282328+0000","last_peered":"2024-06-14T22:54:06.282328+0000","last_clean":"2024-06-14T22:54:06.282328+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T22:54:06.282328+0000","last_undegraded":"2024-06-14T22:54:06.282328+0000","last_fullsized":"2024-06-14T22:54:06.282328+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":480,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279291+0000","last_change":"2024-06-14T22:44:51.840491+0000","last_active":"2024-06-14T22:54:05.279291+0000","last_peered":"2024-06-14T22:54:05.279291+0000","last_clean":"2024-06-14T22:54:05.279291+0000","last_became_active":"2024-06-14T22:44:51.839990+0000","last_became_peered":"2024-06-14T22:44:51.839990+0000","last_unstale":"2024-06-14T22:54:05.279291+0000","last_undegraded":"2024-06-14T22:54:05.279291+0000","last_fullsized":"2024-06-14T22:54:05.279291+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:51:24.139036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281690+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T22:54:05.281690+0000","last_peered":"2024-06-14T22:54:05.281690+0000","last_clean":"2024-06-14T22:54:05.281690+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T22:54:05.281690+0000","last_undegraded":"2024-06-14T22:54:05.281690+0000","last_fullsized":"2024-06-14T22:54:05.281690+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":64,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279205+0000","last_change":"2024-06-14T22:54:06.279205+0000","last_active":"2024-06-14T22:54:06.279205+0000","last_peered":"2024-06-14T22:54:06.279205+0000","last_clean":"2024-06-14T22:54:06.279205+0000","last_became_active":"2024-06-14T22:54:06.278792+0000","last_became_peered":"2024-06-14T22:54:06.278792+0000","last_unstale":"2024-06-14T22:54:06.279205+0000","last_undegraded":"2024-06-14T22:54:06.279205+0000","last_fullsized":"2024-06-14T22:54:06.279205+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:55:02.899941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279434+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:54:05.279434+0000","last_peered":"2024-06-14T22:54:05.279434+0000","last_clean":"2024-06-14T22:54:05.279434+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:54:05.279434+0000","last_undegraded":"2024-06-14T22:54:05.279434+0000","last_fullsized":"2024-06-14T22:54:05.279434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":121,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279201+0000","last_change":"2024-06-14T22:49:25.476578+0000","last_active":"2024-06-14T22:54:05.279201+0000","last_peered":"2024-06-14T22:54:05.279201+0000","last_clean":"2024-06-14T22:54:05.279201+0000","last_became_active":"2024-06-14T22:49:25.476183+0000","last_became_peered":"2024-06-14T22:49:25.476183+0000","last_unstale":"2024-06-14T22:54:05.279201+0000","last_undegraded":"2024-06-14T22:54:05.279201+0000","last_fullsized":"2024-06-14T22:54:05.279201+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:01:44.585320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277248+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T22:54:06.277248+0000","last_peered":"2024-06-14T22:54:06.277248+0000","last_clean":"2024-06-14T22:54:06.277248+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T22:54:06.277248+0000","last_undegraded":"2024-06-14T22:54:06.277248+0000","last_fullsized":"2024-06-14T22:54:06.277248+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278487+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T22:54:05.278487+0000","last_peered":"2024-06-14T22:54:05.278487+0000","last_clean":"2024-06-14T22:54:05.278487+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T22:54:05.278487+0000","last_undegraded":"2024-06-14T22:54:05.278487+0000","last_fullsized":"2024-06-14T22:54:05.278487+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.275775+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T22:54:06.275775+0000","last_peered":"2024-06-14T22:54:06.275775+0000","last_clean":"2024-06-14T22:54:06.275775+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T22:54:06.275775+0000","last_undegraded":"2024-06-14T22:54:06.275775+0000","last_fullsized":"2024-06-14T22:54:06.275775+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270774+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T22:54:05.270774+0000","last_peered":"2024-06-14T22:54:05.270774+0000","last_clean":"2024-06-14T22:54:05.270774+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T22:54:05.270774+0000","last_undegraded":"2024-06-14T22:54:05.270774+0000","last_fullsized":"2024-06-14T22:54:05.270774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270053+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T22:54:05.270053+0000","last_peered":"2024-06-14T22:54:05.270053+0000","last_clean":"2024-06-14T22:54:05.270053+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T22:54:05.270053+0000","last_undegraded":"2024-06-14T22:54:05.270053+0000","last_fullsized":"2024-06-14T22:54:05.270053+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281419+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T22:54:05.281419+0000","last_peered":"2024-06-14T22:54:05.281419+0000","last_clean":"2024-06-14T22:54:05.281419+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T22:54:05.281419+0000","last_undegraded":"2024-06-14T22:54:05.281419+0000","last_fullsized":"2024-06-14T22:54:05.281419+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275641+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T22:54:05.275641+0000","last_peered":"2024-06-14T22:54:05.275641+0000","last_clean":"2024-06-14T22:54:05.275641+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T22:54:05.275641+0000","last_undegraded":"2024-06-14T22:54:05.275641+0000","last_fullsized":"2024-06-14T22:54:05.275641+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":119,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280880+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T22:54:06.280880+0000","last_peered":"2024-06-14T22:54:06.280880+0000","last_clean":"2024-06-14T22:54:06.280880+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T22:54:06.280880+0000","last_undegraded":"2024-06-14T22:54:06.280880+0000","last_fullsized":"2024-06-14T22:54:06.280880+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":117,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280346+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T22:54:06.280346+0000","last_peered":"2024-06-14T22:54:06.280346+0000","last_clean":"2024-06-14T22:54:06.280346+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T22:54:06.280346+0000","last_undegraded":"2024-06-14T22:54:06.280346+0000","last_fullsized":"2024-06-14T22:54:06.280346+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270215+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T22:54:05.270215+0000","last_peered":"2024-06-14T22:54:05.270215+0000","last_clean":"2024-06-14T22:54:05.270215+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T22:54:05.270215+0000","last_undegraded":"2024-06-14T22:54:05.270215+0000","last_fullsized":"2024-06-14T22:54:05.270215+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270541+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T22:54:05.270541+0000","last_peered":"2024-06-14T22:54:05.270541+0000","last_clean":"2024-06-14T22:54:05.270541+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T22:54:05.270541+0000","last_undegraded":"2024-06-14T22:54:05.270541+0000","last_fullsized":"2024-06-14T22:54:05.270541+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270139+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T22:54:05.270139+0000","last_peered":"2024-06-14T22:54:05.270139+0000","last_clean":"2024-06-14T22:54:05.270139+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T22:54:05.270139+0000","last_undegraded":"2024-06-14T22:54:05.270139+0000","last_fullsized":"2024-06-14T22:54:05.270139+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279227+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T22:54:05.279227+0000","last_peered":"2024-06-14T22:54:05.279227+0000","last_clean":"2024-06-14T22:54:05.279227+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T22:54:05.279227+0000","last_undegraded":"2024-06-14T22:54:05.279227+0000","last_fullsized":"2024-06-14T22:54:05.279227+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":74,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.276447+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T22:54:05.276447+0000","last_peered":"2024-06-14T22:54:05.276447+0000","last_clean":"2024-06-14T22:54:05.276447+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T22:54:05.276447+0000","last_undegraded":"2024-06-14T22:54:05.276447+0000","last_fullsized":"2024-06-14T22:54:05.276447+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.284651+0000","last_change":"2024-06-14T22:54:06.284651+0000","last_active":"2024-06-14T22:54:06.284651+0000","last_peered":"2024-06-14T22:54:06.284651+0000","last_clean":"2024-06-14T22:54:06.284651+0000","last_became_active":"2024-06-14T22:54:06.283564+0000","last_became_peered":"2024-06-14T22:54:06.283564+0000","last_unstale":"2024-06-14T22:54:06.284651+0000","last_undegraded":"2024-06-14T22:54:06.284651+0000","last_fullsized":"2024-06-14T22:54:06.284651+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:34.916272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270118+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T22:54:05.270118+0000","last_peered":"2024-06-14T22:54:05.270118+0000","last_clean":"2024-06-14T22:54:05.270118+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T22:54:05.270118+0000","last_undegraded":"2024-06-14T22:54:05.270118+0000","last_fullsized":"2024-06-14T22:54:05.270118+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281419+0000","last_change":"2024-06-14T22:54:06.281419+0000","last_active":"2024-06-14T22:54:06.281419+0000","last_peered":"2024-06-14T22:54:06.281419+0000","last_clean":"2024-06-14T22:54:06.281419+0000","last_became_active":"2024-06-14T22:54:06.279482+0000","last_became_peered":"2024-06-14T22:54:06.279482+0000","last_unstale":"2024-06-14T22:54:06.281419+0000","last_undegraded":"2024-06-14T22:54:06.281419+0000","last_fullsized":"2024-06-14T22:54:06.281419+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:06:37.039006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.277756+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T22:54:05.277756+0000","last_peered":"2024-06-14T22:54:05.277756+0000","last_clean":"2024-06-14T22:54:05.277756+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T22:54:05.277756+0000","last_undegraded":"2024-06-14T22:54:05.277756+0000","last_fullsized":"2024-06-14T22:54:05.277756+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280283+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T22:54:05.280283+0000","last_peered":"2024-06-14T22:54:05.280283+0000","last_clean":"2024-06-14T22:54:05.280283+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T22:54:05.280283+0000","last_undegraded":"2024-06-14T22:54:05.280283+0000","last_fullsized":"2024-06-14T22:54:05.280283+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275645+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:54:05.275645+0000","last_peered":"2024-06-14T22:54:05.275645+0000","last_clean":"2024-06-14T22:54:05.275645+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:54:05.275645+0000","last_undegraded":"2024-06-14T22:54:05.275645+0000","last_fullsized":"2024-06-14T22:54:05.275645+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275589+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T22:54:05.275589+0000","last_peered":"2024-06-14T22:54:05.275589+0000","last_clean":"2024-06-14T22:54:05.275589+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T22:54:05.275589+0000","last_undegraded":"2024-06-14T22:54:05.275589+0000","last_fullsized":"2024-06-14T22:54:05.275589+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":105,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275170+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:54:05.275170+0000","last_peered":"2024-06-14T22:54:05.275170+0000","last_clean":"2024-06-14T22:54:05.275170+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:54:05.275170+0000","last_undegraded":"2024-06-14T22:54:05.275170+0000","last_fullsized":"2024-06-14T22:54:05.275170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270411+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T22:54:05.270411+0000","last_peered":"2024-06-14T22:54:05.270411+0000","last_clean":"2024-06-14T22:54:05.270411+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T22:54:05.270411+0000","last_undegraded":"2024-06-14T22:54:05.270411+0000","last_fullsized":"2024-06-14T22:54:05.270411+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.275561+0000","last_change":"2024-06-14T22:54:06.275561+0000","last_active":"2024-06-14T22:54:06.275561+0000","last_peered":"2024-06-14T22:54:06.275561+0000","last_clean":"2024-06-14T22:54:06.275561+0000","last_became_active":"2024-06-14T22:54:06.274930+0000","last_became_peered":"2024-06-14T22:54:06.274930+0000","last_unstale":"2024-06-14T22:54:06.275561+0000","last_undegraded":"2024-06-14T22:54:06.275561+0000","last_fullsized":"2024-06-14T22:54:06.275561+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:11.022230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281532+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T22:54:05.281532+0000","last_peered":"2024-06-14T22:54:05.281532+0000","last_clean":"2024-06-14T22:54:05.281532+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T22:54:05.281532+0000","last_undegraded":"2024-06-14T22:54:05.281532+0000","last_fullsized":"2024-06-14T22:54:05.281532+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":64,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279332+0000","last_change":"2024-06-14T22:54:06.279332+0000","last_active":"2024-06-14T22:54:06.279332+0000","last_peered":"2024-06-14T22:54:06.279332+0000","last_clean":"2024-06-14T22:54:06.279332+0000","last_became_active":"2024-06-14T22:54:06.279020+0000","last_became_peered":"2024-06-14T22:54:06.279020+0000","last_unstale":"2024-06-14T22:54:06.279332+0000","last_undegraded":"2024-06-14T22:54:06.279332+0000","last_fullsized":"2024-06-14T22:54:06.279332+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:52:57.777668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.285313+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T22:54:06.285313+0000","last_peered":"2024-06-14T22:54:06.285313+0000","last_clean":"2024-06-14T22:54:06.285313+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T22:54:06.285313+0000","last_undegraded":"2024-06-14T22:54:06.285313+0000","last_fullsized":"2024-06-14T22:54:06.285313+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281615+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T22:54:06.281615+0000","last_peered":"2024-06-14T22:54:06.281615+0000","last_clean":"2024-06-14T22:54:06.281615+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T22:54:06.281615+0000","last_undegraded":"2024-06-14T22:54:06.281615+0000","last_fullsized":"2024-06-14T22:54:06.281615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280203+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T22:54:05.280203+0000","last_peered":"2024-06-14T22:54:05.280203+0000","last_clean":"2024-06-14T22:54:05.280203+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T22:54:05.280203+0000","last_undegraded":"2024-06-14T22:54:05.280203+0000","last_fullsized":"2024-06-14T22:54:05.280203+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":108,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282120+0000","last_change":"2024-06-14T22:54:06.282120+0000","last_active":"2024-06-14T22:54:06.282120+0000","last_peered":"2024-06-14T22:54:06.282120+0000","last_clean":"2024-06-14T22:54:06.282120+0000","last_became_active":"2024-06-14T22:54:06.280898+0000","last_became_peered":"2024-06-14T22:54:06.280898+0000","last_unstale":"2024-06-14T22:54:06.282120+0000","last_undegraded":"2024-06-14T22:54:06.282120+0000","last_fullsized":"2024-06-14T22:54:06.282120+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:30:45.544457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.278300+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T22:54:06.278300+0000","last_peered":"2024-06-14T22:54:06.278300+0000","last_clean":"2024-06-14T22:54:06.278300+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T22:54:06.278300+0000","last_undegraded":"2024-06-14T22:54:06.278300+0000","last_fullsized":"2024-06-14T22:54:06.278300+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.284728+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T22:54:06.284728+0000","last_peered":"2024-06-14T22:54:06.284728+0000","last_clean":"2024-06-14T22:54:06.284728+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T22:54:06.284728+0000","last_undegraded":"2024-06-14T22:54:06.284728+0000","last_fullsized":"2024-06-14T22:54:06.284728+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280483+0000","last_change":"2024-06-14T22:54:06.280483+0000","last_active":"2024-06-14T22:54:06.280483+0000","last_peered":"2024-06-14T22:54:06.280483+0000","last_clean":"2024-06-14T22:54:06.280483+0000","last_became_active":"2024-06-14T22:54:06.278985+0000","last_became_peered":"2024-06-14T22:54:06.278985+0000","last_unstale":"2024-06-14T22:54:06.280483+0000","last_undegraded":"2024-06-14T22:54:06.280483+0000","last_fullsized":"2024-06-14T22:54:06.280483+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:43:40.478580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":118,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.274772+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T22:54:05.274772+0000","last_peered":"2024-06-14T22:54:05.274772+0000","last_clean":"2024-06-14T22:54:05.274772+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T22:54:05.274772+0000","last_undegraded":"2024-06-14T22:54:05.274772+0000","last_fullsized":"2024-06-14T22:54:05.274772+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":109,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277110+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T22:54:06.277110+0000","last_peered":"2024-06-14T22:54:06.277110+0000","last_clean":"2024-06-14T22:54:06.277110+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T22:54:06.277110+0000","last_undegraded":"2024-06-14T22:54:06.277110+0000","last_fullsized":"2024-06-14T22:54:06.277110+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":77,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.270191+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T22:54:06.270191+0000","last_peered":"2024-06-14T22:54:06.270191+0000","last_clean":"2024-06-14T22:54:06.270191+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T22:54:06.270191+0000","last_undegraded":"2024-06-14T22:54:06.270191+0000","last_fullsized":"2024-06-14T22:54:06.270191+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281117+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T22:54:06.281117+0000","last_peered":"2024-06-14T22:54:06.281117+0000","last_clean":"2024-06-14T22:54:06.281117+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T22:54:06.281117+0000","last_undegraded":"2024-06-14T22:54:06.281117+0000","last_fullsized":"2024-06-14T22:54:06.281117+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280162+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T22:54:05.280162+0000","last_peered":"2024-06-14T22:54:05.280162+0000","last_clean":"2024-06-14T22:54:05.280162+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T22:54:05.280162+0000","last_undegraded":"2024-06-14T22:54:05.280162+0000","last_fullsized":"2024-06-14T22:54:05.280162+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280576+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T22:54:06.280576+0000","last_peered":"2024-06-14T22:54:06.280576+0000","last_clean":"2024-06-14T22:54:06.280576+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T22:54:06.280576+0000","last_undegraded":"2024-06-14T22:54:06.280576+0000","last_fullsized":"2024-06-14T22:54:06.280576+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":85,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.282162+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:54:05.282162+0000","last_peered":"2024-06-14T22:54:05.282162+0000","last_clean":"2024-06-14T22:54:05.282162+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:54:05.282162+0000","last_undegraded":"2024-06-14T22:54:05.282162+0000","last_fullsized":"2024-06-14T22:54:05.282162+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":123,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282972+0000","last_change":"2024-06-14T22:54:06.282972+0000","last_active":"2024-06-14T22:54:06.282972+0000","last_peered":"2024-06-14T22:54:06.282972+0000","last_clean":"2024-06-14T22:54:06.282972+0000","last_became_active":"2024-06-14T22:54:06.282622+0000","last_became_peered":"2024-06-14T22:54:06.282622+0000","last_unstale":"2024-06-14T22:54:06.282972+0000","last_undegraded":"2024-06-14T22:54:06.282972+0000","last_fullsized":"2024-06-14T22:54:06.282972+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:31:22.051955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":108,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279284+0000","last_change":"2024-06-14T22:54:06.279284+0000","last_active":"2024-06-14T22:54:06.279284+0000","last_peered":"2024-06-14T22:54:06.279284+0000","last_clean":"2024-06-14T22:54:06.279284+0000","last_became_active":"2024-06-14T22:54:06.278861+0000","last_became_peered":"2024-06-14T22:54:06.278861+0000","last_unstale":"2024-06-14T22:54:06.279284+0000","last_undegraded":"2024-06-14T22:54:06.279284+0000","last_fullsized":"2024-06-14T22:54:06.279284+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:17:47.260769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.273550+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T22:54:05.273550+0000","last_peered":"2024-06-14T22:54:05.273550+0000","last_clean":"2024-06-14T22:54:05.273550+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T22:54:05.273550+0000","last_undegraded":"2024-06-14T22:54:05.273550+0000","last_fullsized":"2024-06-14T22:54:05.273550+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275021+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T22:54:05.275021+0000","last_peered":"2024-06-14T22:54:05.275021+0000","last_clean":"2024-06-14T22:54:05.275021+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T22:54:05.275021+0000","last_undegraded":"2024-06-14T22:54:05.275021+0000","last_fullsized":"2024-06-14T22:54:05.275021+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234254,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600240,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":93144912,"statfs":{"total":95995035648,"available":95380389888,"internally_reserved":0,"allocated":2260992,"data_stored":1493426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:55:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.98499999999999999,"15min":1.127},"min":{"1min":0.378,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":0.73799999999999999,"5min":6.9199999999999999,"15min":9.3919999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.1559999999999999,"15min":1.264},"min":{"1min":0.371,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.70399999999999996,"5min":6.8440000000000003,"15min":10.416},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.212,"15min":1.3700000000000001},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80500000000000005,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":1.3560000000000001,"15min":1.464},"min":{"1min":0.38100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.56599999999999995}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.63400000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.67000000000000004,"15min":0.80400000000000005},"min":{"1min":0.27300000000000002,"5min":0.248,"15min":0.214},"max":{"1min":0.67900000000000005,"5min":5.8449999999999998,"15min":9.7430000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.33900000000000002,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":0.64500000000000002,"5min":4.2709999999999999,"15min":6.6660000000000004},"last":0.38600000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.752,"15min":0.77200000000000002},"min":{"1min":0.34999999999999998,"5min":0.26100000000000001,"15min":0.22},"max":{"1min":0.70599999999999996,"5min":3.9860000000000002,"15min":7.9870000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.88,"15min":0.88600000000000001},"min":{"1min":0.33500000000000002,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.76200000000000001,"5min":5.3659999999999997,"15min":9.5120000000000005},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.75800000000000001,"15min":0.81999999999999995},"min":{"1min":0.39400000000000002,"5min":0.308,"15min":0.28100000000000003},"max":{"1min":0.72499999999999998,"5min":5.806,"15min":5.806},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.76700000000000002,"15min":0.71999999999999997},"min":{"1min":0.39000000000000001,"5min":0.311,"15min":0.29899999999999999},"max":{"1min":0.70299999999999996,"5min":4.3090000000000002,"15min":5.7430000000000003},"last":0.55000000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311836,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":93433316,"statfs":{"total":95995035648,"available":95675715584,"internally_reserved":0,"allocated":2244608,"data_stored":1492957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.1739999999999999,"15min":1.1839999999999999},"min":{"1min":0.44900000000000001,"5min":0.224,"15min":0.218},"max":{"1min":0.67700000000000005,"5min":7.1689999999999996,"15min":31.358000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.99299999999999999,"15min":1.411},"min":{"1min":0.38500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.73799999999999999,"5min":4.9180000000000001,"15min":29.399000000000001},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.224,"15min":1.304},"min":{"1min":0.39400000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73399999999999999,"5min":9.9559999999999995,"15min":9.9559999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":1.2769999999999999,"15min":1.276},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.67400000000000004,"5min":9.5510000000000002,"15min":9.5510000000000002},"last":0.46300000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.50600000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":1.2110000000000001,"15min":0.88800000000000001},"min":{"1min":0.35299999999999998,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":0.71999999999999997,"5min":47.698,"15min":47.698},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.496,"5min":1.181,"15min":0.82599999999999996},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.23899999999999999},"max":{"1min":0.72699999999999998,"5min":48.197000000000003,"15min":48.197000000000003},"last":0.39300000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.88600000000000001,"15min":0.79100000000000004},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.22600000000000001},"max":{"1min":0.75800000000000001,"5min":8.6500000000000004,"15min":8.6500000000000004},"last":0.439},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.95399999999999996,"15min":0.86199999999999999},"min":{"1min":0.41199999999999998,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":8.6349999999999998,"15min":9.5609999999999999},"last":0.58099999999999996}]},{"osd":7,"last update":"Fri Jun 14 22:54:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.84199999999999997,"15min":0.82099999999999995},"min":{"1min":0.38100000000000001,"5min":0.31,"15min":0.28799999999999998},"max":{"1min":0.82299999999999995,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.90500000000000003,"15min":0.83299999999999996},"min":{"1min":0.44500000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78100000000000003,"5min":8.673,"15min":8.673},"last":0.60199999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757529,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896852,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":92848300,"statfs":{"total":95995035648,"available":95076659200,"internally_reserved":0,"allocated":1642496,"data_stored":898403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":1.1299999999999999,"15min":1.2270000000000001},"min":{"1min":0.41099999999999998,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.69899999999999995,"5min":6.0369999999999999,"15min":17.766999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.53500000000000003,"5min":1.046,"15min":1.2689999999999999},"min":{"1min":0.38500000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70999999999999996,"5min":5.2439999999999998,"15min":17.806000000000001},"last":0.59999999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.2809999999999999,"15min":1.357},"min":{"1min":0.36099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66400000000000003,"5min":7.694,"15min":7.694},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.501,"5min":1.129,"15min":1.169},"min":{"1min":0.33700000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.72599999999999998,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.67500000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Jun 14 22:55:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.63600000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.60599999999999998,"15min":0.73599999999999999},"min":{"1min":0.313,"5min":0.23300000000000001,"15min":0.214},"max":{"1min":0.64400000000000002,"5min":4.9790000000000001,"15min":11.771000000000001},"last":0.432},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.69699999999999995,"15min":0.74299999999999999},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.23200000000000001},"max":{"1min":0.66100000000000003,"5min":5.8109999999999999,"15min":10.032},"last":0.46500000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.84699999999999998,"15min":0.86599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.23000000000000001},"max":{"1min":0.67900000000000005,"5min":9.6020000000000003,"15min":9.8369999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.85699999999999998,"15min":0.88300000000000001},"min":{"1min":0.33300000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.74399999999999999,"5min":12.339,"15min":12.339},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.75800000000000001,"15min":0.82299999999999995},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":5.2629999999999999,"15min":10.462999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.72999999999999998,"15min":0.83899999999999997},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69299999999999995,"5min":10.638999999999999,"15min":10.638999999999999},"last":0.48299999999999998}]}]},{"osd":4,"up_from":30,"seq":128849019165,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599652,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":93145500,"statfs":{"total":95995035648,"available":95380992000,"internally_reserved":0,"allocated":1658880,"data_stored":902723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.0640000000000001,"15min":1.0609999999999999},"min":{"1min":0.39500000000000002,"5min":0.219,"15min":0.218},"max":{"1min":0.70199999999999996,"5min":4.641,"15min":7.5650000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":1.1499999999999999,"15min":1.365},"min":{"1min":0.39800000000000002,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":0.65400000000000003,"5min":5.7569999999999997,"15min":9.1470000000000002},"last":0.59599999999999997}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":1.1779999999999999,"15min":1.1499999999999999},"min":{"1min":0.34899999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65300000000000002,"5min":5.141,"15min":5.141},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.2030000000000001,"15min":1.3360000000000001},"min":{"1min":0.35299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.70799999999999996,"5min":6.2709999999999999,"15min":6.2709999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.438}]},{"osd":5,"last update":"Fri Jun 14 22:54:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.72599999999999998,"15min":0.749},"min":{"1min":0.31,"5min":0.24199999999999999,"15min":0.21199999999999999},"max":{"1min":0.67900000000000005,"5min":3.964,"15min":5.774},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":0.78200000000000003,"5min":4.1269999999999998,"15min":4.7480000000000002},"last":0.32000000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.65200000000000002,"15min":0.78100000000000003},"min":{"1min":0.34399999999999997,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":3.8929999999999998,"15min":10.904999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61499999999999999,"15min":0.77200000000000002},"min":{"1min":0.371,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":0.64600000000000002,"5min":3.7450000000000001,"15min":9.2390000000000008},"last":0.45000000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:54:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.63,"15min":0.752},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.246},"max":{"1min":0.72399999999999998,"5min":3.7029999999999998,"15min":6.298},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.74299999999999999,"15min":0.76400000000000001},"min":{"1min":0.38300000000000001,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.73499999999999999,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":0.42299999999999999}]}]},{"osd":3,"up_from":138,"seq":592705486864,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77328,"kb_used_data":1604,"kb_used_omap":4,"kb_used_meta":75707,"kb_avail":93667824,"statfs":{"total":95995035648,"available":95915851776,"internally_reserved":0,"allocated":1642496,"data_stored":898176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":77524446},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.253},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.42499999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.37}]},{"osd":2,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.24299999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.48799999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.626}]}]},{"osd":2,"up_from":133,"seq":571230650390,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62544,"kb_used_data":1604,"kb_used_omap":6,"kb_used_meta":60921,"kb_avail":93682608,"statfs":{"total":95995035648,"available":95930990592,"internally_reserved":0,"allocated":1642496,"data_stored":898390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":62383605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.29199999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.22500000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.317},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.33400000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.53500000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.504},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.443}]},{"osd":6,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48199999999999998}]},{"osd":7,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54200000000000004}]}]},{"osd":1,"up_from":116,"seq":498216206408,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":213204,"kb_used_data":1600,"kb_used_omap":5,"kb_used_meta":211578,"kb_avail":93531948,"statfs":{"total":95995035648,"available":95776714752,"internally_reserved":0,"allocated":1638400,"data_stored":898086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":216656152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":1.8759999999999999,"15min":2.3620000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41299999999999998,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.4159999999999999,"15min":1.746},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.433,"5min":10.426,"15min":10.426},"last":0.22500000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.251}]},{"osd":3,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.5229999999999999,"15min":2.0209999999999999},"min":{"1min":0.41799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.95399999999999996,"5min":8.8599999999999994,"15min":8.8599999999999994},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.625,"15min":1.9490000000000001},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88600000000000001,"5min":11.161,"15min":11.161},"last":0.63}]},{"osd":5,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.6719999999999999,"15min":1.8420000000000001},"min":{"1min":0.40799999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":1.52,"15min":2.0760000000000001},"min":{"1min":0.438,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.97899999999999998,"5min":8.125,"15min":8.125},"last":0.495}]},{"osd":6,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":1.7749999999999999,"15min":1.929},"min":{"1min":0.39700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89700000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":1.4870000000000001,"15min":2.0750000000000002},"min":{"1min":0.44,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.92300000000000004,"5min":8.1479999999999997,"15min":8.1479999999999997},"last":0.68400000000000005}]},{"osd":7,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.589,"15min":2.089},"min":{"1min":0.45900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79900000000000004,"5min":9.0960000000000001,"15min":9.0960000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.621,"5min":1.5600000000000001,"15min":1.702},"min":{"1min":0.48199999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96499999999999997,"5min":9.5449999999999999,"15min":9.5449999999999999},"last":0.55700000000000005}]}]},{"osd":0,"up_from":91,"seq":390842024062,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":441492,"kb_used_data":2180,"kb_used_omap":5,"kb_used_meta":439290,"kb_avail":93303660,"statfs":{"total":95995035648,"available":95542947840,"internally_reserved":0,"allocated":2232320,"data_stored":1486453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":449833050},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:54:27 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.7989999999999999,"15min":1.609},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.44600000000000001,"5min":9.6999999999999993,"15min":9.6999999999999993},"last":0.317},{"interface":"front","average":{"1min":0.34999999999999998,"5min":1.895,"15min":2.2869999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.58199999999999996,"5min":10.613,"15min":10.613},"last":0.28699999999999998}]},{"osd":2,"last update":"Fri Jun 14 22:54:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.30099999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:55:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.35199999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.427,"15min":1.6850000000000001},"min":{"1min":0.40000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.83999999999999997,"5min":8.7889999999999997,"15min":8.9510000000000005},"last":0.623},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.2929999999999999,"15min":1.3460000000000001},"min":{"1min":0.436,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.87,"5min":13.574,"15min":13.574},"last":0.51400000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.3999999999999999,"15min":1.391},"min":{"1min":0.41399999999999998,"5min":0.317,"15min":0.29299999999999998},"max":{"1min":0.84099999999999997,"5min":13.616,"15min":13.616},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":1.3620000000000001,"15min":1.4850000000000001},"min":{"1min":0.45300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.0409999999999999,"5min":7.6509999999999998,"15min":9.6620000000000008},"last":0.61099999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":1.4039999999999999,"15min":1.421},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81699999999999995,"5min":13.645,"15min":13.645},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.248,"15min":1.5780000000000001},"min":{"1min":0.44800000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.022,"5min":7.0119999999999996,"15min":9.8599999999999994},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:54:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.2509999999999999,"15min":1.617},"min":{"1min":0.42399999999999999,"5min":0.38700000000000001,"15min":0.33100000000000002},"max":{"1min":1.0009999999999999,"5min":6.9870000000000001,"15min":9.6370000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":1.4339999999999999,"15min":1.7989999999999999},"min":{"1min":0.44400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.84799999999999998,"5min":8.8290000000000006,"15min":8.9900000000000002},"last":0.64400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:55:15.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:55:15.758 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:15.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:15 smithi086 ceph-mon[97913]: pgmap v531: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:15.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:15 smithi086 ceph-mon[102754]: pgmap v531: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:15.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:15 smithi137 ceph-mon[90634]: pgmap v531: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:16.872 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:16 smithi086 ceph-mon[97913]: from='client.45446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:16.872 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:16 smithi086 ceph-mon[102754]: from='client.45446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:16.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:16 smithi137 ceph-mon[90634]: from='client.45446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:17.515 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:55:17.516 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:55:17.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:17 smithi086 ceph-mon[97913]: pgmap v532: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:17.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:17 smithi086 ceph-mon[102754]: pgmap v532: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:17.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:17 smithi137 ceph-mon[90634]: pgmap v532: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:18.197 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":532,"stamp":"2024-06-14T22:55:16.562986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":300,"num_write_kb":2445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3203148,"kb_used_data":14612,"kb_used_omap":29,"kb_used_meta":3188386,"kb_avail":746758068,"statfs":{"total":767960285184,"available":764680261632,"internally_reserved":0,"allocated":14962688,"data_stored":8968614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":3264907941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002711"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279751+0000","last_change":"2024-06-14T22:54:06.279751+0000","last_active":"2024-06-14T22:54:06.279751+0000","last_peered":"2024-06-14T22:54:06.279751+0000","last_clean":"2024-06-14T22:54:06.279751+0000","last_became_active":"2024-06-14T22:54:06.279443+0000","last_became_peered":"2024-06-14T22:54:06.279443+0000","last_unstale":"2024-06-14T22:54:06.279751+0000","last_undegraded":"2024-06-14T22:54:06.279751+0000","last_fullsized":"2024-06-14T22:54:06.279751+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:13:26.144924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278747+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T22:54:05.278747+0000","last_peered":"2024-06-14T22:54:05.278747+0000","last_clean":"2024-06-14T22:54:05.278747+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T22:54:05.278747+0000","last_undegraded":"2024-06-14T22:54:05.278747+0000","last_fullsized":"2024-06-14T22:54:05.278747+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":99,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.274802+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T22:54:06.274802+0000","last_peered":"2024-06-14T22:54:06.274802+0000","last_clean":"2024-06-14T22:54:06.274802+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T22:54:06.274802+0000","last_undegraded":"2024-06-14T22:54:06.274802+0000","last_fullsized":"2024-06-14T22:54:06.274802+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277962+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T22:54:06.277962+0000","last_peered":"2024-06-14T22:54:06.277962+0000","last_clean":"2024-06-14T22:54:06.277962+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T22:54:06.277962+0000","last_undegraded":"2024-06-14T22:54:06.277962+0000","last_fullsized":"2024-06-14T22:54:06.277962+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":89,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.278179+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T22:54:06.278179+0000","last_peered":"2024-06-14T22:54:06.278179+0000","last_clean":"2024-06-14T22:54:06.278179+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T22:54:06.278179+0000","last_undegraded":"2024-06-14T22:54:06.278179+0000","last_fullsized":"2024-06-14T22:54:06.278179+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":83,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278562+0000","last_change":"2024-06-14T22:34:45.813267+0000","last_active":"2024-06-14T22:54:05.278562+0000","last_peered":"2024-06-14T22:54:05.278562+0000","last_clean":"2024-06-14T22:54:05.278562+0000","last_became_active":"2024-06-14T22:34:45.813003+0000","last_became_peered":"2024-06-14T22:34:45.813003+0000","last_unstale":"2024-06-14T22:54:05.278562+0000","last_undegraded":"2024-06-14T22:54:05.278562+0000","last_fullsized":"2024-06-14T22:54:05.278562+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:46.446483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280927+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T22:54:06.280927+0000","last_peered":"2024-06-14T22:54:06.280927+0000","last_clean":"2024-06-14T22:54:06.280927+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T22:54:06.280927+0000","last_undegraded":"2024-06-14T22:54:06.280927+0000","last_fullsized":"2024-06-14T22:54:06.280927+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":108,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280520+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T22:54:05.280520+0000","last_peered":"2024-06-14T22:54:05.280520+0000","last_clean":"2024-06-14T22:54:05.280520+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T22:54:05.280520+0000","last_undegraded":"2024-06-14T22:54:05.280520+0000","last_fullsized":"2024-06-14T22:54:05.280520+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":101,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280442+0000","last_change":"2024-06-14T22:44:51.839766+0000","last_active":"2024-06-14T22:54:05.280442+0000","last_peered":"2024-06-14T22:54:05.280442+0000","last_clean":"2024-06-14T22:54:05.280442+0000","last_became_active":"2024-06-14T22:44:51.838970+0000","last_became_peered":"2024-06-14T22:44:51.838970+0000","last_unstale":"2024-06-14T22:54:05.280442+0000","last_undegraded":"2024-06-14T22:54:05.280442+0000","last_fullsized":"2024-06-14T22:54:05.280442+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:31:59.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280352+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:54:05.280352+0000","last_peered":"2024-06-14T22:54:05.280352+0000","last_clean":"2024-06-14T22:54:05.280352+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:54:05.280352+0000","last_undegraded":"2024-06-14T22:54:05.280352+0000","last_fullsized":"2024-06-14T22:54:05.280352+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280342+0000","last_change":"2024-06-14T22:49:25.474899+0000","last_active":"2024-06-14T22:54:05.280342+0000","last_peered":"2024-06-14T22:54:05.280342+0000","last_clean":"2024-06-14T22:54:05.280342+0000","last_became_active":"2024-06-14T22:49:25.474251+0000","last_became_peered":"2024-06-14T22:49:25.474251+0000","last_unstale":"2024-06-14T22:54:05.280342+0000","last_undegraded":"2024-06-14T22:54:05.280342+0000","last_fullsized":"2024-06-14T22:54:05.280342+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:22:05.971278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281348+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T22:54:06.281348+0000","last_peered":"2024-06-14T22:54:06.281348+0000","last_clean":"2024-06-14T22:54:06.281348+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T22:54:06.281348+0000","last_undegraded":"2024-06-14T22:54:06.281348+0000","last_fullsized":"2024-06-14T22:54:06.281348+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":84,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279141+0000","last_change":"2024-06-14T22:34:45.810186+0000","last_active":"2024-06-14T22:54:05.279141+0000","last_peered":"2024-06-14T22:54:05.279141+0000","last_clean":"2024-06-14T22:54:05.279141+0000","last_became_active":"2024-06-14T22:34:45.809869+0000","last_became_peered":"2024-06-14T22:34:45.809869+0000","last_unstale":"2024-06-14T22:54:05.279141+0000","last_undegraded":"2024-06-14T22:54:05.279141+0000","last_fullsized":"2024-06-14T22:54:05.279141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:03:45.652442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278987+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T22:54:05.278987+0000","last_peered":"2024-06-14T22:54:05.278987+0000","last_clean":"2024-06-14T22:54:05.278987+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T22:54:05.278987+0000","last_undegraded":"2024-06-14T22:54:05.278987+0000","last_fullsized":"2024-06-14T22:54:05.278987+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280349+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T22:54:06.280349+0000","last_peered":"2024-06-14T22:54:06.280349+0000","last_clean":"2024-06-14T22:54:06.280349+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T22:54:06.280349+0000","last_undegraded":"2024-06-14T22:54:06.280349+0000","last_fullsized":"2024-06-14T22:54:06.280349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.276516+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T22:54:05.276516+0000","last_peered":"2024-06-14T22:54:05.276516+0000","last_clean":"2024-06-14T22:54:05.276516+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T22:54:05.276516+0000","last_undegraded":"2024-06-14T22:54:05.276516+0000","last_fullsized":"2024-06-14T22:54:05.276516+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280994+0000","last_change":"2024-06-14T22:54:06.280994+0000","last_active":"2024-06-14T22:54:06.280994+0000","last_peered":"2024-06-14T22:54:06.280994+0000","last_clean":"2024-06-14T22:54:06.280994+0000","last_became_active":"2024-06-14T22:54:06.279883+0000","last_became_peered":"2024-06-14T22:54:06.279883+0000","last_unstale":"2024-06-14T22:54:06.280994+0000","last_undegraded":"2024-06-14T22:54:06.280994+0000","last_fullsized":"2024-06-14T22:54:06.280994+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:51.156630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":99,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280497+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T22:54:06.280497+0000","last_peered":"2024-06-14T22:54:06.280497+0000","last_clean":"2024-06-14T22:54:06.280497+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T22:54:06.280497+0000","last_undegraded":"2024-06-14T22:54:06.280497+0000","last_fullsized":"2024-06-14T22:54:06.280497+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279157+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T22:54:05.279157+0000","last_peered":"2024-06-14T22:54:05.279157+0000","last_clean":"2024-06-14T22:54:05.279157+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T22:54:05.279157+0000","last_undegraded":"2024-06-14T22:54:05.279157+0000","last_fullsized":"2024-06-14T22:54:05.279157+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":109,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275417+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T22:54:05.275417+0000","last_peered":"2024-06-14T22:54:05.275417+0000","last_clean":"2024-06-14T22:54:05.275417+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T22:54:05.275417+0000","last_undegraded":"2024-06-14T22:54:05.275417+0000","last_fullsized":"2024-06-14T22:54:05.275417+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275776+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T22:54:05.275776+0000","last_peered":"2024-06-14T22:54:05.275776+0000","last_clean":"2024-06-14T22:54:05.275776+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T22:54:05.275776+0000","last_undegraded":"2024-06-14T22:54:05.275776+0000","last_fullsized":"2024-06-14T22:54:05.275776+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275483+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T22:54:05.275483+0000","last_peered":"2024-06-14T22:54:05.275483+0000","last_clean":"2024-06-14T22:54:05.275483+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T22:54:05.275483+0000","last_undegraded":"2024-06-14T22:54:05.275483+0000","last_fullsized":"2024-06-14T22:54:05.275483+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":91,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281907+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T22:54:06.281907+0000","last_peered":"2024-06-14T22:54:06.281907+0000","last_clean":"2024-06-14T22:54:06.281907+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T22:54:06.281907+0000","last_undegraded":"2024-06-14T22:54:06.281907+0000","last_fullsized":"2024-06-14T22:54:06.281907+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282328+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T22:54:06.282328+0000","last_peered":"2024-06-14T22:54:06.282328+0000","last_clean":"2024-06-14T22:54:06.282328+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T22:54:06.282328+0000","last_undegraded":"2024-06-14T22:54:06.282328+0000","last_fullsized":"2024-06-14T22:54:06.282328+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":480,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279291+0000","last_change":"2024-06-14T22:44:51.840491+0000","last_active":"2024-06-14T22:54:05.279291+0000","last_peered":"2024-06-14T22:54:05.279291+0000","last_clean":"2024-06-14T22:54:05.279291+0000","last_became_active":"2024-06-14T22:44:51.839990+0000","last_became_peered":"2024-06-14T22:44:51.839990+0000","last_unstale":"2024-06-14T22:54:05.279291+0000","last_undegraded":"2024-06-14T22:54:05.279291+0000","last_fullsized":"2024-06-14T22:54:05.279291+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:51:24.139036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281690+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T22:54:05.281690+0000","last_peered":"2024-06-14T22:54:05.281690+0000","last_clean":"2024-06-14T22:54:05.281690+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T22:54:05.281690+0000","last_undegraded":"2024-06-14T22:54:05.281690+0000","last_fullsized":"2024-06-14T22:54:05.281690+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":64,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279205+0000","last_change":"2024-06-14T22:54:06.279205+0000","last_active":"2024-06-14T22:54:06.279205+0000","last_peered":"2024-06-14T22:54:06.279205+0000","last_clean":"2024-06-14T22:54:06.279205+0000","last_became_active":"2024-06-14T22:54:06.278792+0000","last_became_peered":"2024-06-14T22:54:06.278792+0000","last_unstale":"2024-06-14T22:54:06.279205+0000","last_undegraded":"2024-06-14T22:54:06.279205+0000","last_fullsized":"2024-06-14T22:54:06.279205+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:55:02.899941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279434+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:54:05.279434+0000","last_peered":"2024-06-14T22:54:05.279434+0000","last_clean":"2024-06-14T22:54:05.279434+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:54:05.279434+0000","last_undegraded":"2024-06-14T22:54:05.279434+0000","last_fullsized":"2024-06-14T22:54:05.279434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":121,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279201+0000","last_change":"2024-06-14T22:49:25.476578+0000","last_active":"2024-06-14T22:54:05.279201+0000","last_peered":"2024-06-14T22:54:05.279201+0000","last_clean":"2024-06-14T22:54:05.279201+0000","last_became_active":"2024-06-14T22:49:25.476183+0000","last_became_peered":"2024-06-14T22:49:25.476183+0000","last_unstale":"2024-06-14T22:54:05.279201+0000","last_undegraded":"2024-06-14T22:54:05.279201+0000","last_fullsized":"2024-06-14T22:54:05.279201+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:01:44.585320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":89,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277780+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T22:54:06.277780+0000","last_peered":"2024-06-14T22:54:06.277780+0000","last_clean":"2024-06-14T22:54:06.277780+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T22:54:06.277780+0000","last_undegraded":"2024-06-14T22:54:06.277780+0000","last_fullsized":"2024-06-14T22:54:06.277780+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.278487+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T22:54:05.278487+0000","last_peered":"2024-06-14T22:54:05.278487+0000","last_clean":"2024-06-14T22:54:05.278487+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T22:54:05.278487+0000","last_undegraded":"2024-06-14T22:54:05.278487+0000","last_fullsized":"2024-06-14T22:54:05.278487+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.275775+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T22:54:06.275775+0000","last_peered":"2024-06-14T22:54:06.275775+0000","last_clean":"2024-06-14T22:54:06.275775+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T22:54:06.275775+0000","last_undegraded":"2024-06-14T22:54:06.275775+0000","last_fullsized":"2024-06-14T22:54:06.275775+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270774+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T22:54:05.270774+0000","last_peered":"2024-06-14T22:54:05.270774+0000","last_clean":"2024-06-14T22:54:05.270774+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T22:54:05.270774+0000","last_undegraded":"2024-06-14T22:54:05.270774+0000","last_fullsized":"2024-06-14T22:54:05.270774+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270053+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T22:54:05.270053+0000","last_peered":"2024-06-14T22:54:05.270053+0000","last_clean":"2024-06-14T22:54:05.270053+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T22:54:05.270053+0000","last_undegraded":"2024-06-14T22:54:05.270053+0000","last_fullsized":"2024-06-14T22:54:05.270053+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281419+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T22:54:05.281419+0000","last_peered":"2024-06-14T22:54:05.281419+0000","last_clean":"2024-06-14T22:54:05.281419+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T22:54:05.281419+0000","last_undegraded":"2024-06-14T22:54:05.281419+0000","last_fullsized":"2024-06-14T22:54:05.281419+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275641+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T22:54:05.275641+0000","last_peered":"2024-06-14T22:54:05.275641+0000","last_clean":"2024-06-14T22:54:05.275641+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T22:54:05.275641+0000","last_undegraded":"2024-06-14T22:54:05.275641+0000","last_fullsized":"2024-06-14T22:54:05.275641+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":119,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280880+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T22:54:06.280880+0000","last_peered":"2024-06-14T22:54:06.280880+0000","last_clean":"2024-06-14T22:54:06.280880+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T22:54:06.280880+0000","last_undegraded":"2024-06-14T22:54:06.280880+0000","last_fullsized":"2024-06-14T22:54:06.280880+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":117,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280346+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T22:54:06.280346+0000","last_peered":"2024-06-14T22:54:06.280346+0000","last_clean":"2024-06-14T22:54:06.280346+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T22:54:06.280346+0000","last_undegraded":"2024-06-14T22:54:06.280346+0000","last_fullsized":"2024-06-14T22:54:06.280346+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270215+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T22:54:05.270215+0000","last_peered":"2024-06-14T22:54:05.270215+0000","last_clean":"2024-06-14T22:54:05.270215+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T22:54:05.270215+0000","last_undegraded":"2024-06-14T22:54:05.270215+0000","last_fullsized":"2024-06-14T22:54:05.270215+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":89,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270541+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T22:54:05.270541+0000","last_peered":"2024-06-14T22:54:05.270541+0000","last_clean":"2024-06-14T22:54:05.270541+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T22:54:05.270541+0000","last_undegraded":"2024-06-14T22:54:05.270541+0000","last_fullsized":"2024-06-14T22:54:05.270541+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270139+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T22:54:05.270139+0000","last_peered":"2024-06-14T22:54:05.270139+0000","last_clean":"2024-06-14T22:54:05.270139+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T22:54:05.270139+0000","last_undegraded":"2024-06-14T22:54:05.270139+0000","last_fullsized":"2024-06-14T22:54:05.270139+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.279227+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T22:54:05.279227+0000","last_peered":"2024-06-14T22:54:05.279227+0000","last_clean":"2024-06-14T22:54:05.279227+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T22:54:05.279227+0000","last_undegraded":"2024-06-14T22:54:05.279227+0000","last_fullsized":"2024-06-14T22:54:05.279227+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":75,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.270273+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T22:54:06.270273+0000","last_peered":"2024-06-14T22:54:06.270273+0000","last_clean":"2024-06-14T22:54:06.270273+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T22:54:06.270273+0000","last_undegraded":"2024-06-14T22:54:06.270273+0000","last_fullsized":"2024-06-14T22:54:06.270273+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.284651+0000","last_change":"2024-06-14T22:54:06.284651+0000","last_active":"2024-06-14T22:54:06.284651+0000","last_peered":"2024-06-14T22:54:06.284651+0000","last_clean":"2024-06-14T22:54:06.284651+0000","last_became_active":"2024-06-14T22:54:06.283564+0000","last_became_peered":"2024-06-14T22:54:06.283564+0000","last_unstale":"2024-06-14T22:54:06.284651+0000","last_undegraded":"2024-06-14T22:54:06.284651+0000","last_fullsized":"2024-06-14T22:54:06.284651+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:34.916272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":88,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270118+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T22:54:05.270118+0000","last_peered":"2024-06-14T22:54:05.270118+0000","last_clean":"2024-06-14T22:54:05.270118+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T22:54:05.270118+0000","last_undegraded":"2024-06-14T22:54:05.270118+0000","last_fullsized":"2024-06-14T22:54:05.270118+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281419+0000","last_change":"2024-06-14T22:54:06.281419+0000","last_active":"2024-06-14T22:54:06.281419+0000","last_peered":"2024-06-14T22:54:06.281419+0000","last_clean":"2024-06-14T22:54:06.281419+0000","last_became_active":"2024-06-14T22:54:06.279482+0000","last_became_peered":"2024-06-14T22:54:06.279482+0000","last_unstale":"2024-06-14T22:54:06.281419+0000","last_undegraded":"2024-06-14T22:54:06.281419+0000","last_fullsized":"2024-06-14T22:54:06.281419+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:06:37.039006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.277756+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T22:54:05.277756+0000","last_peered":"2024-06-14T22:54:05.277756+0000","last_clean":"2024-06-14T22:54:05.277756+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T22:54:05.277756+0000","last_undegraded":"2024-06-14T22:54:05.277756+0000","last_fullsized":"2024-06-14T22:54:05.277756+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280283+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T22:54:05.280283+0000","last_peered":"2024-06-14T22:54:05.280283+0000","last_clean":"2024-06-14T22:54:05.280283+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T22:54:05.280283+0000","last_undegraded":"2024-06-14T22:54:05.280283+0000","last_fullsized":"2024-06-14T22:54:05.280283+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275645+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:54:05.275645+0000","last_peered":"2024-06-14T22:54:05.275645+0000","last_clean":"2024-06-14T22:54:05.275645+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:54:05.275645+0000","last_undegraded":"2024-06-14T22:54:05.275645+0000","last_fullsized":"2024-06-14T22:54:05.275645+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275589+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T22:54:05.275589+0000","last_peered":"2024-06-14T22:54:05.275589+0000","last_clean":"2024-06-14T22:54:05.275589+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T22:54:05.275589+0000","last_undegraded":"2024-06-14T22:54:05.275589+0000","last_fullsized":"2024-06-14T22:54:05.275589+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":105,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.275170+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:54:05.275170+0000","last_peered":"2024-06-14T22:54:05.275170+0000","last_clean":"2024-06-14T22:54:05.275170+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:54:05.275170+0000","last_undegraded":"2024-06-14T22:54:05.275170+0000","last_fullsized":"2024-06-14T22:54:05.275170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":104,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.270411+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T22:54:05.270411+0000","last_peered":"2024-06-14T22:54:05.270411+0000","last_clean":"2024-06-14T22:54:05.270411+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T22:54:05.270411+0000","last_undegraded":"2024-06-14T22:54:05.270411+0000","last_fullsized":"2024-06-14T22:54:05.270411+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.275561+0000","last_change":"2024-06-14T22:54:06.275561+0000","last_active":"2024-06-14T22:54:06.275561+0000","last_peered":"2024-06-14T22:54:06.275561+0000","last_clean":"2024-06-14T22:54:06.275561+0000","last_became_active":"2024-06-14T22:54:06.274930+0000","last_became_peered":"2024-06-14T22:54:06.274930+0000","last_unstale":"2024-06-14T22:54:06.275561+0000","last_undegraded":"2024-06-14T22:54:06.275561+0000","last_fullsized":"2024-06-14T22:54:06.275561+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:11.022230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":100,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.281532+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T22:54:05.281532+0000","last_peered":"2024-06-14T22:54:05.281532+0000","last_clean":"2024-06-14T22:54:05.281532+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T22:54:05.281532+0000","last_undegraded":"2024-06-14T22:54:05.281532+0000","last_fullsized":"2024-06-14T22:54:05.281532+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":64,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279332+0000","last_change":"2024-06-14T22:54:06.279332+0000","last_active":"2024-06-14T22:54:06.279332+0000","last_peered":"2024-06-14T22:54:06.279332+0000","last_clean":"2024-06-14T22:54:06.279332+0000","last_became_active":"2024-06-14T22:54:06.279020+0000","last_became_peered":"2024-06-14T22:54:06.279020+0000","last_unstale":"2024-06-14T22:54:06.279332+0000","last_undegraded":"2024-06-14T22:54:06.279332+0000","last_fullsized":"2024-06-14T22:54:06.279332+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:52:57.777668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.285313+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T22:54:06.285313+0000","last_peered":"2024-06-14T22:54:06.285313+0000","last_clean":"2024-06-14T22:54:06.285313+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T22:54:06.285313+0000","last_undegraded":"2024-06-14T22:54:06.285313+0000","last_fullsized":"2024-06-14T22:54:06.285313+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281615+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T22:54:06.281615+0000","last_peered":"2024-06-14T22:54:06.281615+0000","last_clean":"2024-06-14T22:54:06.281615+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T22:54:06.281615+0000","last_undegraded":"2024-06-14T22:54:06.281615+0000","last_fullsized":"2024-06-14T22:54:06.281615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280203+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T22:54:05.280203+0000","last_peered":"2024-06-14T22:54:05.280203+0000","last_clean":"2024-06-14T22:54:05.280203+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T22:54:05.280203+0000","last_undegraded":"2024-06-14T22:54:05.280203+0000","last_fullsized":"2024-06-14T22:54:05.280203+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":108,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282120+0000","last_change":"2024-06-14T22:54:06.282120+0000","last_active":"2024-06-14T22:54:06.282120+0000","last_peered":"2024-06-14T22:54:06.282120+0000","last_clean":"2024-06-14T22:54:06.282120+0000","last_became_active":"2024-06-14T22:54:06.280898+0000","last_became_peered":"2024-06-14T22:54:06.280898+0000","last_unstale":"2024-06-14T22:54:06.282120+0000","last_undegraded":"2024-06-14T22:54:06.282120+0000","last_fullsized":"2024-06-14T22:54:06.282120+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:30:45.544457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.278300+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T22:54:06.278300+0000","last_peered":"2024-06-14T22:54:06.278300+0000","last_clean":"2024-06-14T22:54:06.278300+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T22:54:06.278300+0000","last_undegraded":"2024-06-14T22:54:06.278300+0000","last_fullsized":"2024-06-14T22:54:06.278300+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.284728+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T22:54:06.284728+0000","last_peered":"2024-06-14T22:54:06.284728+0000","last_clean":"2024-06-14T22:54:06.284728+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T22:54:06.284728+0000","last_undegraded":"2024-06-14T22:54:06.284728+0000","last_fullsized":"2024-06-14T22:54:06.284728+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280483+0000","last_change":"2024-06-14T22:54:06.280483+0000","last_active":"2024-06-14T22:54:06.280483+0000","last_peered":"2024-06-14T22:54:06.280483+0000","last_clean":"2024-06-14T22:54:06.280483+0000","last_became_active":"2024-06-14T22:54:06.278985+0000","last_became_peered":"2024-06-14T22:54:06.278985+0000","last_unstale":"2024-06-14T22:54:06.280483+0000","last_undegraded":"2024-06-14T22:54:06.280483+0000","last_fullsized":"2024-06-14T22:54:06.280483+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:43:40.478580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":118,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.274772+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T22:54:05.274772+0000","last_peered":"2024-06-14T22:54:05.274772+0000","last_clean":"2024-06-14T22:54:05.274772+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T22:54:05.274772+0000","last_undegraded":"2024-06-14T22:54:05.274772+0000","last_fullsized":"2024-06-14T22:54:05.274772+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":109,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.277110+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T22:54:06.277110+0000","last_peered":"2024-06-14T22:54:06.277110+0000","last_clean":"2024-06-14T22:54:06.277110+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T22:54:06.277110+0000","last_undegraded":"2024-06-14T22:54:06.277110+0000","last_fullsized":"2024-06-14T22:54:06.277110+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":77,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.270191+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T22:54:06.270191+0000","last_peered":"2024-06-14T22:54:06.270191+0000","last_clean":"2024-06-14T22:54:06.270191+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T22:54:06.270191+0000","last_undegraded":"2024-06-14T22:54:06.270191+0000","last_fullsized":"2024-06-14T22:54:06.270191+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.281117+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T22:54:06.281117+0000","last_peered":"2024-06-14T22:54:06.281117+0000","last_clean":"2024-06-14T22:54:06.281117+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T22:54:06.281117+0000","last_undegraded":"2024-06-14T22:54:06.281117+0000","last_fullsized":"2024-06-14T22:54:06.281117+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":102,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.280162+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T22:54:05.280162+0000","last_peered":"2024-06-14T22:54:05.280162+0000","last_clean":"2024-06-14T22:54:05.280162+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T22:54:05.280162+0000","last_undegraded":"2024-06-14T22:54:05.280162+0000","last_fullsized":"2024-06-14T22:54:05.280162+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.280576+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T22:54:06.280576+0000","last_peered":"2024-06-14T22:54:06.280576+0000","last_clean":"2024-06-14T22:54:06.280576+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T22:54:06.280576+0000","last_undegraded":"2024-06-14T22:54:06.280576+0000","last_fullsized":"2024-06-14T22:54:06.280576+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":85,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.282162+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:54:05.282162+0000","last_peered":"2024-06-14T22:54:05.282162+0000","last_clean":"2024-06-14T22:54:05.282162+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:54:05.282162+0000","last_undegraded":"2024-06-14T22:54:05.282162+0000","last_fullsized":"2024-06-14T22:54:05.282162+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":123,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.282972+0000","last_change":"2024-06-14T22:54:06.282972+0000","last_active":"2024-06-14T22:54:06.282972+0000","last_peered":"2024-06-14T22:54:06.282972+0000","last_clean":"2024-06-14T22:54:06.282972+0000","last_became_active":"2024-06-14T22:54:06.282622+0000","last_became_peered":"2024-06-14T22:54:06.282622+0000","last_unstale":"2024-06-14T22:54:06.282972+0000","last_undegraded":"2024-06-14T22:54:06.282972+0000","last_fullsized":"2024-06-14T22:54:06.282972+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:31:22.051955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":108,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.279284+0000","last_change":"2024-06-14T22:54:06.279284+0000","last_active":"2024-06-14T22:54:06.279284+0000","last_peered":"2024-06-14T22:54:06.279284+0000","last_clean":"2024-06-14T22:54:06.279284+0000","last_became_active":"2024-06-14T22:54:06.278861+0000","last_became_peered":"2024-06-14T22:54:06.278861+0000","last_unstale":"2024-06-14T22:54:06.279284+0000","last_undegraded":"2024-06-14T22:54:06.279284+0000","last_fullsized":"2024-06-14T22:54:06.279284+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:17:47.260769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":99,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-14T22:54:05.273550+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T22:54:05.273550+0000","last_peered":"2024-06-14T22:54:05.273550+0000","last_clean":"2024-06-14T22:54:05.273550+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T22:54:05.273550+0000","last_undegraded":"2024-06-14T22:54:05.273550+0000","last_fullsized":"2024-06-14T22:54:05.273550+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":93,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-14T22:54:06.270485+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T22:54:06.270485+0000","last_peered":"2024-06-14T22:54:06.270485+0000","last_clean":"2024-06-14T22:54:06.270485+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T22:54:06.270485+0000","last_undegraded":"2024-06-14T22:54:06.270485+0000","last_fullsized":"2024-06-14T22:54:06.270485+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234254,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600240,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":93144912,"statfs":{"total":95995035648,"available":95380389888,"internally_reserved":0,"allocated":2260992,"data_stored":1493426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:55:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.98499999999999999,"15min":1.127},"min":{"1min":0.378,"5min":0.20699999999999999,"15min":0.20200000000000001},"max":{"1min":0.73799999999999999,"5min":6.9199999999999999,"15min":9.3919999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.1559999999999999,"15min":1.264},"min":{"1min":0.371,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.70399999999999996,"5min":6.8440000000000003,"15min":10.416},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.212,"15min":1.3700000000000001},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80500000000000005,"5min":6.9500000000000002,"15min":6.9500000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":1.3560000000000001,"15min":1.464},"min":{"1min":0.38100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":12.106999999999999,"15min":12.106999999999999},"last":0.56599999999999995}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.63400000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.67000000000000004,"15min":0.80400000000000005},"min":{"1min":0.27300000000000002,"5min":0.248,"15min":0.214},"max":{"1min":0.67900000000000005,"5min":5.8449999999999998,"15min":9.7430000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.33900000000000002,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":0.64500000000000002,"5min":4.2709999999999999,"15min":6.6660000000000004},"last":0.38600000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.752,"15min":0.77200000000000002},"min":{"1min":0.34999999999999998,"5min":0.26100000000000001,"15min":0.22},"max":{"1min":0.70599999999999996,"5min":3.9860000000000002,"15min":7.9870000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.88,"15min":0.88600000000000001},"min":{"1min":0.33500000000000002,"5min":0.28699999999999998,"15min":0.28299999999999997},"max":{"1min":0.76200000000000001,"5min":5.3659999999999997,"15min":9.5120000000000005},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Jun 14 22:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.75800000000000001,"15min":0.81999999999999995},"min":{"1min":0.39400000000000002,"5min":0.308,"15min":0.28100000000000003},"max":{"1min":0.72499999999999998,"5min":5.806,"15min":5.806},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.76700000000000002,"15min":0.71999999999999997},"min":{"1min":0.39000000000000001,"5min":0.311,"15min":0.29899999999999999},"max":{"1min":0.70299999999999996,"5min":4.3090000000000002,"15min":5.7430000000000003},"last":0.55000000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311836,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":93433316,"statfs":{"total":95995035648,"available":95675715584,"internally_reserved":0,"allocated":2244608,"data_stored":1492957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.1739999999999999,"15min":1.1839999999999999},"min":{"1min":0.44900000000000001,"5min":0.224,"15min":0.218},"max":{"1min":0.67700000000000005,"5min":7.1689999999999996,"15min":31.358000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.99299999999999999,"15min":1.411},"min":{"1min":0.38500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.73799999999999999,"5min":4.9180000000000001,"15min":29.399000000000001},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.224,"15min":1.304},"min":{"1min":0.39400000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73399999999999999,"5min":9.9559999999999995,"15min":9.9559999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":1.2769999999999999,"15min":1.276},"min":{"1min":0.36099999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.67400000000000004,"5min":9.5510000000000002,"15min":9.5510000000000002},"last":0.46300000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.50600000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":1.2110000000000001,"15min":0.88800000000000001},"min":{"1min":0.35299999999999998,"5min":0.25600000000000001,"15min":0.21299999999999999},"max":{"1min":0.71999999999999997,"5min":47.698,"15min":47.698},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.496,"5min":1.181,"15min":0.82599999999999996},"min":{"1min":0.313,"5min":0.24299999999999999,"15min":0.23899999999999999},"max":{"1min":0.72699999999999998,"5min":48.197000000000003,"15min":48.197000000000003},"last":0.39300000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:54:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.88600000000000001,"15min":0.79100000000000004},"min":{"1min":0.33300000000000002,"5min":0.28599999999999998,"15min":0.22600000000000001},"max":{"1min":0.75800000000000001,"5min":8.6500000000000004,"15min":8.6500000000000004},"last":0.439},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.95399999999999996,"15min":0.86199999999999999},"min":{"1min":0.41199999999999998,"5min":0.313,"15min":0.28399999999999997},"max":{"1min":0.749,"5min":8.6349999999999998,"15min":9.5609999999999999},"last":0.58099999999999996}]},{"osd":7,"last update":"Fri Jun 14 22:54:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.84199999999999997,"15min":0.82099999999999995},"min":{"1min":0.38100000000000001,"5min":0.31,"15min":0.28799999999999998},"max":{"1min":0.82299999999999995,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.90500000000000003,"15min":0.83299999999999996},"min":{"1min":0.44500000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78100000000000003,"5min":8.673,"15min":8.673},"last":0.60199999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757529,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896852,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":92848300,"statfs":{"total":95995035648,"available":95076659200,"internally_reserved":0,"allocated":1642496,"data_stored":898403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":1.1299999999999999,"15min":1.2270000000000001},"min":{"1min":0.41099999999999998,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.69899999999999995,"5min":6.0369999999999999,"15min":17.766999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.53500000000000003,"5min":1.046,"15min":1.2689999999999999},"min":{"1min":0.38500000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70999999999999996,"5min":5.2439999999999998,"15min":17.806000000000001},"last":0.59999999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.2809999999999999,"15min":1.357},"min":{"1min":0.36099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66400000000000003,"5min":7.694,"15min":7.694},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.501,"5min":1.129,"15min":1.169},"min":{"1min":0.33700000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.72599999999999998,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.67500000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Jun 14 22:55:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.63600000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.60599999999999998,"15min":0.73599999999999999},"min":{"1min":0.313,"5min":0.23300000000000001,"15min":0.214},"max":{"1min":0.64400000000000002,"5min":4.9790000000000001,"15min":11.771000000000001},"last":0.432},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.69699999999999995,"15min":0.74299999999999999},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.23200000000000001},"max":{"1min":0.66100000000000003,"5min":5.8109999999999999,"15min":10.032},"last":0.46500000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.84699999999999998,"15min":0.86599999999999999},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.23000000000000001},"max":{"1min":0.67900000000000005,"5min":9.6020000000000003,"15min":9.8369999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.85699999999999998,"15min":0.88300000000000001},"min":{"1min":0.33300000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.74399999999999999,"5min":12.339,"15min":12.339},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:54:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.75800000000000001,"15min":0.82299999999999995},"min":{"1min":0.375,"5min":0.31900000000000001,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":5.2629999999999999,"15min":10.462999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.72999999999999998,"15min":0.83899999999999997},"min":{"1min":0.33900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69299999999999995,"5min":10.638999999999999,"15min":10.638999999999999},"last":0.48299999999999998}]}]},{"osd":4,"up_from":30,"seq":128849019165,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599652,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":93145500,"statfs":{"total":95995035648,"available":95380992000,"internally_reserved":0,"allocated":1658880,"data_stored":902723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.0640000000000001,"15min":1.0609999999999999},"min":{"1min":0.39500000000000002,"5min":0.219,"15min":0.218},"max":{"1min":0.70199999999999996,"5min":4.641,"15min":7.5650000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":1.1499999999999999,"15min":1.365},"min":{"1min":0.39800000000000002,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":0.65400000000000003,"5min":5.7569999999999997,"15min":9.1470000000000002},"last":0.59599999999999997}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":1.1779999999999999,"15min":1.1499999999999999},"min":{"1min":0.34899999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65300000000000002,"5min":5.141,"15min":5.141},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.2030000000000001,"15min":1.3360000000000001},"min":{"1min":0.35299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.70799999999999996,"5min":6.2709999999999999,"15min":6.2709999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.438}]},{"osd":5,"last update":"Fri Jun 14 22:54:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.72599999999999998,"15min":0.749},"min":{"1min":0.31,"5min":0.24199999999999999,"15min":0.21199999999999999},"max":{"1min":0.67900000000000005,"5min":3.964,"15min":5.774},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.71899999999999997,"15min":0.71499999999999997},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":0.78200000000000003,"5min":4.1269999999999998,"15min":4.7480000000000002},"last":0.32000000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:54:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.65200000000000002,"15min":0.78100000000000003},"min":{"1min":0.34399999999999997,"5min":0.25800000000000001,"15min":0.254},"max":{"1min":0.70099999999999996,"5min":3.8929999999999998,"15min":10.904999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61499999999999999,"15min":0.77200000000000002},"min":{"1min":0.371,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":0.64600000000000002,"5min":3.7450000000000001,"15min":9.2390000000000008},"last":0.45000000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:54:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.63,"15min":0.752},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.246},"max":{"1min":0.72399999999999998,"5min":3.7029999999999998,"15min":6.298},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.74299999999999999,"15min":0.76400000000000001},"min":{"1min":0.38300000000000001,"5min":0.317,"15min":0.29399999999999998},"max":{"1min":0.73499999999999999,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":0.42299999999999999}]}]},{"osd":3,"up_from":138,"seq":592705486864,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77328,"kb_used_data":1604,"kb_used_omap":4,"kb_used_meta":75707,"kb_avail":93667824,"statfs":{"total":95995035648,"available":95915851776,"internally_reserved":0,"allocated":1642496,"data_stored":898176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":77524446},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.253},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.42499999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.37}]},{"osd":2,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.24299999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.48799999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.626}]}]},{"osd":2,"up_from":133,"seq":571230650391,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62544,"kb_used_data":1604,"kb_used_omap":6,"kb_used_meta":60921,"kb_avail":93682608,"statfs":{"total":95995035648,"available":95930990592,"internally_reserved":0,"allocated":1642496,"data_stored":898390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":62383605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.376},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.41699999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.315}]},{"osd":3,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.46300000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.66800000000000004}]},{"osd":6,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54200000000000004}]}]},{"osd":1,"up_from":116,"seq":498216206408,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":213204,"kb_used_data":1600,"kb_used_omap":5,"kb_used_meta":211578,"kb_avail":93531948,"statfs":{"total":95995035648,"available":95776714752,"internally_reserved":0,"allocated":1638400,"data_stored":898086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":216656152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":1.8759999999999999,"15min":2.3620000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41299999999999998,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.4159999999999999,"15min":1.746},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.433,"5min":10.426,"15min":10.426},"last":0.22500000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:54:32 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.251}]},{"osd":3,"last update":"Fri Jun 14 22:55:05 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.5229999999999999,"15min":2.0209999999999999},"min":{"1min":0.41799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.95399999999999996,"5min":8.8599999999999994,"15min":8.8599999999999994},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.625,"15min":1.9490000000000001},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88600000000000001,"5min":11.161,"15min":11.161},"last":0.63}]},{"osd":5,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.6719999999999999,"15min":1.8420000000000001},"min":{"1min":0.40799999999999997,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":1.52,"15min":2.0760000000000001},"min":{"1min":0.438,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.97899999999999998,"5min":8.125,"15min":8.125},"last":0.495}]},{"osd":6,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":1.7749999999999999,"15min":1.929},"min":{"1min":0.39700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89700000000000002,"5min":9.7279999999999998,"15min":9.7279999999999998},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":1.4870000000000001,"15min":2.0750000000000002},"min":{"1min":0.44,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.92300000000000004,"5min":8.1479999999999997,"15min":8.1479999999999997},"last":0.68400000000000005}]},{"osd":7,"last update":"Fri Jun 14 22:54:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.589,"15min":2.089},"min":{"1min":0.45900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79900000000000004,"5min":9.0960000000000001,"15min":9.0960000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.621,"5min":1.5600000000000001,"15min":1.702},"min":{"1min":0.48199999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96499999999999997,"5min":9.5449999999999999,"15min":9.5449999999999999},"last":0.55700000000000005}]}]},{"osd":0,"up_from":91,"seq":390842024063,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":441492,"kb_used_data":2180,"kb_used_omap":5,"kb_used_meta":439290,"kb_avail":93303660,"statfs":{"total":95995035648,"available":95542947840,"internally_reserved":0,"allocated":2232320,"data_stored":1486453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":449833050},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:54:27 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.7989999999999999,"15min":1.609},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.44600000000000001,"5min":9.6999999999999993,"15min":9.6999999999999993},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":1.895,"15min":2.2869999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.58199999999999996,"5min":10.613,"15min":10.613},"last":0.32000000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:54:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.40999999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:55:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.33400000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.3520000000000001,"15min":1.603},"min":{"1min":0.374,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.86499999999999999,"5min":8.7889999999999997,"15min":8.9510000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.623,"5min":1.2230000000000001,"15min":1.3109999999999999},"min":{"1min":0.41099999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.93999999999999995,"5min":13.574,"15min":13.574},"last":0.878}]},{"osd":5,"last update":"Fri Jun 14 22:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.331,"15min":1.3520000000000001},"min":{"1min":0.432,"5min":0.317,"15min":0.29299999999999998},"max":{"1min":0.77400000000000002,"5min":13.616,"15min":13.616},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":1.236,"15min":1.4410000000000001},"min":{"1min":0.46500000000000002,"5min":0.35699999999999998,"15min":0.31900000000000001},"max":{"1min":0.877,"5min":7.6509999999999998,"15min":9.6620000000000008},"last":0.59799999999999998}]},{"osd":6,"last update":"Fri Jun 14 22:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.3540000000000001,"15min":1.3819999999999999},"min":{"1min":0.36599999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79500000000000004,"5min":13.645,"15min":13.645},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":1.1539999999999999,"15min":1.5169999999999999},"min":{"1min":0.42299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.84299999999999997,"5min":7.0119999999999996,"15min":9.8599999999999994},"last":0.624}]},{"osd":7,"last update":"Fri Jun 14 22:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":1.1619999999999999,"15min":1.5489999999999999},"min":{"1min":0.39200000000000002,"5min":0.38700000000000001,"15min":0.33100000000000002},"max":{"1min":0.78000000000000003,"5min":6.9870000000000001,"15min":9.6370000000000005},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":1.339,"15min":1.71},"min":{"1min":0.45500000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.80500000000000005,"5min":8.8290000000000006,"15min":8.9900000000000002},"last":0.65400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:55:18.201 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-14T22:55:18.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump 2024-06-14T22:55:18.458 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:18.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:18 smithi086 ceph-mon[97913]: from='client.45452 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:18.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:18 smithi086 ceph-mon[102754]: from='client.45452 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:18.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:18 smithi137 ceph-mon[90634]: from='client.45452 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:55:19.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:19 smithi137 ceph-mon[90634]: pgmap v533: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:20.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:19 smithi086 ceph-mon[97913]: pgmap v533: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:20.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:19 smithi086 ceph-mon[102754]: pgmap v533: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:20.377 INFO:teuthology.orchestra.run.smithi086.stdout:epoch 139 2024-06-14T22:55:20.377 INFO:teuthology.orchestra.run.smithi086.stdout:fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 2024-06-14T22:55:20.377 INFO:teuthology.orchestra.run.smithi086.stdout:created 2024-06-14T22:28:02.984269+0000 2024-06-14T22:55:20.377 INFO:teuthology.orchestra.run.smithi086.stdout:modified 2024-06-14T22:54:06.263307+0000 2024-06-14T22:55:20.377 INFO:teuthology.orchestra.run.smithi086.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-06-14T22:55:20.377 INFO:teuthology.orchestra.run.smithi086.stdout:crush_version 22 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:full_ratio 0.95 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:backfillfull_ratio 0.9 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:nearfull_ratio 0.85 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:require_min_compat_client luminous 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:min_compat_client luminous 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:require_osd_release quincy 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:stretch_mode_enabled false 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 application rbd read_balance_score 2.00 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:max_osd 8 2024-06-14T22:55:20.378 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0 up in weight 1 up_from 91 up_thru 138 down_at 84 last_clean_interval [8,83) [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428] [v2:172.21.15.86:6804/2971024428,v1:172.21.15.86:6805/2971024428] exists,up db19a6c7-4c6f-4c95-aa2f-3ddc1e121270 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1 up in weight 1 up_from 116 up_thru 138 down_at 113 last_clean_interval [13,112) [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517] [v2:172.21.15.86:6812/1358489517,v1:172.21.15.86:6813/1358489517] exists,up f021e24a-e7b4-46f0-94ed-4db138b60d64 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2 up in weight 1 up_from 133 up_thru 138 down_at 130 last_clean_interval [18,129) [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022] [v2:172.21.15.86:6820/3223348022,v1:172.21.15.86:6821/3223348022] exists,up 65dbd353-d1a7-4f6d-8112-b77fbc51f9ce 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [25,134) [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285] [v2:172.21.15.86:6828/1034049285,v1:172.21.15.86:6829/1034049285] exists,up 5e141877-8275-4b3a-bb90-277e99a725d8 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4 up in weight 1 up_from 30 up_thru 138 down_at 0 last_clean_interval [0,0) [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531] [v2:172.21.15.137:6802/3721662531,v1:172.21.15.137:6803/3721662531] exists,up 4f0bd061-11f3-4d2c-9395-65c3ae5489f5 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5 up in weight 1 up_from 38 up_thru 138 down_at 0 last_clean_interval [0,0) [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324] [v2:172.21.15.137:6810/2122959324,v1:172.21.15.137:6811/2122959324] exists,up c27edac8-f0e3-4b4c-a1e6-9e57165aea80 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6 up in weight 1 up_from 46 up_thru 135 down_at 0 last_clean_interval [0,0) [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362] [v2:172.21.15.137:6818/2398798362,v1:172.21.15.137:6819/2398798362] exists,up 44e654f2-c297-49dd-94c0-9d0ef90bffe2 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7 up in weight 1 up_from 54 up_thru 138 down_at 0 last_clean_interval [0,0) [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786] [v2:172.21.15.137:6826/298867786,v1:172.21.15.137:6827/298867786] exists,up f867732b-d642-4650-a57e-b01280b47e6e 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:pg_upmap_items 4.6 [1,5] 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:pg_upmap_items 4.d [1,5] 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:pg_upmap_items 4.1d [1,2] 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/3024645077 expires 2024-06-15T22:39:42.195791+0000 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/2298288322 expires 2024-06-15T22:39:42.195791+0000 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6800/1921939101 expires 2024-06-15T22:39:42.195791+0000 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/1328684620 expires 2024-06-15T22:39:42.195791+0000 2024-06-14T22:55:20.379 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/648695835 expires 2024-06-15T22:39:42.195791+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/358698000 expires 2024-06-15T22:39:42.195791+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.137:0/3481534205 expires 2024-06-15T22:38:44.349280+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6801/2254669699 expires 2024-06-15T22:37:41.915228+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/2741142752 expires 2024-06-15T22:33:27.518416+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/418538598 expires 2024-06-15T22:37:41.915228+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/1759721543 expires 2024-06-15T22:33:27.518416+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6800/2254669699 expires 2024-06-15T22:37:41.915228+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.137:6834/3389028913 expires 2024-06-15T22:38:44.349280+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6801/1921939101 expires 2024-06-15T22:39:42.195791+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/2504812592 expires 2024-06-15T22:28:53.238347+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6801/2713243150 expires 2024-06-15T22:28:23.452282+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/3724463773 expires 2024-06-15T22:33:27.518416+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.137:6835/3389028913 expires 2024-06-15T22:38:44.349280+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.137:0/2407056636 expires 2024-06-15T22:38:44.349280+0000 2024-06-14T22:55:20.380 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.137:0/2914169147 expires 2024-06-15T22:38:44.349280+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.137:0/3094754408 expires 2024-06-15T22:38:44.349280+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.137:0/2665974068 expires 2024-06-15T22:38:44.349280+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.137:0/3471286695 expires 2024-06-15T22:38:44.349280+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/3167284010 expires 2024-06-15T22:37:41.915228+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/2027707012 expires 2024-06-15T22:28:23.452282+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6801/3390625982 expires 2024-06-15T22:28:53.238347+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/3161947015 expires 2024-06-15T22:28:23.452282+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/1226112395 expires 2024-06-15T22:28:53.238347+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/1470436259 expires 2024-06-15T22:39:42.195791+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/2139637654 expires 2024-06-15T22:28:23.452282+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/3107951276 expires 2024-06-15T22:37:41.915228+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/3941467574 expires 2024-06-15T22:33:27.518416+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6800/3866885055 expires 2024-06-15T22:33:27.518416+0000 2024-06-14T22:55:20.381 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6800/3390625982 expires 2024-06-15T22:28:53.238347+0000 2024-06-14T22:55:20.382 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/1167860215 expires 2024-06-15T22:37:41.915228+0000 2024-06-14T22:55:20.382 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6801/3866885055 expires 2024-06-15T22:33:27.518416+0000 2024-06-14T22:55:20.382 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6800/2713243150 expires 2024-06-15T22:28:23.452282+0000 2024-06-14T22:55:20.382 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/1554099440 expires 2024-06-15T22:28:53.238347+0000 2024-06-14T22:55:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4106893894' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-14T22:55:21.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4106893894' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-14T22:55:21.027 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4106893894' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-14T22:55:21.051 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 91 up_thru 138 down_at 84 last_clean_interval [8,83) [v2:172.21.15.86:6802/2971024428,v1:172.21.15.86:6803/2971024428] [v2:172.21.15.86:6804/2971024428,v1:172.21.15.86:6805/2971024428] exists,up db19a6c7-4c6f-4c95-aa2f-3ddc1e121270', 'osd.1 up in weight 1 up_from 116 up_thru 138 down_at 113 last_clean_interval [13,112) [v2:172.21.15.86:6810/1358489517,v1:172.21.15.86:6811/1358489517] [v2:172.21.15.86:6812/1358489517,v1:172.21.15.86:6813/1358489517] exists,up f021e24a-e7b4-46f0-94ed-4db138b60d64', 'osd.2 up in weight 1 up_from 133 up_thru 138 down_at 130 last_clean_interval [18,129) [v2:172.21.15.86:6818/3223348022,v1:172.21.15.86:6819/3223348022] [v2:172.21.15.86:6820/3223348022,v1:172.21.15.86:6821/3223348022] exists,up 65dbd353-d1a7-4f6d-8112-b77fbc51f9ce', 'osd.3 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [25,134) [v2:172.21.15.86:6826/1034049285,v1:172.21.15.86:6827/1034049285] [v2:172.21.15.86:6828/1034049285,v1:172.21.15.86:6829/1034049285] exists,up 5e141877-8275-4b3a-bb90-277e99a725d8', 'osd.4 up in weight 1 up_from 30 up_thru 138 down_at 0 last_clean_interval [0,0) [v2:172.21.15.137:6800/3721662531,v1:172.21.15.137:6801/3721662531] [v2:172.21.15.137:6802/3721662531,v1:172.21.15.137:6803/3721662531] exists,up 4f0bd061-11f3-4d2c-9395-65c3ae5489f5', 'osd.5 up in weight 1 up_from 38 up_thru 138 down_at 0 last_clean_interval [0,0) [v2:172.21.15.137:6808/2122959324,v1:172.21.15.137:6809/2122959324] [v2:172.21.15.137:6810/2122959324,v1:172.21.15.137:6811/2122959324] exists,up c27edac8-f0e3-4b4c-a1e6-9e57165aea80', 'osd.6 up in weight 1 up_from 46 up_thru 135 down_at 0 last_clean_interval [0,0) [v2:172.21.15.137:6816/2398798362,v1:172.21.15.137:6817/2398798362] [v2:172.21.15.137:6818/2398798362,v1:172.21.15.137:6819/2398798362] exists,up 44e654f2-c297-49dd-94c0-9d0ef90bffe2', 'osd.7 up in weight 1 up_from 54 up_thru 138 down_at 0 last_clean_interval [0,0) [v2:172.21.15.137:6824/298867786,v1:172.21.15.137:6825/298867786] [v2:172.21.15.137:6826/298867786,v1:172.21.15.137:6827/298867786] exists,up f867732b-d642-4650-a57e-b01280b47e6e'] 2024-06-14T22:55:21.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon mon.a config show 2024-06-14T22:55:21.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:21 smithi086 ceph-mon[97913]: pgmap v534: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:21.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:55:21.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:21 smithi086 ceph-mon[102754]: pgmap v534: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:21.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:55:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:21.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:21 smithi137 ceph-mon[90634]: pgmap v534: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:21.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:55:23.514 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:55:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "mon.a", 2024-06-14T22:55:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster": "ceph", 2024-06-14T22:55:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-06-14T22:55:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "admin_socket_mode": "", 2024-06-14T22:55:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-06-14T22:55:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_client_required": "cephx, none", 2024-06-14T22:55:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_cluster_required": "cephx", 2024-06-14T22:55:23.514 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_debug": "true", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_service_required": "cephx", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_service_ticket_ttl": "240.000000", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_supported": "", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_aio": "true", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_aio_max_queue_depth": "1024", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_aio_poll_ms": "250", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_aio_reap_max": "16", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_async_discard": "false", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_block_size": "4096", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_debug_aio": "false", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-06-14T22:55:23.515 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_debug_inflight_ios": "false", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_enable_discard": "false", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_flock_retry": "3", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_flock_retry_interval": "0.100000", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_inject_crash": "0", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_inject_crash_flush_delay": "2", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_ioring": "false", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_ioring_hipri": "false", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_ioring_sqthread_poll": "false", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_read_buffer_alignment": "4096", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_type": "", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_alloc_size": "1048576", 2024-06-14T22:55:23.516 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_allocator": "hybrid", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_buffered_io": "true", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_check_for_zeros": "false", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_check_volume_selector_often": "false", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_compact_log_sync": "false", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_log_compact_min_size": "16777216", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_log_replay_check_allocations": "true", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_max_log_runway": "4194304", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_max_prefetch": "1048576", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_min_flush_size": "524288", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_min_log_runway": "1048576", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_replay_recovery": "false", 2024-06-14T22:55:23.517 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_shared_alloc_size": "65536", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_sync_write": "false", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_allocation_from_file": "true", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_allocator": "hybrid", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-06-14T22:55:23.518 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blobid_prealloc": "10240", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_create": "true", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_db_create": "false", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_db_path": "", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_db_size": "0", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_path": "", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_preallocate_file": "false", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_size": "107374182400", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_wal_create": "false", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_wal_path": "", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_wal_size": "100663296", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bluefs": "true", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bluefs_env_mirror": "false", 2024-06-14T22:55:23.519 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_autotune": "true", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_size": "0", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-06-14T22:55:23.520 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_type": "2q", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_clone_cow": "true", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_algorithm": "snappy", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_max_blob_size": "0", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_min_blob_size": "0", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_mode": "none", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_csum_type": "crc32c", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_enforce_settings": "default", 2024-06-14T22:55:23.521 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_freelist": "false", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_fsck_abort": "false", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_inject_bug21040": "false", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_inject_read_err": "false", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_legacy_omap": "false", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_prefill": "0.000000", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-06-14T22:55:23.522 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_small_allocations": "0", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_default_buffered_read": "true", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_default_buffered_write": "false", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_deferred_batch_ops": "0", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fail_eio": "false", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-06-14T22:55:23.523 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_mkfs": "true", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_mount": "false", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_umount": "false", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_ignore_data_csum": "false", 2024-06-14T22:55:23.524 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_kvbackend": "rocksdb", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_log_op_age": "5.000000", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_alloc_size": "0", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_blob_size": "0", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_defer_interval": "3.000000", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_deferred_txc": "32", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_min_alloc_size": "0", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-06-14T22:55:23.525 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_nid_prealloc": "1024", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_prefer_deferred_size": "0", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_qfsck_on_mount": "true", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_retry_disk_reads": "3", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_rocksdb_cf": "true", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_rocksdb_options_annex": "", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_spdk_coremask": "0x1", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_spdk_io_sleep": "5", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_spdk_max_io_completion": "0", 2024-06-14T22:55:23.526 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_spdk_mem": "512", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_sync_submit_transaction": "false", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_bytes": "67108864", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_cost_per_io": "0", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_tracing": "false", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_volume_selection_reserved": "0", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-06-14T22:55:23.527 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_zero_block_detection": "false", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_action_update_interval": "2", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_mount_timeout": "10", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-06-14T22:55:23.528 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_cluster_require_signatures": "false", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_cluster_require_version": "2", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_require_signatures": "false", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_require_version": "2", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_service_require_signatures": "false", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_service_require_version": "2", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_sign_messages": "true", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "chdir": "", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "client_acl_type": "", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "client_asio_thread_count": "2", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "client_cache_mid": "0.750000", 2024-06-14T22:55:23.529 INFO:teuthology.orchestra.run.smithi086.stdout: "client_cache_size": "16384", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_caps_release_delay": "5", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_check_pool_perm": "true", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_collect_and_send_global_metrics": "false", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_debug_force_sync_read": "false", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_debug_getattr_caps": "false", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_debug_inject_features": "", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_debug_inject_tick_delay": "0", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_die_on_failed_remount": "false", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_dirsize_rbytes": "true", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_force_lazyio": "false", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_fs": "", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_inject_fixed_oldest_tid": "false", 2024-06-14T22:55:23.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_inject_release_failure": "false", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_max_inline_size": "4096", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_max_retries_on_remount_failure": "5", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mds_namespace": "", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_metadata": "", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mount_gid": "-1", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mount_timeout": "300", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mount_uid": "-1", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mountpoint": "/", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_notify_timeout": "10", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc": "true", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_max_dirty": "104857600", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_max_dirty_age": "5.000000", 2024-06-14T22:55:23.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_max_objects": "1000", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_size": "209715200", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_target_dirty": "8388608", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_permissions": "true", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_quota": "true", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_quota_df": "true", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_readahead_max_bytes": "0", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_readahead_max_periods": "4", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_readahead_min": "131072", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_reconnect_stale": "false", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_shutdown_timeout": "30", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_snapdir": ".snap", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_tick_interval": "1", 2024-06-14T22:55:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client_trace": "", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "client_try_dentry_invalidate": "false", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "client_use_faked_inos": "false", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "client_use_random_mds": "false", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_graylog": "false", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_graylog_port": "12201", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_monitors": "default=true", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_syslog": "false", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_syslog_level": "info", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_addr": "-", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_network": "", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_network_interface": "", 2024-06-14T22:55:23.533 INFO:teuthology.orchestra.run.smithi086.stdout: "compressor_zlib_isal": "false", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "compressor_zlib_level": "5", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "compressor_zlib_winsize": "-15", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "compressor_zstd_level": "1", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_alien_op_num_threads": "6", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_alien_thread_cpu_cores": "", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_osd_obc_lru_size": "10", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_seastar_smp": "1", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location_hook": "", 2024-06-14T22:55:23.534 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location_hook_timeout": "10", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "daemonize": "false", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "daos_pool": "tank", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "dbstore_db_name_prefix": "dbstore", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_alienstore": "0/5", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_allow_any_pool_priority": "false", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_asok": "1/5", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_asok_assert_abort": "false", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_asserts_on_shutdown": "true", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_auth": "1/5", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_bdev": "1/3", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_bluefs": "1/5", 2024-06-14T22:55:23.535 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_bluestore": "1/5", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_buffer": "0/1", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_ceph_exporter": "1/5", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_cephfs_mirror": "0/5", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_cephsqlite": "0/5", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_client": "0/5", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_compressor": "1/5", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_context": "0/1", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_crush": "1/1", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_crypto": "1/5", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_cyanstore": "0/5", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_deliberately_leak_memory": "false", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_disable_randomized_ping": "false", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_dpdk": "1/5", 2024-06-14T22:55:23.536 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_eventtrace": "1/5", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_filer": "0/1", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_filestore": "1/3", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_finisher": "1/1", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_fuse": "1/5", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_heartbeat_testing_span": "0", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_heartbeatmap": "1/5", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_immutable_obj_cache": "0/5", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_javaclient": "1/5", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_journal": "1/3", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_journaler": "0/5", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_kstore": "1/5", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_leveldb": "4/5", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_lockdep": "0/1", 2024-06-14T22:55:23.537 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mclock": "1/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds": "1/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_balancer": "1/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_locker": "1/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_log": "1/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_log_expire": "1/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_migrator": "1/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_memstore": "1/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mgr": "2/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mgrc": "1/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mon": "20/20", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_monc": "0/10", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_ms": "1/1", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_none": "0/5", 2024-06-14T22:55:23.538 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_objclass": "0/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_objectcacher": "0/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_objecter": "0/1", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_optracker": "0/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_osd": "1/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_paxos": "20/20", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_perfcounter": "1/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_prioritycache": "1/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rados": "0/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rbd": "0/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rbd_mirror": "0/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rbd_pwl": "0/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rbd_replay": "0/5", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_refs": "0/0", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_reserver": "1/1", 2024-06-14T22:55:23.539 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw": "1/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_access": "1/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_datacache": "1/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_dbstore": "1/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_flight": "1/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_sync": "1/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rocksdb": "4/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore": "0/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_backref": "0/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_cache": "0/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_cleaner": "0/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_device": "0/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_epm": "0/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_journal": "0/5", 2024-06-14T22:55:23.540 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_lba": "0/5", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_odata": "0/5", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_omap": "0/5", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_onode": "0/5", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_t": "0/5", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_tm": "0/5", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_striper": "0/1", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_test": "0/5", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_throttle": "1/1", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_timer": "0/1", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_tp": "0/5", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "device_failure_prediction_mode": "none", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-06-14T22:55:23.541 INFO:teuthology.orchestra.run.smithi086.stdout: "err_to_graylog": "false", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "err_to_journald": "false", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "err_to_stderr": "true", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "err_to_syslog": "false", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event_tracing": "false", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_addr": "0.0.0.0", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_http_port": "9926", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_prio_limit": "5", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_sort_metrics": "true", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_stats_period": "5", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "fake_statfs_for_testing": "0", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "fatal_signal_handlers": "true", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "filer_max_purge_ops": "10", 2024-06-14T22:55:23.542 INFO:teuthology.orchestra.run.smithi086.stdout: "filer_max_truncate_ops": "128", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_apply_finisher_threads": "1", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_blackhole": "false", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_btrfs_clone_range": "true", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_btrfs_snap": "true", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_caller_concurrency": "10", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_collect_device_partition_information": "true", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_commit_timeout": "600.000000", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_debug_inject_read_err": "false", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_debug_omap_check": "false", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_debug_random_read_err": "0.000000", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_debug_verify_split": "false", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_dump_file": "", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-06-14T22:55:23.543 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fadvise": "true", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fail_eio": "true", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fd_cache_shards": "16", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fd_cache_size": "128", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fiemap": "false", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fiemap_threshold": "4096", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_index_retry_probability": "0.000000", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_inject_stall": "0", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_journal_parallel": "false", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_journal_trailing": "false", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_journal_writeahead": "false", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_kill_at": "0", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattr_size": "0", 2024-06-14T22:55:23.544 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattrs": "0", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattrs_other": "2", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_sync_interval": "5.000000", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_xattr_value_size": "0", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_merge_threshold": "-10", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_min_sync_interval": "0.010000", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_odsync_write": "false", 2024-06-14T22:55:23.545 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_omap_backend": "rocksdb", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_omap_backend_path": "", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_omap_header_cache_size": "1024", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_ondisk_finisher_threads": "1", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_op_thread_timeout": "60", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_op_threads": "2", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_punch_hole": "false", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_bytes": "104857600", 2024-06-14T22:55:23.546 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_ops": "50", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_seek_data_hole": "false", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_sloppy_crc": "false", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_splice": "false", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_split_multiple": "2", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_split_rand_factor": "20", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_update_to": "1000", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-06-14T22:55:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_enable": "true", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_xfs_extsize": "false", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_zfs_snap": "false", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "fio_dir": "/tmp/fio", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "44b2848a-2a9d-11ef-bc9d-c7b262605968", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_allow_other": "true", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_atomic_o_trunc": "true", 2024-06-14T22:55:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_big_writes": "true", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_debug": "false", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_default_permissions": "false", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_disable_pagecache": "false", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_max_write": "0", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_multithreaded": "true", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_require_active_mds": "true", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_set_user_groups": "true", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_splice_move": "true", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_splice_read": "true", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_splice_write": "true", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_syncfs_on_mksnap": "true", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_use_invalidate_cb": "true", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "gss_target_name": "ceph", 2024-06-14T22:55:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "heartbeat_file": "", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "heartbeat_inject_failure": "0", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "heartbeat_interval": "5", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "host": "smithi086", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_max_size": "1073741824", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-06-14T22:55:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_watermark": "0.900000", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "inject_early_sigterm": "false", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "jaeger_agent_port": "6799", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "jaeger_tracing_enable": "false", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_aio": "true", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_align_min_size": "65536", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_block_align": "true", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_block_size": "4096", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_dio": "true", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_discard": "false", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_force_aio": "false", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_ignore_corruption": "false", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_max_write_bytes": "10485760", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_max_write_entries": "100", 2024-06-14T22:55:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_replay_from": "0", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_throttle_high_multiple": "0.000000", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_throttle_max_multiple": "0.000000", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_write_header_frequency": "0", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_zero_on_create": "false", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "journaler_prefetch_periods": "10", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "journaler_prezero_periods": "5", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "journaler_write_head_interval": "15", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "keyfile": "", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_backend": "rocksdb", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_default_stripe_size": "65536", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_fsck_on_mount": "false", 2024-06-14T22:55:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_fsck_on_mount_deep": "true", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_max_bytes": "67108864", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_max_ops": "512", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_nid_prealloc": "1024", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_onode_map_size": "1024", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_sync_submit_transaction": "false", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_sync_transaction": "false", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_block_size": "65536", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_bloom_size": "0", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_cache_size": "536870912", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_compact_on_mount": "false", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_compression": "false", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_log": "", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_log_to_ceph_log": "true", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_max_open_files": "0", 2024-06-14T22:55:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_paranoid": "false", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "leveldb_write_buffer_size": "33554432", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "librados_thread_count": "2", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "lockdep": "false", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "lockdep_force_backtrace": "false", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_coarse_timestamps": "true", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_flush_on_exit": "false", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_graylog_host": "127.0.0.1", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_graylog_port": "12201", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_max_new": "1000", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_max_recent": "10000", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_stderr_prefix": "", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_stop_at_utilization": "0.970000", 2024-06-14T22:55:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_file": "true", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_graylog": "false", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_journald": "false", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_stderr": "false", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_syslog": "false", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "max_rotating_auth_attempts": "10", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_action_on_write_error": "1", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_alternate_name_max": "8192", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_asio_thread_count": "2", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_export_pin": "true", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_fragment_dirs": "true", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_fragment_interval": "5", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_fragment_size_max": "100000", 2024-06-14T22:55:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_idle_threshold": "0.000000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_interval": "10", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_max": "-1", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_max_until": "-1", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_merge_size": "50", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_midchunk": "0.300000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_min_rebalance": "0.100000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_min_start": "0.200000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_minchunk": "0.001000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_mode": "0", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_need_max": "1.200000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_need_min": "0.800000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_sample_interval": "3.000000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_split_bits": "3", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_split_rd": "25000.000000", 2024-06-14T22:55:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_split_size": "10000", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_split_wr": "10000.000000", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_target_decay": "10.000000", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_beacon_grace": "15.000000", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_beacon_interval": "4.000000", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_beacon_mon_down_grace": "60", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_memory_limit": "4294967296", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_mid": "0.700000", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_release_free_interval": "10", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_reservation": "0.050000", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_trim_interval": "1", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_trim_threshold": "262144", 2024-06-14T22:55:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_client_delegate_inos_pct": "50", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_client_prealloc_inos": "1000", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_connect_bootstrapping": "false", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_damage_table_max_entries": "10000", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_debug_auth_pins": "false", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_debug_frag": "false", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_debug_scatterstat": "false", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_debug_subtrees": "false", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_decay_halflife": "5.000000", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_default_dir_hash": "2", 2024-06-14T22:55:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_defer_session_stale": "true", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_deny_all_reconnect": "false", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dir_keys_per_op": "16384", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dir_max_commit_size": "10", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dir_max_entries": "0", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dir_prefetch": "true", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dirstat_min_interval": "1.000000", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dump_cache_after_rejoin": "false", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dump_cache_on_map": "false", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dump_cache_threshold_file": "0", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_early_reply": "true", 2024-06-14T22:55:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_enable_op_tracker": "true", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_enforce_unique_name": "true", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_export_ephemeral_distributed": "true", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_export_ephemeral_random": "true", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_forward_all_requests_to_auth": "false", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_health_cache_threshold": "1.500000", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_health_summarize_threshold": "10", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_heartbeat_grace": "15.000000", 2024-06-14T22:55:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_heartbeat_reset_grace": "1000", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_health_dummy": "false", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_migrator_session_race": "false", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_join_fs": "", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_journal_format": "1", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_create_at": "0", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_export_at": "0", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_import_at": "0", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_journal_at": "0", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_journal_expire_at": "0", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_journal_replay_at": "0", 2024-06-14T22:55:23.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_link_at": "0", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_mdstable_at": "0", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_openc_at": "0", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_rename_at": "0", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_events_per_segment": "1024", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_max_events": "-1", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_max_segments": "128", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_pause": "false", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_segment_size": "0", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_skip_corrupt_events": "false", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_warn_factor": "2.000000", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_caps_per_client": "1048576", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_completed_flushes": "100000", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_completed_requests": "100000", 2024-06-14T22:55:23.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_export_size": "20971520", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_file_recover": "32", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_purge_files": "64", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_purge_ops": "8192", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_snaps_per_dir": "100", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_metrics_update_interval": "2", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_min_caps_per_client": "100", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_min_caps_working_set": "10000", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_numa_node": "-1", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_complaint_time": "30.000000", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_history_duration": "600", 2024-06-14T22:55:23.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_history_size": "20", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_history_slow_op_size": "20", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_history_slow_op_threshold": "10", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_log_threshold": "5", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_ping_grace": "15", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_ping_interval": "5", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_max_caps": "30000", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_max_decay_threshold": "131072", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_warning_threshold": "262144", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_reconnect_timeout": "45.000000", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_replay_interval": "1.000000", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-06-14T22:55:23.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_root_ino_gid": "0", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_root_ino_uid": "0", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_blocklist_on_evict": "true", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_blocklist_on_timeout": "true", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_metadata_threshold": "16777216", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_shutdown_check": "0", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_skip_ino": "0", 2024-06-14T22:55:23.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sleep_rank_change": "0.000000", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_snap_max_uid": "4294967294", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_snap_min_uid": "0", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_snap_rstat": "false", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_standby_replay_damaged": "false", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_symlink_recovery": "true", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_task_status_update_interval": "2.000000", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_thrash_exports": "0", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_thrash_fragments": "0", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_tick_interval": "5.000000", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_valgrind_exit": "false", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_verify_backtrace": "1", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_verify_scatter": "false", 2024-06-14T22:55:23.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_wipe_ino_prealloc": "false", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_wipe_sessions": "false", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool_debug": "false", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "memstore_debug_omit_block_device_write": "false", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "memstore_device_bytes": "1073741824", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "memstore_page_set": "false", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "memstore_page_size": "65536", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_client_bytes": "134217728", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_client_messages": "512", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_connect_retry_interval": "1.000000", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_disabled_modules": "", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-06-14T22:55:23.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_max_pg_num_change": "128", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_mds_bytes": "134217728", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_mds_messages": "128", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_mon_bytes": "134217728", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_mon_messages": "128", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_osd_bytes": "536870912", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_osd_messages": "8192", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_pool": "true", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_service_beacon_grace": "60.000000", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_standby_modules": "true", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_stats_period": "5", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_stats_threshold": "5", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_tick_period": "2", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-06-14T22:55:23.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_accept_timeout_factor": "2.000000", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_allow_pool_delete": "true", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_allow_pool_size_one": "false", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_auth_validate_all_caps": "true", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_bytes": "104857600", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_directed_command_retry": "2", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_interval": "3.000000", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-06-14T22:55:23.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_parallel": "3", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_log_interval": "1.000000", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_ping_interval": "10.000000", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_ping_timeout": "30.000000", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_clock_drift_allowed": "1.000000", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_file_level": "debug", 2024-06-14T22:55:23.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_file": "false", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_graylog": "false", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_journald": "true", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_stderr": "false", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_compact_on_bootstrap": "false", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_compact_on_start": "false", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_compact_on_trim": "true", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_con_tracker_persist_interval": "10", 2024-06-14T22:55:23.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_con_tracker_score_halflife": "43200", 2024-06-14T22:55:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_config_key_max_entry_size": "65536", 2024-06-14T22:55:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cpu_threads": "4", 2024-06-14T22:55:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_crush_min_required_version": "hammer", 2024-06-14T22:55:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_daemon_bytes": "419430400", 2024-06-14T22:55:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-06-14T22:55:23.572 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_data_avail_crit": "5", 2024-06-14T22:55:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_data_avail_warn": "5", 2024-06-14T22:55:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_data_size_warn": "16106127360", 2024-06-14T22:55:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_block_osdmap_trim": "false", 2024-06-14T22:55:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-06-14T22:55:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_dump_json": "false", 2024-06-14T22:55:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-06-14T22:55:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_dump_transactions": "false", 2024-06-14T22:55:23.573 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_extra_checks": "false", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_no_require_quincy": "false", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_no_require_reef": "false", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_delta_reset_interval": "10.000000", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_dns_srv_name": "ceph-mon", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_down_mkfs_grace": "60", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_election_default_strategy": "3", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_election_timeout": "5.000000", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_elector_ping_divisor": "2", 2024-06-14T22:55:23.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_elector_ping_timeout": "2.000000", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_enable_op_tracker": "true", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_fake_pool_delete": "false", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_force_quorum_join": "false", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_globalid_prealloc": "10000", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_detail_to_clog": "true", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_log_update_period": "5", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_max_detail": "50", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_to_clog": "true", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_to_clog_interval": "600", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_host": "[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0]", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_host_override": "", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_initial_members": "", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-06-14T22:55:23.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_keyvaluedb": "rocksdb", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_lease": "5.000000", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_log_full_interval": "50", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_log_max": "10000", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_log_max_summary": "50", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_log_entries_per_event": "4096", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_log_epochs": "500", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_mdsmap_epochs": "500", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_mgrmap_epochs": "500", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_osd": "10000", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_pg_per_osd": "10000", 2024-06-14T22:55:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_pool_pg_num": "65536", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mds_force_trim_to": "0", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mds_skip_sanity": "true", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_memory_autotune": "true", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_memory_target": "2147483648", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_beacon_grace": "30", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_digest_period": "5", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_inactive_grace": "60", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_mkfs_grace": "240", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-06-14T22:55:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_min_osdmap_epochs": "500", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_complaint_time": "30", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_history_duration": "600", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_history_size": "20", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_history_slow_op_size": "20", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_history_slow_op_threshold": "10", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_log_threshold": "5", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_auto_mark_in": "false", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_auto_mark_new_in": "true", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-06-14T22:55:23.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_cache_size": "500", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_cache_size_min": "134217728", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_crush_smoke_test": "true", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_destroyed_out_interval": "600", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_down_out_interval": "600", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_force_trim_to": "0", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_full_ratio": "0.950000", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_laggy_halflife": "3600", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_laggy_max_interval": "300", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_laggy_weight": "0.300000", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_max_creating_pgs": "1024", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_max_initial_pgs": "1024", 2024-06-14T22:55:23.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_min_down_reporters": "2", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_prime_pg_temp": "true", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_report_timeout": "900", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_warn_num_repaired": "10", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_warn_op_age": "32.000000", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osdmap_full_prune_interval": "10", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osdmap_full_prune_min": "10000", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-06-14T22:55:23.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_stuck_threshold": "60", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_warn_min_objects": "10000", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_warn_min_per_osd": "0", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pool_quota_crit_threshold": "0", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pool_quota_warn_threshold": "0", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_probe_timeout": "2.000000", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_reweight_max_change": "0.050000", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_reweight_max_osds": "4", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-06-14T22:55:23.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_interval": "86400", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_max_keys": "100", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_timeout": "300", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_session_timeout": "300", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_smart_report_timeout": "5", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stat_smooth_intervals": "6", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stretch_pool_min_size": "2", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stretch_pool_size": "4", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_subscribe_interval": "86400.000000", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_debug": "false", 2024-06-14T22:55:23.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_max_payload_keys": "2000", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_max_payload_size": "1048576", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_provider_kill_at": "0", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_requester_kill_at": "0", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_timeout": "60.000000", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_target_pg_per_osd": "100", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_tick_interval": "5", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_timecheck_interval": "300.000000", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_use_min_delay_socket": "false", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_older_version_delay": "604800", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_filestore_osds": "true", 2024-06-14T22:55:23.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_misplaced": "false", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_older_version": "true", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_pool_no_app": "true", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_too_few_osds": "false", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-06-14T22:55:23.584 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": "", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_tracing_enabled": "false", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_op_threads": "3", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_buffer_size": "131072", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_cm": "false", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_device_name": "", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_dscp": "96", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_gid_idx": "0", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_local_gid": "", 2024-06-14T22:55:23.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_polling_us": "1000", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_port_num": "1", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_roce_ver": "1", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_send_buffers": "1024", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_sl": "3", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_support_srq": "true", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_type": "ib", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_reap_threshold": "5", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_before_connect": "false", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_ipv4": "true", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_ipv6": "false", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_msgr1": "true", 2024-06-14T22:55:23.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_msgr2": "true", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_port_max": "7568", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_port_min": "6800", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_prefer_ipv4": "false", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_retry_count": "3", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_retry_delay": "5", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_client": "false", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_mds": "false", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_mgr": "false", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_mon": "false", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_osd": "false", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_client_mode": "crc secure", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_cluster_mode": "crc secure", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_cluster_type": "", 2024-06-14T22:55:23.587 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_compress_secure": "false", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_connection_idle_timeout": "900", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_connection_ready_timeout": "10", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_crc_data": "true", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_crc_header": "true", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_bad_msg": "false", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_bug": "true", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_old_message": "true", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_skipped_message": "false", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_unhandled_msg": "false", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_coremask": "0xF", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_devs_allowlist": "", 2024-06-14T22:55:23.588 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_enable_tso": "true", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_hugepages": "", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_hw_flow_control": "true", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_lro": "true", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_memory_channel": "4", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_pmd": "", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_port_id": "0", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dump_corrupt_message_level": "1", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dump_on_send": "false", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_initial_backoff": "0.200000", 2024-06-14T22:55:23.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_delay_max": "1.000000", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_delay_probability": "0.000000", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_delay_type": "", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_internal_delays": "0.000000", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_network_congestion": "0", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_socket_failures": "0", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_learn_addr_from_peer": "true", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_max_accept_failures": "4", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_max_backoff": "15.000000", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_mon_client_mode": "secure crc", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_mon_cluster_mode": "secure crc", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_mon_service_mode": "secure crc", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_osd_compress_min_size": "1024", 2024-06-14T22:55:23.590 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_osd_compress_mode": "none", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_osd_compression_algorithm": "snappy", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_pq_min_cost": "65536", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_public_type": "", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_service_mode": "crc secure", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_tcp_listen_backlog": "512", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_tcp_nodelay": "true", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_tcp_rcvbuf": "0", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_type": "async+posix", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "no_config_file": "false", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_completion_locks_per_session": "32", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_debug_inject_relock_delay": "false", 2024-06-14T22:55:23.591 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_inflight_op_bytes": "104857600", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_inflight_ops": "1024", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_inject_no_watch_ping": "false", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_tick_interval": "5.000000", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_timeout": "10.000000", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "objectstore_blackhole": "false", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "openssl_engine_opts": "", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_delay_time": "5.000000", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_hist_halflife": "1000", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_max_low_ops": "2", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_max_ops": "4", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_quantize_effort": "0.100000", 2024-06-14T22:55:23.592 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_slop": "0.020000", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_allow_recovery_below_min_size": "true", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_asio_thread_count": "2", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_async_recovery_min_cost": "100", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_auto_mark_unfound_lost": "false", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backfill_retry_interval": "30.000000", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backfill_scan_max": "512", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backfill_scan_min": "64", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backoff_on_degraded": "false", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backoff_on_peering": "false", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backoff_on_unfound": "true", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_beacon_report_interval": "300", 2024-06-14T22:55:23.593 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bench_duration": "30", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bench_max_block_size": "67108864", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bench_small_size_max_iops": "100", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_blkin_trace_all": "false", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_blocked_scrub_grace_period": "120", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_check_for_log_corruption": "false", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-06-14T22:55:23.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_class_update_on_start": "true", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_client_message_cap": "256", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_client_message_size_cap": "524288000", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_client_op_priority": "63", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_client_watch_timeout": "30", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_command_max_records": "256", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_compact_on_start": "false", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_crush_chooseleaf_type": "1", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_crush_initial_weight": "-1.000000", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_crush_update_on_start": "true", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_crush_update_weight_set": "true", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-06-14T22:55:23.595 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-06-14T22:55:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-06-14T22:55:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_drop_ping_duration": "0", 2024-06-14T22:55:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-06-14T22:55:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_feed_pullee": "-1", 2024-06-14T22:55:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-06-14T22:55:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-06-14T22:55:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-06-14T22:55:23.596 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_misdirected_ops": "false", 2024-06-14T22:55:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_no_acting_change": "false", 2024-06-14T22:55:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_no_purge_strays": "false", 2024-06-14T22:55:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_op_order": "false", 2024-06-14T22:55:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_pg_log_writeout": "false", 2024-06-14T22:55:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_pretend_recovery_active": "false", 2024-06-14T22:55:23.597 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-06-14T22:55:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-06-14T22:55:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_shutdown": "false", 2024-06-14T22:55:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-06-14T22:55:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-06-14T22:55:23.598 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_verify_cached_snaps": "false", 2024-06-14T22:55:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_verify_missing_on_start": "false", 2024-06-14T22:55:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_verify_snaps": "false", 2024-06-14T22:55:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-06-14T22:55:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-06-14T22:55:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_keys": "1024", 2024-06-14T22:55:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-06-14T22:55:23.599 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-06-14T22:55:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-06-14T22:55:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_stride": "524288", 2024-06-14T22:55:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-06-14T22:55:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_default_data_pool_replay_window": "45", 2024-06-14T22:55:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_default_notify_timeout": "30", 2024-06-14T22:55:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_delete_sleep": "0.000000", 2024-06-14T22:55:23.600 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-06-14T22:55:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-06-14T22:55:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-06-14T22:55:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_discard_disconnected_ops": "true", 2024-06-14T22:55:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_enable_op_tracker": "true", 2024-06-14T22:55:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-06-14T22:55:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_extblkdev_plugins": "vdo", 2024-06-14T22:55:23.601 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-06-14T22:55:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-06-14T22:55:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_info": "true", 2024-06-14T22:55:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_shutdown": "true", 2024-06-14T22:55:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-06-14T22:55:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_shutdown_timeout": "15", 2024-06-14T22:55:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-06-14T22:55:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-06-14T22:55:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-06-14T22:55:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_function_tracing": "false", 2024-06-14T22:55:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_grace": "20", 2024-06-14T22:55:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_interval": "6", 2024-06-14T22:55:23.603 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-06-14T22:55:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_min_peers": "10", 2024-06-14T22:55:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_min_size": "2000", 2024-06-14T22:55:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_stale": "600", 2024-06-14T22:55:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-06-14T22:55:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_hit_set_max_size": "100000", 2024-06-14T22:55:23.604 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_hit_set_min_size": "1000", 2024-06-14T22:55:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-06-14T22:55:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-06-14T22:55:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-06-14T22:55:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-06-14T22:55:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-06-14T22:55:23.605 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_journal_flush_on_shutdown": "true", 2024-06-14T22:55:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_journal_size": "5120", 2024-06-14T22:55:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_kill_backfill_at": "0", 2024-06-14T22:55:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_loop_before_reset_tphandle": "64", 2024-06-14T22:55:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_cache_size": "50", 2024-06-14T22:55:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_dedup": "true", 2024-06-14T22:55:23.606 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_message_max": "40", 2024-06-14T22:55:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_message_max_bytes": "10485760", 2024-06-14T22:55:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_share_max_epochs": "40", 2024-06-14T22:55:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_attr_name_len": "100", 2024-06-14T22:55:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_attr_size": "0", 2024-06-14T22:55:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_backfills": "1", 2024-06-14T22:55:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_markdown_count": "5", 2024-06-14T22:55:23.607 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_markdown_period": "600", 2024-06-14T22:55:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_object_name_len": "2048", 2024-06-14T22:55:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_object_namespace_len": "256", 2024-06-14T22:55:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_object_size": "134217728", 2024-06-14T22:55:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-06-14T22:55:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_omap_entries_per_request": "1024", 2024-06-14T22:55:23.608 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_pg_blocked_by": "16", 2024-06-14T22:55:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_pg_log_entries": "10000", 2024-06-14T22:55:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-06-14T22:55:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_pgls": "1024", 2024-06-14T22:55:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_push_cost": "8388608", 2024-06-14T22:55:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_push_objects": "10", 2024-06-14T22:55:23.609 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_scrubs": "3", 2024-06-14T22:55:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-06-14T22:55:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_trimming_pgs": "2", 2024-06-14T22:55:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_write_op_reply_len": "64", 2024-06-14T22:55:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_write_size": "90", 2024-06-14T22:55:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-06-14T22:55:23.610 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-06-14T22:55:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-06-14T22:55:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-06-14T22:55:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-06-14T22:55:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-06-14T22:55:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-06-14T22:55:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_override_recovery_settings": "false", 2024-06-14T22:55:23.611 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_profile": "balanced", 2024-06-14T22:55:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-06-14T22:55:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-06-14T22:55:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-06-14T22:55:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-06-14T22:55:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-06-14T22:55:23.612 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-06-14T22:55:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-06-14T22:55:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-06-14T22:55:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-06-14T22:55:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-06-14T22:55:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_skip_benchmark": "false", 2024-06-14T22:55:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_base": "805306368", 2024-06-14T22:55:23.613 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_cache_min": "134217728", 2024-06-14T22:55:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-06-14T22:55:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-06-14T22:55:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_target": "4294967296", 2024-06-14T22:55:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_target_autotune": "false", 2024-06-14T22:55:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-06-14T22:55:23.614 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_min_pg_log_entries": "250", 2024-06-14T22:55:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_min_recovery_priority": "0", 2024-06-14T22:55:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_heartbeat_interval": "30", 2024-06-14T22:55:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-06-14T22:55:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_report_interval": "5", 2024-06-14T22:55:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_report_max_in_flight": "2", 2024-06-14T22:55:23.615 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-06-14T22:55:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_num_cache_shards": "32", 2024-06-14T22:55:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_num_op_tracker_shard": "32", 2024-06-14T22:55:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_numa_auto_affinity": "true", 2024-06-14T22:55:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_numa_node": "-1", 2024-06-14T22:55:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_numa_prefer_iface": "true", 2024-06-14T22:55:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-06-14T22:55:23.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_objecter_finishers": "1", 2024-06-14T22:55:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_objectstore": "bluestore", 2024-06-14T22:55:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_objectstore_fuse": "false", 2024-06-14T22:55:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_objectstore_tracing": "false", 2024-06-14T22:55:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_complaint_time": "30.000000", 2024-06-14T22:55:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_history_duration": "600", 2024-06-14T22:55:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_history_size": "20", 2024-06-14T22:55:23.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_history_slow_op_size": "20", 2024-06-14T22:55:23.618 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-06-14T22:55:23.618 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_log_threshold": "5", 2024-06-14T22:55:23.618 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_shards": "0", 2024-06-14T22:55:23.618 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_shards_hdd": "5", 2024-06-14T22:55:23.618 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_shards_ssd": "8", 2024-06-14T22:55:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_threads_per_shard": "0", 2024-06-14T22:55:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-06-14T22:55:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-06-14T22:55:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-06-14T22:55:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_pq_min_cost": "65536", 2024-06-14T22:55:23.619 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_queue": "mclock_scheduler", 2024-06-14T22:55:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_queue_cut_off": "high", 2024-06-14T22:55:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_thread_suicide_timeout": "150", 2024-06-14T22:55:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_thread_timeout": "15", 2024-06-14T22:55:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_open_classes_on_start": "true", 2024-06-14T22:55:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_os_flags": "0", 2024-06-14T22:55:23.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_peering_op_priority": "255", 2024-06-14T22:55:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_delete_cost": "1048576", 2024-06-14T22:55:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_delete_priority": "5", 2024-06-14T22:55:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-06-14T22:55:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-06-14T22:55:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_log_dups_tracked": "3000", 2024-06-14T22:55:23.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_log_trim_max": "10000", 2024-06-14T22:55:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_log_trim_min": "100", 2024-06-14T22:55:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-06-14T22:55:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_object_context_cache_count": "64", 2024-06-14T22:55:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_stat_report_interval_max": "500", 2024-06-14T22:55:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-06-14T22:55:23.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-06-14T22:55:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-06-14T22:55:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-06-14T22:55:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-06-14T22:55:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-06-14T22:55:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_crimson": "false", 2024-06-14T22:55:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_crush_rule": "-1", 2024-06-14T22:55:23.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_ec_fast_read": "false", 2024-06-14T22:55:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-06-14T22:55:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_bulk": "false", 2024-06-14T22:55:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-06-14T22:55:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_nodelete": "false", 2024-06-14T22:55:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-06-14T22:55:23.624 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-06-14T22:55:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flags": "0", 2024-06-14T22:55:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-06-14T22:55:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_min_size": "0", 2024-06-14T22:55:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-06-14T22:55:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_pg_num": "32", 2024-06-14T22:55:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_pgp_num": "0", 2024-06-14T22:55:23.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-06-14T22:55:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_size": "3", 2024-06-14T22:55:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_type": "replicated", 2024-06-14T22:55:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-06-14T22:55:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_use_gmt_hitset": "true", 2024-06-14T22:55:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_push_per_object_cost": "1000", 2024-06-14T22:55:23.626 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_read_ec_check_for_errors": "false", 2024-06-14T22:55:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recover_clone_overlap": "true", 2024-06-14T22:55:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recover_clone_overlap_limit": "10", 2024-06-14T22:55:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_cost": "20971520", 2024-06-14T22:55:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_delay_start": "0.000000", 2024-06-14T22:55:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_active": "0", 2024-06-14T22:55:23.627 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_active_hdd": "3", 2024-06-14T22:55:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_active_ssd": "10", 2024-06-14T22:55:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_chunk": "8388608", 2024-06-14T22:55:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-06-14T22:55:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_single_start": "1", 2024-06-14T22:55:23.628 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_op_priority": "3", 2024-06-14T22:55:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_op_warn_multiple": "16", 2024-06-14T22:55:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_priority": "5", 2024-06-14T22:55:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_retry_interval": "30.000000", 2024-06-14T22:55:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_sleep": "0.000000", 2024-06-14T22:55:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-06-14T22:55:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-06-14T22:55:23.629 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-06-14T22:55:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_repair_during_recovery": "false", 2024-06-14T22:55:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_requested_scrub_priority": "120", 2024-06-14T22:55:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-06-14T22:55:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_rollback_to_cluster_snap": "", 2024-06-14T22:55:23.630 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_auto_repair": "false", 2024-06-14T22:55:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-06-14T22:55:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-06-14T22:55:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_begin_hour": "0", 2024-06-14T22:55:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_begin_week_day": "0", 2024-06-14T22:55:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_chunk_max": "25", 2024-06-14T22:55:23.631 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_chunk_min": "5", 2024-06-14T22:55:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_cost": "52428800", 2024-06-14T22:55:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_during_recovery": "false", 2024-06-14T22:55:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_end_hour": "0", 2024-06-14T22:55:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_end_week_day": "0", 2024-06-14T22:55:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_event_cost": "4096", 2024-06-14T22:55:23.632 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-06-14T22:55:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-06-14T22:55:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_invalid_stats": "true", 2024-06-14T22:55:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_load_threshold": "0.500000", 2024-06-14T22:55:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_max_interval": "604800.000000", 2024-06-14T22:55:23.633 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_max_preemptions": "5", 2024-06-14T22:55:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_min_interval": "86400.000000", 2024-06-14T22:55:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_priority": "5", 2024-06-14T22:55:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_reservation_timeout": "5000", 2024-06-14T22:55:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_sleep": "0.000000", 2024-06-14T22:55:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-06-14T22:55:23.634 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-06-14T22:55:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-06-14T22:55:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_shutdown_pgref_assert": "false", 2024-06-14T22:55:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_skip_data_digest": "false", 2024-06-14T22:55:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_smart_report_timeout": "5", 2024-06-14T22:55:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_cost": "1048576", 2024-06-14T22:55:23.635 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_priority": "5", 2024-06-14T22:55:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_sleep": "0.000000", 2024-06-14T22:55:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-06-14T22:55:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-06-14T22:55:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-06-14T22:55:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-06-14T22:55:23.636 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_stats_update_period_scrubbing": "15", 2024-06-14T22:55:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-06-14T22:55:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_target_transaction_size": "30", 2024-06-14T22:55:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-06-14T22:55:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-06-14T22:55:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-06-14T22:55:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-06-14T22:55:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-06-14T22:55:23.637 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_mode": "writeback", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tracing": "false", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_use_stale_snap": "false", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "osdc_blkin_trace_all": "false", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_kill_at": "0", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_max_join_drift": "10", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_min": "500", 2024-06-14T22:55:23.638 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_min_wait": "0.050000", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_propose_interval": "1.000000", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_service_trim_max": "500", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_service_trim_max_multiplier": "20", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_service_trim_min": "250", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_stash_full_interval": "25", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_trim_max": "500", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_trim_min": "250", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "perf": "true", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "pid_file": "", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "-", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrv": "", 2024-06-14T22:55:23.639 INFO:teuthology.orchestra.run.smithi086.stdout: "public_bind_addr": "-", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "public_network": "172.21.0.0/20", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "public_network_interface": "", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "qat_compressor_enabled": "false", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "qat_compressor_session_max_number": "256", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rados_mon_op_timeout": "0", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rados_osd_op_timeout": "0", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rados_tracing": "false", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_atime_update_interval": "60", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_balance_parent_reads": "false", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_balance_snap_reads": "false", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_blkin_trace_all": "false", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_blocklist_expire_seconds": "0", 2024-06-14T22:55:23.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_blocklist_on_break_lock": "true", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache": "true", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_block_writes_upfront": "false", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_max_dirty": "25165824", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_max_dirty_object": "0", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_policy": "writearound", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_size": "33554432", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_target_dirty": "16777216", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_clone_copy_on_read": "false", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_compression_hint": "none", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_concurrent_management_ops": "10", 2024-06-14T22:55:23.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_clone_format": "auto", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_data_pool": "", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_features": "61", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_format": "2", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_map_options": "", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_order": "22", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_pool": "rbd", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_stripe_count": "0", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_stripe_unit": "0", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_disable_zero_copy_writes": "true", 2024-06-14T22:55:23.642 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_discard_granularity_bytes": "65536", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_enable_alloc_hint": "true", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_io_scheduler": "simple", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_commit_age": "5.000000", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_flush_interval": "0", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_order": "24", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_pool": "", 2024-06-14T22:55:23.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_splay_width": "4", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_localize_parent_reads": "false", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_localize_snap_reads": "false", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_policy_type": "simple", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-06-14T22:55:23.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_autotune": "true", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_base": "805306368", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_target": "4294967296", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirroring_delete_delay": "0", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirroring_replay_delay": "0", 2024-06-14T22:55:23.645 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_move_to_trash_on_remove": "false", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mtime_update_interval": "60", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_non_blocking_aio": "true", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_op_thread_timeout": "60", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_op_threads": "1", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_parent_cache_enabled": "false", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_persistent_cache_mode": "disabled", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_persistent_cache_path": "/tmp", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_persistent_cache_size": "1073741824", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_plugins": "", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_bps_burst": "0", 2024-06-14T22:55:23.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-06-14T22:55:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_bps_limit": "0", 2024-06-14T22:55:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_exclude_ops": "0", 2024-06-14T22:55:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_iops_burst": "0", 2024-06-14T22:55:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-06-14T22:55:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_iops_limit": "0", 2024-06-14T22:55:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_bps_burst": "0", 2024-06-14T22:55:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-06-14T22:55:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_bps_limit": "0", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_iops_burst": "0", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_iops_limit": "0", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_schedule_tick_min": "50", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_bps_burst": "0", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_bps_limit": "0", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_iops_burst": "0", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_iops_limit": "0", 2024-06-14T22:55:23.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_quiesce_notification_attempts": "10", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_read_from_replica_policy": "default", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_readahead_max_bytes": "524288", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_readahead_trigger_requests": "10", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_request_timed_out_seconds": "30", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_skip_partial_discard": "true", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_tracing": "false", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_validate_names": "true", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_validate_pool": "true", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "restapi_base_url": "", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "restapi_log_level": "", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_acl_grants_max_num": "100", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_admin_entry": "admin", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-06-14T22:55:23.649 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_backend_store": "rados", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_barbican_url": "", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_beast_enable_async": "true", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_index_max_aio": "128", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_quota_ttl": "600", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cache_enabled": "true", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cache_expiry_interval": "900", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cache_lru_size": "10000", 2024-06-14T22:55:23.650 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_config_store": "rados", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_content_length_compat": "false", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_copy_obj_progress": "true", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cors_rules_max_num": "100", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cross_domain_policy": "", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_default_encryption_key": "", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_addr": "", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_ca_path": "", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_client_cert": "", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_client_key": "", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_password": "", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_username": "", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_require_ssl": "true", 2024-06-14T22:55:23.651 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_suppress_logs": "true", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_addr": "", 2024-06-14T22:55:23.652 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_auth": "token", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_namespace": "", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_prefix": "", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_token_file": "", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_buffersize": "524288", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_low_speed_limit": "1024", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_low_speed_time": "300", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_tcp_keepalive": "0", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-06-14T22:55:23.653 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_fadvise": "4", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_log_changes_size": "1000", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_log_num_shards": "128", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_log_window": "30", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_notify_interval_msec": "0", 2024-06-14T22:55:23.654 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_sync_poll_interval": "20", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_sync_spawn_window": "20", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_data_log_backing": "fifo", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_region_info_oid": "default.region", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_defer_to_bucket_acls": "", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-06-14T22:55:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_data_res": "500.000000", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dns_name": "", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dns_s3website_name": "", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dynamic_resharding": "true", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_gc_threads": "true", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_lc_threads": "true", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_ops_log": "false", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_quota_threads": "true", 2024-06-14T22:55:23.656 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_static_website": "false", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_usage_log": "false", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enforce_swift_acls": "true", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_exit_timeout_secs": "120", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_expose_bucket": "false", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_extended_http_attrs": "", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_filter": "none", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_frontends": "beast port=7480", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_concurrent_io": "10", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_deferred": "50", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_objs": "32", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_queue_size": "134213632", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_trim_chunk": "16", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_obj_min_wait": "7200", 2024-06-14T22:55:23.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_processor_max_time": "3600", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_processor_period": "3600", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_get_obj_window_size": "16777216", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_healthcheck_disabling_path": "", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ignore_get_invalid_range": "false", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_init_timeout": "300", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_domain": "", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_password": "", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_password_path": "", 2024-06-14T22:55:23.658 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_project": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_tenant": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_token": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_token_path": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_user": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_api_version": "2", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_domain": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_password": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_project": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_tenant": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_user": "", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-06-14T22:55:23.659 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_implicit_tenants": "false", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_service_token_enabled": "false", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_token_cache_size": "10000", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_url": "", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_verify_ssl": "true", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_debug_interval": "-1", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_lock_max_time": "90", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_max_objs": "32", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_max_rules": "1000", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_max_worker": "3", 2024-06-14T22:55:23.660 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_max_wp_worker": "3", 2024-06-14T22:55:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_thread_delay": "0", 2024-06-14T22:55:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-06-14T22:55:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_dnattr": "uid", 2024-06-14T22:55:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-06-14T22:55:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_searchfilter": "", 2024-06-14T22:55:23.661 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-06-14T22:55:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_uri": "ldaps://", 2024-06-14T22:55:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-06-14T22:55:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-06-14T22:55:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-06-14T22:55:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_log_http_headers": "", 2024-06-14T22:55:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_log_nonexistent_bucket": "false", 2024-06-14T22:55:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_log_object_name_utc": "false", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_attr_name_len": "0", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_attr_size": "0", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_attrs_num_in_req": "0", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_chunk_size": "4194304", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_concurrent_requests": "1024", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_dynamic_shards": "1999", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_listing_results": "1000", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_notify_retries": "10", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_objs_per_shard": "100000", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_put_param_size": "1048576", 2024-06-14T22:55:23.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_put_size": "5368709120", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_slo_entries": "1000", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_md_log_max_shards": "64", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_md_notify_interval_msec": "200", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_meta_sync_poll_interval": "20", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_meta_sync_spawn_window": "20", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_mp_lock_max_time": "600", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_multipart_min_part_size": "5242880", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_fhcache_size": "2017", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-06-14T22:55:23.664 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_lru_lanes": "5", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_max_gc": "300", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_run_gc_threads": "false", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_run_lc_threads": "false", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_run_quota_threads": "false", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_run_sync_thread": "false", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_num_async_rados_threads": "32", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_num_control_oids": "8", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_numa_node": "-1", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_obj_stripe_size": "4194304", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_objexp_chunk_size": "100", 2024-06-14T22:55:23.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_objexp_gc_interval": "600", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_objexp_hints_num_shards": "127", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_op_thread_timeout": "600", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_op_tracing": "false", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_opa_token": "", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_opa_url": "", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_opa_verify_ssl": "true", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ops_log_rados": "false", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ops_log_socket_path": "", 2024-06-14T22:55:23.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_period_push_interval": "2.000000", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_period_push_interval_max": "30.000000", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_period_root_pool": ".rgw.root", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_print_continue": "true", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_print_prohibited_content_length": "false", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_rados_tracing": "false", 2024-06-14T22:55:23.667 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_realm": "", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_realm_id": "", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_region": "", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_region_root_pool": ".rgw.root", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_relaxed_region_enforcement": "false", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_request_uri": "", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_batch_size": "64", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_max_aio": "128", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_num_logs": "16", 2024-06-14T22:55:23.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_thread_interval": "600", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_resolve_cname": "false", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_rest_getusage_op_compat": "false", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_run_sync_thread": "true", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_use_keystone": "false", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_use_ldap": "false", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_use_rados": "true", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_use_sts": "false", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_success_create_obj_status": "0", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_scheduler_type": "throttler", 2024-06-14T22:55:23.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_script_uri": "", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_service_provider_name": "", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_client_id": "", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_client_secret": "", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_entry": "sts", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_key": "sts", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_max_session_duration": "43200", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_min_session_duration": "900", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_token_introspection_url": "", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_account_in_url": "false", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_auth_entry": "auth", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_auth_url": "", 2024-06-14T22:55:23.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_custom_header": "", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_enforce_content_length": "false", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_need_stats": "true", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_tenant_name": "", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_token_expiration": "86400", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_url": "", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_url_prefix": "swift", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_versioning_enabled": "false", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_lease_period": "120", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_log_trim_interval": "1200", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-06-14T22:55:23.671 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_obj_etag_verify": "false", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_trace_history_size": "4096", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_thread_pool_size": "512", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_comment": "", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_createby": "", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_encoding": "", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_flag": "false", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_origin": "", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_sha_unit": "524288", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_tracker": "", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_trust_forwarded_https": "false", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-06-14T22:55:23.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_usage_log_tick_interval": "30", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_usage_max_shards": "32", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_usage_max_user_shards": "1", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_use_opa_authz": "false", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_default_quota_max_size": "-1", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_max_buckets": "1000", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_policies_max_num": "100", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_quota_sync_interval": "86400", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_unique_email": "true", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_verify_ssl": "true", 2024-06-14T22:55:23.673 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_website_routing_rules_max_num": "50", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zone": "", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zone_id": "", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zonegroup": "", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zonegroup_id": "", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rgwlc_auto_session_clear": "true", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rgwlc_skip_bucket_step": "false", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_block_size": "4096", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-06-14T22:55:23.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_shard_bits": "4", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_size": "536870912", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_type": "binned_lru", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_collect_compaction_stats": "false", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_collect_extended_stats": "false", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_collect_memory_stats": "false", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_index_type": "binary_search", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_log_to_ceph_log": "true", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_metadata_block_size": "4096", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_partition_filters": "false", 2024-06-14T22:55:23.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_perf": "false", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "rotating_keys_renewal_timeout": "10", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "run_dir": "/var/run/ceph", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_block_create": "true", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_cache_lru_size": "67108864", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_cbjournal_size": "5368709120", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_default_max_object_size": "16777216", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_device_size": "53687091200", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_journal_batch_capacity": "16", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-06-14T22:55:23.676 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_journal_iodepth_limit": "5", 2024-06-14T22:55:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_main_device_type": "SSD", 2024-06-14T22:55:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_max_concurrent_transactions": "8", 2024-06-14T22:55:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-06-14T22:55:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-06-14T22:55:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-06-14T22:55:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-06-14T22:55:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_segment_size": "67108864", 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "set_keepcaps": "false", 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "setgroup": "ceph", 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "setuser": "ceph", 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "setuser_match_path": "", 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "target_max_misplaced_ratio": "0.050000", 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "thp": "false", 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "threadpool_default_timeout": "60", 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "threadpool_empty_queue_max_wait": "2", 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: "throttler_perf_counter": "true" 2024-06-14T22:55:23.678 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:55:23.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:23 smithi086 ceph-mon[97913]: pgmap v535: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:23.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:23 smithi086 ceph-mon[102754]: pgmap v535: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:23.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:23 smithi137 ceph-mon[90634]: pgmap v535: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:24.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-06-14T22:55:25.327 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:25.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:55:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:55:24] "GET /metrics HTTP/1.1" 200 34119 "" "Prometheus/2.43.0" 2024-06-14T22:55:25.807 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:25 smithi086 ceph-mon[97913]: pgmap v536: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:25.807 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:25.807 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:25.807 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:25 smithi086 ceph-mon[102754]: pgmap v536: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:25.807 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:25.808 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:25.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:25 smithi137 ceph-mon[90634]: pgmap v536: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:25.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:25.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:27.889 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:27 smithi086 ceph-mon[102754]: pgmap v537: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:27.889 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:27.889 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:27 smithi086 ceph-mon[97913]: pgmap v537: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:27.889 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:27.910 INFO:teuthology.orchestra.run.smithi086.stdout:mon.a: {} 2024-06-14T22:55:27.910 INFO:teuthology.orchestra.run.smithi086.stdout:mon.b: {} 2024-06-14T22:55:27.910 INFO:teuthology.orchestra.run.smithi086.stdout:mon.c: {} 2024-06-14T22:55:27.910 INFO:teuthology.orchestra.run.smithi086.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-14T22:55:27.910 INFO:teuthology.orchestra.run.smithi086.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-14T22:55:27.910 INFO:teuthology.orchestra.run.smithi086.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-14T22:55:27.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:27 smithi137 ceph-mon[90634]: pgmap v537: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:27.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:28.495 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-06-14T22:55:28.495 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-06-14T22:55:28.495 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T22:55:28.495 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T22:55:28.496 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-06-14T22:55:28.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-06-14T22:55:28.496 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-06-14T22:55:28.496 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T22:55:28.496 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T22:55:28.497 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-06-14T22:55:28.497 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T22:55:28.497 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T22:55:28.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-06-14T22:55:28.497 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-06-14T22:55:28.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:28.498 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-06-14T22:55:28.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:55:28.498 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-06-14T22:55:28.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-14T22:55:28.499 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-06-14T22:55:28.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T22:55:28.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:28.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:28.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:28.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:28.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:28.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:28.795 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2648275502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T22:55:28.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2648275502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T22:55:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:28.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:28.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:28.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:28.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:28.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:28.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2648275502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T22:55:29.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:29.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:29.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:29.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:29.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:29.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:29.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:29.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:29.309 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:29.377 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:29.378 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:29.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:29.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:29.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:29.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:29.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:29.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:29.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:29.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:29 smithi086 ceph-mon[97913]: pgmap v538: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:29.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:29 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:29.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:29 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:29.737 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:29 smithi086 ceph-mon[102754]: pgmap v538: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:29.738 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:29 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:29.738 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:29 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:29.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:29.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:29.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:29.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:29.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:29.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:29.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:29.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:29 smithi137 ceph-mon[90634]: pgmap v538: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:29.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:29 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:29.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:29 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:30.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:30.054 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:30.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:30.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:30.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:30.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:30.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:30.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:30.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:30.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:30.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:30.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:30.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:30.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:30.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:30.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:30.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:30.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:30.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:30.798 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:30.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:30.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:30 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:55:30.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:30 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:55:30.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:30 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:30.819 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:30 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:55:30.819 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:30 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:55:30.819 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:30 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:30.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:30.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:30.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:30.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:30 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:55:30.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:30 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:55:30.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:30 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:31.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:31.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:31.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:31.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:31.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:31.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:31.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:31.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:31.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:31.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:31.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:31.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:31.547 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:31.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:31.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:31.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:31.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:31.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:31.792 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:31.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:31.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:31.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:31.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:31.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:31 smithi086 ceph-mon[97913]: pgmap v539: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:31.976 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:31 smithi086 ceph-mon[102754]: pgmap v539: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:31.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:31 smithi137 ceph-mon[90634]: pgmap v539: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:32.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:32.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:32.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:32.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:32.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:32.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:32.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:32.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:32.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:32.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:32.419 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:32.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:32.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:32.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:32.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:32.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:32.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:32.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:32.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:32.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:32.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:32.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:32.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:33.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:33.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:33.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:33.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:33.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:33.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:33.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:33.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:33.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:33.351 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:55:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T22:55:33.351 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-06-14T22:55:33.352 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:55:33.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:33.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:33.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:33.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:33.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:33.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:33.541 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:33.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:33.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:33.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:33.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:33.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:33.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:33.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:33.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:33.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:33 smithi086 ceph-mon[97913]: pgmap v540: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:33.979 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:33 smithi086 ceph-mon[102754]: pgmap v540: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:33.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-14T22:55:33.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:33 smithi137 ceph-mon[90634]: pgmap v540: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:34.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:34.055 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:34.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:34.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:34.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:34.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:34.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:34.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:34.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:34.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:34.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:34.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:34.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:34.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:34.563 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:34.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:34.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:34.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:34.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:34.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:34.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:34.814 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:34.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:34.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:34.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:34.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:35.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:35.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:35.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:35.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:35.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:35.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:35.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:35.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:35.333 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:55:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:55:34] "GET /metrics HTTP/1.1" 200 34118 "" "Prometheus/2.43.0" 2024-06-14T22:55:35.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:35.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:35.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:35.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:35.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:35.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:35.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:35.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:35.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:35.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:35.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:35.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:35.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:35.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:35.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:35.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:35.972 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:35 smithi086 ceph-mon[97913]: pgmap v541: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:35.972 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:35 smithi086 ceph-mon[102754]: pgmap v541: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:35.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:35 smithi137 ceph-mon[90634]: pgmap v541: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:36.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:36.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:36.069 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:55:36.069 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-14T22:55:36.069 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:55:36.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:36.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:36.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:36.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:36.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:36.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:36.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:36.437 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:36.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:36.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:36.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:36.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:36.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:36.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:36.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:36.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:36.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:36.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:36.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:36.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:36.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:36.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:37.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:37.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:37.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:37.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:37.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:37.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:37.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:37.235 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:55:37.235 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:55:37.235 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:55:37.235 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:55:37.235 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:55:37.235 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:55:37.236 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:55:37.236 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:55:37.236 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:55:37.236 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:55:37.236 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:55:37.236 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:55:37.236 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:55:37.236 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:55:37.236 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:55:37.237 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:55:37.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:37.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:37.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:37.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:37.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:37.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:37.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:37.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:37.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:37.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:37.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:37.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:37.717 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_4' created 2024-06-14T22:55:37.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:37.817 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:37.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:37.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:37 smithi086 ceph-mon[97913]: pgmap v542: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:37.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2226311393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:55:37.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1244960464' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-14T22:55:37.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:37 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-14T22:55:37.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:37 smithi086 ceph-mon[102754]: pgmap v542: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:37.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2226311393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:55:37.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1244960464' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-14T22:55:37.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:37 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-14T22:55:37.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:37.940 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:37.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:37.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:37 smithi137 ceph-mon[90634]: pgmap v542: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:37.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2226311393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:55:37.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1244960464' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-14T22:55:37.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:37 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-14T22:55:38.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:38.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:38.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:38.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:38.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:38.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:38.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:38.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:38.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:38.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:38.445 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:38.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:38.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-06-14T22:55:38.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:38.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:38.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:38.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:38.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:38.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:38.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:38.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:38.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:38 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-06-14T22:55:38.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:38 smithi086 ceph-mon[97913]: osdmap e140: 8 total, 8 up, 8 in 2024-06-14T22:55:38.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:38 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-06-14T22:55:38.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:38 smithi086 ceph-mon[102754]: osdmap e140: 8 total, 8 up, 8 in 2024-06-14T22:55:38.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:38.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:38.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:38.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:38 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-06-14T22:55:38.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:38 smithi137 ceph-mon[90634]: osdmap e140: 8 total, 8 up, 8 in 2024-06-14T22:55:39.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:39.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:39.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:39.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:39.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:39.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:39.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:39.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:39.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:39.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:39.447 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:39.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:39.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:39.569 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:39.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:39.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:39.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:39.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:39.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:39.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:39.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:39.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:39.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:39.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:39.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:39.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:39 smithi086 ceph-mon[97913]: pgmap v544: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:39.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:39 smithi086 ceph-mon[97913]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:55:39.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:39 smithi086 ceph-mon[97913]: osdmap e141: 8 total, 8 up, 8 in 2024-06-14T22:55:39.965 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:39 smithi086 ceph-mon[102754]: pgmap v544: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:39.965 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:39 smithi086 ceph-mon[102754]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:55:39.965 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:39 smithi086 ceph-mon[102754]: osdmap e141: 8 total, 8 up, 8 in 2024-06-14T22:55:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:39 smithi137 ceph-mon[90634]: pgmap v544: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:39 smithi137 ceph-mon[90634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T22:55:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:39 smithi137 ceph-mon[90634]: osdmap e141: 8 total, 8 up, 8 in 2024-06-14T22:55:40.042 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:40.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:40.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:40.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:40.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:40.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:40.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:40.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:40.317 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:40.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:40.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:40.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:40.447 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:40.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:40.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:40.570 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:40.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:40.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:40.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:40.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:40.746 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T22:55:40.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:40.816 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:40.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:40.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:40.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:40.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:40.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:40.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:40 smithi137 ceph-mon[90634]: osdmap e142: 8 total, 8 up, 8 in 2024-06-14T22:55:40.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1944017787' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:55:41.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:41.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:41.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:41.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:40 smithi086 ceph-mon[102754]: osdmap e142: 8 total, 8 up, 8 in 2024-06-14T22:55:41.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1944017787' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:55:41.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:40 smithi086 ceph-mon[97913]: osdmap e142: 8 total, 8 up, 8 in 2024-06-14T22:55:41.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1944017787' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:55:41.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:41.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:41.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:41.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:41.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:41.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:41.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:41.434 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:41.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:41.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:41.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:41.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:41.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:41.679 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:41.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:41.691 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T22:55:41.692 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T22:55:41.692 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T22:55:41.692 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.301053 2024-06-14T22:55:41.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 7 0.3010532216391477 2024-06-14T22:55:41.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:55:41.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:41.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:41.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:41.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:41.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:41.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:41.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:41 smithi137 ceph-mon[90634]: pgmap v547: 89 pgs: 3 creating+activating, 3 creating+peering, 3 unknown, 80 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:41.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:41 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:55:41.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1944017787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:55:41.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:41 smithi137 ceph-mon[90634]: osdmap e143: 8 total, 8 up, 8 in 2024-06-14T22:55:42.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:42.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:42.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:42.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:41 smithi086 ceph-mon[97913]: pgmap v547: 89 pgs: 3 creating+activating, 3 creating+peering, 3 unknown, 80 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:42.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:41 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:55:42.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1944017787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:55:42.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:41 smithi086 ceph-mon[97913]: osdmap e143: 8 total, 8 up, 8 in 2024-06-14T22:55:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:41 smithi086 ceph-mon[102754]: pgmap v547: 89 pgs: 3 creating+activating, 3 creating+peering, 3 unknown, 80 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:41 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:55:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1944017787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:55:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:41 smithi086 ceph-mon[102754]: osdmap e143: 8 total, 8 up, 8 in 2024-06-14T22:55:42.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:42.173 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:42.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:42.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:42.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:42.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:42.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:42.420 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:42.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:42.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:42.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:42.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:42.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:42.665 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:42.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:42.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:42.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:42.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:42.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:42.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:42.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:42.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:43.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:43.020 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:43.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:43.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:43.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:43.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:43.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:43.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:43.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:43.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:43.354 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-06-14T22:55:43.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:43.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:43.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:43.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[97913]: pgmap v549: 89 pgs: 3 creating+activating, 3 creating+peering, 83 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/839086578' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[102754]: pgmap v549: 89 pgs: 3 creating+activating, 3 creating+peering, 83 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/839086578' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:55:43.450 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:43 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:55:43.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:43.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:43.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:43.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:43.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:43.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:43 smithi137 ceph-mon[90634]: pgmap v549: 89 pgs: 3 creating+activating, 3 creating+peering, 83 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/839086578' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:55:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:43 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T22:55:43.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:43.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:43.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:43.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:43.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:43.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:44.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:44.023 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:44.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:44.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:44.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:44.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:44.206 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:44.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:44.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:44.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:44.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:44.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:44.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:44.421 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:44 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:55:44.421 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:44 smithi086 ceph-mon[97913]: osdmap e144: 8 total, 8 up, 8 in 2024-06-14T22:55:44.421 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:44 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:55:44.421 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:44 smithi086 ceph-mon[102754]: osdmap e144: 8 total, 8 up, 8 in 2024-06-14T22:55:44.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:44.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:44.526 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:44.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:44.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:44.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:44.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:44.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:44 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T22:55:44.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:44 smithi137 ceph-mon[90634]: osdmap e144: 8 total, 8 up, 8 in 2024-06-14T22:55:44.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:44.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:44.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:44.882 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:44.882 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:55:44.883 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:55:44.905 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:44.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:44.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:45.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:45.029 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:45.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:45.084 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:55:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:55:44] "GET /metrics HTTP/1.1" 200 36477 "" "Prometheus/2.43.0" 2024-06-14T22:55:45.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:45.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:45.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:45.174 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-06-14T22:55:45.177 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T22:55:45.177 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T22:55:45.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T22:55:45.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T22:55:45.249 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_152062 2024-06-14T22:55:45.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:55:45.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T22:55:45.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:45.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:45.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:45.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:45.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:45.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:45.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:45.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:45.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:45.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:45.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:45.631 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:45 smithi086 ceph-mon[102754]: pgmap v551: 89 pgs: 89 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:45.631 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:45 smithi086 ceph-mon[97913]: pgmap v551: 89 pgs: 89 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:45.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:45.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:45.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:45.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:45 smithi137 ceph-mon[90634]: pgmap v551: 89 pgs: 89 active+clean; 582 KiB data, 3.1 GiB used, 712 GiB / 715 GiB avail 2024-06-14T22:55:45.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:45.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:45.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:45.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:45.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:45.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:46.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:46.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:46.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:46.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:46.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 4239 4223 263.919 263.938 0.00462477 0.00377912 2024-06-14T22:55:46.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:46.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:46.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:46.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:46.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:46.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:46.536 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:46.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:46.551 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:46 smithi086 ceph-mon[97913]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:55:46.551 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:46 smithi086 ceph-mon[102754]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:55:46.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-14T22:55:46.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:46.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:46.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:46.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:46 smithi137 ceph-mon[90634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T22:55:46.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:46.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:46.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:46.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:46.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:46.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:46.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:47.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:47.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:47.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:47.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:47.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:47.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:47.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 16 7936 7920 247.481 231.062 0.00353202 0.00403172 2024-06-14T22:55:47.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:47.357 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:47.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:47.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:47.492 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:47.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:47.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:47.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:47.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:47.701 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:47 smithi086 ceph-mon[97913]: pgmap v552: 89 pgs: 89 active+clean; 54 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.8 MiB/s wr, 124 op/s 2024-06-14T22:55:47.701 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:47 smithi086 ceph-mon[102754]: pgmap v552: 89 pgs: 89 active+clean; 54 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.8 MiB/s wr, 124 op/s 2024-06-14T22:55:47.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:47.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:47 smithi137 ceph-mon[90634]: pgmap v552: 89 pgs: 89 active+clean; 54 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.8 MiB/s wr, 124 op/s 2024-06-14T22:55:47.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:47.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:47.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:47.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:47.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:48.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:48.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:48.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:48.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:48.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:48.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:48.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 16 11606 11590 241.439 229.375 0.00383512 0.00413639 2024-06-14T22:55:48.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:48.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:48.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:48.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:48.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:48.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:48.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:48.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:48.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:48.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:48.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:48.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:48.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:48.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:48.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:48.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:48.984 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:48.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:49.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:49.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:49.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:49.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:49.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:49.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 16 14988 14972 233.918 211.375 0.00847662 0.00426967 2024-06-14T22:55:49.252 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:49.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:49.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:49.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:49.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:49.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:49.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:49.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:49.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:49.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:49.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:49.814 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:49.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:49.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:49.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:49.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:49.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:49 smithi137 ceph-mon[90634]: pgmap v553: 89 pgs: 89 active+clean; 54 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 MiB/s wr, 106 op/s 2024-06-14T22:55:50.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:50.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:49 smithi086 ceph-mon[102754]: pgmap v553: 89 pgs: 89 active+clean; 54 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 MiB/s wr, 106 op/s 2024-06-14T22:55:50.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:49 smithi086 ceph-mon[97913]: pgmap v553: 89 pgs: 89 active+clean; 54 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 MiB/s wr, 106 op/s 2024-06-14T22:55:50.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:50.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:50.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:50.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:50.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:50.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 16 18232 18216 227.681 202.75 0.00319795 0.0043869 2024-06-14T22:55:50.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:50.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:50.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:50.491 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:50.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:50.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:50.606 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:50.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:50.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:50.675 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:55:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:55:50.710 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:55:50.711 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:55:50.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:50.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:50 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3713078393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:55:50.738 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:50 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3713078393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:55:50.746 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:55:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:55:50.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:50.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:50.780 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:55:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:55:50.815 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:55:50.815 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:55:50.852 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:55:50.852 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:55:50.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:50.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:50.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:50.895 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:55:50.895 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:55:50.916 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:50 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3713078393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:55:50.927 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:55:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:55:50.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:51.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:51.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:51.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:51.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:51.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:51.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 20981 20965 218.367 171.812 0.00585328 0.00457486 2024-06-14T22:55:51.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:51.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:51.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:51.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:51.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:51.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:51.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:51.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:51.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:51.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:51.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:51.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:51.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:51.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:51.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:51.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:51.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:51 smithi137 ceph-mon[90634]: pgmap v554: 89 pgs: 89 active+clean; 744 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-14T22:55:52.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:51 smithi086 ceph-mon[97913]: pgmap v554: 89 pgs: 89 active+clean; 744 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-14T22:55:52.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:51 smithi086 ceph-mon[102754]: pgmap v554: 89 pgs: 89 active+clean; 744 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-14T22:55:52.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:52.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:52.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:52.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:52.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 16 23691 23675 211.366 169.375 0.00809356 0.0047258 2024-06-14T22:55:52.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:52.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:52.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:52.405 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:52.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:52.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:52.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:52.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:52.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:52.656 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.7 primary-affinity to 0.301053 (8197292) 2024-06-14T22:55:52.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:52.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:52.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:52.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:52.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:52.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:52.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:52.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:52.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:52.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3907814438' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3010532216391477}]: dispatch 2024-06-14T22:55:52.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:52 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3010532216391477}]: dispatch 2024-06-14T22:55:52.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:52 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:53.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:53.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:53.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3907814438' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3010532216391477}]: dispatch 2024-06-14T22:55:53.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:52 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3010532216391477}]: dispatch 2024-06-14T22:55:53.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:52 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:53.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3907814438' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3010532216391477}]: dispatch 2024-06-14T22:55:53.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:52 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3010532216391477}]: dispatch 2024-06-14T22:55:53.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:52 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:53.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:53.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:53.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:53.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 24796 24780 193.576 69.0625 0.00193359 0.00479274 2024-06-14T22:55:53.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:53.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:53.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:53.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:53.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:53.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:53.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:53.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:53.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:53.651 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T22:55:53.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:53.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:53.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:53.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:53.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:53.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:53.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:53.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:53.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:53 smithi137 ceph-mon[90634]: pgmap v555: 89 pgs: 89 active+clean; 941 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-14T22:55:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:53 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3010532216391477}]': finished 2024-06-14T22:55:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:53 smithi137 ceph-mon[90634]: osdmap e145: 8 total, 8 up, 8 in 2024-06-14T22:55:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/334645244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:55:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:53 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:55:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:53 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:55:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:53 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:55:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:53 smithi137 ceph-mon[90634]: osdmap e146: 8 total, 8 up, 8 in 2024-06-14T22:55:54.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:54.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[102754]: pgmap v555: 89 pgs: 89 active+clean; 941 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-14T22:55:54.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3010532216391477}]': finished 2024-06-14T22:55:54.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[102754]: osdmap e145: 8 total, 8 up, 8 in 2024-06-14T22:55:54.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/334645244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[102754]: osdmap e146: 8 total, 8 up, 8 in 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[97913]: pgmap v555: 89 pgs: 89 active+clean; 941 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3010532216391477}]': finished 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[97913]: osdmap e145: 8 total, 8 up, 8 in 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/334645244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:55:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:53 smithi086 ceph-mon[97913]: osdmap e146: 8 total, 8 up, 8 in 2024-06-14T22:55:54.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:54.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:54.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:54.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:54.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:54.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:54.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 26335 26319 182.754 96.1875 0.00409803 0.00546722 2024-06-14T22:55:54.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:54.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:54.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:54.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:54.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:54.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:54.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:54.635 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:54.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:54.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:54.772 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:54.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:54.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:54.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:54.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:54.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:54 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-06-14T22:55:55.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:55.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:55.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:55.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:55.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:54 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-06-14T22:55:55.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:54 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-06-14T22:55:55.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:55:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:55:54] "GET /metrics HTTP/1.1" 200 36477 "" "Prometheus/2.43.0" 2024-06-14T22:55:55.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:55.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:55.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:55.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 16 28859 28843 180.252 157.75 0.00560463 0.0055426 2024-06-14T22:55:55.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:55.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:55.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:55.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:55.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:55.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:55.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:55.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:55.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:55.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:55.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:55.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:55.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:55.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:55.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:55.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:55.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:55.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:55.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:55.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:55.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:55 smithi137 ceph-mon[90634]: pgmap v558: 89 pgs: 4 peering, 85 active+clean; 1.5 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 186 MiB/s wr, 2.98k op/s 2024-06-14T22:55:55.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:55 smithi086 ceph-mon[97913]: pgmap v558: 89 pgs: 4 peering, 85 active+clean; 1.5 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 186 MiB/s wr, 2.98k op/s 2024-06-14T22:55:55.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:55 smithi086 ceph-mon[102754]: pgmap v558: 89 pgs: 4 peering, 85 active+clean; 1.5 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 186 MiB/s wr, 2.98k op/s 2024-06-14T22:55:56.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:56.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:56.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:56.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:55:56.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:56.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 31286 31270 177.655 151.688 0.00573528 0.00562069 2024-06-14T22:55:56.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:56.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:56.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:56.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:56.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:56.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:56.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:56.562 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:56.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:56.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:56.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:56.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:56.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:56.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:56.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:56.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:56.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:57.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:57.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:57.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:57.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:57.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:57.240 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:57.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:57.250 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 15 33663 33648 175.234 148.625 0.00469311 0.00570141 2024-06-14T22:55:57.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:57.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:57.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:57.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:57.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:57.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:57.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:57.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:57.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:57.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:57 smithi086 ceph-mon[97913]: pgmap v559: 89 pgs: 4 peering, 85 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 201 MiB/s wr, 3.22k op/s 2024-06-14T22:55:57.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:57.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:57.738 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:57 smithi086 ceph-mon[102754]: pgmap v559: 89 pgs: 4 peering, 85 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 201 MiB/s wr, 3.22k op/s 2024-06-14T22:55:57.738 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:57.738 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:57.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:55:57.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:57.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:55:57.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:57.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:57.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:57 smithi137 ceph-mon[90634]: pgmap v559: 89 pgs: 4 peering, 85 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 201 MiB/s wr, 3.22k op/s 2024-06-14T22:55:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:55:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:55:58.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T22:55:58.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:58.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:58.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:58.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:58.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:58.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:58.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 35694 35678 171.504 126.875 0.00883069 0.0058246 2024-06-14T22:55:58.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:58.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:58.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:58.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:58.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:58.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:58.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:55:58.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:58.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:55:58.712 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:55:58.712 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T22:55:58.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34311.0:117617 7.9 7:9efb4cf5:::benchmark_data_smithi086_113105_object4102:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e98)", 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:46:18.050418+0000", 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 580.65661694899995, 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0082925009999999, 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34311", 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/3123058808", 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117617 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.713 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.050418+0000", 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.050418+0000", 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.050417+0000", 2024-06-14T22:55:58.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.050424+0000", 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2810000000000003e-06 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.050439+0000", 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.502e-05 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.050447+0000", 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9980000000000003e-06 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.050545+0000", 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8238000000000004e-05 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.050552+0000", 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8369999999999999e-06 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.065726+0000", 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015173765000000001 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.065736+0000", 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9720000000000004e-06 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.058657+0000", 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99292074100000005 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.058711+0000", 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3955000000000001e-05 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34311.0:117647 7.9 7:992c1295:::benchmark_data_smithi086_113105_object4132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:46:18.077205+0000", 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 580.62982988199997, 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99140176199999996, 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.718 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34311", 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/3123058808", 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117647 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.077205+0000", 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.077205+0000", 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.077205+0000", 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.077207+0000", 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9420000000000002e-06 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.077212+0000", 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8350000000000003e-06 2024-06-14T22:55:58.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.077216+0000", 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0509999999999998e-06 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.077273+0000", 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7543000000000001e-05 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:55:58.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.077284+0000", 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0334e-05 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.058991+0000", 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98170718800000001 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.059084+0000", 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3083000000000003e-05 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,4", 2024-06-14T22:55:58.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.059205+0000", 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012126 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068521+0000", 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0093153999999999997 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068528+0000", 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5920000000000003e-06 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068557+0000", 2024-06-14T22:55:58.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8668000000000001e-05 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068584+0000", 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6917e-05 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068607+0000", 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3249999999999999e-05 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34311.0:117680 7.9 7:9e6b5b6c:::benchmark_data_smithi086_113105_object4165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:46:18.114613+0000", 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 580.59242210399998, 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95414605100000005, 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34311", 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/3123058808", 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117680 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.114613+0000", 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.114613+0000", 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.114611+0000", 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.114619+0000", 2024-06-14T22:55:58.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2479999999999999e-06 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.114637+0000", 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8731000000000001e-05 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.114645+0000", 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8390000000000007e-06 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.114704+0000", 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9242e-05 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.114715+0000", 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0648e-05 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.061556+0000", 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.94684111699999995 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.061666+0000", 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109578 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,4", 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.061776+0000", 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011013700000000001 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068534+0000", 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067578559999999996 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068657+0000", 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123227 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068699+0000", 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1591999999999998e-05 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068728+0000", 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9547e-05 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068759+0000", 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0910000000000001e-05 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34311.0:117698 7.9 7:9fe77050:::benchmark_data_smithi086_113105_object4183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:46:18.133963+0000", 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 580.573072432, 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.941695642, 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.731 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34311", 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/3123058808", 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117698 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.133963+0000", 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.133963+0000", 2024-06-14T22:55:58.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.133961+0000", 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.133969+0000", 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6729999999999992e-06 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.133986+0000", 2024-06-14T22:55:58.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6880000000000001e-05 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.133993+0000", 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3749999999999997e-06 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.136733+0000", 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027400250000000001 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:18.136747+0000", 2024-06-14T22:55:58.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3655999999999999e-05 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.064119+0000", 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92737206400000005 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.064195+0000", 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6532999999999994e-05 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,4", 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.064266+0000", 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0437000000000003e-05 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068666+0000", 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043999360000000001 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.068801+0000", 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013544499999999999 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.075583+0000", 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067818890000000001 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.075623+0000", 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9789999999999997e-05 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:46:19.075658+0000", 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5503000000000002e-05 2024-06-14T22:55:58.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:20569 8.1 8:8a52555f:::benchmark_data_smithi086_122122_object20568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e113)", 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:08.128996+0000", 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 410.57803914900001, 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99832512699999998, 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.738 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:55:58.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:58.739 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:55:58.739 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 20569 2024-06-14T22:55:58.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.739 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.128996+0000", 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.128996+0000", 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.128994+0000", 2024-06-14T22:55:58.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129024+0000", 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9388000000000001e-05 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129046+0000", 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2385e-05 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129058+0000", 2024-06-14T22:55:58.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1654000000000001e-05 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129111+0000", 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3245000000000003e-05 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129122+0000", 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0903e-05 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:09.124962+0000", 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99583976600000002 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:09.125103+0000", 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014192999999999999 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:09.127246+0000", 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021425060000000002 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:09.127321+0000", 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5135999999999993e-05 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55937 8.1 8:8b1d3009:::benchmark_data_smithi086_122122_object55936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e117)", 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:25.469855+0000", 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 393.23718064100001, 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1369695980000001, 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.744 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55937 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469855+0000", 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469855+0000", 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469853+0000", 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469880+0000", 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6359e-05 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469890+0000", 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0591000000000001e-05 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469900+0000", 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3640000000000005e-06 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.470053+0000", 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015309699999999999 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.470062+0000", 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1870000000000003e-06 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.476164+0000", 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006101843 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.476176+0000", 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1759e-05 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.587460+0000", 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1112849090000001 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.587708+0000", 2024-06-14T22:55:58.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000247545 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.587896+0000", 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000187896 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.594356+0000", 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064597040000000001 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.594397+0000", 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0932e-05 2024-06-14T22:55:58.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.604097+0000", 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.009700195 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.604523+0000", 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00042586799999999998 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.606824+0000", 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023015380000000001 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55950 8.1 8:8ad1e56a:::benchmark_data_smithi086_122122_object55949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e117)", 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:25.469906+0000", 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 393.23712955000002, 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.138639081, 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.751 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55950 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469906+0000", 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469906+0000", 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469905+0000", 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469938+0000", 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3105000000000002e-05 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469945+0000", 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.289e-06 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469951+0000", 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8760000000000002e-06 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.470110+0000", 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015907599999999999 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting_for_map not empty", 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.470117+0000", 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6850000000000004e-06 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.476497+0000", 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063794699999999999 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.476506+0000", 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2760000000000006e-06 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.591937+0000", 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1154308989999999 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.592083+0000", 2024-06-14T22:55:58.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014589200000000001 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.592211+0000", 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012861799999999999 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.594817+0000", 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026060699999999998 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.594940+0000", 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012278300000000001 2024-06-14T22:55:58.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.606883+0000", 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011943339000000001 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.607027+0000", 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014385299999999999 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.608545+0000", 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015173859999999999 2024-06-14T22:55:58.757 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:28256 9.b 9:d57e5729:::benchmark_data_smithi086_152062_object28255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.986330+0000", 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.7207047790000001, 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042558327999999999, 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 28256 2024-06-14T22:55:58.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.986330+0000", 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.986330+0000", 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.986330+0000", 2024-06-14T22:55:58.759 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.986375+0000", 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5859000000000003e-05 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.986387+0000", 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2037e-05 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.986394+0000", 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8249999999999999e-06 2024-06-14T22:55:58.760 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.986450+0000", 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5495999999999999e-05 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.986558+0000", 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010837599999999999 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.986676+0000", 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118 2024-06-14T22:55:58.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991749+0000", 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005072572 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991811+0000", 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1883000000000004e-05 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028809+0000", 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036998112999999999 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028851+0000", 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2794999999999999e-05 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028889+0000", 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7225999999999999e-05 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:28257 9.9 9:9acf8b6d:::benchmark_data_smithi086_152062_object28256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-06-14T22:55:58.763 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.988252+0000", 2024-06-14T22:55:58.764 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.718783647, 2024-06-14T22:55:58.764 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.040532849000000003, 2024-06-14T22:55:58.764 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.764 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.764 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.764 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:55:58.764 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:58.764 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 28257 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988252+0000", 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988252+0000", 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988250+0000", 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988300+0000", 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9914999999999999e-05 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988312+0000", 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2054e-05 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988320+0000", 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8180000000000004e-06 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988354+0000", 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3988000000000001e-05 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.767 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988557+0000", 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020305400000000001 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988684+0000", 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012754199999999999 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.992216+0000", 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0035320479999999999 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.768 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.992330+0000", 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114059 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028693+0000", 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036362152000000002 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028747+0000", 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4255999999999997e-05 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.769 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028784+0000", 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7512999999999998e-05 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:28259 9.b 9:d8755eac:::benchmark_data_smithi086_152062_object28258:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.988628+0000", 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.7184067560000003, 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.040436870999999999, 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.770 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 28259 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988628+0000", 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988628+0000", 2024-06-14T22:55:58.771 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988628+0000", 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988677+0000", 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8837e-05 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988687+0000", 2024-06-14T22:55:58.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0453e-05 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988693+0000", 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3069999999999999e-06 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989839+0000", 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001145711 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.990071+0000", 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00023188399999999999 2024-06-14T22:55:58.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.990304+0000", 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000233224 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028949+0000", 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038644666000000001 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028957+0000", 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7570000000000004e-06 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028991+0000", 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4919999999999998e-05 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029032+0000", 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0141000000000001e-05 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029065+0000", 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3680000000000003e-05 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:28261 9.5 9:aaa4b027:::benchmark_data_smithi086_152062_object28260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.988969+0000", 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.7180659710000001, 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.040077208000000003, 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 28261 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988969+0000", 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988969+0000", 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.988968+0000", 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:55:58.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989029+0000", 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1163999999999999e-05 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989040+0000", 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0485e-05 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989162+0000", 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012238599999999999 2024-06-14T22:55:58.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989215+0000", 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2305000000000002e-05 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989330+0000", 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114925 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989458+0000", 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000128419 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991140+0000", 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001682027 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991548+0000", 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00040814399999999999 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028953+0000", 2024-06-14T22:55:58.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037404528999999999 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029013+0000", 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9824000000000002e-05 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029046+0000", 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3884000000000003e-05 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:28263 9.9 9:9cdf384e:::benchmark_data_smithi086_152062_object28262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.989180+0000", 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.7178551190000002, 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039799833, 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 28263 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989180+0000", 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989180+0000", 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989179+0000", 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989220+0000", 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0179000000000003e-05 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989228+0000", 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9859999999999994e-06 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989233+0000", 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9749999999999998e-06 2024-06-14T22:55:58.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989830+0000", 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00059744500000000005 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989909+0000", 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8597000000000007e-05 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.990044+0000", 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013493000000000001 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.992337+0000", 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022932730000000002 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028834+0000", 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036497622 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028896+0000", 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1744999999999999e-05 2024-06-14T22:55:58.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028946+0000", 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9923999999999998e-05 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028980+0000", 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3745000000000003e-05 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.787 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:28264 9.9 9:9687d9c1:::benchmark_data_smithi086_152062_object28263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.989422+0000", 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.717613182, 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039715426999999998, 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 28264 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989422+0000", 2024-06-14T22:55:58.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989422+0000", 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989421+0000", 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989466+0000", 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4209999999999999e-05 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989474+0000", 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9869999999999999e-06 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.989479+0000", 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0320000000000003e-06 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991006+0000", 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015277470000000001 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991124+0000", 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000117743 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991255+0000", 2024-06-14T22:55:58.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000131235 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028865+0000", 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037609800999999998 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029026+0000", 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016039200000000001 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029067+0000", 2024-06-14T22:55:58.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1350999999999998e-05 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029107+0000", 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9745e-05 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029137+0000", 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0684e-05 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:28257 9.9 e146/140 ondisk, result = 0)", 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.990144+0000", 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.7168913969999999, 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038651754000000003, 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.990144+0000", 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.990144+0000", 2024-06-14T22:55:58.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.990143+0000", 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.990149+0000", 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2469999999999997e-06 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.990159+0000", 2024-06-14T22:55:58.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4080000000000004e-06 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.990164+0000", 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5709999999999997e-06 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028673+0000", 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038508449 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028686+0000", 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3363999999999999e-05 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028796+0000", 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109261 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:28270 9.b 9:d2e317a2:::benchmark_data_smithi086_152062_object28269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.991236+0000", 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.715799337, 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038180196999999999, 2024-06-14T22:55:58.797 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 28270 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991236+0000", 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991236+0000", 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991234+0000", 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991281+0000", 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6474000000000003e-05 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991292+0000", 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.164e-05 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991300+0000", 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2130000000000002e-06 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991835+0000", 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00053564800000000001 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991940+0000", 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000105058 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.992031+0000", 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0844e-05 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028961+0000", 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036930136000000002 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029289+0000", 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00032716199999999998 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029342+0000", 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.329e-05 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029383+0000", 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0921999999999999e-05 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029416+0000", 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3259000000000001e-05 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:28263 9.9 e146/140 ondisk, result = 0)", 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.991550+0000", 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.7154854560000001, 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037287012000000001, 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.803 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:55:58.804 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991550+0000", 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991550+0000", 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991549+0000", 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991554+0000", 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4900000000000002e-06 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991563+0000", 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7820000000000005e-06 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.991569+0000", 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5310000000000001e-06 2024-06-14T22:55:58.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028822+0000", 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037253367000000003 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028828+0000", 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1920000000000002e-06 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028837+0000", 2024-06-14T22:55:58.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2670000000000006e-06 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:28264 9.9 e146/140 ondisk, result = 0)", 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.992673+0000", 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.714362473, 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036194670999999998, 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.992673+0000", 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.992673+0000", 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.992672+0000", 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.992676+0000", 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8020000000000002e-06 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028655+0000", 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.035978867999999997 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028664+0000", 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6260000000000007e-06 2024-06-14T22:55:58.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028855+0000", 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019020500000000001 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028860+0000", 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4149999999999998e-06 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028867+0000", 2024-06-14T22:55:58.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2459999999999997e-06 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:28259 9.b e146/140 ondisk, result = 0)", 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.993448+0000", 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.71358685, 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.035627332999999997, 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.993448+0000", 2024-06-14T22:55:58.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.993448+0000", 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.993448+0000", 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.993451+0000", 2024-06-14T22:55:58.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2090000000000001e-06 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028812+0000", 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.035361409000000003 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028817+0000", 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1189999999999996e-06 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028982+0000", 2024-06-14T22:55:58.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016406799999999999 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028987+0000", 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2669999999999997e-06 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.029076+0000", 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8868000000000004e-05 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.815 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:28263 9.9 e146/140 ondisk, result = 0)", 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:54.993919+0000", 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.7131166289999999, 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.035069623000000001, 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.993919+0000", 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.993919+0000", 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.993918+0000", 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:54.993921+0000", 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.247e-06 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028696+0000", 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034774855 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028702+0000", 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6030000000000004e-06 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028886+0000", 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018419500000000001 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028892+0000", 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6999999999999996e-06 2024-06-14T22:55:58.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:55.028988+0000", 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6502999999999994e-05 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:31233 9.9 9:9c77e381:::benchmark_data_smithi086_152062_object31232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:56.218688+0000", 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 2.48834731, 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026793214999999999, 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:55:58.819 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 31233 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.218688+0000", 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:55:58.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.218688+0000", 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.218686+0000", 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.218713+0000", 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6687e-05 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.239852+0000", 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021138606000000001 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.239871+0000", 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9076000000000002e-05 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.240132+0000", 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026129499999999998 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.240274+0000", 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014255100000000001 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.240425+0000", 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015082299999999999 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.244159+0000", 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037336719999999999 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.245326+0000", 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001166968 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.245403+0000", 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6916999999999996e-05 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.245444+0000", 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1289999999999999e-05 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:55:58.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:55:58.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:56.245481+0000", 2024-06-14T22:55:58.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6942e-05 2024-06-14T22:55:58.825 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.825 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.825 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.825 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:55:58.825 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:55:58.825 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:55:58.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:58.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:58.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:58.972 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:58.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:59.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:59.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:59.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:59.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:55:59.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:59.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:55:59.251 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 37460 37444 167.137 110.375 0.00738541 0.00597807 2024-06-14T22:55:59.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:55:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:59.378 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:59.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:55:59.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:55:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:59.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:59.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:55:59.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:55:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:59.646 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:55:59.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:55:59.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:55:59.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:59.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:59.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:55:59.880 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:55:59 smithi086 ceph-mon[97913]: pgmap v560: 89 pgs: 4 peering, 85 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-06-14T22:55:59.880 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:55:59 smithi086 ceph-mon[102754]: pgmap v560: 89 pgs: 4 peering, 85 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-06-14T22:55:59.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:55:59.911 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:55:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:55:59.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:55:59.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-14T22:55:59.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:55:59 smithi137 ceph-mon[90634]: pgmap v560: 89 pgs: 4 peering, 85 active+clean; 1.6 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-06-14T22:56:00.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:00.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:00.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:00.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:00.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:00.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:00.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:00.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 39478 39462 164.397 126.125 0.00728116 0.0060775 2024-06-14T22:56:00.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:00.343 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:00.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:00.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:00.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:00.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:00.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:00.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:00.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:00.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:00.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:00.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:00.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:00.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:00.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:00.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:00.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:01.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:01.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:01.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:00 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-06-14T22:56:01.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:00 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-06-14T22:56:01.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:01.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:01.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:01.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:00 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-06-14T22:56:01.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:01.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 41525 41509 162.118 127.938 0.00433041 0.00616302 2024-06-14T22:56:01.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:01.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:01.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:01.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:01.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:01.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:01.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:01.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:01.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:01.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:01.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:01.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:01.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:01.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:01.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:01.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:01.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:02.053 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:01 smithi086 ceph-mon[97913]: pgmap v561: 89 pgs: 89 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-06-14T22:56:02.054 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:01 smithi086 ceph-mon[102754]: pgmap v561: 89 pgs: 89 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-06-14T22:56:02.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:02.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:02.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:02.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:02.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:02.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:02.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:01 smithi137 ceph-mon[90634]: pgmap v561: 89 pgs: 89 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-06-14T22:56:02.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 16 43617 43601 160.272 130.75 0.00600856 0.00623484 2024-06-14T22:56:02.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:02.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:02.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:02.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:02.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:02.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:02.644 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:02.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:02.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:02.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:02.783 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:02.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:02.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:02.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:02.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:02.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:03.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:03.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:03.101 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T22:56:03.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 0 1 2024-06-14T22:56:03.146 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:02 smithi086 ceph-mon[102754]: pgmap v562: 89 pgs: 89 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-06-14T22:56:03.147 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:02 smithi086 ceph-mon[97913]: pgmap v562: 89 pgs: 89 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-06-14T22:56:03.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:03.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:03.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:02 smithi137 ceph-mon[90634]: pgmap v562: 89 pgs: 89 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-06-14T22:56:03.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 45419 45403 157.625 112.625 0.00926091 0.0063388 2024-06-14T22:56:03.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:03.343 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:03.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:03.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:03.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:03.475 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:03.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:03.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:03.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:03.613 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:03.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:03.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:03.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:03.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:03.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:03.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:03.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:04.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:04.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:04.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:04.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:04.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:04.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:04.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:04.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 47343 47327 155.657 120.25 0.00804632 0.00641966 2024-06-14T22:56:04.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:04.325 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:04.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:04.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:04.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:04.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:04.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:04.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:04.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:04.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:04.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:04.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:04.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:04.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:04.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:04.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:05.019 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:05.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:05.084 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:56:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:56:04] "GET /metrics HTTP/1.1" 200 36534 "" "Prometheus/2.43.0" 2024-06-14T22:56:05.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:05.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:05.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:05.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:56:05.253919+0000 min lat: 0.00125959 max lat: 1.02178 avg lat: 0.00651108 2024-06-14T22:56:05.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:56:05.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 49135 49119 153.474 112 0.00906133 0.00651108 2024-06-14T22:56:05.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:05.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:05.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:05.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:05.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:05.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:05.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:05.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:05.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:05.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:05.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:05.703 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:05 smithi086 ceph-mon[102754]: pgmap v563: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-06-14T22:56:05.704 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:05 smithi086 ceph-mon[97913]: pgmap v563: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-06-14T22:56:05.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:05.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:05.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:05.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:05.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:05.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:05.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:05.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:05 smithi137 ceph-mon[90634]: pgmap v563: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-06-14T22:56:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:05.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:06.083 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:06.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:06.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:06.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:06.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:06.252 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 16 50895 50879 151.403 110 0.014941 0.00659949 2024-06-14T22:56:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:06.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:06.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:06.400 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:06.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:06.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:06.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:06.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:06.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:06.663 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T22:56:06.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:06.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:06.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:06.801 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:06.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:06.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3369764391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:56:06.813 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3369764391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:56:06.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:06.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:06.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:06.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3369764391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:56:07.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:07.074 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:07.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:07.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:07.210 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:07.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:07.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 52708 52692 149.671 113.312 0.00756523 0.00667692 2024-06-14T22:56:07.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:07.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:07.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:07.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:07.481 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:07.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:07.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:07.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:07.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:07.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:07.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:07.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[97913]: pgmap v564: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-06-14T22:56:07.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:56:07.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3369764391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:56:07.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[97913]: osdmap e147: 8 total, 8 up, 8 in 2024-06-14T22:56:07.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:07.727 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[102754]: pgmap v564: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-06-14T22:56:07.727 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:56:07.727 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3369764391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:56:07.727 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[102754]: osdmap e147: 8 total, 8 up, 8 in 2024-06-14T22:56:07.727 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:07 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:07.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:07.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:07.783 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:56:07.783 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:56:07.825 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:56:07.825 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:56:07.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:07.857 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:56:07.858 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:56:07.879 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:07.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:07.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:07.896 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:56:07.896 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:56:07.931 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:56:07.931 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:56:07.962 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:56:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:56:07.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:07 smithi137 ceph-mon[90634]: pgmap v564: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-06-14T22:56:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:07 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:56:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3369764391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:56:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:07 smithi137 ceph-mon[90634]: osdmap e147: 8 total, 8 up, 8 in 2024-06-14T22:56:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:07 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:08.000 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:56:08.000 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:56:08.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:08.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:08.037 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:56:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:56:08.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:08.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:08.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:08.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:08.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 54457 54441 147.916 109.312 0.00566329 0.00675598 2024-06-14T22:56:08.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:08.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:08.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:08.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:08.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:08.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:08.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:08.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:08.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:08.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:08.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:08.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:08.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:08.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:08.909 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/754223457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:56:08.910 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/754223457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:56:08.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:08.983 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:08.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:08.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/754223457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:56:09.065 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:56:09.065 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:56:09.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:09.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:09.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:09.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:09.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 56203 56187 146.299 109.125 0.0119604 0.00683126 2024-06-14T22:56:09.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:09.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:09.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:09.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:09.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:09.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:09.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T22:56:09.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:09.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:09.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:09.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:09.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:09.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:09.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:09.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:09.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:09 smithi086 ceph-mon[97913]: pgmap v566: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-06-14T22:56:09.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:09 smithi086 ceph-mon[102754]: pgmap v566: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-06-14T22:56:09.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:09.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:09.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:09.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:09.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:09.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:09.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:09 smithi137 ceph-mon[90634]: pgmap v566: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-06-14T22:56:10.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:10.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:10.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:10.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:10.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:10.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:10.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 57787 57771 144.407 99 0.016157 0.00692024 2024-06-14T22:56:10.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:10.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:10.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:10.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:10.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:10.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:10.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:10.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:10.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:10.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:10.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:10.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:10.914 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:10.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:11.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:11.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:11.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:11.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:11.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:11.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 59434 59418 142.812 102.938 0.012816 0.00699728 2024-06-14T22:56:11.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:11.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:11.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:11.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:11.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:11.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:11.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:11.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:11.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:11.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:11.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:11.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:11.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:11.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:11.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:11.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:11.913 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:11 smithi086 ceph-mon[102754]: pgmap v567: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-06-14T22:56:11.913 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:11 smithi086 ceph-mon[97913]: pgmap v567: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-06-14T22:56:11.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:11.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:11 smithi137 ceph-mon[90634]: pgmap v567: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-06-14T22:56:11.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:11.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:12.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:12.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:12.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:12.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:12.253 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 60929 60913 140.983 93.4375 0.0110791 0.00708744 2024-06-14T22:56:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:12.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:12.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:12.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:12.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:12.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:12.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:12.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:12.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:12.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:12.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:12.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:12.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:12.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:12.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:12.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:12.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:12.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:13.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:13.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:13.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:13.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:13.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:13.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:13.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:13.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-14T22:56:13.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 62497 62481 139.446 98 0.00594502 0.00716645 2024-06-14T22:56:13.286 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:13.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.0 to 1 (10000) 2024-06-14T22:56:13.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:13.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:13.494 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:13.494 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:13.494 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[97913]: pgmap v568: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-06-14T22:56:13.494 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3101661188' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T22:56:13.494 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T22:56:13.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:13.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:13.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[102754]: pgmap v568: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-06-14T22:56:13.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3101661188' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T22:56:13.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:13 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T22:56:13.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:13.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:13.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:13.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:13.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:13.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:13.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:13 smithi137 ceph-mon[90634]: pgmap v568: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-06-14T22:56:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3101661188' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T22:56:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:13 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T22:56:13.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:13.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:13.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:13.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:13.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:13.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:14.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:14.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:56:14.101 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:14.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:14.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:14.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 64015 63999 137.909 94.875 0.00666791 0.00724626 2024-06-14T22:56:14.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:14.255 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:14.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:14.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 1 1 2024-06-14T22:56:14.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:14.407 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:14 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-14T22:56:14.407 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:14 smithi086 ceph-mon[102754]: osdmap e148: 8 total, 8 up, 8 in 2024-06-14T22:56:14.408 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:14 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-14T22:56:14.408 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:14 smithi086 ceph-mon[97913]: osdmap e148: 8 total, 8 up, 8 in 2024-06-14T22:56:14.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:14.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:14.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:14.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:14.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:14.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:14.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:14.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:14.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:14 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-14T22:56:14.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:14 smithi137 ceph-mon[90634]: osdmap e148: 8 total, 8 up, 8 in 2024-06-14T22:56:14.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:14.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:14.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:14.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:14.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:14.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:15.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:15.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:15.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:15.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:15.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:15.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 65499 65483 136.404 92.75 0.0126917 0.00732636 2024-06-14T22:56:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:15.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:15.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:15.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:56:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:56:14] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.43.0" 2024-06-14T22:56:15.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:15.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:15.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:15.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:15.462 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:15.466 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:15.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:15.569 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:15.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:15.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:15.597 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:15 smithi086 ceph-mon[97913]: pgmap v570: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-06-14T22:56:15.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:15 smithi086 ceph-mon[102754]: pgmap v570: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-06-14T22:56:15.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:15.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:15.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:15.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:15 smithi137 ceph-mon[90634]: pgmap v570: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-06-14T22:56:15.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:15.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:15.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:15.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:15.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:15.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:16.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:16.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:16.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:16.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:16.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 66959 66943 134.947 91.25 0.00824962 0.0074055 2024-06-14T22:56:16.262 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:16.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:16.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:16.400 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:16.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:16.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:16.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:16.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:16.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:16.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:16.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:16.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:16.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:16.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:16.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:16.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:16.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:16.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:17.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:17.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:17.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:17.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:17.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:17.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 68467 68451 133.675 94.25 0.00838994 0.00747604 2024-06-14T22:56:17.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:17.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:17.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:17.422 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:17.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:17.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:17.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:17.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:17.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:17.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:17 smithi086 ceph-mon[97913]: pgmap v571: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-14T22:56:17.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1266328530' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:56:17.654 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:17 smithi086 ceph-mon[102754]: pgmap v571: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-14T22:56:17.654 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1266328530' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:56:17.654 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T22:56:17.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:17.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:17.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:17.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:17.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:17.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:17.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:17.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:17 smithi137 ceph-mon[90634]: pgmap v571: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-14T22:56:17.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1266328530' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:56:18.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:18.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:18.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:18.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:18.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:18.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:18.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 16 69859 69843 132.26 87 0.0136736 0.00755611 2024-06-14T22:56:18.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:18.321 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:18.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:18.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:18.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:18.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:18.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:18.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:18.730 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:18.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:18.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:18.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:18.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:18.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:18.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:18 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:56:18.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:18 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:56:18.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1266328530' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:56:18.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:18 smithi137 ceph-mon[90634]: osdmap e149: 8 total, 8 up, 8 in 2024-06-14T22:56:19.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:19.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:19.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:19.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:18 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:56:19.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:18 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:56:19.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1266328530' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:56:19.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:18 smithi086 ceph-mon[97913]: osdmap e149: 8 total, 8 up, 8 in 2024-06-14T22:56:19.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:18 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:56:19.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:18 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:56:19.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1266328530' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:56:19.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:18 smithi086 ceph-mon[102754]: osdmap e149: 8 total, 8 up, 8 in 2024-06-14T22:56:19.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:19.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:19.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:19.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:19.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 71012 70996 130.49 72.0625 0.0137269 0.00765796 2024-06-14T22:56:19.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:19.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:19.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:19.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:19.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:19.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:19.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:19.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:19.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:19.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:19.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:19.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:19.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:19.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:19 smithi086 ceph-mon[97913]: pgmap v573: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-14T22:56:19.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:19 smithi086 ceph-mon[102754]: pgmap v573: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-14T22:56:19.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:19.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:19.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:19.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:19 smithi137 ceph-mon[90634]: pgmap v573: 89 pgs: 89 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-14T22:56:20.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:20.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:20.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:20.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:20.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:20.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:20.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:20.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 72356 72340 129.161 84 0.00646504 0.00773718 2024-06-14T22:56:20.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:20.378 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:20.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:20.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:20.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:20.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:20.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:20.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:20.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:20.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:20.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:20.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:20.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:20.910 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:20.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:21.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:21.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:21.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:21.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:21.185 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:21.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:21.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 73761 73745 128.013 87.8125 0.0151594 0.0078071 2024-06-14T22:56:21.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:21.320 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:21.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:21.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:21.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:21.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:21.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:21.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:21.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:21.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:21.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:21.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:21.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:21.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:21.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:21.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:21.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:21.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:21 smithi137 ceph-mon[90634]: pgmap v574: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-06-14T22:56:22.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:22.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:21 smithi086 ceph-mon[97913]: pgmap v574: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-06-14T22:56:22.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:21 smithi086 ceph-mon[102754]: pgmap v574: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-06-14T22:56:22.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:22.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:22.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:22.246 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:22.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:22.254 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 75050 75034 126.73 80.5625 0.00682943 0.00788584 2024-06-14T22:56:22.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:22.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:22.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:22.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:22.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:22.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:22.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:22.654 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:22.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:22.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:22.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:22.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:22.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:22.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:22.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:23.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:23.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:23.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:23.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:23.202 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:23.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:23.255 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 76547 76531 125.857 93.5625 0.00772526 0.00794026 2024-06-14T22:56:23.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:23.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:23.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:23.457 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:23.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:23.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:23.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:23.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:23.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:23.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:23.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:23.758 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:23.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:23.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:23 smithi086 ceph-mon[97913]: pgmap v575: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-06-14T22:56:23.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:23 smithi086 ceph-mon[102754]: pgmap v575: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-06-14T22:56:23.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:23.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:23.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:23.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:23.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:23 smithi137 ceph-mon[90634]: pgmap v575: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-06-14T22:56:24.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:24.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:24.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:24.158 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:24.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:24.255 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 16 77924 77908 124.836 86.0625 0.0149682 0.00800549 2024-06-14T22:56:24.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:24.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:24.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:24.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:24.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:24.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:24.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:24.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:24.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:24.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:24.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:24.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:24.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:24.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:24.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:24.847 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:56:24.847 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:56:24.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:24.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:24.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:25.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:25.084 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:56:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:56:24] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.43.0" 2024-06-14T22:56:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:25.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:25.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:25.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:25.255 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:56:25.256501+0000 min lat: 0.00125959 max lat: 1.02178 avg lat: 0.00806294 2024-06-14T22:56:25.255 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:56:25.255 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 79347 79331 123.937 88.9375 0.00965438 0.00806294 2024-06-14T22:56:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:25.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:25.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:25.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:25.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:25.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:25.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:25.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:25.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:25.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:25.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:25.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:25.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:25.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:25.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:25.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:25.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:25.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:25 smithi086 ceph-mon[97913]: pgmap v576: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-14T22:56:25.940 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:25 smithi086 ceph-mon[102754]: pgmap v576: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-14T22:56:25.960 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:25.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:25.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:25 smithi137 ceph-mon[90634]: pgmap v576: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-14T22:56:26.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:26.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:26.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:26.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:26.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:26.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:26.255 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 80767 80751 123.079 88.75 0.014401 0.00812007 2024-06-14T22:56:26.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:26.370 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:26.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:26.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-14T22:56:26.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:26.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:26.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:26.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:26.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:26.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:26.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:26.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:26.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:26.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:26.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:27.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:27.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:27.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:27.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:27.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:27.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:27.257 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 16 82061 82045 122.074 80.875 0.00977619 0.00818638 2024-06-14T22:56:27.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:27.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:27.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:27.508 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:27.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:27.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:27.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:27.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:27.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:27.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T22:56:27.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:27.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:27.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[97913]: pgmap v577: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-14T22:56:27.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:27.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:27.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/813019493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T22:56:27.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T22:56:27.865 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[102754]: pgmap v577: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-14T22:56:27.866 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:27.866 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:27.866 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/813019493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T22:56:27.866 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:27 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T22:56:27.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:27.892 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:56:27.892 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:56:27.892 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:56:27.893 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:56:27.893 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:56:27.893 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:56:27.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:27.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:27.906 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:56:27.907 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:56:27.942 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:56:27.942 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:56:27.975 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:56:27.975 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:56:27.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:27 smithi137 ceph-mon[90634]: pgmap v577: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-14T22:56:27.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:27.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:27.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/813019493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T22:56:27.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:27 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T22:56:27.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:28.012 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:56:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:56:28.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:28.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:28.048 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:56:28.048 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:56:28.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:28.203 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:28.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:28.255 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 83119 83103 120.773 66.125 0.00772547 0.00827543 2024-06-14T22:56:28.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:28.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:28.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.1 to 1 (10000) 2024-06-14T22:56:28.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:28.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:28.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:28.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:28.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:28.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:28.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:28.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:28.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:28.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:28.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:28.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:28.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:28.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4080456219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:56:28.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:28 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-14T22:56:28.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:28 smithi137 ceph-mon[90634]: osdmap e150: 8 total, 8 up, 8 in 2024-06-14T22:56:28.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:29.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:29.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4080456219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:56:29.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:28 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-14T22:56:29.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:28 smithi086 ceph-mon[102754]: osdmap e150: 8 total, 8 up, 8 in 2024-06-14T22:56:29.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4080456219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:56:29.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:28 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-14T22:56:29.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:28 smithi086 ceph-mon[97913]: osdmap e150: 8 total, 8 up, 8 in 2024-06-14T22:56:29.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:29.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:29.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:29.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:29.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:29.255 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 16 84004 83988 119.285 55.3125 0.145614 0.00835415 2024-06-14T22:56:29.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:29.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:29.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:29.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:29.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:29.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:29.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:29.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:29.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:29.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:29.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:29.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:29.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:29.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:29.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:29.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:29 smithi137 ceph-mon[90634]: pgmap v579: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-14T22:56:30.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:30.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:30.059 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:30.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:30.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:29 smithi086 ceph-mon[97913]: pgmap v579: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-14T22:56:30.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:29 smithi086 ceph-mon[102754]: pgmap v579: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-14T22:56:30.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:30.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:30.256 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 85014 84998 118.037 63.125 0.00997388 0.00846722 2024-06-14T22:56:30.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:30.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:30.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:30.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:30.457 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:30.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:30.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:30.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:30.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:30.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:30.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:30.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:30.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:30.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:30 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:56:30.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:30 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:56:30.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:30.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:30.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 2 1 2024-06-14T22:56:30.941 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:30 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:56:30.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:31.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:31.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:31.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:31.160 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:31.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:31.256 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 16 86263 86247 117.167 78.0625 0.00596183 0.00852986 2024-06-14T22:56:31.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:31.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:31.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:31.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:31.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:31.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:31.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:31.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:31.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:31.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:31.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:31.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:31.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:31.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:31.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:31.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:31.947 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:31 smithi137 ceph-mon[90634]: pgmap v580: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-14T22:56:31.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:32.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:32.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:32.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:32.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:31 smithi086 ceph-mon[102754]: pgmap v580: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-14T22:56:32.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:31 smithi086 ceph-mon[97913]: pgmap v580: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-14T22:56:32.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:32.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:32.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:32.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:32.256 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 87436 87420 116.233 73.3125 0.0155819 0.00859844 2024-06-14T22:56:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:32.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:32.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:32.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:32.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:32.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:32.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:32.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:32.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:32.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:32.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:32.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:32.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:32.834 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:32.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:32.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:32.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:32.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:32.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:33.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:33.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:33.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:33.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:33.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:33.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:33.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:33.257 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 88562 88546 115.276 70.375 0.0121182 0.00866974 2024-06-14T22:56:33.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:33.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:33.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:56:33.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:33.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:33.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:33.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:33.680 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:33.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:33.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:33 smithi086 ceph-mon[97913]: pgmap v581: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-14T22:56:33.698 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:33 smithi086 ceph-mon[102754]: pgmap v581: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-14T22:56:33.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:33.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:33.904 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:56:33.904 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55923 8.3 8.11c3dc33 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:24.482881+0000", 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 429.41755796400003, 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98888014700000004, 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55923 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.482881+0000", 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.482881+0000", 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.482880+0000", 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T22:56:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.482959+0000", 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9635000000000006e-05 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.482971+0000", 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1813e-05 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.482981+0000", 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0069e-05 2024-06-14T22:56:33.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.483332+0000", 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035074999999999999 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.483342+0000", 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8859999999999999e-06 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.471748+0000", 2024-06-14T22:56:33.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98840659399999997 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.471761+0000", 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3236e-05 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55930 8.a 8.445d7bca (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:24.482995+0000", 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 429.41744387599999, 2024-06-14T22:56:33.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98625170399999995, 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55930 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.482995+0000", 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.482995+0000", 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.482995+0000", 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.483040+0000", 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4767e-05 2024-06-14T22:56:33.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.483047+0000", 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6669999999999996e-06 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.483053+0000", 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2029999999999997e-06 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.489215+0000", 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061611349999999999 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.489224+0000", 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8039999999999996e-06 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469236+0000", 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98001182799999997 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469247+0000", 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0896e-05 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55936 8.7 8.156cc3f7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:24.488924+0000", 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 429.41151506900002, 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98218318000000004, 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55936 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.488924+0000", 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.488924+0000", 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.488923+0000", 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.488979+0000", 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6712999999999997e-05 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.488995+0000", 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5639999999999999e-05 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.489005+0000", 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5510000000000005e-06 2024-06-14T22:56:33.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490361+0000", 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013565879999999999 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490370+0000", 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1609999999999992e-06 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.471098+0000", 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98072737300000001 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.471107+0000", 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7869999999999996e-06 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55937 8.1 8.900cb8d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:24.490006+0000", 2024-06-14T22:56:33.918 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 429.41043341, 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97707022300000002, 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55937 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490006+0000", 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490006+0000", 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490005+0000", 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490053+0000", 2024-06-14T22:56:33.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7948000000000002e-05 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490063+0000", 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0084999999999999e-05 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490070+0000", 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1130000000000003e-06 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490144+0000", 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3736e-05 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.490153+0000", 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8249999999999994e-06 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.467061+0000", 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97690866099999996 2024-06-14T22:56:33.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.467076+0000", 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4771e-05 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55944 8.a 8.f7b2062a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:24.495487+0000", 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 429.40495215800001, 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97379240700000003, 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.923 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55944 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495487+0000", 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495487+0000", 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495485+0000", 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495623+0000", 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013770399999999999 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495637+0000", 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3957e-05 2024-06-14T22:56:33.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495646+0000", 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0880000000000001e-06 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495691+0000", 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4916999999999997e-05 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495703+0000", 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1559000000000001e-05 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469271+0000", 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97356817900000003 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469280+0000", 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6829999999999997e-06 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55946 8.a 8.a6d184a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-06-14T22:56:33.927 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:24.495660+0000", 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 429.404779711, 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97365209799999997, 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55946 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495660+0000", 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495660+0000", 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495659+0000", 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495704+0000", 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5235999999999999e-05 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495712+0000", 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3010000000000007e-06 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495718+0000", 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8449999999999999e-06 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495764+0000", 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5810999999999999e-05 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.495772+0000", 2024-06-14T22:56:33.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0090000000000007e-06 2024-06-14T22:56:33.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469302+0000", 2024-06-14T22:56:33.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97353049400000002 2024-06-14T22:56:33.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:33.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469312+0000", 2024-06-14T22:56:33.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1959999999999997e-06 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55950 8.1 8.56a78b51 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:24.498878+0000", 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 429.40156174999998, 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96847474600000005, 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.933 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55950 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.498878+0000", 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.498878+0000", 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.498874+0000", 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999962 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.498935+0000", 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1470000000000006e-05 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.498949+0000", 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.384e-05 2024-06-14T22:56:33.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.498955+0000", 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.981e-06 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.499007+0000", 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1301000000000002e-05 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:24.499019+0000", 2024-06-14T22:56:33.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2351000000000001e-05 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.467341+0000", 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96832236400000005 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.467352+0000", 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1042e-05 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98108 8.a 8:580c610c:::benchmark_data_smithi086_122122_object98107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.663895+0000", 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 387.23654382799998, 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.007937861, 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98108 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.938 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663895+0000", 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663895+0000", 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663893+0000", 2024-06-14T22:56:33.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663938+0000", 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4446000000000001e-05 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663958+0000", 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.984e-05 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663967+0000", 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6819999999999998e-06 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.664096+0000", 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000128489 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.667900+0000", 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038044369999999999 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.662435+0000", 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.994535222 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.662563+0000", 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012799 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.662679+0000", 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011518099999999999 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.669323+0000", 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066442560000000003 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671664+0000", 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002341278 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671736+0000", 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1549000000000002e-05 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671787+0000", 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1718999999999997e-05 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671833+0000", 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6004999999999997e-05 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.944 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98111 8.a 8:52d8109b:::benchmark_data_smithi086_122122_object98110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.666047+0000", 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 387.23439205699998, 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0059651590000001, 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98111 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.945 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666047+0000", 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666047+0000", 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666045+0000", 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666073+0000", 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7528000000000001e-05 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666087+0000", 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4144999999999999e-05 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666103+0000", 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6232999999999999e-05 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.668274+0000", 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002170834 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.668286+0000", 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1773999999999999e-05 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.665038+0000", 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99675228999999999 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.665152+0000", 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114057 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.665289+0000", 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136279 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671689+0000", 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006400579 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671888+0000", 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000198803 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671934+0000", 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5759e-05 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671974+0000", 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0763000000000003e-05 2024-06-14T22:56:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672012+0000", 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7978000000000001e-05 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98134 8.a 8:5734d724:::benchmark_data_smithi086_122122_object98133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.691770+0000", 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 387.20866937, 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98507571699999996, 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98134 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691770+0000", 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691770+0000", 2024-06-14T22:56:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691768+0000", 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691802+0000", 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3120999999999997e-05 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691817+0000", 2024-06-14T22:56:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5747e-05 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691824+0000", 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7830000000000001e-06 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691873+0000", 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9345000000000003e-05 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691884+0000", 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0575e-05 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.667450+0000", 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97556580900000001 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.667567+0000", 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011715 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:56:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.667680+0000", 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011291899999999999 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671898+0000", 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042183409999999996 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672064+0000", 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166167 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676751+0000", 2024-06-14T22:56:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046863879999999997 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676805+0000", 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4749e-05 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676846+0000", 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0166000000000001e-05 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98150 8.a 8:58ff2b8c:::benchmark_data_smithi086_122122_object98149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.706092+0000", 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 387.19434767000001, 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97089046899999998, 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98150 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706092+0000", 2024-06-14T22:56:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706092+0000", 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706090+0000", 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706136+0000", 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5948999999999997e-05 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706194+0000", 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7394e-05 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706212+0000", 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8150000000000001e-05 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706306+0000", 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4178000000000005e-05 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706317+0000", 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.135e-05 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.669367+0000", 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96304950300000003 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.669484+0000", 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000117681 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.669627+0000", 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000142724 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672073+0000", 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024456830000000001 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672097+0000", 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4059000000000001e-05 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676894+0000", 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004797 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676954+0000", 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0337999999999997e-05 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676982+0000", 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7861e-05 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98157 8.a 8:5815c7bb:::benchmark_data_smithi086_122122_object98156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.713784+0000", 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 387.186655358, 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96892899099999996, 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98157 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713784+0000", 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713784+0000", 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713783+0000", 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713807+0000", 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4148999999999999e-05 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713825+0000", 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8187999999999999e-05 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713834+0000", 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6710000000000005e-06 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713891+0000", 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7528000000000002e-05 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713906+0000", 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4440000000000001e-05 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672124+0000", 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95821818800000003 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672245+0000", 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000121508 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672386+0000", 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000141218 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.679864+0000", 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074774819999999997 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.682556+0000", 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026924309999999999 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.682587+0000", 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0913e-05 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.682653+0000", 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5442000000000001e-05 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.682713+0000", 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0180000000000003e-05 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:129201 8.3 8:c7ab2e14:::benchmark_data_smithi086_122122_object8612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-06-14T22:56:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:43.140054+0000", 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 350.76038568799999, 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97529428799999995, 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 129201 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140054+0000", 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140054+0000", 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140052+0000", 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140060+0000", 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3979999999999999e-06 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140079+0000", 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8173e-05 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140087+0000", 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9200000000000004e-06 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140149+0000", 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2477000000000005e-05 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140161+0000", 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1712999999999999e-05 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.108330+0000", 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96816938500000005 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.108456+0000", 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012608199999999999 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.109046+0000", 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00058971700000000002 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115204+0000", 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061580799999999998 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115254+0000", 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9706999999999997e-05 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115265+0000", 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0859999999999999e-05 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115307+0000", 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2867999999999997e-05 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115348+0000", 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0429999999999997e-05 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:129202 8.3 8:cccec3f8:::benchmark_data_smithi086_122122_object8613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:43.140692+0000", 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 350.75974773299998, 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98892060800000003, 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 129202 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140692+0000", 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140692+0000", 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140691+0000", 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140696+0000", 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5260000000000001e-06 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140704+0000", 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9330000000000001e-06 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140710+0000", 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0170000000000002e-06 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140783+0000", 2024-06-14T22:56:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3145999999999994e-05 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140806+0000", 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3059000000000001e-05 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115379+0000", 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97457263599999999 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115465+0000", 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5741999999999993e-05 2024-06-14T22:56:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115543+0000", 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8623999999999998e-05 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.123140+0000", 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0075970279999999996 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.123181+0000", 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0568000000000002e-05 2024-06-14T22:56:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129524+0000", 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063430120000000003 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129572+0000", 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8378999999999999e-05 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129612+0000", 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9752999999999997e-05 2024-06-14T22:56:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:129204 8.3 8:c65a5a5e:::benchmark_data_smithi086_122122_object8615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:43.142040+0000", 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 350.758399489, 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98773785999999997, 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 129204 2024-06-14T22:56:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142040+0000", 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142040+0000", 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142039+0000", 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:56:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142046+0000", 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.781e-06 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142062+0000", 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6713000000000001e-05 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142070+0000", 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2050000000000002e-06 2024-06-14T22:56:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.163576+0000", 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021505791 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.163587+0000", 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0869999999999999e-05 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.117985+0000", 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95439831399999997 2024-06-14T22:56:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.118089+0000", 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103247 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.118203+0000", 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011443 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.126190+0000", 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079872650000000003 2024-06-14T22:56:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129473+0000", 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0032825979999999999 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129702+0000", 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00022937899999999999 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129745+0000", 2024-06-14T22:56:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2685000000000002e-05 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129778+0000", 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2648999999999999e-05 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24715 9.d 9:b7390049:::benchmark_data_smithi086_152062_object24714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.661292+0000", 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.239147731999999, 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.018053425, 2024-06-14T22:56:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24715 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661292+0000", 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661292+0000", 2024-06-14T22:56:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661290+0000", 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661315+0000", 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5228999999999998e-05 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661325+0000", 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8689999999999999e-06 2024-06-14T22:56:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661332+0000", 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8990000000000004e-06 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661364+0000", 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661375+0000", 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0645999999999999e-05 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.665805+0000", 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.004429807 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.666631+0000", 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082573 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.666788+0000", 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000157484 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673421+0000", 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066326279999999998 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673465+0000", 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4218999999999998e-05 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679261+0000", 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057959120000000003 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679308+0000", 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6998e-05 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679345+0000", 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7234999999999998e-05 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24723 9.d 9:b2f904ff:::benchmark_data_smithi086_152062_object24722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.664488+0000", 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.235950955, 2024-06-14T22:56:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0150036360000001, 2024-06-14T22:56:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24723 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664488+0000", 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664488+0000", 2024-06-14T22:56:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664487+0000", 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664523+0000", 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5485000000000003e-05 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664538+0000", 2024-06-14T22:56:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5412000000000001e-05 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664545+0000", 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.781e-06 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664598+0000", 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2976000000000002e-05 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664607+0000", 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6290000000000005e-06 2024-06-14T22:56:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668290+0000", 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0036830910000001 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668426+0000", 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013543800000000001 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668573+0000", 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147498 2024-06-14T22:56:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673521+0000", 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049472129999999998 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.676833+0000", 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0033120250000000001 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679379+0000", 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025462699999999998 2024-06-14T22:56:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679445+0000", 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6476999999999994e-05 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679492+0000", 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6640000000000001e-05 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24729 9.d 9:b6335b33:::benchmark_data_smithi086_152062_object24728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.667823+0000", 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.232615981999999, 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0118812850000001, 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24729 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667823+0000", 2024-06-14T22:56:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667823+0000", 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667822+0000", 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667847+0000", 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4666000000000001e-05 2024-06-14T22:56:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667864+0000", 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7611000000000001e-05 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667878+0000", 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3182e-05 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668333+0000", 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000454931 2024-06-14T22:56:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668342+0000", 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1630000000000002e-06 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.669852+0000", 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.00151035 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.670010+0000", 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000158257 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.670183+0000", 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017219199999999999 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.677648+0000", 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007465546 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679516+0000", 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001868429 2024-06-14T22:56:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679566+0000", 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9395999999999999e-05 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679670+0000", 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010412900000000001 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679705+0000", 2024-06-14T22:56:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4591999999999997e-05 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24730 9.d 9:be481d2c:::benchmark_data_smithi086_152062_object24729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.668425+0000", 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.232014106000001, 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0114436330000001, 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24730 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668447+0000", 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2699000000000001e-05 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668454+0000", 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6880000000000002e-06 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668459+0000", 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4940000000000003e-06 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668500+0000", 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0577000000000001e-05 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668508+0000", 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5229999999999998e-06 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.671828+0000", 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.003320011 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.671965+0000", 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000137081 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.672100+0000", 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013549800000000001 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.677685+0000", 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055849710000000002 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679521+0000", 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001835761 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679784+0000", 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000263465 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679825+0000", 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0980999999999999e-05 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679869+0000", 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3461000000000003e-05 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24738 9.d 9:b3d2a961:::benchmark_data_smithi086_152062_object24737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.671698+0000", 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.228740989000002, 2024-06-14T22:56:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.009193183, 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24738 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671698+0000", 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:56:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671698+0000", 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671697+0000", 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671720+0000", 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3345000000000001e-05 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:56:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671734+0000", 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3669000000000001e-05 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671742+0000", 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7470000000000005e-06 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671822+0000", 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.051e-05 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671834+0000", 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1847e-05 2024-06-14T22:56:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673548+0000", 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.001713718 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673671+0000", 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012297600000000001 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673796+0000", 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012489900000000001 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679254+0000", 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054580119999999999 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679525+0000", 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00027125800000000002 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680809+0000", 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012839290000000001 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680856+0000", 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7094e-05 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680891+0000", 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5438999999999997e-05 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:56:34.022 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:56:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:34.023 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:34.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:34.023 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:33 smithi137 ceph-mon[90634]: pgmap v581: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-14T22:56:34.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:34.166 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:34.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:34.257 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 89721 89705 114.402 72.4375 0.0111844 0.00873621 2024-06-14T22:56:34.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:34.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:34.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:34.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:34.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:34.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:34.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:34.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:34.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:34.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:34.662 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:34.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:34.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:34.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:34.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:34.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:34.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:34.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:34.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:34.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:34.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:35.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:35.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:35.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:35.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:35.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:35.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 90943 90927 113.641 76.375 0.00723448 0.00879476 2024-06-14T22:56:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:35.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:35.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:35.274 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:56:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:56:34] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.43.0" 2024-06-14T22:56:35.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T22:56:35.323 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:35.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:35.448 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:35.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:35.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:35.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:35.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:35.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:35.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:35.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:35.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:35.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:35.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:35.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:35 smithi086 ceph-mon[97913]: pgmap v582: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-14T22:56:35.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:35.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:35 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:35.981 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:35 smithi086 ceph-mon[102754]: pgmap v582: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-14T22:56:35.982 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:35.982 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:35 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:35.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:35 smithi137 ceph-mon[90634]: pgmap v582: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-14T22:56:35.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:35.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:35 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:36.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:36.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:36.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:36.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:36.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:36.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 92057 92041 112.778 69.625 0.00811819 0.00886265 2024-06-14T22:56:36.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:36.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:36.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:36.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:36.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:36.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:36.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:36.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:36.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:36.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:36.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:36.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:36.829 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:36.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:36.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:36.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:36.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:37.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:37.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:37.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:37.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:37.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:37.258 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 16 93300 93284 112.103 77.6875 0.0147673 0.00891584 2024-06-14T22:56:37.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:37.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:37.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:37.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:37.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:37.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:37.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:37.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:37.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:37.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:37.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:37.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:37 smithi086 ceph-mon[97913]: pgmap v583: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-06-14T22:56:37.816 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:37 smithi086 ceph-mon[102754]: pgmap v583: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-06-14T22:56:37.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:37.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:37.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:37.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:37 smithi137 ceph-mon[90634]: pgmap v583: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-06-14T22:56:38.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:38.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:38.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:38.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:38.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:38.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:38.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 16 94515 94499 111.418 75.9375 0.0151837 0.00897008 2024-06-14T22:56:38.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:38.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:38.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:38.501 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:38.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:38.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:38.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:38.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:38.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:38.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:38.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:38.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:38.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:39.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:39.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:39.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:39.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:39.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:39.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:39.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 95632 95616 110.647 69.8125 0.0116979 0.00903275 2024-06-14T22:56:39.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:39.320 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:39.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:39.358 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:39.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:39.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:39.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:39.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:39.608 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:39.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:39.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:39.740 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T22:56:39.740 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.2 to 1 (10000) 2024-06-14T22:56:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:39.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:39.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:39.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:39.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:39.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:39 smithi137 ceph-mon[90634]: pgmap v584: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:39 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3075175999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:56:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:39 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1675304027' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T22:56:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:39 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T22:56:40.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:39 smithi086 ceph-mon[102754]: pgmap v584: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:40.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:39 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3075175999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:56:40.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:39 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1675304027' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T22:56:40.003 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:39 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T22:56:40.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:39 smithi086 ceph-mon[97913]: pgmap v584: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:40.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:39 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3075175999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:56:40.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:39 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1675304027' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T22:56:40.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:39 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T22:56:40.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:40.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:40.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:40.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:40.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:40.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:40.250 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:40.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 96716 96700 109.868 67.75 0.0127646 0.00909642 2024-06-14T22:56:40.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:40.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:40.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:40.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:40.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:40.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:40.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:40.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:40.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:40.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 3 1 2024-06-14T22:56:40.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:40.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:40.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:40.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:40.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:40.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:40.937 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:40 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:56:40.937 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3075175999' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:56:40.937 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:40 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T22:56:40.937 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:40 smithi137 ceph-mon[90634]: osdmap e151: 8 total, 8 up, 8 in 2024-06-14T22:56:40.954 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:40 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:56:40.954 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3075175999' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:56:40.954 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:40 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T22:56:40.954 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:40 smithi086 ceph-mon[102754]: osdmap e151: 8 total, 8 up, 8 in 2024-06-14T22:56:40.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:40 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:56:40.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3075175999' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:56:40.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:40 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T22:56:40.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:40 smithi086 ceph-mon[97913]: osdmap e151: 8 total, 8 up, 8 in 2024-06-14T22:56:41.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:41.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:41.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:41.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:41.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:41.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 97843 97827 109.163 70.4375 0.00855964 0.00915538 2024-06-14T22:56:41.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:41.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:41.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:41.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:41.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:41.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:41.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:41.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:41.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:41.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:41.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:41 smithi086 ceph-mon[97913]: pgmap v586: 89 pgs: 89 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:41.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:41.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:41.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:41.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:41.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:41.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:41.989 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:41 smithi086 ceph-mon[102754]: pgmap v586: 89 pgs: 89 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:41.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:41 smithi137 ceph-mon[90634]: pgmap v586: 89 pgs: 89 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:42.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:42.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:42.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:42.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:42.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:42.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:42.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 98951 98935 108.463 69.25 0.00912878 0.00921534 2024-06-14T22:56:42.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:42.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:42.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:42.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:42.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:42.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:42.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:42.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:42.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:42.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:42.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:42.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:42.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:42.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:42.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:42.874 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:42.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:42.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:43.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:43.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:43.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:43.165 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:43.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:43.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 16 100138 100122 107.872 74.1875 0.0203562 0.00926567 2024-06-14T22:56:43.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:43.300 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:43.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:43.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:43.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:43.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:43.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:43.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T22:56:43.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:43.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:43.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:43.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:43.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:43.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:43.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:43 smithi137 ceph-mon[90634]: pgmap v587: 89 pgs: 89 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:43.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:43.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:43.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:43.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:43 smithi086 ceph-mon[102754]: pgmap v587: 89 pgs: 89 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:43.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:43.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:43.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:43 smithi086 ceph-mon[97913]: pgmap v587: 89 pgs: 89 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:43.856 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:43.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:43.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:44.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:44.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:44.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:44.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:44.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:44.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:44.259 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 101245 101229 107.216 69.1875 0.0139366 0.00932224 2024-06-14T22:56:44.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:44.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:44.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:44.433 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:44.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:44.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:44.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:44.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:44.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:44.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:44.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:44.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:44.865 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:44.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:44.932 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:56:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:56:44] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.43.0" 2024-06-14T22:56:44.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:44.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:44.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:45.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:45.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:45.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:45.261 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:56:45.261926+0000 min lat: 0.00125959 max lat: 1.02178 avg lat: 0.00937799 2024-06-14T22:56:45.261 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:56:45.261 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 102341 102325 106.569 68.5 0.0130723 0.00937799 2024-06-14T22:56:45.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:45.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:45.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:45.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:45.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:45.498 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:56:45.499 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:56:45.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:45.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:45.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:45.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:45.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:56:45.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:56:45.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:56:45.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:56:45.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:56:45.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:56:45.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:56:45.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:56:45.663 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:56:45.663 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:56:45.663 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:56:45.663 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:56:45.663 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:56:45.663 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:56:45.663 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:56:45.663 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:56:45.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:45.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:45.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:45.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:45.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:45.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:45 smithi086 ceph-mon[102754]: pgmap v588: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:45.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2131005576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:56:45.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:45 smithi086 ceph-mon[97913]: pgmap v588: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:45.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2131005576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:56:45.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:45.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:45.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:45.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:45 smithi137 ceph-mon[90634]: pgmap v588: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:45.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2131005576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:56:46.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:46.084 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:46.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:46.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:46.229 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:46.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:46.261 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 103434 103418 105.942 68.3125 0.00518668 0.00943426 2024-06-14T22:56:46.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:46.365 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:46.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:46.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:46.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:46.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:46.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:46.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:46.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:46.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:46.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:46.773 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:46.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:46.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:46.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:47.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:47.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:47.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:47.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:47.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:47.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:47.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:47.261 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 104597 104581 105.405 72.6875 0.00843131 0.00948217 2024-06-14T22:56:47.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:47.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:47.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:47.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:47.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:47.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:47.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:47.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:47.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:47.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:47.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:47.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:47.742 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:47 smithi086 ceph-mon[102754]: pgmap v589: 89 pgs: 89 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:47.742 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:47 smithi086 ceph-mon[97913]: pgmap v589: 89 pgs: 89 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:47.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:47.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:47.872 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:47.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:47.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:47.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:47 smithi137 ceph-mon[90634]: pgmap v589: 89 pgs: 89 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:48.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:48.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:48.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:48.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:48.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:48.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:48.261 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 105740 105724 104.866 71.4375 0.00665154 0.00953129 2024-06-14T22:56:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:48.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:48.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:48.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:48.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:48.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:48.561 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:48.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:48.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-14T22:56:48.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:48.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:48.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:48.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:48.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:48.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:48.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:49.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:49.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:49.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:49.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:49.263 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 106800 106784 104.259 66.25 0.0173461 0.00958581 2024-06-14T22:56:49.268 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:49.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:49.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:49.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:49.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:49.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:49.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:49.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:49.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:49.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:49.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:49.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:49.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:56:49.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:49 smithi086 ceph-mon[97913]: pgmap v590: 89 pgs: 89 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:49.786 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:49 smithi086 ceph-mon[102754]: pgmap v590: 89 pgs: 89 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:49.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:49.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:49.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:49.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:49.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:49.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:49.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:49 smithi137 ceph-mon[90634]: pgmap v590: 89 pgs: 89 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T22:56:50.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:50.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:50.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:50.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:50.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:50.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:50.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:50.263 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 107896 107880 103.709 68.5 0.00700257 0.00963747 2024-06-14T22:56:50.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:50.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:50.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:50.419 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:50.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:50.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:50.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:50.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:50.636 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:50.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:50.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:50.766 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:50.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:50.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:50.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:50.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:51.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:51.054 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:51.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:51.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:51.180 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:51.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:51.263 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 108883 108867 103.072 61.6875 0.00587606 0.0096969 2024-06-14T22:56:51.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:51.320 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:51.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:51.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:51.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:51.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:51.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:51.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:51.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:51.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:51.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:51.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:51.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:51.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:51.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:51 smithi137 ceph-mon[90634]: pgmap v591: 89 pgs: 89 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-14T22:56:51.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:51.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:51.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:51.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:51 smithi086 ceph-mon[97913]: pgmap v591: 89 pgs: 89 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-14T22:56:51.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:51.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:51.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:51 smithi086 ceph-mon[102754]: pgmap v591: 89 pgs: 89 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-14T22:56:51.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:51.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:51.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:51.991 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:51.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:52.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:52.131 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:52.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:52.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:52.263 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 109977 109961 102.554 68.375 0.0205954 0.00974531 2024-06-14T22:56:52.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:52.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:52.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:52.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:52.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:52.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:52.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:52.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:52.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:52.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:52.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:52.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:52.816 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:52.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:52.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:52.962 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:52.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:52.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:53.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:53.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:53.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:53.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:53.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:53.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:53.264 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 111075 111059 102.055 68.625 0.039245 0.00979385 2024-06-14T22:56:53.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:53.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:53.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:53.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:53.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:53.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:53.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:53.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:53.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:53.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:53.682 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:53 smithi086 ceph-mon[97913]: pgmap v592: 89 pgs: 89 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-14T22:56:53.682 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:53 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:56:53.682 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:53 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:56:53.682 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:53 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:53.682 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:53 smithi086 ceph-mon[102754]: pgmap v592: 89 pgs: 89 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-14T22:56:53.682 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:53 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:56:53.682 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:53 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:56:53.682 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:53 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:53.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:53.792 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:53.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:53.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:53.926 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:53.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:53 smithi137 ceph-mon[90634]: pgmap v592: 89 pgs: 89 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-14T22:56:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:53 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:56:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:53 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:56:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:53 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:56:54.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:54.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:54.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:54.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:54.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:54.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:54.264 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 112127 112111 101.529 65.75 0.0143946 0.00984449 2024-06-14T22:56:54.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:54.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:54.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:54.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:54.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:54.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:54.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:54.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:54.615 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:54.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:54.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:54.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:54.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:54.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:54.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:54.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:54.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:55.023 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:55.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:55.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:55.159 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:55.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:55.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:55.180 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:56:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:56:54] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.43.0" 2024-06-14T22:56:55.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:55.264 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 113194 113178 101.031 66.6875 0.0161728 0.00989329 2024-06-14T22:56:55.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:55.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:55.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:55.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:55.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:55.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:55.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:55.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:55.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:55.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:55.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:55.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:55.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:55.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:55 smithi086 ceph-mon[102754]: pgmap v593: 89 pgs: 89 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-14T22:56:55.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:55 smithi086 ceph-mon[97913]: pgmap v593: 89 pgs: 89 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-14T22:56:55.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:55.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:56:55.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:55.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:55.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:55.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:55 smithi137 ceph-mon[90634]: pgmap v593: 89 pgs: 89 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-14T22:56:56.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:56.122 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:56.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:56.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:56.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:56.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:56.264 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 114229 114213 100.519 64.6875 0.00990458 0.0099435 2024-06-14T22:56:56.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:56.398 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:56.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:56.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:56.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:56.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:56.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:56.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:56.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:56.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:56.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:56.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:56.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:56.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:56.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:56.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:57.000 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:56:57.000 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:56:57.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:57.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:57.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:57.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:57.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:57.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:57.264 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 115214 115198 99.9781 61.5625 0.0123967 0.00999711 2024-06-14T22:56:57.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:57.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:57.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:57.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:57.492 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:57.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:57.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:57.633 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:57.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:57.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:57.665 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.3 to 1 (10000) 2024-06-14T22:56:57.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:57.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:57.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:57.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:57 smithi086 ceph-mon[97913]: pgmap v594: 89 pgs: 89 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 995 op/s 2024-06-14T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3304134860' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:57 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:57.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:57 smithi086 ceph-mon[102754]: pgmap v594: 89 pgs: 89 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 995 op/s 2024-06-14T22:56:57.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3304134860' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T22:56:57.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:57 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T22:56:57.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:57.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:57.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:57.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:57 smithi137 ceph-mon[90634]: pgmap v594: 89 pgs: 89 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 995 op/s 2024-06-14T22:56:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3304134860' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T22:56:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:57 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T22:56:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:56:58.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:58.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:58.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:58.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:58.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:58.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:58.264 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 116177 116161 99.433 60.1875 0.0243575 0.0100523 2024-06-14T22:56:58.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:58.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:58.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:58.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:58.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:58.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:58.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:58.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:58.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:58.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:58.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:56:58.693 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T22:56:58.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:58.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:56:58.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:56:58.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:58.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:58.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:56:58.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:56:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:58.992 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:58.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:56:58.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:58 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T22:56:58.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:58 smithi137 ceph-mon[90634]: osdmap e152: 8 total, 8 up, 8 in 2024-06-14T22:56:58.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2950723931' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:56:59.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:58 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T22:56:59.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:58 smithi086 ceph-mon[102754]: osdmap e152: 8 total, 8 up, 8 in 2024-06-14T22:56:59.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2950723931' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:56:59.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:58 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T22:56:59.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:58 smithi086 ceph-mon[97913]: osdmap e152: 8 total, 8 up, 8 in 2024-06-14T22:56:59.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2950723931' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:56:59.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:56:59.132 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:59.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:56:59.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:59.262 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:59.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:59.264 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 15 117119 117104 98.8858 58.9375 0.021863 0.0101078 2024-06-14T22:56:59.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:56:59.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:59.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:56:59.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:59.534 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:59.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:59.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:56:59.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:56:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:59.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:56:59.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:56:59.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:59.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:56:59.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:59 smithi086 ceph-mon[102754]: pgmap v596: 89 pgs: 89 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-14T22:56:59.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:59 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:56:59.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:59 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2950723931' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:56:59.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:56:59 smithi086 ceph-mon[102754]: osdmap e153: 8 total, 8 up, 8 in 2024-06-14T22:56:59.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:59 smithi086 ceph-mon[97913]: pgmap v596: 89 pgs: 89 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-14T22:56:59.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:59 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:56:59.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:59 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2950723931' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:56:59.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:56:59 smithi086 ceph-mon[97913]: osdmap e153: 8 total, 8 up, 8 in 2024-06-14T22:56:59.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 4 1 2024-06-14T22:56:59.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:56:59.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:56:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:56:59.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:56:59.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:59 smithi137 ceph-mon[90634]: pgmap v596: 89 pgs: 89 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-14T22:56:59.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:59 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:56:59.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:59 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2950723931' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:56:59.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:56:59 smithi137 ceph-mon[90634]: osdmap e153: 8 total, 8 up, 8 in 2024-06-14T22:57:00.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:00.104 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:00.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:00.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:00.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:00.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:00.264 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 118019 118003 98.3165 56.1875 0.0382175 0.0101664 2024-06-14T22:57:00.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:00.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:00.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:00.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-14T22:57:00.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:00.506 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:00.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:00.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:00.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:00.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:00.710 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:57:00.739 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:57:00.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:57:00.781 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:57:00.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:00.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:57:00.815 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:57:00.851 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:57:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:57:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:57:00.887 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:57:00.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:00.919 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:57:00.919 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:57:00.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:00.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:00.952 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:57:00.952 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:57:00.985 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:57:00.986 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:57:01.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:01.065 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:01.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:01.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:01.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:01.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:01.264 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 118972 118956 97.8065 59.5625 0.00889883 0.0102192 2024-06-14T22:57:01.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:01.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:01.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:01.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:01.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:01.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:01.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:01.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:01.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:01.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:01.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:01.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:01.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:01.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:01.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:01.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:01 smithi137 ceph-mon[90634]: pgmap v598: 89 pgs: 89 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-14T22:57:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3355457311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:01.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:01 smithi086 ceph-mon[97913]: pgmap v598: 89 pgs: 89 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-14T22:57:01.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3355457311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:01.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:01 smithi086 ceph-mon[102754]: pgmap v598: 89 pgs: 89 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-14T22:57:01.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3355457311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:02.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:02.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:02.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:02.157 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:02.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:02.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:02.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:02.265 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 16 119980 119964 97.3541 63 0.0152089 0.0102672 2024-06-14T22:57:02.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:02.336 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:02.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:02.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:02.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:02.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:02.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:02.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:02.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:02.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:02.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:02.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:02.750 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:02.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:02.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:02.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:02.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:02.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:03.023 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:03.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:03.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:03.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:03.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:03.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:03.265 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 120983 120967 96.909 62.6875 0.0120036 0.0103141 2024-06-14T22:57:03.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:03.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:03.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:03.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:03.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:03.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T22:57:03.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:03.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:03.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:03.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:03.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:03.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:03.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:03.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:03.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:03.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:03.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:03 smithi086 ceph-mon[97913]: pgmap v599: 89 pgs: 89 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 749 op/s 2024-06-14T22:57:03.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:03 smithi086 ceph-mon[102754]: pgmap v599: 89 pgs: 89 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 749 op/s 2024-06-14T22:57:03.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:57:03.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:03.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:03.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:03.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:03 smithi137 ceph-mon[90634]: pgmap v599: 89 pgs: 89 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 749 op/s 2024-06-14T22:57:04.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:04.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:04.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:04.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:04.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:04.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:04.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 121901 121885 96.4085 57.375 0.0068049 0.0103676 2024-06-14T22:57:04.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:04.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:04.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:04.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:04.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:04.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:04.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:04.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:04.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:04.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:04.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:04.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:04.975 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:57:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:57:04] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.43.0" 2024-06-14T22:57:05.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:05.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:05.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:05.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:05.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T22:57:05.267022+0000 min lat: 0.00125959 max lat: 1.02178 avg lat: 0.0104124 2024-06-14T22:57:05.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T22:57:05.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 122918 122902 95.9979 63.5625 0.00635092 0.0104124 2024-06-14T22:57:05.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:05.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:05.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:05.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:05.503 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:05.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:05.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:05.658 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:05.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:05.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:05.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:05.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:05 smithi086 ceph-mon[97913]: pgmap v600: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-06-14T22:57:05.807 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:05 smithi086 ceph-mon[102754]: pgmap v600: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-06-14T22:57:05.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:05.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:05.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:05.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:05 smithi137 ceph-mon[90634]: pgmap v600: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-06-14T22:57:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:06.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:06.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:06.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:06.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:06.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:06.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:06.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 123895 123879 95.5666 61.0625 0.0160372 0.0104588 2024-06-14T22:57:06.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:06.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:06.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:06.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:06.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:06.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:06.539 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:06.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:06.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:06.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:06.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:06.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:06.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:06.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:06.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:07.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:07.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:07.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:07.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:07.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:07.229 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:07.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:07.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 124823 124807 95.1084 58 0.011199 0.0105093 2024-06-14T22:57:07.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:07.362 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:07.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:07.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:07.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:07.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:07.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:07.631 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:07.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.4 to 1 (10000) 2024-06-14T22:57:07.732 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:07.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:07.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:07.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:07.906 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:07.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:07 smithi137 ceph-mon[90634]: pgmap v601: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-14T22:57:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/740783767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T22:57:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:07 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T22:57:08.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:08.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:08.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:08.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:07 smithi086 ceph-mon[97913]: pgmap v601: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-14T22:57:08.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/740783767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T22:57:08.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:07 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T22:57:08.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:07 smithi086 ceph-mon[102754]: pgmap v601: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-14T22:57:08.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/740783767' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T22:57:08.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:07 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T22:57:08.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:08.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:08.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:08.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 16 125785 125769 94.6869 60.125 0.018085 0.0105566 2024-06-14T22:57:08.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:08.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:08.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:08.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:08.448 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:08.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:08.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:08.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:08.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:08.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:08.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:08.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:08.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:08.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:08.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:08.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:08.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:08 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T22:57:08.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:08 smithi137 ceph-mon[90634]: osdmap e154: 8 total, 8 up, 8 in 2024-06-14T22:57:09.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:09.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:09.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:08 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T22:57:09.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:08 smithi086 ceph-mon[97913]: osdmap e154: 8 total, 8 up, 8 in 2024-06-14T22:57:09.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:08 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T22:57:09.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:08 smithi086 ceph-mon[102754]: osdmap e154: 8 total, 8 up, 8 in 2024-06-14T22:57:09.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:09.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:09.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:09.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:09.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 126792 126776 94.3089 62.9375 0.0149072 0.0105989 2024-06-14T22:57:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:09.280 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:09.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:09.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:09.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:09.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:09.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:09.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:09.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:09.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:09.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:09.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:09.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:09.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:09.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:09.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:09.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:09.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:09 smithi137 ceph-mon[90634]: pgmap v603: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-06-14T22:57:10.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:10.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:09 smithi086 ceph-mon[102754]: pgmap v603: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-06-14T22:57:10.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:09 smithi086 ceph-mon[97913]: pgmap v603: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-06-14T22:57:10.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:10.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:10.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:10.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:10.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:10.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:10.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 127735 127719 93.8928 58.9375 0.0167283 0.0106452 2024-06-14T22:57:10.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:10.364 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:10.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:10.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:10.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:10.497 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:10.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:10.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:10.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:10.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:10.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:10.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:10.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:10.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:10.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:10.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:10.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:11.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:11.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:11.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:11.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:11.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:11.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:11.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 128741 128725 93.5321 62.875 0.00729512 0.0106865 2024-06-14T22:57:11.275 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:57:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T22:57:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T22:57:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T22:57:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:91055 9.0 e150/140 9:09c35380:::benchmark_data_smithi086_152062_object91054:head v 150'5771, mlcod=150'5768)", 2024-06-14T22:57:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:35.395076+0000", 2024-06-14T22:57:11.275 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.391049836999997, 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024041976, 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.395076+0000", 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.395076+0000", 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.395075+0000", 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.395101+0000", 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5624999999999999e-05 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.395119+0000", 2024-06-14T22:57:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8785000000000001e-05 2024-06-14T22:57:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.395127+0000", 2024-06-14T22:57:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.554e-06 2024-06-14T22:57:11.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.395234+0000", 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010754199999999999 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.395280+0000", 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5139000000000003e-05 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.418961+0000", 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023681890000000001 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:35.419118+0000", 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015637299999999999 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.280 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:93340 9.0 e150/140 9:06114669:::benchmark_data_smithi086_152062_object93339:head v 150'5892, mlcod=150'5889)", 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:37.299057+0000", 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 30.487068301000001, 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018480358999999998, 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.299057+0000", 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.299057+0000", 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.299056+0000", 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.299082+0000", 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6256000000000001e-05 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.299099+0000", 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6341000000000002e-05 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.299106+0000", 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7039999999999994e-06 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.299215+0000", 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109063 2024-06-14T22:57:11.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.299286+0000", 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0090000000000001e-05 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.317490+0000", 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018204623 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:37.317538+0000", 2024-06-14T22:57:11.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7620999999999997e-05 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:95722 9.2 e150/140 9:4c1e2103:::benchmark_data_smithi086_152062_object95721:head v 150'6013, mlcod=150'6009)", 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:39.348179+0000", 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 28.437947116, 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024845190999999999, 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.348179+0000", 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.348179+0000", 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.348177+0000", 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.348345+0000", 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016810300000000001 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.348367+0000", 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2095000000000001e-05 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.348376+0000", 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4780000000000008e-06 2024-06-14T22:57:11.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.351678+0000", 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0033022580000000002 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.351747+0000", 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9338000000000006e-05 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.371990+0000", 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020242722000000001 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.373024+0000", 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010335730000000001 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:96143 9.6 e151/140 9:6de9922f:::benchmark_data_smithi086_152062_object96142:head v 151'6030, mlcod=150'6027)", 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:39.755009+0000", 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 28.031116254000001, 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027870290999999998, 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.755009+0000", 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.755009+0000", 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.755009+0000", 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.755051+0000", 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1987999999999998e-05 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.755069+0000", 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7869000000000001e-05 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.755076+0000", 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2289999999999997e-06 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.767909+0000", 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012832913 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.767989+0000", 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9988999999999996e-05 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.782814+0000", 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014825031000000001 2024-06-14T22:57:11.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.782880+0000", 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5602e-05 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:96212 9.0 e151/140 9:08c7d87f:::benchmark_data_smithi086_152062_object96211:head v 151'6072, mlcod=151'6070)", 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:39.832360+0000", 2024-06-14T22:57:11.294 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.953765522000001, 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020185906999999999, 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.832360+0000", 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.832360+0000", 2024-06-14T22:57:11.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.832360+0000", 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.832386+0000", 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6171999999999999e-05 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.832398+0000", 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2401e-05 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.832404+0000", 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5799999999999999e-06 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.832535+0000", 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013092600000000001 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.832577+0000", 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2676999999999998e-05 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.852428+0000", 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019850519 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:39.852546+0000", 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118311 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.298 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:99877 9.0 e151/140 9:0423d3ac:::benchmark_data_smithi086_152062_object99876:head v 151'6315, mlcod=151'6313)", 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:43.031760+0000", 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 24.754365467, 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019041969999999998, 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031760+0000", 2024-06-14T22:57:11.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031760+0000", 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031759+0000", 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031789+0000", 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0474999999999999e-05 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031815+0000", 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6128000000000001e-05 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031824+0000", 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7970000000000006e-06 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.033036+0000", 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001211686 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.033130+0000", 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3993999999999995e-05 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.050719+0000", 2024-06-14T22:57:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017588616000000001 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.050802+0000", 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3681000000000001e-05 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:99875 9.6 e151/140 9:69ad53b0:::benchmark_data_smithi086_152062_object99874:head v 151'6275, mlcod=151'6273)", 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:43.031848+0000", 2024-06-14T22:57:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 24.754277332000001, 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019133614, 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031848+0000", 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031848+0000", 2024-06-14T22:57:11.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031848+0000", 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031881+0000", 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.328e-05 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031900+0000", 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8660000000000001e-05 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.031908+0000", 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8439999999999998e-06 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.032095+0000", 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000187238 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.032151+0000", 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6171000000000001e-05 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.050939+0000", 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018787979 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:43.050982+0000", 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2920000000000002e-05 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.307 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121598 9.6 e153/140 9:67b8e956:::benchmark_data_smithi086_152062_object121597:head v 153'7596, mlcod=153'7593)", 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.940526+0000", 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.8455993309999998, 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032672680000000003, 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.940526+0000", 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.940526+0000", 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.940526+0000", 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.940571+0000", 2024-06-14T22:57:11.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5049000000000002e-05 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.940581+0000", 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0735e-05 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.940587+0000", 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7849999999999997e-06 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.941508+0000", 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092063900000000001 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.941574+0000", 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6340999999999994e-05 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.973154+0000", 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031580270000000001 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.973199+0000", 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4629999999999998e-05 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121603 9.6 e153/140 9:61336adf:::benchmark_data_smithi086_152062_object121602:head v 153'7597, mlcod=153'7593)", 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.945536+0000", 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.8405900310000001, 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030996955, 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.312 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.945536+0000", 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.945536+0000", 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.945535+0000", 2024-06-14T22:57:11.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.945595+0000", 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0179000000000001e-05 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.945613+0000", 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8559e-05 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.945621+0000", 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3889999999999999e-06 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.953401+0000", 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00778066 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.953451+0000", 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9703999999999997e-05 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.976444+0000", 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022992756999999999 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.976533+0000", 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8795000000000006e-05 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121608 9.6 e153/140 9:62ce2266:::benchmark_data_smithi086_152062_object121607:head v 153'7598, mlcod=153'7593)", 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.949687+0000", 2024-06-14T22:57:11.316 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.8364382539999999, 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026898611999999999, 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.949687+0000", 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.949687+0000", 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.949686+0000", 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.949725+0000", 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9075000000000001e-05 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.949746+0000", 2024-06-14T22:57:11.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0506e-05 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.949754+0000", 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2700000000000004e-06 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957048+0000", 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0072940519999999997 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957070+0000", 2024-06-14T22:57:11.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2246999999999999e-05 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.976537+0000", 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019466441000000001 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.976586+0000", 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9397000000000002e-05 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.320 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121612 9.6 e153/140 9:62343ee0:::benchmark_data_smithi086_152062_object121611:head v 153'7599, mlcod=153'7593)", 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.953900+0000", 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.8322256750000001, 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022713811, 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.953900+0000", 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.953900+0000", 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.953897+0000", 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999967 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.953958+0000", 2024-06-14T22:57:11.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1808000000000002e-05 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.953975+0000", 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6872e-05 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.953983+0000", 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6210000000000004e-06 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.966428+0000", 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01244545 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.966472+0000", 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4131999999999997e-05 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.976592+0000", 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010119203 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.976614+0000", 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2220000000000001e-05 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121618 9.2 e153/140 9:46508b2e:::benchmark_data_smithi086_152062_object121617:head v 153'7668, mlcod=153'7664)", 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.957153+0000", 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.8289731040000001, 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028463200000000001, 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957153+0000", 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957153+0000", 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957151+0000", 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:57:11.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957178+0000", 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6145000000000001e-05 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957241+0000", 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3088000000000003e-05 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957250+0000", 2024-06-14T22:57:11.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7720000000000006e-06 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.958003+0000", 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00075284600000000005 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.958028+0000", 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4978e-05 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.985493+0000", 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027464358000000001 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.985616+0000", 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012312299999999999 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121615 9.6 e153/140 9:649ee72a:::benchmark_data_smithi086_152062_object121614:head v 153'7600, mlcod=153'7593)", 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.957175+0000", 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.8289502620000002, 2024-06-14T22:57:11.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024771855999999998, 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957175+0000", 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957175+0000", 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957175+0000", 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957309+0000", 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013454100000000001 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957318+0000", 2024-06-14T22:57:11.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8880000000000003e-06 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.957323+0000", 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5909999999999998e-06 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.973233+0000", 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015910757000000001 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.973258+0000", 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4807000000000002e-05 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.981799+0000", 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085407739999999992 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.981947+0000", 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014833000000000001 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.333 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121619 9.2 e153/140 9:4bce839c:::benchmark_data_smithi086_152062_object121618:head v 153'7669, mlcod=153'7665)", 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.964034+0000", 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.8220919320000002, 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.045358066000000002, 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.964034+0000", 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.964034+0000", 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.964032+0000", 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.964091+0000", 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8118000000000002e-05 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.964111+0000", 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0458999999999998e-05 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.964119+0000", 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.208e-06 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.969573+0000", 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054538269999999996 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.969627+0000", 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4435999999999999e-05 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009290+0000", 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039662290000000003 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009392+0000", 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010206599999999999 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121623 9.2 e153/140 9:4ee31fb9:::benchmark_data_smithi086_152062_object121622:head v 153'7670, mlcod=153'7665)", 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.966989+0000", 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.819137059, 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042542185000000003, 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.338 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.966989+0000", 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.966989+0000", 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.966988+0000", 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:57:11.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.967011+0000", 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2626e-05 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.967025+0000", 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4834000000000001e-05 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.967033+0000", 2024-06-14T22:57:11.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4529999999999996e-06 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.973674+0000", 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066407289999999997 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.973726+0000", 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.291e-05 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009396+0000", 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.035669827000000001 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009531+0000", 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013452900000000001 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121628 9.2 e153/140 9:49e916b5:::benchmark_data_smithi086_152062_object121627:head v 153'7671, mlcod=153'7666)", 2024-06-14T22:57:11.342 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.972120+0000", 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.814005871, 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037516496000000003, 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.972120+0000", 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.972120+0000", 2024-06-14T22:57:11.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.972119+0000", 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.972145+0000", 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5899999999999999e-05 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.972160+0000", 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5367e-05 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.972168+0000", 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6129999999999998e-06 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.981425+0000", 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.009257454 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.981449+0000", 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4090000000000001e-05 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009533+0000", 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028084034000000001 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009636+0000", 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102954 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.346 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121630 9.2 e153/140 9:4e8b2838:::benchmark_data_smithi086_152062_object121629:head v 153'7672, mlcod=153'7666)", 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.975837+0000", 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.8102887709999997, 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033919751999999997, 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.975837+0000", 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.975837+0000", 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.975836+0000", 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.975860+0000", 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4397e-05 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.975879+0000", 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8870000000000001e-05 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.975887+0000", 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4170000000000003e-06 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.985678+0000", 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0097908119999999994 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.985703+0000", 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5103e-05 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009641+0000", 2024-06-14T22:57:11.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023938329000000001 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009757+0000", 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115668 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121631 9.2 e153/140 9:47079681:::benchmark_data_smithi086_152062_object121630:head v 153'7673, mlcod=153'7666)", 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.980458+0000", 2024-06-14T22:57:11.351 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.805667584, 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029401465000000002, 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.980458+0000", 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.980458+0000", 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.980457+0000", 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.980482+0000", 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5162999999999999e-05 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.980503+0000", 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1064999999999999e-05 2024-06-14T22:57:11.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.980515+0000", 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2121e-05 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.989095+0000", 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0085802750000000001 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.989114+0000", 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8255999999999999e-05 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009759+0000", 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.020645266999999998 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009860+0000", 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010071100000000001 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121632 9.2 e153/140 9:44baf893:::benchmark_data_smithi086_152062_object121631:head v 153'7674, mlcod=153'7666)", 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.988874+0000", 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.7972518709999998, 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021106630000000001, 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988874+0000", 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988874+0000", 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988873+0000", 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988905+0000", 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2048999999999998e-05 2024-06-14T22:57:11.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988930+0000", 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5343000000000001e-05 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988939+0000", 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6570000000000003e-06 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.993278+0000", 2024-06-14T22:57:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043390729999999997 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.993321+0000", 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3129e-05 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009862+0000", 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016540777999999999 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009980+0000", 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118659 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:121644 9.2 e153/140 9:4ed5c520:::benchmark_data_smithi086_152062_object121643:head v 153'7675, mlcod=153'7666)", 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:57:03.988948+0000", 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.7971780100000001, 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021154051, 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:57:11.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988948+0000", 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988948+0000", 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988947+0000", 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:57:11.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988969+0000", 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2164999999999999e-05 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988977+0000", 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3189999999999997e-06 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:57:11.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.988982+0000", 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8420000000000004e-06 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.997380+0000", 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083981750000000008 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:03.997400+0000", 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9888e-05 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.009983+0000", 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012582881000000001 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:57:04.010102+0000", 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119165 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:57:11.364 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:57:11.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:11.396 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:11.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:11.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:11.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:11.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:11.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:11.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:11.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:11.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:11.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:11.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:11.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:11.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:11.947 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:11.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:12.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:12.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:12.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:12.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:11 smithi086 ceph-mon[102754]: pgmap v604: 89 pgs: 89 active+clean; 7.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-06-14T22:57:12.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:11 smithi086 ceph-mon[97913]: pgmap v604: 89 pgs: 89 active+clean; 7.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-06-14T22:57:12.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 5 1 2024-06-14T22:57:12.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:12.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:11 smithi137 ceph-mon[90634]: pgmap v604: 89 pgs: 89 active+clean; 7.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-06-14T22:57:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:12.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:12.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 129739 129723 93.1739 62.375 0.0196072 0.0107272 2024-06-14T22:57:12.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-14T22:57:12.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:12.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:12.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:12.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:12.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:12.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:12.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:12.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:12.733 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:12.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:12.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:12.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:12.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:12.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:13.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:13.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:13.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:13.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:13.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:13.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:13.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:13.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:13.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:13.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:13.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 130603 130587 92.7288 54 0.00966205 0.0107785 2024-06-14T22:57:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:13.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:13.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:13.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:13.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:13.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:13.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:13.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:13.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:13.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:13.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:13.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:13.773 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T22:57:13.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:13.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:13.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:13.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:14.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:13 smithi086 ceph-mon[97913]: pgmap v605: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 61 MiB/s wr, 977 op/s 2024-06-14T22:57:14.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/217343014' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:57:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:13 smithi086 ceph-mon[102754]: pgmap v605: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 61 MiB/s wr, 977 op/s 2024-06-14T22:57:14.015 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/217343014' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:57:14.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:14.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:14.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:14.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:14.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:14.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:13 smithi137 ceph-mon[90634]: pgmap v605: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 61 MiB/s wr, 977 op/s 2024-06-14T22:57:14.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/217343014' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:57:14.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:14.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 131566 131550 92.3631 60.1875 0.0112345 0.0108219 2024-06-14T22:57:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:14.295 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:14.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:14.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:14.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:14.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:14.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:14.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:14.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:14.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:14.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:14.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:14.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:14.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:14.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:14.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:14 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:57:14.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/217343014' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:57:14.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:14 smithi086 ceph-mon[97913]: osdmap e155: 8 total, 8 up, 8 in 2024-06-14T22:57:14.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3061442558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:14.869 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:14 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:57:14.869 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/217343014' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:57:14.869 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:14 smithi086 ceph-mon[102754]: osdmap e155: 8 total, 8 up, 8 in 2024-06-14T22:57:14.870 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3061442558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:57:14.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:57:14.930 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:57:14.930 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:57:14.930 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:57:14.930 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:57:14.930 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:57:14.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:15.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:15.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:15.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:15.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:15.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:15.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:15.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:15.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:14 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:57:15.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/217343014' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:57:15.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:14 smithi137 ceph-mon[90634]: osdmap e155: 8 total, 8 up, 8 in 2024-06-14T22:57:15.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3061442558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:15.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:15.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0089 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 132479 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 91.9902 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 42.8674 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 263.938 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 54 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1471 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 685.878 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 4223 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 864 2024-06-14T22:57:15.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0108664 2024-06-14T22:57:15.268 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.0131606 2024-06-14T22:57:15.268 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 1.02178 2024-06-14T22:57:15.268 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00125959 2024-06-14T22:57:15.272 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T22:57:15.303 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:57:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:57:14] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.43.0" 2024-06-14T22:57:15.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:15.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:15.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:15.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:15.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:15.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:15.653 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:15.654 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:15.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:15.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:15.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:15.806 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:15.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:15.876 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:15 smithi086 ceph-mon[102754]: pgmap v607: 89 pgs: 89 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-14T22:57:15.876 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:15 smithi086 ceph-mon[97913]: pgmap v607: 89 pgs: 89 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-14T22:57:15.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:15.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:15.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:16.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:16.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:16.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:16.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:16.211 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:16.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:16.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:15 smithi137 ceph-mon[90634]: pgmap v607: 89 pgs: 89 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-14T22:57:16.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:16.346 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:16.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:16.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:16.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:16.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:16.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:16.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:16.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:16.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:16.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T22:57:16.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:16.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:16.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:17.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:17.053 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:17.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:17.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:17.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:17.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:17.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:17.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:17.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:17.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:17.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:17.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:17.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:17.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:17.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:57:17.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:17.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:17.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:17.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:17.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:17.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:17.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:17.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:17.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:18.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:17 smithi086 ceph-mon[97913]: pgmap v608: 89 pgs: 89 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 67 MiB/s wr, 1.09k op/s 2024-06-14T22:57:18.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:17 smithi086 ceph-mon[102754]: pgmap v608: 89 pgs: 89 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 67 MiB/s wr, 1.09k op/s 2024-06-14T22:57:18.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:18.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:18.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:18.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:17 smithi137 ceph-mon[90634]: pgmap v608: 89 pgs: 89 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 67 MiB/s wr, 1.09k op/s 2024-06-14T22:57:18.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:18.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:18.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:18.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:18.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:18.414 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:18.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:18.493 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:18.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:18.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:18.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:18.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:18.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:18.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:18.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:18.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:18.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:18.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:18.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:19.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:19.136 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:19.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:19.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:19.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:19.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:19.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:19.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:19.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:19.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:19.567 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:19.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:19.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:19.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:19.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:19.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:19.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:19.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:19.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:19.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:20.022 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:19 smithi086 ceph-mon[97913]: pgmap v609: 89 pgs: 89 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 59 MiB/s wr, 971 op/s 2024-06-14T22:57:20.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:19 smithi086 ceph-mon[102754]: pgmap v609: 89 pgs: 89 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 59 MiB/s wr, 971 op/s 2024-06-14T22:57:20.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:20.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:20.178 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:57:20.178 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:57:20.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:20.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:19 smithi137 ceph-mon[90634]: pgmap v609: 89 pgs: 89 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 59 MiB/s wr, 971 op/s 2024-06-14T22:57:20.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:20.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:20.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:20.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:20.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:20.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:20.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:20.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:20.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:20.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:20.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:20.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:20.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:20.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:20.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:20.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:20.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:20.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:21.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:21.069 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:21.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:21.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:21.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:21.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:21.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:21.336 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:21.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:21.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:21.470 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:21.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:21.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:21.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:21.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:21.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:21.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:21.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:21.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:21.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-14T22:57:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:21.879 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:21.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:21.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:22.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:22.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:22.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:21 smithi086 ceph-mon[97913]: pgmap v610: 89 pgs: 89 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 40 MiB/s wr, 969 op/s 2024-06-14T22:57:22.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:21 smithi086 ceph-mon[102754]: pgmap v610: 89 pgs: 89 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 40 MiB/s wr, 969 op/s 2024-06-14T22:57:22.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:22.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:22.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:22.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:21 smithi137 ceph-mon[90634]: pgmap v610: 89 pgs: 89 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 40 MiB/s wr, 969 op/s 2024-06-14T22:57:22.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:22.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:22.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:22.420 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:22.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:22.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:22.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:22.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:22.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:22.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:22.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:22.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:22.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:22.812 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.5 to 1 (10000) 2024-06-14T22:57:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:22.827 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:22.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:22.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/105722874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T22:57:22.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:22 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T22:57:22.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:22.966 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:22.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:23.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:23.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/105722874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T22:57:23.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:22 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T22:57:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:23.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:23.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:23.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:23.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:23.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/105722874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T22:57:23.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:22 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T22:57:23.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:23.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:23.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:23.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:23.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:23.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:23.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:23.762 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:23.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:23.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:23.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:23.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:23.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:23.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:23 smithi086 ceph-mon[102754]: pgmap v611: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 963 op/s 2024-06-14T22:57:23.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:23 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T22:57:23.999 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:23 smithi086 ceph-mon[102754]: osdmap e156: 8 total, 8 up, 8 in 2024-06-14T22:57:23.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:23 smithi086 ceph-mon[97913]: pgmap v611: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 963 op/s 2024-06-14T22:57:23.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:23 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T22:57:24.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:23 smithi086 ceph-mon[97913]: osdmap e156: 8 total, 8 up, 8 in 2024-06-14T22:57:24.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:24.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:24.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:24.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:24.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:24.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:24.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:23 smithi137 ceph-mon[90634]: pgmap v611: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 963 op/s 2024-06-14T22:57:24.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:23 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T22:57:24.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:23 smithi137 ceph-mon[90634]: osdmap e156: 8 total, 8 up, 8 in 2024-06-14T22:57:24.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:24.337 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:24.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:24.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:24.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:24.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:24.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:24.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:24.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:24.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:24.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:24.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:24.824 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:24 smithi086 ceph-mon[97913]: pgmap v613: 89 pgs: 89 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 9.5 MiB/s wr, 966 op/s 2024-06-14T22:57:24.824 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:24 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3959894831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:57:24.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:24.835 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T22:57:24.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:24.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:24.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:24.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:24.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 6 1 2024-06-14T22:57:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:24.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:24.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:25.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:24 smithi086 ceph-mon[102754]: pgmap v613: 89 pgs: 89 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 9.5 MiB/s wr, 966 op/s 2024-06-14T22:57:25.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:24 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3959894831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:57:25.084 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:57:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:57:24] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.43.0" 2024-06-14T22:57:25.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:25.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:25.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:24 smithi137 ceph-mon[90634]: pgmap v613: 89 pgs: 89 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 9.5 MiB/s wr, 966 op/s 2024-06-14T22:57:25.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:24 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3959894831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:57:25.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:25.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:25.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:25.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:25.427 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:25.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:25.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:25.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:25.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:25.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:25.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:25.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:25.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:25.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:25.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:25.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:25.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:25.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:25.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:26.049 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:26.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:26.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:25 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:57:26.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:25 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:57:26.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3959894831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:57:26.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:25 smithi086 ceph-mon[97913]: osdmap e157: 8 total, 8 up, 8 in 2024-06-14T22:57:26.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:25 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:57:26.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:25 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:57:26.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3959894831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:57:26.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:25 smithi086 ceph-mon[102754]: osdmap e157: 8 total, 8 up, 8 in 2024-06-14T22:57:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:26.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:26.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:26.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:25 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:57:26.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:25 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:57:26.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3959894831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:57:26.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:25 smithi137 ceph-mon[90634]: osdmap e157: 8 total, 8 up, 8 in 2024-06-14T22:57:26.258 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:26.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:26.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:26.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:26.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:26.552 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:26.559 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:26.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:26.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:26.703 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:26.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:26.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:26.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:26.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:26.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:26.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:26.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:27.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:27.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:26 smithi086 ceph-mon[102754]: pgmap v615: 89 pgs: 89 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 6.7 MiB/s wr, 1.22k op/s 2024-06-14T22:57:27.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:26 smithi086 ceph-mon[97913]: pgmap v615: 89 pgs: 89 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 6.7 MiB/s wr, 1.22k op/s 2024-06-14T22:57:27.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:27.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:27.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:27.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:26 smithi137 ceph-mon[90634]: pgmap v615: 89 pgs: 89 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 6.7 MiB/s wr, 1.22k op/s 2024-06-14T22:57:27.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:27.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:27.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:27.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:27.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:27.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:27.519 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:27.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:27.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:27.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:27.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:27.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:27.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:27.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:27.847 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:57:27.847 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:57:27.882 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:57:27.882 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:57:27.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:27.917 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:57:27.917 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:57:27.918 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:27.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:27.954 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:57:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:57:27.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T22:57:27.984 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:57:27.985 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:57:28.017 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:57:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:57:28.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:28.056 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:57:28.056 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:57:28.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:28.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:28.097 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:57:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:57:28.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:28.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:28.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:28.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:28.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:28.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:28.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:28.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:28.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:28.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:28.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:28.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:28.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:28.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:57:28.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:28.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2375271433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:28.740 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:57:28.740 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:28.740 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2375271433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:28.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:57:28.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:28.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2375271433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:28.774 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:28.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:28.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:28.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:28.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:29.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:29.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:29.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:29.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:29.166 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:29.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:29.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:29.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:29.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:29.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:29.441 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:29.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:29.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:29.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:29.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:29.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:29 smithi086 ceph-mon[102754]: pgmap v616: 89 pgs: 89 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-06-14T22:57:29.584 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:29 smithi086 ceph-mon[97913]: pgmap v616: 89 pgs: 89 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-06-14T22:57:29.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:29.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:29.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:29.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:29 smithi137 ceph-mon[90634]: pgmap v616: 89 pgs: 89 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-06-14T22:57:29.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:29.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:29.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:29.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:29.980 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:29.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:30.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:30.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:30.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:30.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:30.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:30.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:30.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:30.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:30.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:30.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:30.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:30.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:30.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:30.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:30.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:30.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:30.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:30.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:30.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:30.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:30.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:31.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:31.051 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:57:31.052 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T22:57:31.052 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T22:57:31.052 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T22:57:31.052 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:57:31.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:31.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:31.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:31.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:31.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:31.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:31.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:57:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:31.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:31.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:31.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:31.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:31.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:31.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:31.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:31.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:31.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:31.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:31.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:31.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:31.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:31.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:31 smithi137 ceph-mon[90634]: pgmap v617: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:57:31.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3666500965' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T22:57:31.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:31 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T22:57:32.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:32.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:32.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:31 smithi086 ceph-mon[102754]: pgmap v617: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:57:32.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3666500965' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T22:57:32.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:31 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T22:57:32.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:31 smithi086 ceph-mon[97913]: pgmap v617: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T22:57:32.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3666500965' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T22:57:32.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:31 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T22:57:32.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:32.153 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:32.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:32.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:32.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:32.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:32.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:32.424 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:32.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:32.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:32.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:32.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:32.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:32.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:32.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:32.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:32.763 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.6 to 1 (10000) 2024-06-14T22:57:32.788 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:32.816 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:32.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:32.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:32.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:32.953 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:32.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:32.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:32.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:32 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T22:57:32.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:32 smithi137 ceph-mon[90634]: osdmap e158: 8 total, 8 up, 8 in 2024-06-14T22:57:33.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:33.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:32 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T22:57:33.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:32 smithi086 ceph-mon[97913]: osdmap e158: 8 total, 8 up, 8 in 2024-06-14T22:57:33.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:32 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T22:57:33.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:32 smithi086 ceph-mon[102754]: osdmap e158: 8 total, 8 up, 8 in 2024-06-14T22:57:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:33.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:33.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:33.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:33.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:33.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:33.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:33.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:33.503 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:33.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:33.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:33.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:33.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:33.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 7 1 2024-06-14T22:57:33.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:33.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:33.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:33.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:33.961 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:33.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:33.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:33 smithi086 ceph-mon[97913]: pgmap v619: 89 pgs: 89 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-06-14T22:57:33.982 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:33 smithi086 ceph-mon[102754]: pgmap v619: 89 pgs: 89 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-06-14T22:57:33.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:33 smithi137 ceph-mon[90634]: pgmap v619: 89 pgs: 89 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 703 op/s 2024-06-14T22:57:34.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:34.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:34.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:34.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:34.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:34.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:34.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-14T22:57:34.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:34.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:34.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:34.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:34.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:34.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:34.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:34.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:34.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:34.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:34.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:34.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:34.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:34.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:34.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:34.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:35.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:35.063 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:35.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:35.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:35.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:35.084 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:57:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:57:34] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.43.0" 2024-06-14T22:57:35.084 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:35.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:35.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:35.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:35.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:35.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:35.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:35.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:35.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:35.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:35.650 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:35.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:35.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:35.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:35.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:35.823 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:35 smithi086 ceph-mon[97913]: pgmap v620: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 962 op/s 2024-06-14T22:57:35.823 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:35 smithi086 ceph-mon[102754]: pgmap v620: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 962 op/s 2024-06-14T22:57:35.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:35.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:35.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:35 smithi137 ceph-mon[90634]: pgmap v620: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 962 op/s 2024-06-14T22:57:36.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:36.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:36.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:36.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:36.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:36.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:36.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:36.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:36.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:36.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:36.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:36.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:36.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:36.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:36.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:36.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:36.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:36.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:36.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:36.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:37.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:37.050 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:37.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:37.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:37.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:37.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:37.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:37.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:37.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:37.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:37.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:37.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:37.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:37.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:37.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:37.726 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:37.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:37.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:37.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:37.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:37.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:37 smithi086 ceph-mon[97913]: pgmap v621: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-06-14T22:57:37.981 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:37 smithi086 ceph-mon[102754]: pgmap v621: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-06-14T22:57:37.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:37 smithi137 ceph-mon[90634]: pgmap v621: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-06-14T22:57:37.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:37.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:38.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:38.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:38.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:38.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:38.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:38.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:38.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:38.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:38.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:38.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:38.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:38.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:38.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:38.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:38.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:38.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:38.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:38.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:38.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:38.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:38.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:38.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:38 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1934343448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:57:38.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:38 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:57:39.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:39.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:38 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1934343448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:57:39.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:38 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:57:39.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:38 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1934343448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:57:39.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:38 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:57:39.093 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:39.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:39.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:39.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:39.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:39.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:39.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:39.370 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:39.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:39.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:39.481 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:39.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:39.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:39.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:39.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:39.717 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T22:57:39.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:39 smithi086 ceph-mon[102754]: pgmap v622: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-06-14T22:57:39.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:39 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:57:39.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:39 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:57:39.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:39 smithi086 ceph-mon[102754]: osdmap e159: 8 total, 8 up, 8 in 2024-06-14T22:57:39.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:39 smithi086 ceph-mon[97913]: pgmap v622: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-06-14T22:57:39.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:39 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:57:39.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:39 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:57:39.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:39 smithi086 ceph-mon[97913]: osdmap e159: 8 total, 8 up, 8 in 2024-06-14T22:57:39.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:39.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:39.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:39.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:39.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:39.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:39 smithi137 ceph-mon[90634]: pgmap v622: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-06-14T22:57:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:39 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:57:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:39 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:57:39.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:39 smithi137 ceph-mon[90634]: osdmap e159: 8 total, 8 up, 8 in 2024-06-14T22:57:40.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:40.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:40.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:40.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:40.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:40.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:40.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:40.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:40.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:40.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:40.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:40.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:40.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:40.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:40.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:40.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:40.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:40.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:40.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:40.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:40.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:40.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:41.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:41.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:41.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:41.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:41.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:41.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:41.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:41.238 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:57:41.239 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:57:41.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:41.273 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:57:41.274 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:57:41.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:41.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:41.325 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:57:41.325 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:57:41.360 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:57:41.360 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:57:41.393 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:57:41.393 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:57:41.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:41.422 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:57:41.422 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:57:41.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:41.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:41.470 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:57:41.470 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:57:41.512 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:57:41.512 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:57:41.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:41.588 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:41.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:41.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:41.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:41.736 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.7 to 1 (10000) 2024-06-14T22:57:41.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:41.864 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:41.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:41.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:41.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:41 smithi137 ceph-mon[90634]: pgmap v624: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 948 op/s 2024-06-14T22:57:41.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3196073533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:41.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1493947386' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T22:57:42.008 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:42.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:42.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:41 smithi086 ceph-mon[102754]: pgmap v624: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 948 op/s 2024-06-14T22:57:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3196073533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1493947386' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T22:57:42.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:41 smithi086 ceph-mon[97913]: pgmap v624: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 948 op/s 2024-06-14T22:57:42.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3196073533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:42.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1493947386' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T22:57:42.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:42.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:42.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:42.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:42.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:42.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:42.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:42.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:42.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:42.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:42.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:42.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:42.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:42.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T22:57:42.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:42.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:42.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:42.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:42.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:42.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:42.960 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:42.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:42.975 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:42 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1493947386' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T22:57:42.975 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:42 smithi086 ceph-mon[97913]: osdmap e160: 8 total, 8 up, 8 in 2024-06-14T22:57:42.975 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:57:42.975 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:42.976 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:42 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1493947386' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T22:57:42.976 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:42 smithi086 ceph-mon[102754]: osdmap e160: 8 total, 8 up, 8 in 2024-06-14T22:57:42.976 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:57:42.976 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:42.977 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-14T22:57:42.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:57:42.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:42 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1493947386' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T22:57:42.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:42 smithi137 ceph-mon[90634]: osdmap e160: 8 total, 8 up, 8 in 2024-06-14T22:57:42.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:57:42.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:43.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:43.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:43.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:43.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:43.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:43.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:43.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:43.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:43.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:43.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:43.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:43.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:43.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:43.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:43.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:43.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:43.771 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:43.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:43.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:43 smithi086 ceph-mon[102754]: pgmap v626: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 697 op/s 2024-06-14T22:57:43.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:43 smithi086 ceph-mon[97913]: pgmap v626: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 697 op/s 2024-06-14T22:57:43.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:43.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:43.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:43.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:43 smithi137 ceph-mon[90634]: pgmap v626: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 697 op/s 2024-06-14T22:57:44.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:44.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:44.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:44.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:44.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:44.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:44.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:44.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:44.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:44.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:44.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:44.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:44.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:44.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:44.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:44.751 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:44.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:44.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:44.879 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:44.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:44.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:45.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:45.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:45.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:45.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:45.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:45.150 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:57:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:57:44] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T22:57:45.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:45.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:45.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:45.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:45.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:45.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:45.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:45.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:45.622 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:57:45.623 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:57:45.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:45.694 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:45.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:45.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:45.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:45.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:45.900 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:45.908 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:45.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:45.983 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:45.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:45.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:45 smithi137 ceph-mon[90634]: pgmap v627: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:57:46.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:45 smithi086 ceph-mon[102754]: pgmap v627: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:57:46.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:45 smithi086 ceph-mon[97913]: pgmap v627: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-06-14T22:57:46.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:46.114 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:46.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:46.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:46.251 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:46.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:46.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:46.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:46.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-14T22:57:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:46.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:46.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:46.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:46.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:46.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:46.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:46.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:46.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:46.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:46.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:46.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:46.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:46.958 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:57:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:46.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:46.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:46.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:57:47.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:47.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:47.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:47.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:47.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:47.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:47.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:47.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:47.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:47.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:47.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:47.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:47.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:47.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:47.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:47.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:47.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:47.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:47.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:47.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:47.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:47.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:48.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:47 smithi086 ceph-mon[97913]: pgmap v628: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-06-14T22:57:48.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:47 smithi086 ceph-mon[102754]: pgmap v628: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-06-14T22:57:48.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:48.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:48.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:48.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:48.217 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:48.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:48.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:47 smithi137 ceph-mon[90634]: pgmap v628: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s 2024-06-14T22:57:48.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:48.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:48.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:48.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:48.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:48.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:48.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:48.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:48.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:48.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:48.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:48.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:48.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:48.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:48.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:48.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:48.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:48.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:49.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:49.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:49.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:49.045 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T22:57:49.045 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:57:49.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:49.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:49.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:49.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:49.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:49.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:49.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:49.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:49.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:49.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:49.581 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:49.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:49.593 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:57:49.596 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:57:49.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:49.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:49.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:49.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:49.844 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:49.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:49.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:49.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:49.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-14T22:57:49.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:49.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:50.012 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:49 smithi137 ceph-mon[90634]: pgmap v629: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 912 op/s 2024-06-14T22:57:50.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:50.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:49 smithi086 ceph-mon[97913]: pgmap v629: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 912 op/s 2024-06-14T22:57:50.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:49 smithi086 ceph-mon[102754]: pgmap v629: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 912 op/s 2024-06-14T22:57:50.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:50.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:50.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:50.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:50.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:50.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:50.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:50.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:50.414 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:57:50.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:50.547 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:50.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:50.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:50.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:50.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:50.798 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:50.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:50.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:50.856 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:50 smithi137 ceph-mon[90634]: from='client.35709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:57:50.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:50.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:50.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:51.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:51.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:50 smithi086 ceph-mon[97913]: from='client.35709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:57:51.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:50 smithi086 ceph-mon[102754]: from='client.35709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:57:51.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:51.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:51.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:51.227 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:51.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:51.285 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":629,"stamp":"2024-06-14T22:57:48.613449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6863787998,"num_objects":104749,"num_object_clones":0,"num_object_copies":314247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104749,"num_whiteouts":0,"num_read":487,"num_read_kb":913,"num_write":160535,"num_write_kb":8481102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138451,"ondisk_log_size":138451,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24037644,"kb_used_data":20301952,"kb_used_omap":29,"kb_used_meta":3735266,"kb_avail":725923572,"statfs":{"total":767960285184,"available":743345737728,"internally_reserved":0,"allocated":20789198848,"data_stored":20782536118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":3824913061},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-591790080,"num_objects":-9030,"num_object_clones":0,"num_object_copies":-27090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9030,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.900134"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":112,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.785025+0000","last_change":"2024-06-14T22:54:06.279751+0000","last_active":"2024-06-14T22:57:38.785025+0000","last_peered":"2024-06-14T22:57:38.785025+0000","last_clean":"2024-06-14T22:57:38.785025+0000","last_became_active":"2024-06-14T22:54:06.279443+0000","last_became_peered":"2024-06-14T22:54:06.279443+0000","last_unstale":"2024-06-14T22:57:38.785025+0000","last_undegraded":"2024-06-14T22:57:38.785025+0000","last_fullsized":"2024-06-14T22:57:38.785025+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:13:26.144924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":123,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.729725+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T22:57:38.729725+0000","last_peered":"2024-06-14T22:57:38.729725+0000","last_clean":"2024-06-14T22:57:38.729725+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T22:57:38.729725+0000","last_undegraded":"2024-06-14T22:57:38.729725+0000","last_fullsized":"2024-06-14T22:57:38.729725+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.719903+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T22:57:38.719903+0000","last_peered":"2024-06-14T22:57:38.719903+0000","last_clean":"2024-06-14T22:57:38.719903+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T22:57:38.719903+0000","last_undegraded":"2024-06-14T22:57:38.719903+0000","last_fullsized":"2024-06-14T22:57:38.719903+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":110,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722212+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T22:57:38.722212+0000","last_peered":"2024-06-14T22:57:38.722212+0000","last_clean":"2024-06-14T22:57:38.722212+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T22:57:38.722212+0000","last_undegraded":"2024-06-14T22:57:38.722212+0000","last_fullsized":"2024-06-14T22:57:38.722212+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722350+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T22:57:38.722350+0000","last_peered":"2024-06-14T22:57:38.722350+0000","last_clean":"2024-06-14T22:57:38.722350+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T22:57:38.722350+0000","last_undegraded":"2024-06-14T22:57:38.722350+0000","last_fullsized":"2024-06-14T22:57:38.722350+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731481+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T22:57:38.731481+0000","last_peered":"2024-06-14T22:57:38.731481+0000","last_clean":"2024-06-14T22:57:38.731481+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T22:57:38.731481+0000","last_undegraded":"2024-06-14T22:57:38.731481+0000","last_fullsized":"2024-06-14T22:57:38.731481+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":112,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.785707+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T22:57:38.785707+0000","last_peered":"2024-06-14T22:57:38.785707+0000","last_clean":"2024-06-14T22:57:38.785707+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T22:57:38.785707+0000","last_undegraded":"2024-06-14T22:57:38.785707+0000","last_fullsized":"2024-06-14T22:57:38.785707+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":129,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.788947+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T22:57:38.788947+0000","last_peered":"2024-06-14T22:57:38.788947+0000","last_clean":"2024-06-14T22:57:38.788947+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T22:57:38.788947+0000","last_undegraded":"2024-06-14T22:57:38.788947+0000","last_fullsized":"2024-06-14T22:57:38.788947+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"160'10416","reported_seq":10445,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:46.479415+0000","last_change":"2024-06-14T22:55:38.730745+0000","last_active":"2024-06-14T22:57:46.479415+0000","last_peered":"2024-06-14T22:57:46.479415+0000","last_clean":"2024-06-14T22:57:46.479415+0000","last_became_active":"2024-06-14T22:55:38.730320+0000","last_became_peered":"2024-06-14T22:55:38.730320+0000","last_unstale":"2024-06-14T22:57:46.479415+0000","last_undegraded":"2024-06-14T22:57:46.479415+0000","last_fullsized":"2024-06-14T22:57:46.479415+0000","mapping_epoch":140,"log_start":"144'400","ondisk_log_start":"144'400","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:29.533721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430833664,"num_objects":6574,"num_object_clones":0,"num_object_copies":19722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10416,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":38,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.732075+0000","last_change":"2024-06-14T22:55:53.663882+0000","last_active":"2024-06-14T22:57:38.732075+0000","last_peered":"2024-06-14T22:57:38.732075+0000","last_clean":"2024-06-14T22:57:38.732075+0000","last_became_active":"2024-06-14T22:55:53.663509+0000","last_became_peered":"2024-06-14T22:55:53.663509+0000","last_unstale":"2024-06-14T22:57:38.732075+0000","last_undegraded":"2024-06-14T22:57:38.732075+0000","last_fullsized":"2024-06-14T22:57:38.732075+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:36.354907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":103,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.719569+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:57:38.719569+0000","last_peered":"2024-06-14T22:57:38.719569+0000","last_clean":"2024-06-14T22:57:38.719569+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:57:38.719569+0000","last_undegraded":"2024-06-14T22:57:38.719569+0000","last_fullsized":"2024-06-14T22:57:38.719569+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"160'9866","reported_seq":9896,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.609726+0000","last_change":"2024-06-14T22:55:38.730175+0000","last_active":"2024-06-14T22:57:44.609726+0000","last_peered":"2024-06-14T22:57:44.609726+0000","last_clean":"2024-06-14T22:57:44.609726+0000","last_became_active":"2024-06-14T22:55:38.729339+0000","last_became_peered":"2024-06-14T22:55:38.729339+0000","last_unstale":"2024-06-14T22:57:44.609726+0000","last_undegraded":"2024-06-14T22:57:44.609726+0000","last_fullsized":"2024-06-14T22:57:44.609726+0000","mapping_epoch":140,"log_start":"146'2500","ondisk_log_start":"146'2500","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:40:50.307712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424542208,"num_objects":6478,"num_object_clones":0,"num_object_copies":19434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9866,"num_write_kb":523008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":38,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.734352+0000","last_change":"2024-06-14T22:55:53.664736+0000","last_active":"2024-06-14T22:57:38.734352+0000","last_peered":"2024-06-14T22:57:38.734352+0000","last_clean":"2024-06-14T22:57:38.734352+0000","last_became_active":"2024-06-14T22:55:53.664141+0000","last_became_peered":"2024-06-14T22:55:53.664141+0000","last_unstale":"2024-06-14T22:57:38.734352+0000","last_undegraded":"2024-06-14T22:57:38.734352+0000","last_fullsized":"2024-06-14T22:57:38.734352+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:52.824254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":137,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731545+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T22:57:38.731545+0000","last_peered":"2024-06-14T22:57:38.731545+0000","last_clean":"2024-06-14T22:57:38.731545+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T22:57:38.731545+0000","last_undegraded":"2024-06-14T22:57:38.731545+0000","last_fullsized":"2024-06-14T22:57:38.731545+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"160'9942","reported_seq":9968,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:43.936461+0000","last_change":"2024-06-14T22:55:38.730856+0000","last_active":"2024-06-14T22:57:43.936461+0000","last_peered":"2024-06-14T22:57:43.936461+0000","last_clean":"2024-06-14T22:57:43.936461+0000","last_became_active":"2024-06-14T22:55:38.730542+0000","last_became_peered":"2024-06-14T22:55:38.730542+0000","last_unstale":"2024-06-14T22:57:43.936461+0000","last_undegraded":"2024-06-14T22:57:43.936461+0000","last_fullsized":"2024-06-14T22:57:43.936461+0000","mapping_epoch":140,"log_start":"144'200","ondisk_log_start":"144'200","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:16:51.828869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426246144,"num_objects":6504,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9942,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":38,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.719803+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T22:57:38.719803+0000","last_peered":"2024-06-14T22:57:38.719803+0000","last_clean":"2024-06-14T22:57:38.719803+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T22:57:38.719803+0000","last_undegraded":"2024-06-14T22:57:38.719803+0000","last_fullsized":"2024-06-14T22:57:38.719803+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.786900+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T22:57:38.786900+0000","last_peered":"2024-06-14T22:57:38.786900+0000","last_clean":"2024-06-14T22:57:38.786900+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T22:57:38.786900+0000","last_undegraded":"2024-06-14T22:57:38.786900+0000","last_fullsized":"2024-06-14T22:57:38.786900+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"160'9954","reported_seq":9982,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:43.933433+0000","last_change":"2024-06-14T22:55:39.130045+0000","last_active":"2024-06-14T22:57:43.933433+0000","last_peered":"2024-06-14T22:57:43.933433+0000","last_clean":"2024-06-14T22:57:43.933433+0000","last_became_active":"2024-06-14T22:55:39.129733+0000","last_became_peered":"2024-06-14T22:55:39.129733+0000","last_unstale":"2024-06-14T22:57:43.933433+0000","last_undegraded":"2024-06-14T22:57:43.933433+0000","last_fullsized":"2024-06-14T22:57:43.933433+0000","mapping_epoch":140,"log_start":"144'200","ondisk_log_start":"144'200","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:34:14.279180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434765824,"num_objects":6634,"num_object_clones":0,"num_object_copies":19902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9954,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.719703+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T22:57:38.719703+0000","last_peered":"2024-06-14T22:57:38.719703+0000","last_clean":"2024-06-14T22:57:38.719703+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T22:57:38.719703+0000","last_undegraded":"2024-06-14T22:57:38.719703+0000","last_fullsized":"2024-06-14T22:57:38.719703+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.782795+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T22:57:38.782795+0000","last_peered":"2024-06-14T22:57:38.782795+0000","last_clean":"2024-06-14T22:57:38.782795+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T22:57:38.782795+0000","last_undegraded":"2024-06-14T22:57:38.782795+0000","last_fullsized":"2024-06-14T22:57:38.782795+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"160'9888","reported_seq":9917,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.257626+0000","last_change":"2024-06-14T22:55:38.974910+0000","last_active":"2024-06-14T22:57:44.257626+0000","last_peered":"2024-06-14T22:57:44.257626+0000","last_clean":"2024-06-14T22:57:44.257626+0000","last_became_active":"2024-06-14T22:55:38.974574+0000","last_became_peered":"2024-06-14T22:55:38.974574+0000","last_unstale":"2024-06-14T22:57:44.257626+0000","last_undegraded":"2024-06-14T22:57:44.257626+0000","last_fullsized":"2024-06-14T22:57:44.257626+0000","mapping_epoch":140,"log_start":"146'2100","ondisk_log_start":"146'2100","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:48:06.634805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428867584,"num_objects":6544,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":525824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":132,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.784845+0000","last_change":"2024-06-14T22:54:06.280994+0000","last_active":"2024-06-14T22:57:38.784845+0000","last_peered":"2024-06-14T22:57:38.784845+0000","last_clean":"2024-06-14T22:57:38.784845+0000","last_became_active":"2024-06-14T22:54:06.279883+0000","last_became_peered":"2024-06-14T22:54:06.279883+0000","last_unstale":"2024-06-14T22:57:38.784845+0000","last_undegraded":"2024-06-14T22:57:38.784845+0000","last_fullsized":"2024-06-14T22:57:38.784845+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:51.156630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.732014+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T22:57:38.732014+0000","last_peered":"2024-06-14T22:57:38.732014+0000","last_clean":"2024-06-14T22:57:38.732014+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T22:57:38.732014+0000","last_undegraded":"2024-06-14T22:57:38.732014+0000","last_fullsized":"2024-06-14T22:57:38.732014+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"160'10146","reported_seq":10176,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.596050+0000","last_change":"2024-06-14T22:55:39.129787+0000","last_active":"2024-06-14T22:57:44.596050+0000","last_peered":"2024-06-14T22:57:44.596050+0000","last_clean":"2024-06-14T22:57:44.596050+0000","last_became_active":"2024-06-14T22:55:39.129528+0000","last_became_peered":"2024-06-14T22:55:39.129528+0000","last_unstale":"2024-06-14T22:57:44.596050+0000","last_undegraded":"2024-06-14T22:57:44.596050+0000","last_fullsized":"2024-06-14T22:57:44.596050+0000","mapping_epoch":140,"log_start":"146'2800","ondisk_log_start":"146'2800","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7346,"log_dups_size":0,"ondisk_log_size":7346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:24:05.362187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431226880,"num_objects":6580,"num_object_clones":0,"num_object_copies":19740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10146,"num_write_kb":535232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.787007+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T22:57:38.787007+0000","last_peered":"2024-06-14T22:57:38.787007+0000","last_clean":"2024-06-14T22:57:38.787007+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T22:57:38.787007+0000","last_undegraded":"2024-06-14T22:57:38.787007+0000","last_fullsized":"2024-06-14T22:57:38.787007+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":130,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.732126+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T22:57:38.732126+0000","last_peered":"2024-06-14T22:57:38.732126+0000","last_clean":"2024-06-14T22:57:38.732126+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T22:57:38.732126+0000","last_undegraded":"2024-06-14T22:57:38.732126+0000","last_fullsized":"2024-06-14T22:57:38.732126+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"160'9916","reported_seq":9951,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.237113+0000","last_change":"2024-06-14T22:55:53.668083+0000","last_active":"2024-06-14T22:57:44.237113+0000","last_peered":"2024-06-14T22:57:44.237113+0000","last_clean":"2024-06-14T22:57:44.237113+0000","last_became_active":"2024-06-14T22:55:53.664556+0000","last_became_peered":"2024-06-14T22:55:53.664556+0000","last_unstale":"2024-06-14T22:57:44.237113+0000","last_undegraded":"2024-06-14T22:57:44.237113+0000","last_fullsized":"2024-06-14T22:57:44.237113+0000","mapping_epoch":145,"log_start":"146'2200","ondisk_log_start":"146'2200","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:50:22.930473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430440448,"num_objects":6568,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9916,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731571+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T22:57:38.731571+0000","last_peered":"2024-06-14T22:57:38.731571+0000","last_clean":"2024-06-14T22:57:38.731571+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T22:57:38.731571+0000","last_undegraded":"2024-06-14T22:57:38.731571+0000","last_fullsized":"2024-06-14T22:57:38.731571+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.734368+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T22:57:38.734368+0000","last_peered":"2024-06-14T22:57:38.734368+0000","last_clean":"2024-06-14T22:57:38.734368+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T22:57:38.734368+0000","last_undegraded":"2024-06-14T22:57:38.734368+0000","last_fullsized":"2024-06-14T22:57:38.734368+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"160'10088","reported_seq":10118,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.253401+0000","last_change":"2024-06-14T22:55:39.129746+0000","last_active":"2024-06-14T22:57:44.253401+0000","last_peered":"2024-06-14T22:57:44.253401+0000","last_clean":"2024-06-14T22:57:44.253401+0000","last_became_active":"2024-06-14T22:55:39.129442+0000","last_became_peered":"2024-06-14T22:55:39.129442+0000","last_unstale":"2024-06-14T22:57:44.253401+0000","last_undegraded":"2024-06-14T22:57:44.253401+0000","last_fullsized":"2024-06-14T22:57:44.253401+0000","mapping_epoch":140,"log_start":"146'2300","ondisk_log_start":"146'2300","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:36:21.422229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435945472,"num_objects":6652,"num_object_clones":0,"num_object_copies":19956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10088,"num_write_kb":535680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":113,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.784582+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T22:57:38.784582+0000","last_peered":"2024-06-14T22:57:38.784582+0000","last_clean":"2024-06-14T22:57:38.784582+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T22:57:38.784582+0000","last_undegraded":"2024-06-14T22:57:38.784582+0000","last_fullsized":"2024-06-14T22:57:38.784582+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.785454+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T22:57:38.785454+0000","last_peered":"2024-06-14T22:57:38.785454+0000","last_clean":"2024-06-14T22:57:38.785454+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T22:57:38.785454+0000","last_undegraded":"2024-06-14T22:57:38.785454+0000","last_fullsized":"2024-06-14T22:57:38.785454+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"160'9803","reported_seq":9832,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.240935+0000","last_change":"2024-06-14T22:55:38.729850+0000","last_active":"2024-06-14T22:57:44.240935+0000","last_peered":"2024-06-14T22:57:44.240935+0000","last_clean":"2024-06-14T22:57:44.240935+0000","last_became_active":"2024-06-14T22:55:38.729536+0000","last_became_peered":"2024-06-14T22:55:38.729536+0000","last_unstale":"2024-06-14T22:57:44.240935+0000","last_undegraded":"2024-06-14T22:57:44.240935+0000","last_fullsized":"2024-06-14T22:57:44.240935+0000","mapping_epoch":140,"log_start":"146'2100","ondisk_log_start":"146'2100","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:47:35.019362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427098112,"num_objects":6517,"num_object_clones":0,"num_object_copies":19551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9803,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":425,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722354+0000","last_change":"2024-06-14T22:55:53.665022+0000","last_active":"2024-06-14T22:57:38.722354+0000","last_peered":"2024-06-14T22:57:38.722354+0000","last_clean":"2024-06-14T22:57:38.722354+0000","last_became_active":"2024-06-14T22:55:53.664025+0000","last_became_peered":"2024-06-14T22:55:53.664025+0000","last_unstale":"2024-06-14T22:57:38.722354+0000","last_undegraded":"2024-06-14T22:57:38.722354+0000","last_fullsized":"2024-06-14T22:57:38.722354+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.106147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.721853+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T22:57:38.721853+0000","last_peered":"2024-06-14T22:57:38.721853+0000","last_clean":"2024-06-14T22:57:38.721853+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T22:57:38.721853+0000","last_undegraded":"2024-06-14T22:57:38.721853+0000","last_fullsized":"2024-06-14T22:57:38.721853+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":86,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.784391+0000","last_change":"2024-06-14T22:54:06.279205+0000","last_active":"2024-06-14T22:57:38.784391+0000","last_peered":"2024-06-14T22:57:38.784391+0000","last_clean":"2024-06-14T22:57:38.784391+0000","last_became_active":"2024-06-14T22:54:06.278792+0000","last_became_peered":"2024-06-14T22:54:06.278792+0000","last_unstale":"2024-06-14T22:57:38.784391+0000","last_undegraded":"2024-06-14T22:57:38.784391+0000","last_fullsized":"2024-06-14T22:57:38.784391+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:55:02.899941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":103,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.787176+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:57:38.787176+0000","last_peered":"2024-06-14T22:57:38.787176+0000","last_clean":"2024-06-14T22:57:38.787176+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:57:38.787176+0000","last_undegraded":"2024-06-14T22:57:38.787176+0000","last_fullsized":"2024-06-14T22:57:38.787176+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"160'10258","reported_seq":10286,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:46.450966+0000","last_change":"2024-06-14T22:55:38.974930+0000","last_active":"2024-06-14T22:57:46.450966+0000","last_peered":"2024-06-14T22:57:46.450966+0000","last_clean":"2024-06-14T22:57:46.450966+0000","last_became_active":"2024-06-14T22:55:38.974625+0000","last_became_peered":"2024-06-14T22:55:38.974625+0000","last_unstale":"2024-06-14T22:57:46.450966+0000","last_undegraded":"2024-06-14T22:57:46.450966+0000","last_fullsized":"2024-06-14T22:57:46.450966+0000","mapping_epoch":140,"log_start":"144'200","ondisk_log_start":"144'200","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:44:40.336597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423624704,"num_objects":6464,"num_object_clones":0,"num_object_copies":19392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":535104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":116,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.782306+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T22:57:38.782306+0000","last_peered":"2024-06-14T22:57:38.782306+0000","last_clean":"2024-06-14T22:57:38.782306+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T22:57:38.782306+0000","last_undegraded":"2024-06-14T22:57:38.782306+0000","last_fullsized":"2024-06-14T22:57:38.782306+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.782993+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T22:57:38.782993+0000","last_peered":"2024-06-14T22:57:38.782993+0000","last_clean":"2024-06-14T22:57:38.782993+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T22:57:38.782993+0000","last_undegraded":"2024-06-14T22:57:38.782993+0000","last_fullsized":"2024-06-14T22:57:38.782993+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":124,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722487+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T22:57:38.722487+0000","last_peered":"2024-06-14T22:57:38.722487+0000","last_clean":"2024-06-14T22:57:38.722487+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T22:57:38.722487+0000","last_undegraded":"2024-06-14T22:57:38.722487+0000","last_fullsized":"2024-06-14T22:57:38.722487+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"160'10100","reported_seq":10128,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:43.921276+0000","last_change":"2024-06-14T22:55:38.731601+0000","last_active":"2024-06-14T22:57:43.921276+0000","last_peered":"2024-06-14T22:57:43.921276+0000","last_clean":"2024-06-14T22:57:43.921276+0000","last_became_active":"2024-06-14T22:55:38.731200+0000","last_became_peered":"2024-06-14T22:55:38.731200+0000","last_unstale":"2024-06-14T22:57:43.921276+0000","last_undegraded":"2024-06-14T22:57:43.921276+0000","last_fullsized":"2024-06-14T22:57:43.921276+0000","mapping_epoch":140,"log_start":"144'400","ondisk_log_start":"144'400","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9700,"log_dups_size":0,"ondisk_log_size":9700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:53:02.395452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429260800,"num_objects":6550,"num_object_clones":0,"num_object_copies":19650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":532800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":112,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:40.127003+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T22:57:40.127003+0000","last_peered":"2024-06-14T22:57:40.127003+0000","last_clean":"2024-06-14T22:57:40.127003+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T22:57:40.127003+0000","last_undegraded":"2024-06-14T22:57:40.127003+0000","last_fullsized":"2024-06-14T22:57:40.127003+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":127,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.782925+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T22:57:38.782925+0000","last_peered":"2024-06-14T22:57:38.782925+0000","last_clean":"2024-06-14T22:57:38.782925+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T22:57:38.782925+0000","last_undegraded":"2024-06-14T22:57:38.782925+0000","last_fullsized":"2024-06-14T22:57:38.782925+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":111,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.790197+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T22:57:38.790197+0000","last_peered":"2024-06-14T22:57:38.790197+0000","last_clean":"2024-06-14T22:57:38.790197+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T22:57:38.790197+0000","last_undegraded":"2024-06-14T22:57:38.790197+0000","last_fullsized":"2024-06-14T22:57:38.790197+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"160'10021","reported_seq":10050,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:46.455636+0000","last_change":"2024-06-14T22:55:38.729565+0000","last_active":"2024-06-14T22:57:46.455636+0000","last_peered":"2024-06-14T22:57:46.455636+0000","last_clean":"2024-06-14T22:57:46.455636+0000","last_became_active":"2024-06-14T22:55:38.729284+0000","last_became_peered":"2024-06-14T22:55:38.729284+0000","last_unstale":"2024-06-14T22:57:46.455636+0000","last_undegraded":"2024-06-14T22:57:46.455636+0000","last_fullsized":"2024-06-14T22:57:46.455636+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:15.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422248448,"num_objects":6443,"num_object_clones":0,"num_object_copies":19329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.720407+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T22:57:38.720407+0000","last_peered":"2024-06-14T22:57:38.720407+0000","last_clean":"2024-06-14T22:57:38.720407+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T22:57:38.720407+0000","last_undegraded":"2024-06-14T22:57:38.720407+0000","last_fullsized":"2024-06-14T22:57:38.720407+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731818+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T22:57:38.731818+0000","last_peered":"2024-06-14T22:57:38.731818+0000","last_clean":"2024-06-14T22:57:38.731818+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T22:57:38.731818+0000","last_undegraded":"2024-06-14T22:57:38.731818+0000","last_fullsized":"2024-06-14T22:57:38.731818+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":142,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739615+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T22:57:41.739615+0000","last_peered":"2024-06-14T22:57:41.739615+0000","last_clean":"2024-06-14T22:57:41.739615+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T22:57:41.739615+0000","last_undegraded":"2024-06-14T22:57:41.739615+0000","last_fullsized":"2024-06-14T22:57:41.739615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"160'9956","reported_seq":9984,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:43.917606+0000","last_change":"2024-06-14T22:55:38.729989+0000","last_active":"2024-06-14T22:57:43.917606+0000","last_peered":"2024-06-14T22:57:43.917606+0000","last_clean":"2024-06-14T22:57:43.917606+0000","last_became_active":"2024-06-14T22:55:38.729695+0000","last_became_peered":"2024-06-14T22:55:38.729695+0000","last_unstale":"2024-06-14T22:57:43.917606+0000","last_undegraded":"2024-06-14T22:57:43.917606+0000","last_fullsized":"2024-06-14T22:57:43.917606+0000","mapping_epoch":140,"log_start":"144'200","ondisk_log_start":"144'200","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9756,"log_dups_size":0,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:44:38.218727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430309376,"num_objects":6566,"num_object_clones":0,"num_object_copies":19698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9956,"num_write_kb":528704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":138,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.720126+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T22:57:38.720126+0000","last_peered":"2024-06-14T22:57:38.720126+0000","last_clean":"2024-06-14T22:57:38.720126+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T22:57:38.720126+0000","last_undegraded":"2024-06-14T22:57:38.720126+0000","last_fullsized":"2024-06-14T22:57:38.720126+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":117,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739407+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T22:57:41.739407+0000","last_peered":"2024-06-14T22:57:41.739407+0000","last_clean":"2024-06-14T22:57:41.739407+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T22:57:41.739407+0000","last_undegraded":"2024-06-14T22:57:41.739407+0000","last_fullsized":"2024-06-14T22:57:41.739407+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":112,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739781+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T22:57:41.739781+0000","last_peered":"2024-06-14T22:57:41.739781+0000","last_clean":"2024-06-14T22:57:41.739781+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T22:57:41.739781+0000","last_undegraded":"2024-06-14T22:57:41.739781+0000","last_fullsized":"2024-06-14T22:57:41.739781+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"160'9941","reported_seq":9978,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:43.607846+0000","last_change":"2024-06-14T22:55:53.672630+0000","last_active":"2024-06-14T22:57:43.607846+0000","last_peered":"2024-06-14T22:57:43.607846+0000","last_clean":"2024-06-14T22:57:43.607846+0000","last_became_active":"2024-06-14T22:55:53.663693+0000","last_became_peered":"2024-06-14T22:55:53.663693+0000","last_unstale":"2024-06-14T22:57:43.607846+0000","last_undegraded":"2024-06-14T22:57:43.607846+0000","last_fullsized":"2024-06-14T22:57:43.607846+0000","mapping_epoch":145,"log_start":"146'1600","ondisk_log_start":"146'1600","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":8341,"log_dups_size":0,"ondisk_log_size":8341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:20:34.289185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432734208,"num_objects":6603,"num_object_clones":0,"num_object_copies":19809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9941,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":127,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739343+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T22:57:41.739343+0000","last_peered":"2024-06-14T22:57:41.739343+0000","last_clean":"2024-06-14T22:57:41.739343+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T22:57:41.739343+0000","last_undegraded":"2024-06-14T22:57:41.739343+0000","last_fullsized":"2024-06-14T22:57:41.739343+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.729647+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T22:57:38.729647+0000","last_peered":"2024-06-14T22:57:38.729647+0000","last_clean":"2024-06-14T22:57:38.729647+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T22:57:38.729647+0000","last_undegraded":"2024-06-14T22:57:38.729647+0000","last_fullsized":"2024-06-14T22:57:38.729647+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":97,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722669+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T22:57:38.722669+0000","last_peered":"2024-06-14T22:57:38.722669+0000","last_clean":"2024-06-14T22:57:38.722669+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T22:57:38.722669+0000","last_undegraded":"2024-06-14T22:57:38.722669+0000","last_fullsized":"2024-06-14T22:57:38.722669+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"160'10057","reported_seq":10094,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.558300+0000","last_change":"2024-06-14T22:55:53.668144+0000","last_active":"2024-06-14T22:57:44.558300+0000","last_peered":"2024-06-14T22:57:44.558300+0000","last_clean":"2024-06-14T22:57:44.558300+0000","last_became_active":"2024-06-14T22:55:53.663690+0000","last_became_peered":"2024-06-14T22:55:53.663690+0000","last_unstale":"2024-06-14T22:57:44.558300+0000","last_undegraded":"2024-06-14T22:57:44.558300+0000","last_fullsized":"2024-06-14T22:57:44.558300+0000","mapping_epoch":145,"log_start":"146'2700","ondisk_log_start":"146'2700","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7357,"log_dups_size":0,"ondisk_log_size":7357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:52:03.686394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427163672,"num_objects":6519,"num_object_clones":0,"num_object_copies":19557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6519,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10057,"num_write_kb":530369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":132,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.785896+0000","last_change":"2024-06-14T22:54:06.284651+0000","last_active":"2024-06-14T22:57:38.785896+0000","last_peered":"2024-06-14T22:57:38.785896+0000","last_clean":"2024-06-14T22:57:38.785896+0000","last_became_active":"2024-06-14T22:54:06.283564+0000","last_became_peered":"2024-06-14T22:54:06.283564+0000","last_unstale":"2024-06-14T22:57:38.785896+0000","last_undegraded":"2024-06-14T22:57:38.785896+0000","last_fullsized":"2024-06-14T22:57:38.785896+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:34.916272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":110,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.787999+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T22:57:38.787999+0000","last_peered":"2024-06-14T22:57:38.787999+0000","last_clean":"2024-06-14T22:57:38.787999+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T22:57:38.787999+0000","last_undegraded":"2024-06-14T22:57:38.787999+0000","last_fullsized":"2024-06-14T22:57:38.787999+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":129,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.784340+0000","last_change":"2024-06-14T22:54:06.281419+0000","last_active":"2024-06-14T22:57:38.784340+0000","last_peered":"2024-06-14T22:57:38.784340+0000","last_clean":"2024-06-14T22:57:38.784340+0000","last_became_active":"2024-06-14T22:54:06.279482+0000","last_became_peered":"2024-06-14T22:54:06.279482+0000","last_unstale":"2024-06-14T22:57:38.784340+0000","last_undegraded":"2024-06-14T22:57:38.784340+0000","last_fullsized":"2024-06-14T22:57:38.784340+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:06:37.039006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"160'9883","reported_seq":9911,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.224395+0000","last_change":"2024-06-14T22:55:38.729250+0000","last_active":"2024-06-14T22:57:44.224395+0000","last_peered":"2024-06-14T22:57:44.224395+0000","last_clean":"2024-06-14T22:57:44.224395+0000","last_became_active":"2024-06-14T22:55:38.728938+0000","last_became_peered":"2024-06-14T22:55:38.728938+0000","last_unstale":"2024-06-14T22:57:44.224395+0000","last_undegraded":"2024-06-14T22:57:44.224395+0000","last_fullsized":"2024-06-14T22:57:44.224395+0000","mapping_epoch":140,"log_start":"146'2100","ondisk_log_start":"146'2100","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:32:07.069919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427884544,"num_objects":6529,"num_object_clones":0,"num_object_copies":19587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9883,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":114,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722330+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T22:57:38.722330+0000","last_peered":"2024-06-14T22:57:38.722330+0000","last_clean":"2024-06-14T22:57:38.722330+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T22:57:38.722330+0000","last_undegraded":"2024-06-14T22:57:38.722330+0000","last_fullsized":"2024-06-14T22:57:38.722330+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.787262+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T22:57:38.787262+0000","last_peered":"2024-06-14T22:57:38.787262+0000","last_clean":"2024-06-14T22:57:38.787262+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T22:57:38.787262+0000","last_undegraded":"2024-06-14T22:57:38.787262+0000","last_fullsized":"2024-06-14T22:57:38.787262+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":103,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731454+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:57:38.731454+0000","last_peered":"2024-06-14T22:57:38.731454+0000","last_clean":"2024-06-14T22:57:38.731454+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:57:38.731454+0000","last_undegraded":"2024-06-14T22:57:38.731454+0000","last_fullsized":"2024-06-14T22:57:38.731454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731798+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T22:57:38.731798+0000","last_peered":"2024-06-14T22:57:38.731798+0000","last_clean":"2024-06-14T22:57:38.731798+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T22:57:38.731798+0000","last_undegraded":"2024-06-14T22:57:38.731798+0000","last_fullsized":"2024-06-14T22:57:38.731798+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.732097+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:57:38.732097+0000","last_peered":"2024-06-14T22:57:38.732097+0000","last_clean":"2024-06-14T22:57:38.732097+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:57:38.732097+0000","last_undegraded":"2024-06-14T22:57:38.732097+0000","last_fullsized":"2024-06-14T22:57:38.732097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.783101+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T22:57:38.783101+0000","last_peered":"2024-06-14T22:57:38.783101+0000","last_clean":"2024-06-14T22:57:38.783101+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T22:57:38.783101+0000","last_undegraded":"2024-06-14T22:57:38.783101+0000","last_fullsized":"2024-06-14T22:57:38.783101+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":37,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.785249+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T22:57:38.785249+0000","last_peered":"2024-06-14T22:57:38.785249+0000","last_clean":"2024-06-14T22:57:38.785249+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T22:57:38.785249+0000","last_undegraded":"2024-06-14T22:57:38.785249+0000","last_fullsized":"2024-06-14T22:57:38.785249+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":121,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.720085+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T22:57:38.720085+0000","last_peered":"2024-06-14T22:57:38.720085+0000","last_clean":"2024-06-14T22:57:38.720085+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T22:57:38.720085+0000","last_undegraded":"2024-06-14T22:57:38.720085+0000","last_fullsized":"2024-06-14T22:57:38.720085+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":86,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.784061+0000","last_change":"2024-06-14T22:54:06.279332+0000","last_active":"2024-06-14T22:57:38.784061+0000","last_peered":"2024-06-14T22:57:38.784061+0000","last_clean":"2024-06-14T22:57:38.784061+0000","last_became_active":"2024-06-14T22:54:06.279020+0000","last_became_peered":"2024-06-14T22:54:06.279020+0000","last_unstale":"2024-06-14T22:57:38.784061+0000","last_undegraded":"2024-06-14T22:57:38.784061+0000","last_fullsized":"2024-06-14T22:57:38.784061+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:52:57.777668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":110,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722430+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T22:57:38.722430+0000","last_peered":"2024-06-14T22:57:38.722430+0000","last_clean":"2024-06-14T22:57:38.722430+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T22:57:38.722430+0000","last_undegraded":"2024-06-14T22:57:38.722430+0000","last_fullsized":"2024-06-14T22:57:38.722430+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":112,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.784260+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T22:57:38.784260+0000","last_peered":"2024-06-14T22:57:38.784260+0000","last_clean":"2024-06-14T22:57:38.784260+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T22:57:38.784260+0000","last_undegraded":"2024-06-14T22:57:38.784260+0000","last_fullsized":"2024-06-14T22:57:38.784260+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.787202+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T22:57:38.787202+0000","last_peered":"2024-06-14T22:57:38.787202+0000","last_clean":"2024-06-14T22:57:38.787202+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T22:57:38.787202+0000","last_undegraded":"2024-06-14T22:57:38.787202+0000","last_fullsized":"2024-06-14T22:57:38.787202+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":130,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.784505+0000","last_change":"2024-06-14T22:54:06.282120+0000","last_active":"2024-06-14T22:57:38.784505+0000","last_peered":"2024-06-14T22:57:38.784505+0000","last_clean":"2024-06-14T22:57:38.784505+0000","last_became_active":"2024-06-14T22:54:06.280898+0000","last_became_peered":"2024-06-14T22:54:06.280898+0000","last_unstale":"2024-06-14T22:57:38.784505+0000","last_undegraded":"2024-06-14T22:57:38.784505+0000","last_fullsized":"2024-06-14T22:57:38.784505+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:30:45.544457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":112,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722405+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T22:57:38.722405+0000","last_peered":"2024-06-14T22:57:38.722405+0000","last_clean":"2024-06-14T22:57:38.722405+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T22:57:38.722405+0000","last_undegraded":"2024-06-14T22:57:38.722405+0000","last_fullsized":"2024-06-14T22:57:38.722405+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.782590+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T22:57:38.782590+0000","last_peered":"2024-06-14T22:57:38.782590+0000","last_clean":"2024-06-14T22:57:38.782590+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T22:57:38.782590+0000","last_undegraded":"2024-06-14T22:57:38.782590+0000","last_fullsized":"2024-06-14T22:57:38.782590+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":138,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.784223+0000","last_change":"2024-06-14T22:54:06.280483+0000","last_active":"2024-06-14T22:57:38.784223+0000","last_peered":"2024-06-14T22:57:38.784223+0000","last_clean":"2024-06-14T22:57:38.784223+0000","last_became_active":"2024-06-14T22:54:06.278985+0000","last_became_peered":"2024-06-14T22:54:06.278985+0000","last_unstale":"2024-06-14T22:57:38.784223+0000","last_undegraded":"2024-06-14T22:57:38.784223+0000","last_fullsized":"2024-06-14T22:57:38.784223+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:43:40.478580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":139,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731703+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T22:57:38.731703+0000","last_peered":"2024-06-14T22:57:38.731703+0000","last_clean":"2024-06-14T22:57:38.731703+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T22:57:38.731703+0000","last_undegraded":"2024-06-14T22:57:38.731703+0000","last_fullsized":"2024-06-14T22:57:38.731703+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":131,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.711766+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T22:57:38.711766+0000","last_peered":"2024-06-14T22:57:38.711766+0000","last_clean":"2024-06-14T22:57:38.711766+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T22:57:38.711766+0000","last_undegraded":"2024-06-14T22:57:38.711766+0000","last_fullsized":"2024-06-14T22:57:38.711766+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":99,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722640+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T22:57:38.722640+0000","last_peered":"2024-06-14T22:57:38.722640+0000","last_clean":"2024-06-14T22:57:38.722640+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T22:57:38.722640+0000","last_undegraded":"2024-06-14T22:57:38.722640+0000","last_fullsized":"2024-06-14T22:57:38.722640+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":137,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731775+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T22:57:38.731775+0000","last_peered":"2024-06-14T22:57:38.731775+0000","last_clean":"2024-06-14T22:57:38.731775+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T22:57:38.731775+0000","last_undegraded":"2024-06-14T22:57:38.731775+0000","last_fullsized":"2024-06-14T22:57:38.731775+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":123,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.719522+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T22:57:38.719522+0000","last_peered":"2024-06-14T22:57:38.719522+0000","last_clean":"2024-06-14T22:57:38.719522+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T22:57:38.719522+0000","last_undegraded":"2024-06-14T22:57:38.719522+0000","last_fullsized":"2024-06-14T22:57:38.719522+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":137,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731653+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T22:57:38.731653+0000","last_peered":"2024-06-14T22:57:38.731653+0000","last_clean":"2024-06-14T22:57:38.731653+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T22:57:38.731653+0000","last_undegraded":"2024-06-14T22:57:38.731653+0000","last_fullsized":"2024-06-14T22:57:38.731653+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":106,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.720047+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:57:38.720047+0000","last_peered":"2024-06-14T22:57:38.720047+0000","last_clean":"2024-06-14T22:57:38.720047+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:57:38.720047+0000","last_undegraded":"2024-06-14T22:57:38.720047+0000","last_fullsized":"2024-06-14T22:57:38.720047+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":145,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.784361+0000","last_change":"2024-06-14T22:54:06.282972+0000","last_active":"2024-06-14T22:57:38.784361+0000","last_peered":"2024-06-14T22:57:38.784361+0000","last_clean":"2024-06-14T22:57:38.784361+0000","last_became_active":"2024-06-14T22:54:06.282622+0000","last_became_peered":"2024-06-14T22:54:06.282622+0000","last_unstale":"2024-06-14T22:57:38.784361+0000","last_undegraded":"2024-06-14T22:57:38.784361+0000","last_fullsized":"2024-06-14T22:57:38.784361+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:31:22.051955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":130,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.785105+0000","last_change":"2024-06-14T22:54:06.279284+0000","last_active":"2024-06-14T22:57:38.785105+0000","last_peered":"2024-06-14T22:57:38.785105+0000","last_clean":"2024-06-14T22:57:38.785105+0000","last_became_active":"2024-06-14T22:54:06.278861+0000","last_became_peered":"2024-06-14T22:54:06.278861+0000","last_unstale":"2024-06-14T22:57:38.785105+0000","last_undegraded":"2024-06-14T22:57:38.785105+0000","last_fullsized":"2024-06-14T22:57:38.785105+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:17:47.260769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":120,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.731363+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T22:57:38.731363+0000","last_peered":"2024-06-14T22:57:38.731363+0000","last_clean":"2024-06-14T22:57:38.731363+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T22:57:38.731363+0000","last_undegraded":"2024-06-14T22:57:38.731363+0000","last_fullsized":"2024-06-14T22:57:38.731363+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":115,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-06-14T22:57:38.722380+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T22:57:38.722380+0000","last_peered":"2024-06-14T22:57:38.722380+0000","last_clean":"2024-06-14T22:57:38.722380+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T22:57:38.722380+0000","last_undegraded":"2024-06-14T22:57:38.722380+0000","last_fullsized":"2024-06-14T22:57:38.722380+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6863192088,"num_objects":104725,"num_object_clones":0,"num_object_copies":314175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160235,"num_write_kb":8478657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20641099776,"data_stored":20641087560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138235,"ondisk_log_size":138235,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234284,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4400812,"kb_used_data":3815072,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":89344340,"statfs":{"total":95995035648,"available":91488604160,"internally_reserved":0,"allocated":3906633728,"data_stored":3905780035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:10 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":0.86299999999999999,"15min":1.103},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":4.0330000000000004,"5min":4.0330000000000004,"15min":9.3919999999999995},"last":3.2160000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.80600000000000005,"15min":1.2130000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.6190000000000002,"5min":4.4649999999999999,"15min":10.416},"last":2.4460000000000002}]},{"osd":1,"last update":"Fri Jun 14 22:57:40 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":2.3660000000000001,"15min":1.772},"min":{"1min":0.38600000000000001,"5min":0.28899999999999998,"15min":0.28100000000000003},"max":{"1min":4.0629999999999997,"5min":83.295000000000002,"15min":83.295000000000002},"last":2.395},{"interface":"front","average":{"1min":1.5149999999999999,"5min":2.3330000000000002,"15min":1.843},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.223},"max":{"1min":3.9820000000000002,"5min":82.972999999999999,"15min":82.972999999999999},"last":3.2469999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:57:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.76900000000000002,"15min":0.64400000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.0540000000000003,"5min":4.0540000000000003,"15min":4.0540000000000003},"last":3.2679999999999998},{"interface":"front","average":{"1min":1.4370000000000001,"5min":0.876,"15min":0.67300000000000004},"min":{"1min":0.33400000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":8.1150000000000002,"5min":8.1150000000000002,"15min":8.1150000000000002},"last":3.2890000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:57:07 2024","interfaces":[{"interface":"back","average":{"1min":3.8399999999999999,"5min":1.246,"15min":0.77300000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":55.683999999999997,"5min":55.683999999999997,"15min":55.683999999999997},"last":2.4849999999999999},{"interface":"front","average":{"1min":4.2060000000000004,"5min":1.359,"15min":0.85199999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":55.533999999999999,"5min":55.533999999999999,"15min":55.533999999999999},"last":2.3639999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.70099999999999996,"15min":0.65100000000000002},"min":{"1min":0.31,"5min":0.245,"15min":0.214},"max":{"1min":2.7240000000000002,"5min":5.8449999999999998,"15min":9.7430000000000003},"last":6.7610000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.70099999999999996,"15min":0.64500000000000002},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":3.3050000000000002,"5min":4.2709999999999999,"15min":6.6660000000000004},"last":2.3839999999999999}]},{"osd":5,"last update":"Fri Jun 14 22:56:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":0.72899999999999998,"15min":0.65100000000000002},"min":{"1min":0.30099999999999999,"5min":0.26100000000000001,"15min":0.22},"max":{"1min":3.9140000000000001,"5min":3.9140000000000001,"15min":3.9860000000000002},"last":3.3069999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.86299999999999999,"15min":0.77300000000000002},"min":{"1min":0.379,"5min":0.31900000000000001,"15min":0.28699999999999998},"max":{"1min":5.0030000000000001,"5min":5.3659999999999997,"15min":5.3659999999999997},"last":6.8529999999999998}]},{"osd":6,"last update":"Fri Jun 14 22:56:53 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":0.80800000000000005,"15min":0.72899999999999998},"min":{"1min":0.32900000000000001,"5min":0.31900000000000001,"15min":0.28100000000000003},"max":{"1min":2.7530000000000001,"5min":5.806,"15min":5.806},"last":2.4199999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.751,"15min":0.70299999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29899999999999999},"max":{"1min":3.355,"5min":4.3090000000000002,"15min":5.7430000000000003},"last":2.0720000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495920,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1866388,"kb_used_data":1280652,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91878764,"statfs":{"total":95995035648,"available":94083854336,"internally_reserved":0,"allocated":1311387648,"data_stored":1310553942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:10 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":0.96199999999999997,"15min":1.1930000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":6.0549999999999997,"5min":7.1689999999999996,"15min":31.358000000000001},"last":0.748},{"interface":"front","average":{"1min":1.2809999999999999,"5min":0.79100000000000004,"15min":1.3620000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":29.399000000000001},"last":1.3360000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":0.89900000000000002,"15min":1.2310000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.9269999999999996,"5min":4.9269999999999996,"15min":9.9559999999999995},"last":1.897},{"interface":"front","average":{"1min":1.2529999999999999,"5min":0.92300000000000004,"15min":1.2070000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.247},"max":{"1min":3.9790000000000001,"5min":5.2960000000000003,"15min":9.5510000000000002},"last":1.4279999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":0.72199999999999998,"15min":0.623},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.5190000000000001,"5min":4.5190000000000001,"15min":4.5190000000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.087,"5min":0.747,"15min":0.65300000000000002},"min":{"1min":0.33500000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.4580000000000002,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":0.38700000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:57:10 2024","interfaces":[{"interface":"back","average":{"1min":2.9020000000000001,"5min":1.042,"15min":0.68200000000000005},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":29.152000000000001,"5min":29.152000000000001,"15min":29.152000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":2.734,"5min":1.089,"15min":0.746},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":25.864000000000001,"5min":25.864000000000001,"15min":25.864000000000001},"last":0.372}]},{"osd":4,"last update":"Fri Jun 14 22:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.76800000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":2.7149999999999999,"5min":5.327,"15min":47.698},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53000000000000003,"15min":0.74099999999999999},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.23899999999999999},"max":{"1min":1.8939999999999999,"5min":2.2010000000000001,"15min":48.197000000000003},"last":0.33600000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.751,"15min":0.69799999999999995},"min":{"1min":0.308,"5min":0.308,"15min":0.245},"max":{"1min":2.3159999999999998,"5min":8.6500000000000004,"15min":8.6500000000000004},"last":0.318},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73199999999999998,"15min":0.73699999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.6480000000000001,"5min":8.6349999999999998,"15min":8.6349999999999998},"last":0.60399999999999998}]},{"osd":7,"last update":"Fri Jun 14 22:56:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.72199999999999998,"15min":0.71999999999999997},"min":{"1min":0.33000000000000002,"5min":0.314,"15min":0.28799999999999998},"max":{"1min":0.95599999999999996,"5min":8.6950000000000003,"15min":8.6950000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.86499999999999999,"15min":0.78000000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.28899999999999998},"max":{"1min":2.9510000000000001,"5min":8.673,"15min":8.673},"last":0.5}]}]},{"osd":5,"up_from":38,"seq":163208757559,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4711372,"kb_used_data":3824324,"kb_used_omap":1,"kb_used_meta":887038,"kb_avail":89033780,"statfs":{"total":95995035648,"available":91170590720,"internally_reserved":0,"allocated":3916107776,"data_stored":3915277556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908327256},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.92900000000000005,"15min":1.194},"min":{"1min":0.376,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":4.3470000000000004,"5min":6.0369999999999999,"15min":17.766999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.3380000000000001,"5min":0.94199999999999995,"15min":1.224},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":4.3659999999999997,"5min":4.4429999999999996,"15min":17.806000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Jun 14 22:57:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.92300000000000004,"15min":1.274},"min":{"1min":0.41899999999999998,"5min":0.24199999999999999,"15min":0.23300000000000001},"max":{"1min":2.8889999999999998,"5min":3.7759999999999998,"15min":7.694},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.8759999999999999,"5min":1.069,"15min":1.1719999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":3.9470000000000001,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":0.64700000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.77400000000000002,"15min":0.622},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.3620000000000001,"5min":5.9379999999999997,"15min":5.9379999999999997},"last":0.502},{"interface":"front","average":{"1min":1.079,"5min":0.73399999999999999,"15min":0.624},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.677,"5min":3.677,"15min":3.677},"last":0.75}]},{"osd":3,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":4.274,"5min":1.335,"15min":0.79700000000000004},"min":{"1min":0.39200000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":45.744,"5min":45.744,"15min":45.744},"last":0.441},{"interface":"front","average":{"1min":4.4210000000000003,"5min":1.4370000000000001,"15min":0.90500000000000003},"min":{"1min":0.48199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":45.838000000000001,"5min":45.838000000000001,"15min":45.838000000000001},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.59899999999999998,"15min":0.56299999999999994},"min":{"1min":0.26300000000000001,"5min":0.24399999999999999,"15min":0.214},"max":{"1min":3.3039999999999998,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":1.077},{"interface":"front","average":{"1min":0.622,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.335,"5min":3.8900000000000001,"15min":6.0149999999999997},"last":0.378}]},{"osd":6,"last update":"Fri Jun 14 22:57:29 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.64300000000000002,"15min":0.72999999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":3.1389999999999998,"5min":4.7370000000000001,"15min":9.6020000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.71899999999999997},"min":{"1min":0.318,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":1.7789999999999999,"5min":5.2329999999999997,"15min":12.339},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Jun 14 22:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.76700000000000002,"15min":0.71999999999999997},"min":{"1min":0.39300000000000002,"5min":0.31900000000000001,"15min":0.27900000000000003},"max":{"1min":3.0539999999999998,"5min":5.2629999999999999,"15min":5.2629999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.77700000000000002,"15min":0.72399999999999998},"min":{"1min":0.31900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.3399999999999999,"5min":10.638999999999999,"15min":10.638999999999999},"last":1.1439999999999999}]}]},{"osd":4,"up_from":30,"seq":128849019195,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1869720,"kb_used_data":1271632,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":91875432,"statfs":{"total":95995035648,"available":94080442368,"internally_reserved":0,"allocated":1302151168,"data_stored":1301312956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73699999999999999,"15min":1.002},"min":{"1min":0.216,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.9950000000000001,"5min":4.359,"15min":7.5650000000000004},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.82699999999999996,"15min":1.2050000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":3.718,"5min":5.6130000000000004,"15min":9.1470000000000002},"last":0.85899999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":0.83499999999999996,"15min":1.0920000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.915,"5min":4.8250000000000002,"15min":5.141},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.7949999999999999,"5min":1.0600000000000001,"15min":1.274},"min":{"1min":0.39900000000000002,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":6.5549999999999997,"5min":6.5549999999999997,"15min":6.5549999999999997},"last":0.77600000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.59999999999999998,"15min":0.57699999999999996},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.0179999999999998,"5min":4.0179999999999998,"15min":4.0179999999999998},"last":0.307},{"interface":"front","average":{"1min":1.143,"5min":0.70399999999999996,"15min":0.61799999999999999},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.0490000000000004,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":1.081}]},{"osd":3,"last update":"Fri Jun 14 22:57:15 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":0.78000000000000003,"15min":0.623},"min":{"1min":0.30199999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.9329999999999998,"5min":3.9329999999999998,"15min":3.9329999999999998},"last":0.879},{"interface":"front","average":{"1min":1.4179999999999999,"5min":0.80700000000000005,"15min":0.66500000000000004},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.9580000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":1.115}]},{"osd":5,"last update":"Fri Jun 14 22:56:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.73899999999999999,"15min":0.68999999999999995},"min":{"1min":0.44400000000000001,"5min":0.24199999999999999,"15min":0.21199999999999999},"max":{"1min":2.3010000000000002,"5min":3.9289999999999998,"15min":5.7030000000000003},"last":0.371},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.67500000000000004,"15min":0.65800000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":1.7529999999999999,"5min":4.0369999999999999,"15min":4.7480000000000002},"last":0.80200000000000005}]},{"osd":6,"last update":"Fri Jun 14 22:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.60099999999999998,"15min":0.63500000000000001},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.254},"max":{"1min":1.1539999999999999,"5min":3.8929999999999998,"15min":4.5960000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.53800000000000003,"15min":0.61099999999999999},"min":{"1min":0.28899999999999998,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":1.0680000000000001,"5min":1.647,"15min":5.0960000000000001},"last":0.34000000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.63900000000000001,"15min":0.65200000000000002},"min":{"1min":0.377,"5min":0.316,"15min":0.246},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":5.0389999999999997},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77700000000000002,"15min":0.68600000000000005},"min":{"1min":0.308,"5min":0.308,"15min":0.29399999999999998},"max":{"1min":2.7069999999999999,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":0.91400000000000003}]}]},{"osd":3,"up_from":138,"seq":592705486894,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3518348,"kb_used_data":3366080,"kb_used_omap":4,"kb_used_meta":151995,"kb_avail":90226804,"statfs":{"total":95995035648,"available":92392247296,"internally_reserved":0,"allocated":3446865920,"data_stored":3446039545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":155643358},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.56399999999999995,"15min":0.40400000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":2.9660000000000002,"5min":3.4580000000000002,"15min":3.4580000000000002},"last":3.2290000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.51200000000000001,"15min":0.40899999999999997},"min":{"1min":0.24399999999999999,"5min":0.185,"15min":0.185},"max":{"1min":1.2490000000000001,"5min":3.484,"15min":3.484},"last":2.5009999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":0.64800000000000002,"15min":0.45200000000000001},"min":{"1min":0.48299999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":2.4209999999999998},{"interface":"front","average":{"1min":1.2030000000000001,"5min":0.626,"15min":0.47099999999999997},"min":{"1min":0.36499999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.6890000000000001,"5min":3.6890000000000001,"15min":3.6890000000000001},"last":2.4820000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.56599999999999995,"15min":0.46200000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":2.4529999999999998},{"interface":"front","average":{"1min":1.224,"5min":0.59099999999999997,"15min":0.435},"min":{"1min":0.35499999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.8070000000000004,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":1.3500000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":0.78400000000000003,"15min":0.68200000000000005},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.139,"5min":0.78600000000000003,"15min":0.68200000000000005},"min":{"1min":0.39800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.2720000000000002,"5min":5.2720000000000002,"15min":5.2720000000000002},"last":3.952}]},{"osd":5,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":0.85799999999999998,"15min":0.71099999999999997},"min":{"1min":0.44600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.3559999999999999,"5min":5.3559999999999999,"15min":5.3559999999999999},"last":3.919},{"interface":"front","average":{"1min":1.161,"5min":0.77800000000000002,"15min":0.66700000000000004},"min":{"1min":0.46200000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":2.7530000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":0.77400000000000002,"15min":0.68700000000000006},"min":{"1min":0.46300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.536,"5min":2.536,"15min":2.536},"last":2.726},{"interface":"front","average":{"1min":1.1839999999999999,"5min":0.78100000000000003,"15min":0.66800000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.8259999999999996,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.49299999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":0.94799999999999995,"15min":0.75},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":6.0369999999999999,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.591,"5min":0.89200000000000002,"15min":0.71899999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":3.9849999999999999}]}]},{"osd":2,"up_from":133,"seq":571230650421,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1353736,"kb_used_data":1270656,"kb_used_omap":6,"kb_used_meta":83065,"kb_avail":92391416,"statfs":{"total":95995035648,"available":94608809984,"internally_reserved":0,"allocated":1301151744,"data_stored":1300325583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":85059061},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.53300000000000003,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.587,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.46100000000000002,"15min":0.39400000000000002},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.448,"5min":2.448,"15min":2.448},"last":0.19700000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":0.96199999999999997,"15min":0.57399999999999995},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":5.5149999999999997,"5min":21.920000000000002,"15min":21.920000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.417,"5min":0.96199999999999997,"15min":0.56100000000000005},"min":{"1min":0.27300000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.4509999999999996,"5min":21.573,"15min":21.573},"last":0.41999999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":0.71699999999999997,"15min":0.48099999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":21.66,"5min":21.66,"15min":21.66},"last":0.315},{"interface":"front","average":{"1min":2.145,"5min":0.91000000000000003,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.252},"max":{"1min":22.341000000000001,"5min":22.341000000000001,"15min":22.341000000000001},"last":0.33900000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.69199999999999995,"15min":0.62},"min":{"1min":0.32300000000000001,"5min":0.318,"15min":0.318},"max":{"1min":4.2149999999999999,"5min":4.2149999999999999,"15min":4.2149999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.68799999999999994,"15min":0.61499999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":4.1890000000000001,"5min":4.1890000000000001,"15min":4.1890000000000001},"last":0.57499999999999996}]},{"osd":5,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.64200000000000002,"15min":0.58699999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.649,"5min":3.8700000000000001,"15min":3.8700000000000001},"last":3.137},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.60099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.4710000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.55600000000000005}]},{"osd":6,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.60599999999999998,"15min":0.57799999999999996},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.8320000000000001,"5min":3.915,"15min":3.915},"last":3.0209999999999999},{"interface":"front","average":{"1min":1.163,"5min":0.71199999999999997,"15min":0.622},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":5.1829999999999998,"5min":5.1829999999999998,"15min":5.1829999999999998},"last":0.44600000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.73699999999999999,"15min":0.64500000000000002},"min":{"1min":0.42799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":4.3259999999999996},"last":2.5819999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.66000000000000003,"15min":0.60999999999999999},"min":{"1min":0.45500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.056,"5min":3.9529999999999998,"15min":3.9529999999999998},"last":1.095}]}]},{"osd":1,"up_from":116,"seq":498216206438,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4177356,"kb_used_data":3803328,"kb_used_omap":5,"kb_used_meta":374010,"kb_avail":89567796,"statfs":{"total":95995035648,"available":91717423104,"internally_reserved":0,"allocated":3894607872,"data_stored":3893781431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":382986520},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":0.97899999999999998,"15min":2.0339999999999998},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.3860000000000001,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":1.6319999999999999},{"interface":"front","average":{"1min":1.512,"5min":0.79300000000000004,"15min":1.5509999999999999},"min":{"1min":0.248,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.0220000000000002,"5min":5.1390000000000002,"15min":10.426},"last":0.374}]},{"osd":2,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":0.78100000000000003,"15min":0.48399999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":5.3330000000000002,"5min":8.2880000000000003,"15min":8.2880000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.458,"5min":0.66300000000000003,"15min":0.44500000000000001},"min":{"1min":0.29799999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":5.077,"5min":5.077,"15min":5.077},"last":0.41599999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:57:06 2024","interfaces":[{"interface":"back","average":{"1min":3.359,"5min":1.0209999999999999,"15min":0.56100000000000005},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":21.422000000000001,"5min":21.422000000000001,"15min":21.422000000000001},"last":2.8010000000000002},{"interface":"front","average":{"1min":2.9020000000000001,"5min":1.02,"15min":0.58599999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":21.016999999999999,"5min":21.016999999999999,"15min":21.016999999999999},"last":2.8410000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.0860000000000001,"15min":1.8069999999999999},"min":{"1min":0.309,"5min":0.28899999999999998,"15min":0.28399999999999997},"max":{"1min":4.3639999999999999,"5min":7.673,"15min":8.8599999999999994},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.696,"5min":1.1200000000000001,"15min":1.79},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.3570000000000002,"5min":8.3620000000000001,"15min":11.161},"last":0.77100000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.0569999999999999,"15min":1.6819999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.30599999999999999},"max":{"1min":5.3399999999999999,"5min":6.9020000000000001,"15min":9.7539999999999996},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.694,"5min":1.117,"15min":1.869},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.4160000000000004,"5min":7.7439999999999998,"15min":8.125},"last":0.64600000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.0820000000000001,"15min":1.7290000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.32100000000000001},"max":{"1min":5.3849999999999998,"5min":6.9290000000000003,"15min":9.7279999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.081,"15min":1.8360000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.899,"5min":7.7060000000000004,"15min":8.1479999999999997},"last":0.48399999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.0720000000000001,"15min":1.873},"min":{"1min":0.435,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":5.4080000000000004,"5min":8.3369999999999997,"15min":9.0960000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.496,"5min":1.0429999999999999,"15min":1.5720000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":9.5449999999999999},"last":0.66100000000000003}]}]},{"osd":0,"up_from":91,"seq":390842024093,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2139912,"kb_used_data":1670208,"kb_used_omap":5,"kb_used_meta":469690,"kb_avail":91605240,"statfs":{"total":95995035648,"available":93803765760,"internally_reserved":0,"allocated":1710292992,"data_stored":1709465070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":480962650},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.2450000000000001,"15min":1.5549999999999999},"min":{"1min":0.188,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7709999999999999,"5min":16.305,"15min":16.305},"last":1.718},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.486,"15min":2.0950000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.9649999999999999,"5min":16.184999999999999,"15min":16.184999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:57:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":0.79900000000000004,"15min":0.505},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.7679999999999998,"5min":16.757000000000001,"15min":16.757000000000001},"last":4.1230000000000002},{"interface":"front","average":{"1min":1.4830000000000001,"5min":0.85399999999999998,"15min":0.55000000000000004},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":4.7350000000000003,"5min":16.055,"15min":16.055},"last":4.1539999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:57:09 2024","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.81999999999999995,"15min":0.51600000000000001},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.8090000000000002,"5min":16.873999999999999,"15min":16.873999999999999},"last":1.98},{"interface":"front","average":{"1min":1.373,"5min":0.76400000000000001,"15min":0.49399999999999999},"min":{"1min":0.27300000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.9969999999999999,"5min":16.085999999999999,"15min":16.085999999999999},"last":5.2329999999999997}]},{"osd":4,"last update":"Fri Jun 14 22:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.1970000000000001,"15min":1.514},"min":{"1min":0.308,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":3.2610000000000001,"5min":16.018000000000001,"15min":16.018000000000001},"last":2.0920000000000001},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.117,"15min":1.3109999999999999},"min":{"1min":0.36099999999999999,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":2.8700000000000001,"5min":15.976000000000001,"15min":15.976000000000001},"last":0.69699999999999995}]},{"osd":5,"last update":"Fri Jun 14 22:57:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.335,"15min":1.407},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29299999999999998},"max":{"1min":7.1269999999999998,"5min":16.108000000000001,"15min":16.108000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.284,"15min":1.478},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":3.9239999999999999,"5min":16.006,"15min":16.006},"last":4.2030000000000003}]},{"osd":6,"last update":"Fri Jun 14 22:57:19 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1639999999999999,"15min":1.387},"min":{"1min":0.38800000000000001,"5min":0.35699999999999998,"15min":0.34300000000000003},"max":{"1min":2.895,"5min":16.152000000000001,"15min":16.152000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.179,"15min":1.4990000000000001},"min":{"1min":0.40500000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.4790000000000001,"5min":16.227,"15min":16.227},"last":4.2240000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:57:19 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.22,"15min":1.528},"min":{"1min":0.503,"5min":0.39000000000000001,"15min":0.33100000000000002},"max":{"1min":3.9550000000000001,"5min":16.199999999999999,"15min":16.199999999999999},"last":4.1820000000000004},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2669999999999999,"15min":1.625},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.314},"max":{"1min":3.3039999999999998,"5min":16.259,"15min":16.259},"last":2.1349999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1695350784,"data_stored":1695350784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3861975040,"data_stored":3861970968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1290665984,"data_stored":1290665984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3430809600,"data_stored":3430809600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1291386880,"data_stored":1291386880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3884453888,"data_stored":3884449816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1302003712,"data_stored":1302003712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3884453888,"data_stored":3884449816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:57:51.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:57:51.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:51.362 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:51.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:51.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:51.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:51.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:51.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:51.624 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:51.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:51.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:51.776 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T22:57:51.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:51.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:51.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:51 smithi086 ceph-mon[97913]: pgmap v630: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 924 op/s 2024-06-14T22:57:51.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3895752263' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:57:51.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:57:51.799 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:51 smithi086 ceph-mon[102754]: pgmap v630: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 924 op/s 2024-06-14T22:57:51.799 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3895752263' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:57:51.799 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:57:51.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:51.914 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:51.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:51.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:51.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:51 smithi137 ceph-mon[90634]: pgmap v630: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 924 op/s 2024-06-14T22:57:51.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3895752263' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:57:51.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:57:52.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:52.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:52.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:52.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:52.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:52.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:52.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:52.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:52.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:52.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:52.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:52.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:52.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:52.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:52.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:52.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:52.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:52.754 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T22:57:52.755 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:57:52.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:52.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:52.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:52.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:52.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:52 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:57:52.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:52 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:57:52.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:52 smithi137 ceph-mon[90634]: osdmap e161: 8 total, 8 up, 8 in 2024-06-14T22:57:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:53.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:53.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:52 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:57:53.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:52 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:57:53.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:52 smithi086 ceph-mon[97913]: osdmap e161: 8 total, 8 up, 8 in 2024-06-14T22:57:53.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:52 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:57:53.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:52 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:57:53.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:52 smithi086 ceph-mon[102754]: osdmap e161: 8 total, 8 up, 8 in 2024-06-14T22:57:53.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:53.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:53.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:53.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:53.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:53.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:53.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:53.390 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:53.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:53.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-14T22:57:53.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:53.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:53.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:53.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:53.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:53.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:53.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:53.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:53.885 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:57:53.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:53.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:53.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:54.041 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:53 smithi137 ceph-mon[90634]: pgmap v632: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 922 op/s 2024-06-14T22:57:54.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:53 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:57:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:54.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:54.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:54.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:53 smithi086 ceph-mon[97913]: pgmap v632: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 922 op/s 2024-06-14T22:57:54.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:53 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:57:54.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:53 smithi086 ceph-mon[102754]: pgmap v632: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 922 op/s 2024-06-14T22:57:54.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:53 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:57:54.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:54.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:54.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:54.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:54.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:54.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:54.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:54.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:54.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:54.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:54.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:54.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:54.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:54.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:54.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:54.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:54.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T22:57:54.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:54.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:54.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:55.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:55.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:55.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:55.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:55.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:55.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:55.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:55.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:55.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:57:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:57:54] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T22:57:55.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:55.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:55.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:55.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:55.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:55.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:55.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:55.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:55.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:55.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:55.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:55.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:55.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:55.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:55 smithi137 ceph-mon[90634]: pgmap v633: 89 pgs: 89 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 826 op/s 2024-06-14T22:57:56.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:56.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:56.063 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:55 smithi086 ceph-mon[97913]: pgmap v633: 89 pgs: 89 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 826 op/s 2024-06-14T22:57:56.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:55 smithi086 ceph-mon[102754]: pgmap v633: 89 pgs: 89 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 826 op/s 2024-06-14T22:57:56.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:56.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:56.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:56.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:56.222 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:57:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:57:56.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:56.252 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:57:56.252 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:57:56.280 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:57:56.281 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:57:56.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:56.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:56.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:56.311 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:57:56.311 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:57:56.341 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T22:57:56.341 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:57:56.344 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:57:56.345 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:57:56.378 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:57:56.378 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:57:56.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:56.409 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:57:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:57:56.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:56.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:56.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:56.442 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:57:56.442 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:57:56.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:56.551 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:56.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:56.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:56.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:56.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:56.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:56.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:56.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:56.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:56.958 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:56.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:56.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2942092278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:57.044 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2942092278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:57.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2942092278' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:57:57.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:57.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:57.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:57.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:57.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:57.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:57.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:57.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:57.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:57.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:57.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:57.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:57.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:57.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:57.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:57.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:57.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:57.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:57.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:57.809 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:57:57.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:57.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:57.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:58.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:58.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:58.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:58.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:57 smithi086 ceph-mon[97913]: pgmap v634: 89 pgs: 89 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 905 op/s 2024-06-14T22:57:58.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:58.069 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:57 smithi086 ceph-mon[102754]: pgmap v634: 89 pgs: 89 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 905 op/s 2024-06-14T22:57:58.069 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:58.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:57:58.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:58.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:57:58.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:57 smithi137 ceph-mon[90634]: pgmap v634: 89 pgs: 89 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 905 op/s 2024-06-14T22:57:58.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:57:58.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:58.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:58.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:58.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:57:58.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:58.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:57:58.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:58.585 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:58.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:58.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:57:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:58.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:58.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:57:58.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:57:58.876 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:57:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:58.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:58.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:57:58.878 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:57:58.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:59.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:59.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:59.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T22:57:59.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:59.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:59.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:59.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:57:59.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:59.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:59.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:57:59.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:59.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:59.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:59.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:57:59.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:59.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:59.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:57:59.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:57:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:59.758 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:57:59.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:57:59.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:57:59.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:57:59.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:57:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:57:59.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:57:59.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:00.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:00.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:00.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:00.046 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:58:00.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:57:59 smithi086 ceph-mon[102754]: pgmap v635: 89 pgs: 89 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 905 op/s 2024-06-14T22:58:00.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:57:59 smithi086 ceph-mon[97913]: pgmap v635: 89 pgs: 89 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 905 op/s 2024-06-14T22:58:00.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:00.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:00.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:00.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:00.198 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:57:59 smithi137 ceph-mon[90634]: pgmap v635: 89 pgs: 89 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 905 op/s 2024-06-14T22:58:00.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:00.291 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:00.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:00.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:00.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:00.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:00.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:00.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:00.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:00.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:00.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:00.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:00.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:00.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:00.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:00.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:00.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:01.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:01.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":635,"stamp":"2024-06-14T22:57:58.620720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6270228446,"num_objects":95692,"num_object_clones":0,"num_object_copies":287076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95692,"num_whiteouts":0,"num_read":487,"num_read_kb":913,"num_write":169592,"num_write_kb":8481102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138508,"ondisk_log_size":138508,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22637552,"kb_used_data":18558496,"kb_used_omap":29,"kb_used_meta":4078818,"kb_avail":727323664,"statfs":{"total":767960285184,"available":744779431936,"internally_reserved":0,"allocated":19003899904,"data_stored":18997205542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":4176710309},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-593559552,"num_objects":-9057,"num_object_clones":0,"num_object_copies":-27171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9057,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007271"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":113,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.744388+0000","last_change":"2024-06-14T22:54:06.279751+0000","last_active":"2024-06-14T22:57:41.744388+0000","last_peered":"2024-06-14T22:57:41.744388+0000","last_clean":"2024-06-14T22:57:41.744388+0000","last_became_active":"2024-06-14T22:54:06.279443+0000","last_became_peered":"2024-06-14T22:54:06.279443+0000","last_unstale":"2024-06-14T22:57:41.744388+0000","last_undegraded":"2024-06-14T22:57:41.744388+0000","last_fullsized":"2024-06-14T22:57:41.744388+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:13:26.144924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":124,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.744106+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T22:57:41.744106+0000","last_peered":"2024-06-14T22:57:41.744106+0000","last_clean":"2024-06-14T22:57:41.744106+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T22:57:41.744106+0000","last_undegraded":"2024-06-14T22:57:41.744106+0000","last_fullsized":"2024-06-14T22:57:41.744106+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.746705+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T22:57:41.746705+0000","last_peered":"2024-06-14T22:57:41.746705+0000","last_clean":"2024-06-14T22:57:41.746705+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T22:57:41.746705+0000","last_undegraded":"2024-06-14T22:57:41.746705+0000","last_fullsized":"2024-06-14T22:57:41.746705+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":111,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.745143+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T22:57:41.745143+0000","last_peered":"2024-06-14T22:57:41.745143+0000","last_clean":"2024-06-14T22:57:41.745143+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T22:57:41.745143+0000","last_undegraded":"2024-06-14T22:57:41.745143+0000","last_fullsized":"2024-06-14T22:57:41.745143+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":112,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.745323+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T22:57:41.745323+0000","last_peered":"2024-06-14T22:57:41.745323+0000","last_clean":"2024-06-14T22:57:41.745323+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T22:57:41.745323+0000","last_undegraded":"2024-06-14T22:57:41.745323+0000","last_fullsized":"2024-06-14T22:57:41.745323+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":28,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.736910+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T22:57:41.736910+0000","last_peered":"2024-06-14T22:57:41.736910+0000","last_clean":"2024-06-14T22:57:41.736910+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T22:57:41.736910+0000","last_undegraded":"2024-06-14T22:57:41.736910+0000","last_fullsized":"2024-06-14T22:57:41.736910+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":113,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.742991+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T22:57:41.742991+0000","last_peered":"2024-06-14T22:57:41.742991+0000","last_clean":"2024-06-14T22:57:41.742991+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T22:57:41.742991+0000","last_undegraded":"2024-06-14T22:57:41.742991+0000","last_fullsized":"2024-06-14T22:57:41.742991+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":130,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.742005+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T22:57:41.742005+0000","last_peered":"2024-06-14T22:57:41.742005+0000","last_clean":"2024-06-14T22:57:41.742005+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T22:57:41.742005+0000","last_undegraded":"2024-06-14T22:57:41.742005+0000","last_fullsized":"2024-06-14T22:57:41.742005+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"161'11002","reported_seq":11033,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:56.471859+0000","last_change":"2024-06-14T22:55:38.730745+0000","last_active":"2024-06-14T22:57:56.471859+0000","last_peered":"2024-06-14T22:57:56.471859+0000","last_clean":"2024-06-14T22:57:56.471859+0000","last_became_active":"2024-06-14T22:55:38.730320+0000","last_became_peered":"2024-06-14T22:55:38.730320+0000","last_unstale":"2024-06-14T22:57:56.471859+0000","last_undegraded":"2024-06-14T22:57:56.471859+0000","last_fullsized":"2024-06-14T22:57:56.471859+0000","mapping_epoch":140,"log_start":"144'1000","ondisk_log_start":"144'1000","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:29.533721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392429568,"num_objects":5988,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":39,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737687+0000","last_change":"2024-06-14T22:55:53.663882+0000","last_active":"2024-06-14T22:57:41.737687+0000","last_peered":"2024-06-14T22:57:41.737687+0000","last_clean":"2024-06-14T22:57:41.737687+0000","last_became_active":"2024-06-14T22:55:53.663509+0000","last_became_peered":"2024-06-14T22:55:53.663509+0000","last_unstale":"2024-06-14T22:57:41.737687+0000","last_undegraded":"2024-06-14T22:57:41.737687+0000","last_fullsized":"2024-06-14T22:57:41.737687+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:36.354907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":104,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.743398+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:57:41.743398+0000","last_peered":"2024-06-14T22:57:41.743398+0000","last_clean":"2024-06-14T22:57:41.743398+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:57:41.743398+0000","last_undegraded":"2024-06-14T22:57:41.743398+0000","last_fullsized":"2024-06-14T22:57:41.743398+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"161'10423","reported_seq":10454,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:54.605072+0000","last_change":"2024-06-14T22:55:38.730175+0000","last_active":"2024-06-14T22:57:54.605072+0000","last_peered":"2024-06-14T22:57:54.605072+0000","last_clean":"2024-06-14T22:57:54.605072+0000","last_became_active":"2024-06-14T22:55:38.729339+0000","last_became_peered":"2024-06-14T22:55:38.729339+0000","last_unstale":"2024-06-14T22:57:54.605072+0000","last_undegraded":"2024-06-14T22:57:54.605072+0000","last_fullsized":"2024-06-14T22:57:54.605072+0000","mapping_epoch":140,"log_start":"146'3100","ondisk_log_start":"146'3100","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7323,"log_dups_size":0,"ondisk_log_size":7323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:40:50.307712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388038656,"num_objects":5921,"num_object_clones":0,"num_object_copies":17763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10423,"num_write_kb":523008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":39,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737922+0000","last_change":"2024-06-14T22:55:53.664736+0000","last_active":"2024-06-14T22:57:41.737922+0000","last_peered":"2024-06-14T22:57:41.737922+0000","last_clean":"2024-06-14T22:57:41.737922+0000","last_became_active":"2024-06-14T22:55:53.664141+0000","last_became_peered":"2024-06-14T22:55:53.664141+0000","last_unstale":"2024-06-14T22:57:41.737922+0000","last_undegraded":"2024-06-14T22:57:41.737922+0000","last_fullsized":"2024-06-14T22:57:41.737922+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:52.824254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":138,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737101+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T22:57:41.737101+0000","last_peered":"2024-06-14T22:57:41.737101+0000","last_clean":"2024-06-14T22:57:41.737101+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T22:57:41.737101+0000","last_undegraded":"2024-06-14T22:57:41.737101+0000","last_fullsized":"2024-06-14T22:57:41.737101+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"161'10519","reported_seq":10548,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:53.927074+0000","last_change":"2024-06-14T22:55:38.730856+0000","last_active":"2024-06-14T22:57:53.927074+0000","last_peered":"2024-06-14T22:57:53.927074+0000","last_clean":"2024-06-14T22:57:53.927074+0000","last_became_active":"2024-06-14T22:55:38.730542+0000","last_became_peered":"2024-06-14T22:55:38.730542+0000","last_unstale":"2024-06-14T22:57:53.927074+0000","last_undegraded":"2024-06-14T22:57:53.927074+0000","last_fullsized":"2024-06-14T22:57:53.927074+0000","mapping_epoch":140,"log_start":"144'800","ondisk_log_start":"144'800","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9719,"log_dups_size":0,"ondisk_log_size":9719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:16:51.828869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388431872,"num_objects":5927,"num_object_clones":0,"num_object_copies":17781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10519,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":39,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.744226+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T22:57:41.744226+0000","last_peered":"2024-06-14T22:57:41.744226+0000","last_clean":"2024-06-14T22:57:41.744226+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T22:57:41.744226+0000","last_undegraded":"2024-06-14T22:57:41.744226+0000","last_fullsized":"2024-06-14T22:57:41.744226+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739369+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T22:57:41.739369+0000","last_peered":"2024-06-14T22:57:41.739369+0000","last_clean":"2024-06-14T22:57:41.739369+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T22:57:41.739369+0000","last_undegraded":"2024-06-14T22:57:41.739369+0000","last_fullsized":"2024-06-14T22:57:41.739369+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"161'10516","reported_seq":10545,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:53.938664+0000","last_change":"2024-06-14T22:55:39.130045+0000","last_active":"2024-06-14T22:57:53.938664+0000","last_peered":"2024-06-14T22:57:53.938664+0000","last_clean":"2024-06-14T22:57:53.938664+0000","last_became_active":"2024-06-14T22:55:39.129733+0000","last_became_peered":"2024-06-14T22:55:39.129733+0000","last_unstale":"2024-06-14T22:57:53.938664+0000","last_undegraded":"2024-06-14T22:57:53.938664+0000","last_fullsized":"2024-06-14T22:57:53.938664+0000","mapping_epoch":140,"log_start":"144'800","ondisk_log_start":"144'800","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:34:14.279180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397934592,"num_objects":6072,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10516,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":122,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.744116+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T22:57:41.744116+0000","last_peered":"2024-06-14T22:57:41.744116+0000","last_clean":"2024-06-14T22:57:41.744116+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T22:57:41.744116+0000","last_undegraded":"2024-06-14T22:57:41.744116+0000","last_fullsized":"2024-06-14T22:57:41.744116+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":112,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.745691+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T22:57:41.745691+0000","last_peered":"2024-06-14T22:57:41.745691+0000","last_clean":"2024-06-14T22:57:41.745691+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T22:57:41.745691+0000","last_undegraded":"2024-06-14T22:57:41.745691+0000","last_fullsized":"2024-06-14T22:57:41.745691+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"161'10461","reported_seq":10492,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:54.286269+0000","last_change":"2024-06-14T22:55:38.974910+0000","last_active":"2024-06-14T22:57:54.286269+0000","last_peered":"2024-06-14T22:57:54.286269+0000","last_clean":"2024-06-14T22:57:54.286269+0000","last_became_active":"2024-06-14T22:55:38.974574+0000","last_became_peered":"2024-06-14T22:55:38.974574+0000","last_unstale":"2024-06-14T22:57:54.286269+0000","last_undegraded":"2024-06-14T22:57:54.286269+0000","last_fullsized":"2024-06-14T22:57:54.286269+0000","mapping_epoch":140,"log_start":"146'2700","ondisk_log_start":"146'2700","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7761,"log_dups_size":0,"ondisk_log_size":7761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:48:06.634805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391315456,"num_objects":5971,"num_object_clones":0,"num_object_copies":17913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":525824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":133,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.742740+0000","last_change":"2024-06-14T22:54:06.280994+0000","last_active":"2024-06-14T22:57:41.742740+0000","last_peered":"2024-06-14T22:57:41.742740+0000","last_clean":"2024-06-14T22:57:41.742740+0000","last_became_active":"2024-06-14T22:54:06.279883+0000","last_became_peered":"2024-06-14T22:54:06.279883+0000","last_unstale":"2024-06-14T22:57:41.742740+0000","last_undegraded":"2024-06-14T22:57:41.742740+0000","last_fullsized":"2024-06-14T22:57:41.742740+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:51.156630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737522+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T22:57:41.737522+0000","last_peered":"2024-06-14T22:57:41.737522+0000","last_clean":"2024-06-14T22:57:41.737522+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T22:57:41.737522+0000","last_undegraded":"2024-06-14T22:57:41.737522+0000","last_fullsized":"2024-06-14T22:57:41.737522+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"161'10717","reported_seq":10748,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:54.604964+0000","last_change":"2024-06-14T22:55:39.129787+0000","last_active":"2024-06-14T22:57:54.604964+0000","last_peered":"2024-06-14T22:57:54.604964+0000","last_clean":"2024-06-14T22:57:54.604964+0000","last_became_active":"2024-06-14T22:55:39.129528+0000","last_became_peered":"2024-06-14T22:55:39.129528+0000","last_unstale":"2024-06-14T22:57:54.604964+0000","last_undegraded":"2024-06-14T22:57:54.604964+0000","last_fullsized":"2024-06-14T22:57:54.604964+0000","mapping_epoch":140,"log_start":"146'3300","ondisk_log_start":"146'3300","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:24:05.362187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393805824,"num_objects":6009,"num_object_clones":0,"num_object_copies":18027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":535232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":122,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739498+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T22:57:41.739498+0000","last_peered":"2024-06-14T22:57:41.739498+0000","last_clean":"2024-06-14T22:57:41.739498+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T22:57:41.739498+0000","last_undegraded":"2024-06-14T22:57:41.739498+0000","last_fullsized":"2024-06-14T22:57:41.739498+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":131,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737759+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T22:57:41.737759+0000","last_peered":"2024-06-14T22:57:41.737759+0000","last_clean":"2024-06-14T22:57:41.737759+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T22:57:41.737759+0000","last_undegraded":"2024-06-14T22:57:41.737759+0000","last_fullsized":"2024-06-14T22:57:41.737759+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"161'10459","reported_seq":10495,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:54.286709+0000","last_change":"2024-06-14T22:55:53.668083+0000","last_active":"2024-06-14T22:57:54.286709+0000","last_peered":"2024-06-14T22:57:54.286709+0000","last_clean":"2024-06-14T22:57:54.286709+0000","last_became_active":"2024-06-14T22:55:53.664556+0000","last_became_peered":"2024-06-14T22:55:53.664556+0000","last_unstale":"2024-06-14T22:57:54.286709+0000","last_undegraded":"2024-06-14T22:57:54.286709+0000","last_fullsized":"2024-06-14T22:57:54.286709+0000","mapping_epoch":145,"log_start":"146'2700","ondisk_log_start":"146'2700","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7759,"log_dups_size":0,"ondisk_log_size":7759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:50:22.930473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394854400,"num_objects":6025,"num_object_clones":0,"num_object_copies":18075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":122,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737167+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T22:57:41.737167+0000","last_peered":"2024-06-14T22:57:41.737167+0000","last_clean":"2024-06-14T22:57:41.737167+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T22:57:41.737167+0000","last_undegraded":"2024-06-14T22:57:41.737167+0000","last_fullsized":"2024-06-14T22:57:41.737167+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737993+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T22:57:41.737993+0000","last_peered":"2024-06-14T22:57:41.737993+0000","last_clean":"2024-06-14T22:57:41.737993+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T22:57:41.737993+0000","last_undegraded":"2024-06-14T22:57:41.737993+0000","last_fullsized":"2024-06-14T22:57:41.737993+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"161'10677","reported_seq":10707,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:54.310701+0000","last_change":"2024-06-14T22:55:39.129746+0000","last_active":"2024-06-14T22:57:54.310701+0000","last_peered":"2024-06-14T22:57:54.310701+0000","last_clean":"2024-06-14T22:57:54.310701+0000","last_became_active":"2024-06-14T22:55:39.129442+0000","last_became_peered":"2024-06-14T22:55:39.129442+0000","last_unstale":"2024-06-14T22:57:54.310701+0000","last_undegraded":"2024-06-14T22:57:54.310701+0000","last_fullsized":"2024-06-14T22:57:54.310701+0000","mapping_epoch":140,"log_start":"146'2900","ondisk_log_start":"146'2900","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7777,"log_dups_size":0,"ondisk_log_size":7777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:36:21.422229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397344768,"num_objects":6063,"num_object_clones":0,"num_object_copies":18189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10677,"num_write_kb":535680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":114,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.740443+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T22:57:41.740443+0000","last_peered":"2024-06-14T22:57:41.740443+0000","last_clean":"2024-06-14T22:57:41.740443+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T22:57:41.740443+0000","last_undegraded":"2024-06-14T22:57:41.740443+0000","last_fullsized":"2024-06-14T22:57:41.740443+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":127,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737122+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T22:57:41.737122+0000","last_peered":"2024-06-14T22:57:41.737122+0000","last_clean":"2024-06-14T22:57:41.737122+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T22:57:41.737122+0000","last_undegraded":"2024-06-14T22:57:41.737122+0000","last_fullsized":"2024-06-14T22:57:41.737122+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"161'10344","reported_seq":10375,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:54.225991+0000","last_change":"2024-06-14T22:55:38.729850+0000","last_active":"2024-06-14T22:57:54.225991+0000","last_peered":"2024-06-14T22:57:54.225991+0000","last_clean":"2024-06-14T22:57:54.225991+0000","last_became_active":"2024-06-14T22:55:38.729536+0000","last_became_peered":"2024-06-14T22:55:38.729536+0000","last_unstale":"2024-06-14T22:57:54.225991+0000","last_undegraded":"2024-06-14T22:57:54.225991+0000","last_fullsized":"2024-06-14T22:57:54.225991+0000","mapping_epoch":140,"log_start":"146'2600","ondisk_log_start":"146'2600","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7744,"log_dups_size":0,"ondisk_log_size":7744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:47:35.019362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391643136,"num_objects":5976,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10344,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":427,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:51.797165+0000","last_change":"2024-06-14T22:55:53.665022+0000","last_active":"2024-06-14T22:57:51.797165+0000","last_peered":"2024-06-14T22:57:51.797165+0000","last_clean":"2024-06-14T22:57:51.797165+0000","last_became_active":"2024-06-14T22:55:53.664025+0000","last_became_peered":"2024-06-14T22:55:53.664025+0000","last_unstale":"2024-06-14T22:57:51.797165+0000","last_undegraded":"2024-06-14T22:57:51.797165+0000","last_fullsized":"2024-06-14T22:57:51.797165+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.106147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.744115+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T22:57:41.744115+0000","last_peered":"2024-06-14T22:57:41.744115+0000","last_clean":"2024-06-14T22:57:41.744115+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T22:57:41.744115+0000","last_undegraded":"2024-06-14T22:57:41.744115+0000","last_fullsized":"2024-06-14T22:57:41.744115+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":87,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739424+0000","last_change":"2024-06-14T22:54:06.279205+0000","last_active":"2024-06-14T22:57:41.739424+0000","last_peered":"2024-06-14T22:57:41.739424+0000","last_clean":"2024-06-14T22:57:41.739424+0000","last_became_active":"2024-06-14T22:54:06.278792+0000","last_became_peered":"2024-06-14T22:54:06.278792+0000","last_unstale":"2024-06-14T22:57:41.739424+0000","last_undegraded":"2024-06-14T22:57:41.739424+0000","last_fullsized":"2024-06-14T22:57:41.739424+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:55:02.899941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":104,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739751+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:57:41.739751+0000","last_peered":"2024-06-14T22:57:41.739751+0000","last_clean":"2024-06-14T22:57:41.739751+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:57:41.739751+0000","last_undegraded":"2024-06-14T22:57:41.739751+0000","last_fullsized":"2024-06-14T22:57:41.739751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"161'10828","reported_seq":10857,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:56.338578+0000","last_change":"2024-06-14T22:55:38.974930+0000","last_active":"2024-06-14T22:57:56.338578+0000","last_peered":"2024-06-14T22:57:56.338578+0000","last_clean":"2024-06-14T22:57:56.338578+0000","last_became_active":"2024-06-14T22:55:38.974625+0000","last_became_peered":"2024-06-14T22:55:38.974625+0000","last_unstale":"2024-06-14T22:57:56.338578+0000","last_undegraded":"2024-06-14T22:57:56.338578+0000","last_fullsized":"2024-06-14T22:57:56.338578+0000","mapping_epoch":140,"log_start":"144'800","ondisk_log_start":"144'800","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:44:40.336597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386269184,"num_objects":5894,"num_object_clones":0,"num_object_copies":17682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":535104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":118,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:51.798470+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T22:57:51.798470+0000","last_peered":"2024-06-14T22:57:51.798470+0000","last_clean":"2024-06-14T22:57:51.798470+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T22:57:51.798470+0000","last_undegraded":"2024-06-14T22:57:51.798470+0000","last_fullsized":"2024-06-14T22:57:51.798470+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":112,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.745208+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T22:57:41.745208+0000","last_peered":"2024-06-14T22:57:41.745208+0000","last_clean":"2024-06-14T22:57:41.745208+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T22:57:41.745208+0000","last_undegraded":"2024-06-14T22:57:41.745208+0000","last_fullsized":"2024-06-14T22:57:41.745208+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:51.501143+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T22:57:51.501143+0000","last_peered":"2024-06-14T22:57:51.501143+0000","last_clean":"2024-06-14T22:57:51.501143+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T22:57:51.501143+0000","last_undegraded":"2024-06-14T22:57:51.501143+0000","last_fullsized":"2024-06-14T22:57:51.501143+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"161'10654","reported_seq":10683,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:53.969031+0000","last_change":"2024-06-14T22:55:38.731601+0000","last_active":"2024-06-14T22:57:53.969031+0000","last_peered":"2024-06-14T22:57:53.969031+0000","last_clean":"2024-06-14T22:57:53.969031+0000","last_became_active":"2024-06-14T22:55:38.731200+0000","last_became_peered":"2024-06-14T22:55:38.731200+0000","last_unstale":"2024-06-14T22:57:53.969031+0000","last_undegraded":"2024-06-14T22:57:53.969031+0000","last_fullsized":"2024-06-14T22:57:53.969031+0000","mapping_epoch":140,"log_start":"144'900","ondisk_log_start":"144'900","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:53:02.395452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392953856,"num_objects":5996,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10654,"num_write_kb":532800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":113,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737865+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T22:57:41.737865+0000","last_peered":"2024-06-14T22:57:41.737865+0000","last_clean":"2024-06-14T22:57:41.737865+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T22:57:41.737865+0000","last_undegraded":"2024-06-14T22:57:41.737865+0000","last_fullsized":"2024-06-14T22:57:41.737865+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":128,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.735862+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T22:57:41.735862+0000","last_peered":"2024-06-14T22:57:41.735862+0000","last_clean":"2024-06-14T22:57:41.735862+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T22:57:41.735862+0000","last_undegraded":"2024-06-14T22:57:41.735862+0000","last_fullsized":"2024-06-14T22:57:41.735862+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":112,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.735673+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T22:57:41.735673+0000","last_peered":"2024-06-14T22:57:41.735673+0000","last_clean":"2024-06-14T22:57:41.735673+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T22:57:41.735673+0000","last_undegraded":"2024-06-14T22:57:41.735673+0000","last_fullsized":"2024-06-14T22:57:41.735673+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"161'10560","reported_seq":10591,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:56.501507+0000","last_change":"2024-06-14T22:55:38.729565+0000","last_active":"2024-06-14T22:57:56.501507+0000","last_peered":"2024-06-14T22:57:56.501507+0000","last_clean":"2024-06-14T22:57:56.501507+0000","last_became_active":"2024-06-14T22:55:38.729284+0000","last_became_peered":"2024-06-14T22:55:38.729284+0000","last_unstale":"2024-06-14T22:57:56.501507+0000","last_undegraded":"2024-06-14T22:57:56.501507+0000","last_fullsized":"2024-06-14T22:57:56.501507+0000","mapping_epoch":140,"log_start":"144'500","ondisk_log_start":"144'500","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:15.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386924544,"num_objects":5904,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10560,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.746559+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T22:57:41.746559+0000","last_peered":"2024-06-14T22:57:41.746559+0000","last_clean":"2024-06-14T22:57:41.746559+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T22:57:41.746559+0000","last_undegraded":"2024-06-14T22:57:41.746559+0000","last_fullsized":"2024-06-14T22:57:41.746559+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":122,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737351+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T22:57:41.737351+0000","last_peered":"2024-06-14T22:57:41.737351+0000","last_clean":"2024-06-14T22:57:41.737351+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T22:57:41.737351+0000","last_undegraded":"2024-06-14T22:57:41.737351+0000","last_fullsized":"2024-06-14T22:57:41.737351+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":143,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.609997+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T22:57:44.609997+0000","last_peered":"2024-06-14T22:57:44.609997+0000","last_clean":"2024-06-14T22:57:44.609997+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T22:57:44.609997+0000","last_undegraded":"2024-06-14T22:57:44.609997+0000","last_fullsized":"2024-06-14T22:57:44.609997+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"161'10521","reported_seq":10550,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:53.938539+0000","last_change":"2024-06-14T22:55:38.729989+0000","last_active":"2024-06-14T22:57:53.938539+0000","last_peered":"2024-06-14T22:57:53.938539+0000","last_clean":"2024-06-14T22:57:53.938539+0000","last_became_active":"2024-06-14T22:55:38.729695+0000","last_became_peered":"2024-06-14T22:55:38.729695+0000","last_unstale":"2024-06-14T22:57:53.938539+0000","last_undegraded":"2024-06-14T22:57:53.938539+0000","last_fullsized":"2024-06-14T22:57:53.938539+0000","mapping_epoch":140,"log_start":"144'800","ondisk_log_start":"144'800","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9721,"log_dups_size":0,"ondisk_log_size":9721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:44:38.218727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393281536,"num_objects":6001,"num_object_clones":0,"num_object_copies":18003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10521,"num_write_kb":528704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":139,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.746851+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T22:57:41.746851+0000","last_peered":"2024-06-14T22:57:41.746851+0000","last_clean":"2024-06-14T22:57:41.746851+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T22:57:41.746851+0000","last_undegraded":"2024-06-14T22:57:41.746851+0000","last_fullsized":"2024-06-14T22:57:41.746851+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":118,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.610026+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T22:57:44.610026+0000","last_peered":"2024-06-14T22:57:44.610026+0000","last_clean":"2024-06-14T22:57:44.610026+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T22:57:44.610026+0000","last_undegraded":"2024-06-14T22:57:44.610026+0000","last_fullsized":"2024-06-14T22:57:44.610026+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":113,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.609967+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T22:57:44.609967+0000","last_peered":"2024-06-14T22:57:44.609967+0000","last_clean":"2024-06-14T22:57:44.609967+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T22:57:44.609967+0000","last_undegraded":"2024-06-14T22:57:44.609967+0000","last_fullsized":"2024-06-14T22:57:44.609967+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"161'10514","reported_seq":10550,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:53.622313+0000","last_change":"2024-06-14T22:55:53.672630+0000","last_active":"2024-06-14T22:57:53.622313+0000","last_peered":"2024-06-14T22:57:53.622313+0000","last_clean":"2024-06-14T22:57:53.622313+0000","last_became_active":"2024-06-14T22:55:53.663693+0000","last_became_peered":"2024-06-14T22:55:53.663693+0000","last_unstale":"2024-06-14T22:57:53.622313+0000","last_undegraded":"2024-06-14T22:57:53.622313+0000","last_fullsized":"2024-06-14T22:57:53.622313+0000","mapping_epoch":145,"log_start":"146'2100","ondisk_log_start":"146'2100","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":8414,"log_dups_size":0,"ondisk_log_size":8414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:20:34.289185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395182080,"num_objects":6030,"num_object_clones":0,"num_object_copies":18090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":128,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:44.610011+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T22:57:44.610011+0000","last_peered":"2024-06-14T22:57:44.610011+0000","last_clean":"2024-06-14T22:57:44.610011+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T22:57:44.610011+0000","last_undegraded":"2024-06-14T22:57:44.610011+0000","last_fullsized":"2024-06-14T22:57:44.610011+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.746272+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T22:57:41.746272+0000","last_peered":"2024-06-14T22:57:41.746272+0000","last_clean":"2024-06-14T22:57:41.746272+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T22:57:41.746272+0000","last_undegraded":"2024-06-14T22:57:41.746272+0000","last_fullsized":"2024-06-14T22:57:41.746272+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":98,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.745549+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T22:57:41.745549+0000","last_peered":"2024-06-14T22:57:41.745549+0000","last_clean":"2024-06-14T22:57:41.745549+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T22:57:41.745549+0000","last_undegraded":"2024-06-14T22:57:41.745549+0000","last_fullsized":"2024-06-14T22:57:41.745549+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"161'10625","reported_seq":10663,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:54.610369+0000","last_change":"2024-06-14T22:55:53.668144+0000","last_active":"2024-06-14T22:57:54.610369+0000","last_peered":"2024-06-14T22:57:54.610369+0000","last_clean":"2024-06-14T22:57:54.610369+0000","last_became_active":"2024-06-14T22:55:53.663690+0000","last_became_peered":"2024-06-14T22:55:53.663690+0000","last_unstale":"2024-06-14T22:57:54.610369+0000","last_undegraded":"2024-06-14T22:57:54.610369+0000","last_fullsized":"2024-06-14T22:57:54.610369+0000","mapping_epoch":145,"log_start":"147'3300","ondisk_log_start":"147'3300","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7325,"log_dups_size":0,"ondisk_log_size":7325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:52:03.686394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389939224,"num_objects":5951,"num_object_clones":0,"num_object_copies":17853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10625,"num_write_kb":530369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":133,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.743153+0000","last_change":"2024-06-14T22:54:06.284651+0000","last_active":"2024-06-14T22:57:41.743153+0000","last_peered":"2024-06-14T22:57:41.743153+0000","last_clean":"2024-06-14T22:57:41.743153+0000","last_became_active":"2024-06-14T22:54:06.283564+0000","last_became_peered":"2024-06-14T22:54:06.283564+0000","last_unstale":"2024-06-14T22:57:41.743153+0000","last_undegraded":"2024-06-14T22:57:41.743153+0000","last_fullsized":"2024-06-14T22:57:41.743153+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:34.916272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":111,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.743723+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T22:57:41.743723+0000","last_peered":"2024-06-14T22:57:41.743723+0000","last_clean":"2024-06-14T22:57:41.743723+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T22:57:41.743723+0000","last_undegraded":"2024-06-14T22:57:41.743723+0000","last_fullsized":"2024-06-14T22:57:41.743723+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":130,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.740282+0000","last_change":"2024-06-14T22:54:06.281419+0000","last_active":"2024-06-14T22:57:41.740282+0000","last_peered":"2024-06-14T22:57:41.740282+0000","last_clean":"2024-06-14T22:57:41.740282+0000","last_became_active":"2024-06-14T22:54:06.279482+0000","last_became_peered":"2024-06-14T22:54:06.279482+0000","last_unstale":"2024-06-14T22:57:41.740282+0000","last_undegraded":"2024-06-14T22:57:41.740282+0000","last_fullsized":"2024-06-14T22:57:41.740282+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:06:37.039006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"161'10472","reported_seq":10502,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-06-14T22:57:54.288285+0000","last_change":"2024-06-14T22:55:38.729250+0000","last_active":"2024-06-14T22:57:54.288285+0000","last_peered":"2024-06-14T22:57:54.288285+0000","last_clean":"2024-06-14T22:57:54.288285+0000","last_became_active":"2024-06-14T22:55:38.728938+0000","last_became_peered":"2024-06-14T22:55:38.728938+0000","last_unstale":"2024-06-14T22:57:54.288285+0000","last_undegraded":"2024-06-14T22:57:54.288285+0000","last_fullsized":"2024-06-14T22:57:54.288285+0000","mapping_epoch":140,"log_start":"146'2700","ondisk_log_start":"146'2700","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7772,"log_dups_size":0,"ondisk_log_size":7772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:32:07.069919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389283840,"num_objects":5940,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10472,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":116,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:51.501101+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T22:57:51.501101+0000","last_peered":"2024-06-14T22:57:51.501101+0000","last_clean":"2024-06-14T22:57:51.501101+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T22:57:51.501101+0000","last_undegraded":"2024-06-14T22:57:51.501101+0000","last_fullsized":"2024-06-14T22:57:51.501101+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739822+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T22:57:41.739822+0000","last_peered":"2024-06-14T22:57:41.739822+0000","last_clean":"2024-06-14T22:57:41.739822+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T22:57:41.739822+0000","last_undegraded":"2024-06-14T22:57:41.739822+0000","last_fullsized":"2024-06-14T22:57:41.739822+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":104,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.736990+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:57:41.736990+0000","last_peered":"2024-06-14T22:57:41.736990+0000","last_clean":"2024-06-14T22:57:41.736990+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:57:41.736990+0000","last_undegraded":"2024-06-14T22:57:41.736990+0000","last_fullsized":"2024-06-14T22:57:41.736990+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737313+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T22:57:41.737313+0000","last_peered":"2024-06-14T22:57:41.737313+0000","last_clean":"2024-06-14T22:57:41.737313+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T22:57:41.737313+0000","last_undegraded":"2024-06-14T22:57:41.737313+0000","last_fullsized":"2024-06-14T22:57:41.737313+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":127,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737573+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:57:41.737573+0000","last_peered":"2024-06-14T22:57:41.737573+0000","last_clean":"2024-06-14T22:57:41.737573+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:57:41.737573+0000","last_undegraded":"2024-06-14T22:57:41.737573+0000","last_fullsized":"2024-06-14T22:57:41.737573+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":127,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.738912+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T22:57:41.738912+0000","last_peered":"2024-06-14T22:57:41.738912+0000","last_clean":"2024-06-14T22:57:41.738912+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T22:57:41.738912+0000","last_undegraded":"2024-06-14T22:57:41.738912+0000","last_fullsized":"2024-06-14T22:57:41.738912+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":38,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.742266+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T22:57:41.742266+0000","last_peered":"2024-06-14T22:57:41.742266+0000","last_clean":"2024-06-14T22:57:41.742266+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T22:57:41.742266+0000","last_undegraded":"2024-06-14T22:57:41.742266+0000","last_fullsized":"2024-06-14T22:57:41.742266+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":122,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.744578+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T22:57:41.744578+0000","last_peered":"2024-06-14T22:57:41.744578+0000","last_clean":"2024-06-14T22:57:41.744578+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T22:57:41.744578+0000","last_undegraded":"2024-06-14T22:57:41.744578+0000","last_fullsized":"2024-06-14T22:57:41.744578+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":87,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.743788+0000","last_change":"2024-06-14T22:54:06.279332+0000","last_active":"2024-06-14T22:57:41.743788+0000","last_peered":"2024-06-14T22:57:41.743788+0000","last_clean":"2024-06-14T22:57:41.743788+0000","last_became_active":"2024-06-14T22:54:06.279020+0000","last_became_peered":"2024-06-14T22:54:06.279020+0000","last_unstale":"2024-06-14T22:57:41.743788+0000","last_undegraded":"2024-06-14T22:57:41.743788+0000","last_fullsized":"2024-06-14T22:57:41.743788+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:52:57.777668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":111,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.745623+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T22:57:41.745623+0000","last_peered":"2024-06-14T22:57:41.745623+0000","last_clean":"2024-06-14T22:57:41.745623+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T22:57:41.745623+0000","last_undegraded":"2024-06-14T22:57:41.745623+0000","last_fullsized":"2024-06-14T22:57:41.745623+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":113,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.740320+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T22:57:41.740320+0000","last_peered":"2024-06-14T22:57:41.740320+0000","last_clean":"2024-06-14T22:57:41.740320+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T22:57:41.740320+0000","last_undegraded":"2024-06-14T22:57:41.740320+0000","last_fullsized":"2024-06-14T22:57:41.740320+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.739750+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T22:57:41.739750+0000","last_peered":"2024-06-14T22:57:41.739750+0000","last_clean":"2024-06-14T22:57:41.739750+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T22:57:41.739750+0000","last_undegraded":"2024-06-14T22:57:41.739750+0000","last_fullsized":"2024-06-14T22:57:41.739750+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":131,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.740396+0000","last_change":"2024-06-14T22:54:06.282120+0000","last_active":"2024-06-14T22:57:41.740396+0000","last_peered":"2024-06-14T22:57:41.740396+0000","last_clean":"2024-06-14T22:57:41.740396+0000","last_became_active":"2024-06-14T22:54:06.280898+0000","last_became_peered":"2024-06-14T22:54:06.280898+0000","last_unstale":"2024-06-14T22:57:41.740396+0000","last_undegraded":"2024-06-14T22:57:41.740396+0000","last_fullsized":"2024-06-14T22:57:41.740396+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:30:45.544457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":113,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.745411+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T22:57:41.745411+0000","last_peered":"2024-06-14T22:57:41.745411+0000","last_clean":"2024-06-14T22:57:41.745411+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T22:57:41.745411+0000","last_undegraded":"2024-06-14T22:57:41.745411+0000","last_fullsized":"2024-06-14T22:57:41.745411+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":127,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.735469+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T22:57:41.735469+0000","last_peered":"2024-06-14T22:57:41.735469+0000","last_clean":"2024-06-14T22:57:41.735469+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T22:57:41.735469+0000","last_undegraded":"2024-06-14T22:57:41.735469+0000","last_fullsized":"2024-06-14T22:57:41.735469+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":139,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.742145+0000","last_change":"2024-06-14T22:54:06.280483+0000","last_active":"2024-06-14T22:57:41.742145+0000","last_peered":"2024-06-14T22:57:41.742145+0000","last_clean":"2024-06-14T22:57:41.742145+0000","last_became_active":"2024-06-14T22:54:06.278985+0000","last_became_peered":"2024-06-14T22:54:06.278985+0000","last_unstale":"2024-06-14T22:57:41.742145+0000","last_undegraded":"2024-06-14T22:57:41.742145+0000","last_fullsized":"2024-06-14T22:57:41.742145+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:43:40.478580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":140,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737131+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T22:57:41.737131+0000","last_peered":"2024-06-14T22:57:41.737131+0000","last_clean":"2024-06-14T22:57:41.737131+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T22:57:41.737131+0000","last_undegraded":"2024-06-14T22:57:41.737131+0000","last_fullsized":"2024-06-14T22:57:41.737131+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":132,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.738214+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T22:57:41.738214+0000","last_peered":"2024-06-14T22:57:41.738214+0000","last_clean":"2024-06-14T22:57:41.738214+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T22:57:41.738214+0000","last_undegraded":"2024-06-14T22:57:41.738214+0000","last_fullsized":"2024-06-14T22:57:41.738214+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":100,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.745481+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T22:57:41.745481+0000","last_peered":"2024-06-14T22:57:41.745481+0000","last_clean":"2024-06-14T22:57:41.745481+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T22:57:41.745481+0000","last_undegraded":"2024-06-14T22:57:41.745481+0000","last_fullsized":"2024-06-14T22:57:41.745481+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":138,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737282+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T22:57:41.737282+0000","last_peered":"2024-06-14T22:57:41.737282+0000","last_clean":"2024-06-14T22:57:41.737282+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T22:57:41.737282+0000","last_undegraded":"2024-06-14T22:57:41.737282+0000","last_fullsized":"2024-06-14T22:57:41.737282+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":124,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.744047+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T22:57:41.744047+0000","last_peered":"2024-06-14T22:57:41.744047+0000","last_clean":"2024-06-14T22:57:41.744047+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T22:57:41.744047+0000","last_undegraded":"2024-06-14T22:57:41.744047+0000","last_fullsized":"2024-06-14T22:57:41.744047+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":138,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.737370+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T22:57:41.737370+0000","last_peered":"2024-06-14T22:57:41.737370+0000","last_clean":"2024-06-14T22:57:41.737370+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T22:57:41.737370+0000","last_undegraded":"2024-06-14T22:57:41.737370+0000","last_fullsized":"2024-06-14T22:57:41.737370+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":107,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.746784+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:57:41.746784+0000","last_peered":"2024-06-14T22:57:41.746784+0000","last_clean":"2024-06-14T22:57:41.746784+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:57:41.746784+0000","last_undegraded":"2024-06-14T22:57:41.746784+0000","last_fullsized":"2024-06-14T22:57:41.746784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":146,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.744188+0000","last_change":"2024-06-14T22:54:06.282972+0000","last_active":"2024-06-14T22:57:41.744188+0000","last_peered":"2024-06-14T22:57:41.744188+0000","last_clean":"2024-06-14T22:57:41.744188+0000","last_became_active":"2024-06-14T22:54:06.282622+0000","last_became_peered":"2024-06-14T22:54:06.282622+0000","last_unstale":"2024-06-14T22:57:41.744188+0000","last_undegraded":"2024-06-14T22:57:41.744188+0000","last_fullsized":"2024-06-14T22:57:41.744188+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:31:22.051955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":131,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.742481+0000","last_change":"2024-06-14T22:54:06.279284+0000","last_active":"2024-06-14T22:57:41.742481+0000","last_peered":"2024-06-14T22:57:41.742481+0000","last_clean":"2024-06-14T22:57:41.742481+0000","last_became_active":"2024-06-14T22:54:06.278861+0000","last_became_peered":"2024-06-14T22:54:06.278861+0000","last_unstale":"2024-06-14T22:57:41.742481+0000","last_undegraded":"2024-06-14T22:57:41.742481+0000","last_fullsized":"2024-06-14T22:57:41.742481+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:17:47.260769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":121,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.736886+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T22:57:41.736886+0000","last_peered":"2024-06-14T22:57:41.736886+0000","last_clean":"2024-06-14T22:57:41.736886+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T22:57:41.736886+0000","last_undegraded":"2024-06-14T22:57:41.736886+0000","last_fullsized":"2024-06-14T22:57:41.736886+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":116,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-06-14T22:57:41.745548+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T22:57:41.745548+0000","last_peered":"2024-06-14T22:57:41.745548+0000","last_clean":"2024-06-14T22:57:41.745548+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T22:57:41.745548+0000","last_undegraded":"2024-06-14T22:57:41.745548+0000","last_fullsized":"2024-06-14T22:57:41.745548+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6269632536,"num_objects":95668,"num_object_clones":0,"num_object_copies":287004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95668,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169292,"num_write_kb":8478657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18854522880,"data_stored":18854510664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138292,"ondisk_log_size":138292,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234286,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4395832,"kb_used_data":3492780,"kb_used_omap":1,"kb_used_meta":903038,"kb_avail":89349320,"statfs":{"total":95995035648,"available":91493703680,"internally_reserved":0,"allocated":3576606720,"data_stored":3575749073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924711256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:10 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":0.86299999999999999,"15min":1.103},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":4.0330000000000004,"5min":4.0330000000000004,"15min":9.3919999999999995},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.80600000000000005,"15min":1.2130000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.6190000000000002,"5min":4.4649999999999999,"15min":10.416},"last":5.0220000000000002}]},{"osd":1,"last update":"Fri Jun 14 22:57:40 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":2.3660000000000001,"15min":1.772},"min":{"1min":0.38600000000000001,"5min":0.28899999999999998,"15min":0.28100000000000003},"max":{"1min":4.0629999999999997,"5min":83.295000000000002,"15min":83.295000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.5149999999999999,"5min":2.3330000000000002,"15min":1.843},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.223},"max":{"1min":3.9820000000000002,"5min":82.972999999999999,"15min":82.972999999999999},"last":3.8170000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:57:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.76900000000000002,"15min":0.64400000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.0540000000000003,"5min":4.0540000000000003,"15min":4.0540000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.4370000000000001,"5min":0.876,"15min":0.67300000000000004},"min":{"1min":0.33400000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":8.1150000000000002,"5min":8.1150000000000002,"15min":8.1150000000000002},"last":0.70099999999999996}]},{"osd":3,"last update":"Fri Jun 14 22:57:07 2024","interfaces":[{"interface":"back","average":{"1min":3.8399999999999999,"5min":1.246,"15min":0.77300000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":55.683999999999997,"5min":55.683999999999997,"15min":55.683999999999997},"last":3.544},{"interface":"front","average":{"1min":4.2060000000000004,"5min":1.359,"15min":0.85199999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":55.533999999999999,"5min":55.533999999999999,"15min":55.533999999999999},"last":0.82299999999999995}]},{"osd":4,"last update":"Fri Jun 14 22:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.70099999999999996,"15min":0.65100000000000002},"min":{"1min":0.31,"5min":0.245,"15min":0.214},"max":{"1min":2.7240000000000002,"5min":5.8449999999999998,"15min":9.7430000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.70099999999999996,"15min":0.64500000000000002},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.24399999999999999},"max":{"1min":3.3050000000000002,"5min":4.2709999999999999,"15min":6.6660000000000004},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:56:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":0.72899999999999998,"15min":0.65100000000000002},"min":{"1min":0.30099999999999999,"5min":0.26100000000000001,"15min":0.22},"max":{"1min":3.9140000000000001,"5min":3.9140000000000001,"15min":3.9860000000000002},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.196,"5min":0.86299999999999999,"15min":0.77300000000000002},"min":{"1min":0.379,"5min":0.31900000000000001,"15min":0.28699999999999998},"max":{"1min":5.0030000000000001,"5min":5.3659999999999997,"15min":5.3659999999999997},"last":0.41299999999999998}]},{"osd":6,"last update":"Fri Jun 14 22:56:53 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":0.80800000000000005,"15min":0.72899999999999998},"min":{"1min":0.32900000000000001,"5min":0.31900000000000001,"15min":0.28100000000000003},"max":{"1min":2.7530000000000001,"5min":5.806,"15min":5.806},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.751,"15min":0.70299999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.29899999999999999},"max":{"1min":3.355,"5min":4.3090000000000002,"15min":5.7430000000000003},"last":0.49099999999999999}]}]},{"osd":6,"up_from":46,"seq":197568495922,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1763876,"kb_used_data":1169944,"kb_used_omap":1,"kb_used_meta":593918,"kb_avail":91981276,"statfs":{"total":95995035648,"available":94188826624,"internally_reserved":0,"allocated":1198022656,"data_stored":1197184996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608172376},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:10 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":0.96199999999999997,"15min":1.1930000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":6.0549999999999997,"5min":7.1689999999999996,"15min":31.358000000000001},"last":1.252},{"interface":"front","average":{"1min":1.2809999999999999,"5min":0.79100000000000004,"15min":1.3620000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":29.399000000000001},"last":1.234}]},{"osd":1,"last update":"Fri Jun 14 22:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":0.89900000000000002,"15min":1.2310000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.9269999999999996,"5min":4.9269999999999996,"15min":9.9559999999999995},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":0.92300000000000004,"15min":1.2070000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.247},"max":{"1min":3.9790000000000001,"5min":5.2960000000000003,"15min":9.5510000000000002},"last":0.67100000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:57:35 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":0.72199999999999998,"15min":0.623},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.5190000000000001,"5min":4.5190000000000001,"15min":4.5190000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.087,"5min":0.747,"15min":0.65300000000000002},"min":{"1min":0.33500000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.4580000000000002,"5min":6.4580000000000002,"15min":6.4580000000000002},"last":0.61299999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:57:10 2024","interfaces":[{"interface":"back","average":{"1min":2.9020000000000001,"5min":1.042,"15min":0.68200000000000005},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":29.152000000000001,"5min":29.152000000000001,"15min":29.152000000000001},"last":1.4139999999999999},{"interface":"front","average":{"1min":2.734,"5min":1.089,"15min":0.746},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":25.864000000000001,"5min":25.864000000000001,"15min":25.864000000000001},"last":3.0259999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.59699999999999998,"15min":0.76800000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":2.7149999999999999,"5min":5.327,"15min":47.698},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53000000000000003,"15min":0.74099999999999999},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.23899999999999999},"max":{"1min":1.8939999999999999,"5min":2.2010000000000001,"15min":48.197000000000003},"last":0.27600000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.751,"15min":0.69799999999999995},"min":{"1min":0.308,"5min":0.308,"15min":0.245},"max":{"1min":2.3159999999999998,"5min":8.6500000000000004,"15min":8.6500000000000004},"last":1.403},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73199999999999998,"15min":0.73699999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.6480000000000001,"5min":8.6349999999999998,"15min":8.6349999999999998},"last":0.63200000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":0.755,"15min":0.77500000000000002},"min":{"1min":0.32500000000000001,"5min":0.314,"15min":0.28799999999999998},"max":{"1min":5.9039999999999999,"5min":5.9039999999999999,"15min":8.6950000000000003},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.73799999999999999,"15min":0.79500000000000004},"min":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.28899999999999998},"max":{"1min":3.496,"5min":3.496,"15min":8.673},"last":0.59199999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757561,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4385884,"kb_used_data":3498832,"kb_used_omap":1,"kb_used_meta":887038,"kb_avail":89359268,"statfs":{"total":95995035648,"available":91503890432,"internally_reserved":0,"allocated":3582803968,"data_stored":3581969794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908327256},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.92900000000000005,"15min":1.194},"min":{"1min":0.376,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":4.3470000000000004,"5min":6.0369999999999999,"15min":17.766999999999999},"last":1.292},{"interface":"front","average":{"1min":1.3380000000000001,"5min":0.94199999999999995,"15min":1.224},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":4.3659999999999997,"5min":4.4429999999999996,"15min":17.806000000000001},"last":1.5920000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:57:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":0.92300000000000004,"15min":1.274},"min":{"1min":0.41899999999999998,"5min":0.24199999999999999,"15min":0.23300000000000001},"max":{"1min":2.8889999999999998,"5min":3.7759999999999998,"15min":7.694},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.8759999999999999,"5min":1.069,"15min":1.1719999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":3.9470000000000001,"5min":5.3259999999999996,"15min":5.3259999999999996},"last":1.718}]},{"osd":2,"last update":"Fri Jun 14 22:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.77400000000000002,"15min":0.622},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.3620000000000001,"5min":5.9379999999999997,"15min":5.9379999999999997},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.079,"5min":0.73399999999999999,"15min":0.624},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.677,"5min":3.677,"15min":3.677},"last":1.49}]},{"osd":3,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":4.274,"5min":1.335,"15min":0.79700000000000004},"min":{"1min":0.39200000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":45.744,"5min":45.744,"15min":45.744},"last":1.464},{"interface":"front","average":{"1min":4.4210000000000003,"5min":1.4370000000000001,"15min":0.90500000000000003},"min":{"1min":0.48199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":45.838000000000001,"5min":45.838000000000001,"15min":45.838000000000001},"last":1.341}]},{"osd":4,"last update":"Fri Jun 14 22:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.59899999999999998,"15min":0.56299999999999994},"min":{"1min":0.26300000000000001,"5min":0.24399999999999999,"15min":0.214},"max":{"1min":3.3039999999999998,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.335,"5min":3.8900000000000001,"15min":6.0149999999999997},"last":2.8199999999999998}]},{"osd":6,"last update":"Fri Jun 14 22:57:29 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.64300000000000002,"15min":0.72999999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":3.1389999999999998,"5min":4.7370000000000001,"15min":9.6020000000000003},"last":1.6319999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.71899999999999997},"min":{"1min":0.318,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":1.7789999999999999,"5min":5.2329999999999997,"15min":12.339},"last":1.23}]},{"osd":7,"last update":"Fri Jun 14 22:56:56 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.76700000000000002,"15min":0.71999999999999997},"min":{"1min":0.39300000000000002,"5min":0.31900000000000001,"15min":0.27900000000000003},"max":{"1min":3.0539999999999998,"5min":5.2629999999999999,"15min":5.2629999999999999},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.77700000000000002,"15min":0.72399999999999998},"min":{"1min":0.31900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.3399999999999999,"5min":10.638999999999999,"15min":10.638999999999999},"last":1.377}]}]},{"osd":4,"up_from":30,"seq":128849019197,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1750888,"kb_used_data":1160988,"kb_used_omap":1,"kb_used_meta":589822,"kb_avail":91994264,"statfs":{"total":95995035648,"available":94202126336,"internally_reserved":0,"allocated":1188851712,"data_stored":1188009546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":603978072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73699999999999999,"15min":1.002},"min":{"1min":0.216,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.9950000000000001,"5min":4.359,"15min":7.5650000000000004},"last":0.255},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.82699999999999996,"15min":1.2050000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":3.718,"5min":5.6130000000000004,"15min":9.1470000000000002},"last":0.44800000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:57:34 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":0.83499999999999996,"15min":1.0920000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.915,"5min":4.8250000000000002,"15min":5.141},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.7949999999999999,"5min":1.0600000000000001,"15min":1.274},"min":{"1min":0.39900000000000002,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":6.5549999999999997,"5min":6.5549999999999997,"15min":6.5549999999999997},"last":5.4160000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.59999999999999998,"15min":0.57699999999999996},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.0179999999999998,"5min":4.0179999999999998,"15min":4.0179999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.143,"5min":0.70399999999999996,"15min":0.61799999999999999},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.0490000000000004,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:57:15 2024","interfaces":[{"interface":"back","average":{"1min":1.647,"5min":0.78000000000000003,"15min":0.623},"min":{"1min":0.30199999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.9329999999999998,"5min":3.9329999999999998,"15min":3.9329999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.4179999999999999,"5min":0.80700000000000005,"15min":0.66500000000000004},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.9580000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":1.278}]},{"osd":5,"last update":"Fri Jun 14 22:56:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.73899999999999999,"15min":0.68999999999999995},"min":{"1min":0.44400000000000001,"5min":0.24199999999999999,"15min":0.21199999999999999},"max":{"1min":2.3010000000000002,"5min":3.9289999999999998,"15min":5.7030000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.67500000000000004,"15min":0.65800000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":1.7529999999999999,"5min":4.0369999999999999,"15min":4.7480000000000002},"last":0.313}]},{"osd":6,"last update":"Fri Jun 14 22:57:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.60099999999999998,"15min":0.63500000000000001},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.254},"max":{"1min":1.1539999999999999,"5min":3.8929999999999998,"15min":4.5960000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.53800000000000003,"15min":0.61099999999999999},"min":{"1min":0.28899999999999998,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":1.0680000000000001,"5min":1.647,"15min":5.0960000000000001},"last":0.53100000000000003}]},{"osd":7,"last update":"Fri Jun 14 22:57:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.65600000000000003,"15min":0.65000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.246},"max":{"1min":3.0950000000000002,"5min":3.0950000000000002,"15min":5.0389999999999997},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.173,"5min":0.86399999999999999,"15min":0.72599999999999998},"min":{"1min":0.372,"5min":0.308,"15min":0.29399999999999998},"max":{"1min":4.4029999999999996,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":0.50700000000000001}]}]},{"osd":3,"up_from":138,"seq":592705486896,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3228244,"kb_used_data":3076236,"kb_used_omap":4,"kb_used_meta":151995,"kb_avail":90516908,"statfs":{"total":95995035648,"available":92689313792,"internally_reserved":0,"allocated":3150065664,"data_stored":3149235335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":155643358},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.56399999999999995,"15min":0.40400000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":2.9660000000000002,"5min":3.4580000000000002,"15min":3.4580000000000002},"last":0.224},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.51200000000000001,"15min":0.40899999999999997},"min":{"1min":0.24399999999999999,"5min":0.185,"15min":0.185},"max":{"1min":1.2490000000000001,"5min":3.484,"15min":3.484},"last":0.23599999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":0.64800000000000002,"15min":0.45200000000000001},"min":{"1min":0.48299999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.2030000000000001,"5min":0.626,"15min":0.47099999999999997},"min":{"1min":0.36499999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.6890000000000001,"5min":3.6890000000000001,"15min":3.6890000000000001},"last":0.32200000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.56599999999999995,"15min":0.46200000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.224,"5min":0.59099999999999997,"15min":0.435},"min":{"1min":0.35499999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.8070000000000004,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.27700000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":0.78400000000000003,"15min":0.68200000000000005},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.7960000000000003,"5min":5.7960000000000003,"15min":5.7960000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.139,"5min":0.78600000000000003,"15min":0.68200000000000005},"min":{"1min":0.39800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.2720000000000002,"5min":5.2720000000000002,"15min":5.2720000000000002},"last":0.378}]},{"osd":5,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":0.85799999999999998,"15min":0.71099999999999997},"min":{"1min":0.44600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.3559999999999999,"5min":5.3559999999999999,"15min":5.3559999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.161,"5min":0.77800000000000002,"15min":0.66700000000000004},"min":{"1min":0.46200000000000002,"5min":0.439,"15min":0.439},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":0.57999999999999996}]},{"osd":6,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":0.77400000000000002,"15min":0.68700000000000006},"min":{"1min":0.46300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.536,"5min":2.536,"15min":2.536},"last":0.441},{"interface":"front","average":{"1min":1.1839999999999999,"5min":0.78100000000000003,"15min":0.66800000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.8259999999999996,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.46899999999999997}]},{"osd":7,"last update":"Fri Jun 14 22:57:08 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":0.94799999999999995,"15min":0.75},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":6.0369999999999999,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.591,"5min":0.89200000000000002,"15min":0.71899999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":5.3300000000000001,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":2.2189999999999999}]}]},{"osd":2,"up_from":133,"seq":571230650423,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1268756,"kb_used_data":1159436,"kb_used_omap":6,"kb_used_meta":109305,"kb_avail":92476396,"statfs":{"total":95995035648,"available":94695829504,"internally_reserved":0,"allocated":1187262464,"data_stored":1186432349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":111928821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.53300000000000003,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.587,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.627,"5min":0.46100000000000002,"15min":0.39400000000000002},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.448,"5min":2.448,"15min":2.448},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":0.96199999999999997,"15min":0.57399999999999995},"min":{"1min":0.248,"5min":0.218,"15min":0.218},"max":{"1min":5.5149999999999997,"5min":21.920000000000002,"15min":21.920000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.417,"5min":0.96199999999999997,"15min":0.56100000000000005},"min":{"1min":0.27300000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.4509999999999996,"5min":21.573,"15min":21.573},"last":0.29499999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:57:12 2024","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":0.71699999999999997,"15min":0.48099999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":21.66,"5min":21.66,"15min":21.66},"last":1.044},{"interface":"front","average":{"1min":2.145,"5min":0.91000000000000003,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.252},"max":{"1min":22.341000000000001,"5min":22.341000000000001,"15min":22.341000000000001},"last":0.84299999999999997}]},{"osd":4,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.69199999999999995,"15min":0.62},"min":{"1min":0.32300000000000001,"5min":0.318,"15min":0.318},"max":{"1min":4.2149999999999999,"5min":4.2149999999999999,"15min":4.2149999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.68799999999999994,"15min":0.61499999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":4.1890000000000001,"5min":4.1890000000000001,"15min":4.1890000000000001},"last":0.372}]},{"osd":5,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.64200000000000002,"15min":0.58699999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.649,"5min":3.8700000000000001,"15min":3.8700000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.60099999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.4710000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.42799999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.60599999999999998,"15min":0.57799999999999996},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.8320000000000001,"5min":3.915,"15min":3.915},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.163,"5min":0.71199999999999997,"15min":0.622},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":5.1829999999999998,"5min":5.1829999999999998,"15min":5.1829999999999998},"last":0.47699999999999998}]},{"osd":7,"last update":"Fri Jun 14 22:57:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.73699999999999999,"15min":0.64500000000000002},"min":{"1min":0.42799999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":4.3259999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.66000000000000003,"15min":0.60999999999999999},"min":{"1min":0.45500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.056,"5min":3.9529999999999998,"15min":3.9529999999999998},"last":0.52300000000000002}]}]},{"osd":1,"up_from":116,"seq":498216206440,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3853780,"kb_used_data":3475596,"kb_used_omap":5,"kb_used_meta":378106,"kb_avail":89891372,"statfs":{"total":95995035648,"available":92048764928,"internally_reserved":0,"allocated":3559010304,"data_stored":3558179909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":387180824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":0.97899999999999998,"15min":2.0339999999999998},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.3860000000000001,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.512,"5min":0.79300000000000004,"15min":1.5509999999999999},"min":{"1min":0.248,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.0220000000000002,"5min":5.1390000000000002,"15min":10.426},"last":4.5330000000000004}]},{"osd":2,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":0.78100000000000003,"15min":0.48399999999999999},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":5.3330000000000002,"5min":8.2880000000000003,"15min":8.2880000000000003},"last":2.589},{"interface":"front","average":{"1min":1.458,"5min":0.66300000000000003,"15min":0.44500000000000001},"min":{"1min":0.29799999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":5.077,"5min":5.077,"15min":5.077},"last":5.5119999999999996}]},{"osd":3,"last update":"Fri Jun 14 22:57:06 2024","interfaces":[{"interface":"back","average":{"1min":3.359,"5min":1.0209999999999999,"15min":0.56100000000000005},"min":{"1min":0.28100000000000003,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":21.422000000000001,"5min":21.422000000000001,"15min":21.422000000000001},"last":2.7719999999999998},{"interface":"front","average":{"1min":2.9020000000000001,"5min":1.02,"15min":0.58599999999999997},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":21.016999999999999,"5min":21.016999999999999,"15min":21.016999999999999},"last":3.3860000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.0860000000000001,"15min":1.8069999999999999},"min":{"1min":0.309,"5min":0.28899999999999998,"15min":0.28399999999999997},"max":{"1min":4.3639999999999999,"5min":7.673,"15min":8.8599999999999994},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.696,"5min":1.1200000000000001,"15min":1.79},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.3570000000000002,"5min":8.3620000000000001,"15min":11.161},"last":0.48799999999999999}]},{"osd":5,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.0569999999999999,"15min":1.6819999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.30599999999999999},"max":{"1min":5.3399999999999999,"5min":6.9020000000000001,"15min":9.7539999999999996},"last":4.5780000000000003},{"interface":"front","average":{"1min":1.694,"5min":1.117,"15min":1.869},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.4160000000000004,"5min":7.7439999999999998,"15min":8.125},"last":0.77300000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.0820000000000001,"15min":1.7290000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.32100000000000001},"max":{"1min":5.3849999999999998,"5min":6.9290000000000003,"15min":9.7279999999999998},"last":4.6449999999999996},{"interface":"front","average":{"1min":1.337,"5min":1.081,"15min":1.8360000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.899,"5min":7.7060000000000004,"15min":8.1479999999999997},"last":0.754}]},{"osd":7,"last update":"Fri Jun 14 22:57:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.0720000000000001,"15min":1.873},"min":{"1min":0.435,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":5.4080000000000004,"5min":8.3369999999999997,"15min":9.0960000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.496,"5min":1.0429999999999999,"15min":1.5720000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":5.3650000000000002,"5min":5.3650000000000002,"15min":9.5449999999999999},"last":4.617}]}]},{"osd":0,"up_from":91,"seq":390842024095,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1990292,"kb_used_data":1524684,"kb_used_omap":5,"kb_used_meta":465594,"kb_avail":91754860,"statfs":{"total":95995035648,"available":93956976640,"internally_reserved":0,"allocated":1561276416,"data_stored":1560444540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":476768346},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:57:33 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.2450000000000001,"15min":1.5549999999999999},"min":{"1min":0.188,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":5.7709999999999999,"5min":16.305,"15min":16.305},"last":1.0900000000000001},{"interface":"front","average":{"1min":2.3479999999999999,"5min":1.486,"15min":2.0950000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.9649999999999999,"5min":16.184999999999999,"15min":16.184999999999999},"last":4.1799999999999997}]},{"osd":2,"last update":"Fri Jun 14 22:57:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":0.79900000000000004,"15min":0.505},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.7679999999999998,"5min":16.757000000000001,"15min":16.757000000000001},"last":1.413},{"interface":"front","average":{"1min":1.4830000000000001,"5min":0.85399999999999998,"15min":0.55000000000000004},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":4.7350000000000003,"5min":16.055,"15min":16.055},"last":6.1219999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:57:09 2024","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.81999999999999995,"15min":0.51600000000000001},"min":{"1min":0.254,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.8090000000000002,"5min":16.873999999999999,"15min":16.873999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.373,"5min":0.76400000000000001,"15min":0.49399999999999999},"min":{"1min":0.27300000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.9969999999999999,"5min":16.085999999999999,"15min":16.085999999999999},"last":1.3540000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:57:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.1970000000000001,"15min":1.514},"min":{"1min":0.308,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":3.2610000000000001,"5min":16.018000000000001,"15min":16.018000000000001},"last":1.425},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.117,"15min":1.3109999999999999},"min":{"1min":0.36099999999999999,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":2.8700000000000001,"5min":15.976000000000001,"15min":15.976000000000001},"last":0.66300000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:57:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.335,"15min":1.407},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29299999999999998},"max":{"1min":7.1269999999999998,"5min":16.108000000000001,"15min":16.108000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.284,"15min":1.478},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.31900000000000001},"max":{"1min":3.9239999999999999,"5min":16.006,"15min":16.006},"last":1.2889999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:57:19 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1639999999999999,"15min":1.387},"min":{"1min":0.38800000000000001,"5min":0.35699999999999998,"15min":0.34300000000000003},"max":{"1min":2.895,"5min":16.152000000000001,"15min":16.152000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.179,"15min":1.4990000000000001},"min":{"1min":0.40500000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.4790000000000001,"5min":16.227,"15min":16.227},"last":1.3280000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:57:19 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.22,"15min":1.528},"min":{"1min":0.503,"5min":0.39000000000000001,"15min":0.33100000000000002},"max":{"1min":3.9550000000000001,"5min":16.199999999999999,"15min":16.199999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.2669999999999999,"15min":1.625},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.314},"max":{"1min":3.3039999999999998,"5min":16.259,"15min":16.259},"last":1.4510000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1550123008,"data_stored":1550123008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3529052160,"data_stored":3529048088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1179385856,"data_stored":1179385856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3131899904,"data_stored":3131899904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1178271744,"data_stored":1178271744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3549696000,"data_stored":3549691928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1186136064,"data_stored":1186136064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3549958144,"data_stored":3549954072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:58:01.018 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-14T22:58:01.018 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T22:58:01.019 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T22:58:01.019 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.387560 2024-06-14T22:58:01.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 3 0.3875596431595302 2024-06-14T22:58:01.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:00 smithi086 ceph-mon[97913]: from='client.35721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:58:01.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:00 smithi086 ceph-mon[102754]: from='client.35721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:58:01.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:01.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:01.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:01.160 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:00 smithi137 ceph-mon[90634]: from='client.35721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:58:01.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:01.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:01.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:01.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:01.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:01.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:01.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:01.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:01.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:01.694 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:01.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:01.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:01.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:01.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:01.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:01.981 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:01.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:02.073 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:01 smithi137 ceph-mon[90634]: pgmap v636: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-06-14T22:58:02.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:02.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:01 smithi086 ceph-mon[97913]: pgmap v636: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-06-14T22:58:02.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:01 smithi086 ceph-mon[102754]: pgmap v636: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-06-14T22:58:02.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:02.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T22:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T22:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:175743 9.7 9:e38634e6:::benchmark_data_smithi086_152062_object43261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-06-14T22:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:02.123365+0000", 2024-06-14T22:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0084428330000000003, 2024-06-14T22:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0084807140000000003, 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 175743 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.123365+0000", 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.123365+0000", 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.123364+0000", 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.123370+0000", 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4790000000000004e-06 2024-06-14T22:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.123381+0000", 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1379999999999999e-05 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.123386+0000", 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1660000000000002e-06 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.123463+0000", 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6483999999999995e-05 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.123601+0000", 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013817300000000001 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.123729+0000", 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012778400000000001 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:175744 9.d e161/145 9:bec51cc7:::benchmark_data_smithi086_152062_object43262:head v 161'11032, mlcod=161'11029)", 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:02.127110+0000", 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0046982289999999999, 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048199890000000002, 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127110+0000", 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:02.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127110+0000", 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127109+0000", 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127116+0000", 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7950000000000002e-06 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127128+0000", 2024-06-14T22:58:02.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2057e-05 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127134+0000", 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3929999999999999e-06 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127926+0000", 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000792912 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:02.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127948+0000", 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1481999999999999e-05 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:175748 9.b e161/140)", 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:02.127393+0000", 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0044150259999999998, 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045890790000000002, 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127393+0000", 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127393+0000", 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127393+0000", 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127398+0000", 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0989999999999998e-06 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127408+0000", 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0774000000000001e-05 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.127413+0000", 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0200000000000002e-06 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:175739 9.b e161/140 9:de4e6bbc:::benchmark_data_smithi086_152062_object43257:head v 161'11113, mlcod=161'11111)", 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:02.122488+0000", 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0093199059999999993, 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095396820000000007, 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.122488+0000", 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.122488+0000", 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.122487+0000", 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.122494+0000", 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9750000000000001e-06 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:02.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.122511+0000", 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6676000000000001e-05 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.122517+0000", 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1920000000000002e-06 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.122634+0000", 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011640499999999999 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:02.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:02.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:02.122653+0000", 2024-06-14T22:58:02.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8782000000000001e-05 2024-06-14T22:58:02.147 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:02.147 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:02.147 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:02.147 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:02.147 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T22:58:02.147 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 4 2024-06-14T22:58:02.147 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T22:58:02.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:02.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:02.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:02.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:02.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:02.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:02.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:02.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:02.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:02.730 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:02.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:02.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:02.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:02.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:02.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:03.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:03.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:03.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:03.036 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:03.040 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:03.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-14T22:58:03.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:03.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:03.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:03.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:03.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:03.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:03.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:03.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:03.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:03.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:03.661 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:58:03.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:03.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:03.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:03.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:03.850 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T22:58:03.868 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:03.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:03.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:04.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:04.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:04.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:03 smithi086 ceph-mon[102754]: pgmap v637: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-06-14T22:58:04.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3065902752' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:58:04.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:03 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:58:04.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:03 smithi086 ceph-mon[97913]: pgmap v637: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-06-14T22:58:04.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3065902752' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:58:04.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:03 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:58:04.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:04.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:04.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:04.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:03 smithi137 ceph-mon[90634]: pgmap v637: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-06-14T22:58:04.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3065902752' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:58:04.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:03 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:58:04.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:04.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:04.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:04.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:04.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:04.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:04.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:04.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:04.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:04.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:04.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:04.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:04.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:04.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:04.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:04.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:04.970 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:04 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:04.970 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:04 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:58:04.970 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:04 smithi137 ceph-mon[90634]: osdmap e162: 8 total, 8 up, 8 in 2024-06-14T22:58:04.970 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:04 smithi137 ceph-mon[90634]: pgmap v639: 89 pgs: 89 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-06-14T22:58:05.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:05.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:04 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:05.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:04 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:58:05.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:04 smithi086 ceph-mon[102754]: osdmap e162: 8 total, 8 up, 8 in 2024-06-14T22:58:05.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:04 smithi086 ceph-mon[102754]: pgmap v639: 89 pgs: 89 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-06-14T22:58:05.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:04 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:05.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:04 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:58:05.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:04 smithi086 ceph-mon[97913]: osdmap e162: 8 total, 8 up, 8 in 2024-06-14T22:58:05.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:04 smithi086 ceph-mon[97913]: pgmap v639: 89 pgs: 89 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-06-14T22:58:05.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:58:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:58:04] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.43.0" 2024-06-14T22:58:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:05.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:05.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:05.226 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:05.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:05.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:05.375 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:05.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:05.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:05.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:05.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:05.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:05.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:05.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:05.783 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:05.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:05.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:05.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:05.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:06.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:06.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:06.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:06.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:06.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:06.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:06.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:06.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:06.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:06.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:06.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:06.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:06.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:06.615 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:06.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:06.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:06.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:06.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:06.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:06.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:06.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:06.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:07.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:07.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:07.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:07.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:07.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:07.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:07.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:07.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:07.341 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T22:58:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T22:58:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T22:58:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T22:58:07.342 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T22:58:07.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:07.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:07.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:07.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:07.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:07.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:07.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:07.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:07.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:07.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:07.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:07 smithi086 ceph-mon[97913]: pgmap v640: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-06-14T22:58:07.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:07 smithi086 ceph-mon[102754]: pgmap v640: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-06-14T22:58:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:07.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:07.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:07.939 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:58:07.940 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:58:07.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:07.973 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:58:07.973 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:58:07.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:07 smithi137 ceph-mon[90634]: pgmap v640: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-06-14T22:58:08.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:08.004 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:08.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:08.009 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:58:08.010 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:58:08.051 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:58:08.051 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:58:08.095 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:58:08.095 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:58:08.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:08.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T22:58:08.132 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:58:08.132 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:58:08.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:08.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:08.179 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:58:08.179 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:58:08.221 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:58:08.222 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:58:08.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:08.270 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:08.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:08.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-14T22:58:08.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:08.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:08.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:08.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:08.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:08.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:08.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:08.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:08.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:08.753 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.3 primary-affinity to 0.38756 (8253992) 2024-06-14T22:58:08.766 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:58:08.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:08.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:08.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:08.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:08.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:08.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:08.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:08.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:08 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:08.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:08 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:08.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2366890515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:08.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1867559722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.3875596431595302}]: dispatch 2024-06-14T22:58:08.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:08 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.3875596431595302}]: dispatch 2024-06-14T22:58:08.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:08.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:08.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2366890515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:08.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1867559722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.3875596431595302}]: dispatch 2024-06-14T22:58:08.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.3875596431595302}]: dispatch 2024-06-14T22:58:08.998 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:08.998 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:08.998 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2366890515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:08.998 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1867559722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.3875596431595302}]: dispatch 2024-06-14T22:58:08.998 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:08 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.3875596431595302}]: dispatch 2024-06-14T22:58:09.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:09.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:09.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:09.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:09.198 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:09.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:09.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:09.321 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:09.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:09.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:09.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:09.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:09.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:09.568 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:09.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:09.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:09.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:09.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:09.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:09.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:09.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:09.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:09.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:09.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:09.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:09.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:10.020 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:09 smithi137 ceph-mon[90634]: pgmap v641: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-06-14T22:58:10.020 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:09 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.3875596431595302}]': finished 2024-06-14T22:58:10.021 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:09 smithi137 ceph-mon[90634]: osdmap e163: 8 total, 8 up, 8 in 2024-06-14T22:58:10.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:10.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:09 smithi086 ceph-mon[97913]: pgmap v641: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-06-14T22:58:10.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:09 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.3875596431595302}]': finished 2024-06-14T22:58:10.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:09 smithi086 ceph-mon[97913]: osdmap e163: 8 total, 8 up, 8 in 2024-06-14T22:58:10.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:09 smithi086 ceph-mon[102754]: pgmap v641: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-06-14T22:58:10.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:09 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.3875596431595302}]': finished 2024-06-14T22:58:10.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:09 smithi086 ceph-mon[102754]: osdmap e163: 8 total, 8 up, 8 in 2024-06-14T22:58:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:10.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:10.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:10.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:10.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:10.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:10.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:10.409 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:10.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:10.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:10.499 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T22:58:10.499 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:58:10.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:10.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:10.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:10.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:10.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:10.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:10.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:10.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:10.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:10.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:10.958 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:10.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:11.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:11.077 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:10 smithi137 ceph-mon[90634]: osdmap e164: 8 total, 8 up, 8 in 2024-06-14T22:58:11.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:10 smithi086 ceph-mon[97913]: osdmap e164: 8 total, 8 up, 8 in 2024-06-14T22:58:11.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:10 smithi086 ceph-mon[102754]: osdmap e164: 8 total, 8 up, 8 in 2024-06-14T22:58:11.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:11.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:11.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:11.234 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:11.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:11.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:11.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:11.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:11.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:11.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T22:58:11.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:11.525 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:11.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:11.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:11.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:11.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:11.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:11.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:11.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:11.860 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:58:11.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:11.981 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:11.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:12.078 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:11 smithi086 ceph-mon[102754]: pgmap v644: 89 pgs: 4 peering, 85 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 965 op/s 2024-06-14T22:58:12.078 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:11 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T22:58:12.078 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:11 smithi086 ceph-mon[97913]: pgmap v644: 89 pgs: 4 peering, 85 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 965 op/s 2024-06-14T22:58:12.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:11 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T22:58:12.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:12.113 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:12.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:12.115 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:11 smithi137 ceph-mon[90634]: pgmap v644: 89 pgs: 4 peering, 85 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 965 op/s 2024-06-14T22:58:12.115 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:11 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T22:58:12.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:12.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:12.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:12.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:12.378 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:12.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:12.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:12.510 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:12.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:12.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:12.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:12.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:12.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:12.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:58:12.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:12.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:12.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:12.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:12.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:13.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:13.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:13.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:13.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:13.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:13.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:13.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:13.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:13.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:13.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:13.479 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:13.479 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:13.479 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:13 smithi086 ceph-mon[97913]: pgmap v645: 89 pgs: 4 peering, 85 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 638 op/s 2024-06-14T22:58:13.479 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:13.480 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:13.480 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:13 smithi086 ceph-mon[102754]: pgmap v645: 89 pgs: 4 peering, 85 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 638 op/s 2024-06-14T22:58:13.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:13.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:13.573 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:13.573 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:13.573 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:13 smithi137 ceph-mon[90634]: pgmap v645: 89 pgs: 4 peering, 85 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 638 op/s 2024-06-14T22:58:13.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:13.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:13.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:13.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:13.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:13.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:13.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:13.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:13.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:14.004 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T22:58:14.004 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:58:14.004 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:14.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:14.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:14.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:14.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:14.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:14.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:14.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:14.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:14.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:14.451 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:14.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:14.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:14.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:14.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:14.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-14T22:58:14.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:14.744 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T22:58:14.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:14.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:14.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:14.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3852445814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3852445814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:58:14.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:58:14.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:14.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:14.882 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T22:58:14.882 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T22:58:14.882 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-06-14T22:58:14.882 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T22:58:14.882 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T22:58:14.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-06-14T22:58:14.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:14.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:14.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:14 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:14.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:14 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:14.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3852445814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:14.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:14 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:14.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:14 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:58:14.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:14 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:58:14.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:14 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:15.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:15.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:15.130 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:58:15.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:15.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:15.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:15.249 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:58:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:58:14] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T22:58:15.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:15.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:15.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:15.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:15.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:15.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:15.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:15.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:15.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:15.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:15.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:15.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:15.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:15.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:15.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:15.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:15.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:15 smithi137 ceph-mon[90634]: pgmap v646: 89 pgs: 3 peering, 86 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 684 op/s 2024-06-14T22:58:15.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:15 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:58:15.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:15 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:58:15.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:15 smithi137 ceph-mon[90634]: osdmap e165: 8 total, 8 up, 8 in 2024-06-14T22:58:16.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:16.054 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:16.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:16.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:15 smithi086 ceph-mon[97913]: pgmap v646: 89 pgs: 3 peering, 86 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 684 op/s 2024-06-14T22:58:16.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:15 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:58:16.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:15 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:58:16.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:15 smithi086 ceph-mon[97913]: osdmap e165: 8 total, 8 up, 8 in 2024-06-14T22:58:16.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:15 smithi086 ceph-mon[102754]: pgmap v646: 89 pgs: 3 peering, 86 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 684 op/s 2024-06-14T22:58:16.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:15 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:58:16.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:15 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:58:16.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:15 smithi086 ceph-mon[102754]: osdmap e165: 8 total, 8 up, 8 in 2024-06-14T22:58:16.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:16.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:16.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:16.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:16.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:16.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:16.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:16.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:16.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:16.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:16.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:16.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:16.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:16.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:16.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:16.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:16.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:16.899 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:16.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:16.964 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:16 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T22:58:16.965 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:16 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:58:16.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:16 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T22:58:16.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:16 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:58:16.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:16 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T22:58:16.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:16 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:58:17.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:17.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:17.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:17.031 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:58:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:58:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:17.073 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:58:17.073 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:58:17.104 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:58:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:58:17.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:17.136 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:58:17.136 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:58:17.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:17.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:17.165 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:58:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:58:17.195 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:58:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:58:17.229 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:58:17.229 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:58:17.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:17.268 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:58:17.269 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:58:17.300 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:17.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:17.399 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T22:58:17.400 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T22:58:17.400 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T22:58:17.400 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T22:58:17.400 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T22:58:17.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:17.439 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:17.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:17.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:17.581 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:17.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:17.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:17.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:17.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:17.860 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:17.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:17.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:17.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:17.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:18.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:17 smithi086 ceph-mon[102754]: pgmap v648: 89 pgs: 89 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 968 op/s 2024-06-14T22:58:18.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/389423341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:18.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:17 smithi086 ceph-mon[97913]: pgmap v648: 89 pgs: 89 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 968 op/s 2024-06-14T22:58:18.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/389423341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:18.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:18.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:18.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:18.144 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:17 smithi137 ceph-mon[90634]: pgmap v648: 89 pgs: 89 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 968 op/s 2024-06-14T22:58:18.144 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/389423341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:18.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T22:58:18.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-14T22:58:18.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:18.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:18.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:18.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:18.408 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:18.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:18.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:18.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:18.628 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:58:18.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:18.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:18.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:18.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:18.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:18.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:18.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:18.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:18.934 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:18.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:19.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:19.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:19.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:19.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:19.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:19.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:19.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:19.336 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:19.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:19.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:19.470 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:19.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:19.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:19.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:19.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:19.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:19.726 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:58:19.726 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-14T22:58:19.726 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:58:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:19.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:19.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:19 smithi086 ceph-mon[97913]: pgmap v649: 89 pgs: 89 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 611 op/s 2024-06-14T22:58:19.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:19 smithi086 ceph-mon[102754]: pgmap v649: 89 pgs: 89 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 611 op/s 2024-06-14T22:58:19.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:19.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:19.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:19.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:20.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:20.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:20.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:20.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:20.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:20.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:20.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:20.188 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:19 smithi137 ceph-mon[90634]: pgmap v649: 89 pgs: 89 active+clean; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 611 op/s 2024-06-14T22:58:20.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:58:20.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:20.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:20.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:20.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:20.439 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:20.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:20.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:20.587 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:20.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:20.599 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:20.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:20.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:20.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:20.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:20.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:20.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:20.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:21.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:21.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:21.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:21.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:21.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:21.158 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T22:58:21.158 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T22:58:21.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:187999 9.1 9:84850bcf:::benchmark_data_smithi086_152062_object55517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:17.253091+0000", 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.8996884180000002, 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033320536999999997, 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 187999 2024-06-14T22:58:21.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.253091+0000", 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.253091+0000", 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.253089+0000", 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.253097+0000", 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1210000000000007e-06 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.253116+0000", 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8581e-05 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.253122+0000", 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2720000000000003e-06 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.253256+0000", 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013349499999999999 2024-06-14T22:58:21.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.253587+0000", 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033114899999999998 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.253880+0000", 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000293164 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.258409+0000", 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045285200000000003 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.261512+0000", 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031037870000000002 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.286132+0000", 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024619285000000001 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.286289+0000", 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015745700000000001 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.286412+0000", 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012257500000000001 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:188601 9.1 9:81481142:::benchmark_data_smithi086_152062_object56119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:17.928244+0000", 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.22453601, 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028903893, 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188601 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.166 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.928244+0000", 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.928244+0000", 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.928242+0000", 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.928250+0000", 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8130000000000002e-06 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.928273+0000", 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2719000000000001e-05 2024-06-14T22:58:21.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.928281+0000", 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3819999999999998e-06 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.928332+0000", 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.113e-05 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.928646+0000", 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031421699999999999 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.928953+0000", 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000307411 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.937526+0000", 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0085726910000000003 2024-06-14T22:58:21.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.941720+0000", 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041938160000000004 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.957047+0000", 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015327431000000001 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.957091+0000", 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3291999999999998e-05 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.957147+0000", 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6904000000000003e-05 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:188606 9.1 9:8d87f240:::benchmark_data_smithi086_152062_object56124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:17.933670+0000", 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.219109037, 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037619342, 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188606 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.933670+0000", 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.933670+0000", 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.933669+0000", 2024-06-14T22:58:21.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.933673+0000", 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2320000000000003e-06 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.933681+0000", 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3250000000000002e-06 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.933685+0000", 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8449999999999996e-06 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.937547+0000", 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038623469999999999 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.937647+0000", 2024-06-14T22:58:21.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010021100000000001 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.937752+0000", 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010499499999999999 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.946993+0000", 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0092410259999999994 2024-06-14T22:58:21.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.947017+0000", 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3810000000000001e-05 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.971153+0000", 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024135663000000002 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.971246+0000", 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3844999999999993e-05 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.971290+0000", 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3398e-05 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:188610 9.1 9:8b4e40a5:::benchmark_data_smithi086_152062_object56128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.179 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:17.940005+0000", 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.2127747520000001, 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031411447000000002, 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188610 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.940005+0000", 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.940005+0000", 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.940003+0000", 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T22:58:21.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.940010+0000", 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9689999999999997e-06 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.940026+0000", 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6025999999999999e-05 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.940033+0000", 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6610000000000003e-06 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.941811+0000", 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017779860000000001 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.942164+0000", 2024-06-14T22:58:21.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000353543 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.942491+0000", 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032685599999999999 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.947039+0000", 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045479689999999998 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.951806+0000", 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047675260000000002 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.971319+0000", 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019513044 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.971383+0000", 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3354000000000004e-05 2024-06-14T22:58:21.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.971416+0000", 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3368000000000003e-05 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:188615 9.1 9:8ee13e2c:::benchmark_data_smithi086_152062_object56133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:17.942157+0000", 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.2106221349999999, 2024-06-14T22:58:21.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029960166, 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188615 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.942157+0000", 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.942157+0000", 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.942156+0000", 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.942161+0000", 2024-06-14T22:58:21.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2279999999999998e-06 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.942174+0000", 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2676e-05 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.942180+0000", 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6339999999999998e-06 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.947058+0000", 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048785850000000004 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.947162+0000", 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103293 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.947257+0000", 2024-06-14T22:58:21.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5748000000000006e-05 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.956988+0000", 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097305489999999998 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.957021+0000", 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.311e-05 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.972036+0000", 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015014949 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.972077+0000", 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1338000000000003e-05 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:17.972118+0000", 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0246000000000001e-05 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:188772 9.1 9:8f0f1bd9:::benchmark_data_smithi086_152062_object56290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.111757+0000", 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.0410230170000001, 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030416647000000002, 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.193 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188772 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.111757+0000", 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.111757+0000", 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.111755+0000", 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.111762+0000", 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2859999999999996e-06 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.111773+0000", 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1865000000000001e-05 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.111779+0000", 2024-06-14T22:58:21.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9190000000000004e-06 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.111904+0000", 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124843 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.112259+0000", 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035432600000000002 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.112626+0000", 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036699900000000001 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.121760+0000", 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091347259999999993 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.121798+0000", 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7441000000000003e-05 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.141897+0000", 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020099625999999999 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.142048+0000", 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015057100000000001 2024-06-14T22:58:21.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.142173+0000", 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125244 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:188774 9.1 9:87d1e397:::benchmark_data_smithi086_152062_object56292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.112029+0000", 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.040750133, 2024-06-14T22:58:21.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030561839, 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188774 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.112029+0000", 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.112029+0000", 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.112029+0000", 2024-06-14T22:58:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.112033+0000", 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0679999999999998e-06 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.112043+0000", 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0010000000000001e-05 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.112047+0000", 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7040000000000002e-06 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117300+0000", 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052520839999999997 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117431+0000", 2024-06-14T22:58:21.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013112399999999999 2024-06-14T22:58:21.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117555+0000", 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012461300000000001 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.121967+0000", 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044117749999999997 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.132448+0000", 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010481214000000001 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.142343+0000", 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098942739999999998 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.142470+0000", 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012749199999999999 2024-06-14T22:58:21.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.142591+0000", 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012116 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:188780 9.1 9:869d37b3:::benchmark_data_smithi086_152062_object56298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.117536+0000", 2024-06-14T22:58:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.0352437069999998, 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047024269, 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188780 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117536+0000", 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117536+0000", 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117535+0000", 2024-06-14T22:58:21.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117540+0000", 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3109999999999996e-06 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117559+0000", 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9017000000000001e-05 2024-06-14T22:58:21.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117564+0000", 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2909999999999998e-06 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.122079+0000", 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045148719999999996 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.122375+0000", 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029629800000000001 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.122674+0000", 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029894600000000002 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.132567+0000", 2024-06-14T22:58:21.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098928149999999992 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.132615+0000", 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7886000000000002e-05 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.164290+0000", 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031675165999999998 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.164440+0000", 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000149727 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.164560+0000", 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120161 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:188781 9.1 9:8539e34d:::benchmark_data_smithi086_152062_object56299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.117571+0000", 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.0352085670000002, 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047510191, 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188781 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.216 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117571+0000", 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117571+0000", 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117571+0000", 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117573+0000", 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4219999999999999e-06 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117579+0000", 2024-06-14T22:58:21.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5589999999999996e-06 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.117582+0000", 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4230000000000001e-06 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.127245+0000", 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096633009999999991 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.127516+0000", 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027064 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.127820+0000", 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000303968 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.132632+0000", 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004812017 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.132885+0000", 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025338200000000002 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.164673+0000", 2024-06-14T22:58:21.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031788163000000001 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.164808+0000", 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013488500000000001 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.165081+0000", 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027283299999999999 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.222 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:188797 9.1 9:8a4520b6:::benchmark_data_smithi086_152062_object56315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.136947+0000", 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.0158323450000002, 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032218038999999997, 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.223 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188797 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.136947+0000", 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.136947+0000", 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.136945+0000", 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.136953+0000", 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7689999999999996e-06 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.136973+0000", 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0015999999999999e-05 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.136980+0000", 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.002e-06 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.137044+0000", 2024-06-14T22:58:21.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3479999999999995e-05 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.137171+0000", 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127348 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.137286+0000", 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115388 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.142724+0000", 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054381050000000004 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.150149+0000", 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0074249670000000002 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.168905+0000", 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018755794999999999 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.169040+0000", 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013437799999999999 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.169165+0000", 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012564799999999999 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189013 9.1 9:88577d5b:::benchmark_data_smithi086_152062_object56531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.390664+0000", 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.7621155580000001, 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032374552000000001, 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189013 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.230 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.390664+0000", 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.390664+0000", 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.390664+0000", 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.390667+0000", 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.287e-06 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.390673+0000", 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4370000000000004e-06 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.390677+0000", 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7019999999999999e-06 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.398148+0000", 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0074714969999999997 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.398358+0000", 2024-06-14T22:58:21.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000209605 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.398483+0000", 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012462700000000001 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.404336+0000", 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005853713 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.404365+0000", 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8755000000000002e-05 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.422866+0000", 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018500989999999998 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.422990+0000", 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012411200000000001 2024-06-14T22:58:21.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.423039+0000", 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8270999999999999e-05 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189119 9.1 9:8aa38c87:::benchmark_data_smithi086_152062_object56637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.516117+0000", 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.6366624299999999, 2024-06-14T22:58:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031010658999999999, 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189119 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.516117+0000", 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.516117+0000", 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.516116+0000", 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.516121+0000", 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8999999999999997e-06 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.516133+0000", 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2148e-05 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.516139+0000", 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8610000000000003e-06 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.520622+0000", 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044825220000000001 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.520904+0000", 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028274099999999998 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.521216+0000", 2024-06-14T22:58:21.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031193799999999999 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.526955+0000", 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057386579999999998 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.527054+0000", 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8889999999999997e-05 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.547033+0000", 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019979203000000001 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.547087+0000", 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3919000000000003e-05 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.547128+0000", 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0896999999999998e-05 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189734 9.1 9:8ff7ef26:::benchmark_data_smithi086_152062_object57252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:19.288072+0000", 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.8647072769999999, 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050300967000000002, 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189734 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288072+0000", 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288072+0000", 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288071+0000", 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288076+0000", 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.437e-06 2024-06-14T22:58:21.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288089+0000", 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2792000000000001e-05 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288094+0000", 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9609999999999996e-06 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.302023+0000", 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013928954 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.302121+0000", 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8340999999999999e-05 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.302214+0000", 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3197999999999998e-05 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.327446+0000", 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025231593 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.332902+0000", 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054563620000000002 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.338318+0000", 2024-06-14T22:58:21.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054156949999999999 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.338349+0000", 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0892000000000002e-05 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.338373+0000", 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4530999999999999e-05 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.249 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189735 9.1 9:8c974211:::benchmark_data_smithi086_152062_object57253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:19.288420+0000", 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.8643592849999999, 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.060468734000000003, 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.250 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189735 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288420+0000", 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288420+0000", 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288419+0000", 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288425+0000", 2024-06-14T22:58:21.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5589999999999996e-06 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288441+0000", 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6191000000000001e-05 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288447+0000", 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9580000000000004e-06 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.306503+0000", 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018056493999999999 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.306775+0000", 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000271439 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.307076+0000", 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030119800000000003 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.327473+0000", 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020397408999999998 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.332910+0000", 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054364610000000001 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.348788+0000", 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015878006 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.348846+0000", 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8294999999999997e-05 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.348889+0000", 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2718e-05 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189739 9.1 9:8f3b18eb:::benchmark_data_smithi086_152062_object57257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:19.293852+0000", 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.858927239, 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059466487999999998, 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.257 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189739 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293852+0000", 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293852+0000", 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293851+0000", 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293859+0000", 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8529999999999992e-06 2024-06-14T22:58:21.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293877+0000", 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8148e-05 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293883+0000", 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7320000000000001e-06 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.311872+0000", 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017989092000000002 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.312176+0000", 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000303696 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.312487+0000", 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000310365 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.327500+0000", 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015013253000000001 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.332925+0000", 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054252670000000001 2024-06-14T22:58:21.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.353087+0000", 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020162269999999999 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.353204+0000", 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011688099999999999 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.353319+0000", 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114548 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189748 9.1 9:8e23fb81:::benchmark_data_smithi086_152062_object57266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:19.301971+0000", 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.8508088599999999, 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.051732593, 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.264 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189748 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.301971+0000", 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.301971+0000", 2024-06-14T22:58:21.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.301969+0000", 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.301976+0000", 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4409999999999998e-06 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.303219+0000", 2024-06-14T22:58:21.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001243383 2024-06-14T22:58:21.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:21.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.303229+0000", 2024-06-14T22:58:21.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7089999999999997e-06 2024-06-14T22:58:21.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.316975+0000", 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01374614 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.317090+0000", 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114838 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.317195+0000", 2024-06-14T22:58:21.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105105 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.327523+0000", 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01032858 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.332930+0000", 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054065099999999998 2024-06-14T22:58:21.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.353428+0000", 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020498512 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.353590+0000", 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016175200000000001 2024-06-14T22:58:21.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.353703+0000", 2024-06-14T22:58:21.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113136 2024-06-14T22:58:21.273 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.273 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189766 9.1 9:894c26ac:::benchmark_data_smithi086_152062_object57284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:19.326604+0000", 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.826175377, 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033192684, 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.274 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189766 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.326604+0000", 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.326604+0000", 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.326603+0000", 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.326610+0000", 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4599999999999997e-06 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.326627+0000", 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6962000000000001e-05 2024-06-14T22:58:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.326633+0000", 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.998e-06 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.327546+0000", 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00091345 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.327690+0000", 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014359799999999999 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.327820+0000", 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013007900000000001 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.333240+0000", 2024-06-14T22:58:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054202449999999998 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.338381+0000", 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051410350000000004 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.359679+0000", 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021297528 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.359756+0000", 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7526000000000003e-05 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.359797+0000", 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0448000000000003e-05 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189912 9.1 9:8a12fff3:::benchmark_data_smithi086_152062_object57430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.280 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:19.486454+0000", 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.666325544, 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029994907000000001, 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189912 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.486454+0000", 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.486454+0000", 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.486452+0000", 2024-06-14T22:58:21.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.486459+0000", 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7819999999999996e-06 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.486477+0000", 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7385e-05 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.486483+0000", 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3409999999999999e-06 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.486825+0000", 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034215800000000001 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.487707+0000", 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00088231500000000005 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.487778+0000", 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1056000000000005e-05 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.496514+0000", 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0087358479999999992 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.511248+0000", 2024-06-14T22:58:21.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014733507999999999 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.516168+0000", 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0049203140000000003 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.516328+0000", 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015991000000000001 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.516449+0000", 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120868 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:191119 9.1 9:87c85257:::benchmark_data_smithi086_152062_object58637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:20.833923+0000", 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.31885681799999999, 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041507448000000002, 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.287 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191119 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.833923+0000", 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.833923+0000", 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.833914+0000", 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999909 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.833930+0000", 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5849e-05 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.833951+0000", 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1829000000000001e-05 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.833958+0000", 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7789999999999998e-06 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.834027+0000", 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8449999999999997e-05 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.834135+0000", 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108515 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.834248+0000", 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112633 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.841062+0000", 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006814184 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.844726+0000", 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036638920000000002 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.875108+0000", 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030382501999999999 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.875290+0000", 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018128799999999999 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.875430+0000", 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000140517 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.293 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:191131 9.1 9:81e94d9f:::benchmark_data_smithi086_152062_object58649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:20.846561+0000", 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.30621860699999998, 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029142102, 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:21.294 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191131 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.846561+0000", 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.846561+0000", 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.846559+0000", 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.846569+0000", 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0295000000000001e-05 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.846591+0000", 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2127e-05 2024-06-14T22:58:21.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.846599+0000", 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1429999999999998e-06 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.846741+0000", 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141395 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T22:58:21.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.847012+0000", 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027125599999999999 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.847304+0000", 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000292324 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.860043+0000", 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012738803 2024-06-14T22:58:21.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.860075+0000", 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1773999999999998e-05 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.875580+0000", 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015504789 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T22:58:21.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.875661+0000", 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1218999999999996e-05 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:20.875703+0000", 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2067999999999998e-05 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T22:58:21.300 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T22:58:21.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:21.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:21.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:21.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:21.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:21.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:21.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:21.690 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:21.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:21.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:21.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:21.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:21.889 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:21.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:21.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:21.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:21 smithi137 ceph-mon[90634]: pgmap v650: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-06-14T22:58:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:22.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:22.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:21 smithi086 ceph-mon[97913]: pgmap v650: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-06-14T22:58:22.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:21 smithi086 ceph-mon[102754]: pgmap v650: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 865 op/s 2024-06-14T22:58:22.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-14T22:58:22.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:22.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:22.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:22.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:22.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:22.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:22.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:22.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:22.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:22.532 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:58:22.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:22.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:22.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:22.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:22.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:22.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:22.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:22.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:22.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:22.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:22.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:23.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:23.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:23.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:23.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:23.240 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:23.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:23.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:23.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:23.372 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:23.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:23.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:23.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:23.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:23.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:23.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:23.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:23.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:23.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:23.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:23.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:23.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:23.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:23 smithi086 ceph-mon[97913]: pgmap v651: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 874 op/s 2024-06-14T22:58:23.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:23.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:23 smithi086 ceph-mon[102754]: pgmap v651: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 874 op/s 2024-06-14T22:58:23.855 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:23.855 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:23 smithi137 ceph-mon[90634]: pgmap v651: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 874 op/s 2024-06-14T22:58:23.856 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 22:58:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=cleanup t=2024-06-14T22:58:23.767703359Z level=info msg="Completed cleanup jobs" duration=13.641425ms 2024-06-14T22:58:23.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:23.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:23.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:24.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:24.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:24.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:24.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:24.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:24.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:24.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:24.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:24.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:24.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:24.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:24.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:24.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:24.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:24.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:24.665 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:24.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:24.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:24.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:24.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:24.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:24.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:24.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:25.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:25.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:25.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:25.074 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:58:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:58:24] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T22:58:25.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:25.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:25.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:25.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:25.333 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:25.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:25.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:25.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:25.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:25.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:25.596 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:58:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:58:25.598 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T22:58:25.598 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T22:58:25.598 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T22:58:25.598 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.396210 2024-06-14T22:58:25.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 7 0.3962101288436226 2024-06-14T22:58:25.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:25.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:25.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:25.653 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:58:25.653 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:58:25.694 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:58:25.694 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:58:25.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:25.718 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T22:58:25.744 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:58:25.744 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:58:25.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:25.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:25.787 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:58:25.788 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:58:25.819 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:58:25.819 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:58:25.860 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:58:25.860 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:58:25.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:25.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:25.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:25.894 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:58:25.894 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:58:25.907 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:25 smithi086 ceph-mon[97913]: pgmap v652: 89 pgs: 89 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 862 op/s 2024-06-14T22:58:25.907 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/862323823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:25.907 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3787935914' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:58:25.907 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:25 smithi086 ceph-mon[102754]: pgmap v652: 89 pgs: 89 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 862 op/s 2024-06-14T22:58:25.908 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/862323823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:25.908 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3787935914' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:58:25.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:25.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:25 smithi137 ceph-mon[90634]: pgmap v652: 89 pgs: 89 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 862 op/s 2024-06-14T22:58:25.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/862323823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:25.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3787935914' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:58:26.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:26.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:26.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:26.143 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T22:58:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T22:58:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-06-14T22:58:26.143 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T22:58:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:26.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:26.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:26.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:26.286 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:26.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:26.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:26.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:26.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:26.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:26.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:26.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:26.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:26.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:26.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:26.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:26.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:26.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:26.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:26.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:26.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:26 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:26.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3787935914' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:58:26.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:26 smithi137 ceph-mon[90634]: osdmap e166: 8 total, 8 up, 8 in 2024-06-14T22:58:27.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-14T22:58:27.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:27.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:26 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:27.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3787935914' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:58:27.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:26 smithi086 ceph-mon[97913]: osdmap e166: 8 total, 8 up, 8 in 2024-06-14T22:58:27.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:26 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:27.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3787935914' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:58:27.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:26 smithi086 ceph-mon[102754]: osdmap e166: 8 total, 8 up, 8 in 2024-06-14T22:58:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:27.096 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:27.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:27.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:27.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:27.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:27.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:27.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:27.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:27.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:27.516 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:58:27.526 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:27.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:27.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:27.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:27.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:27.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:27 smithi086 ceph-mon[97913]: pgmap v654: 89 pgs: 89 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-06-14T22:58:27.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:27.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:27.732 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:27 smithi086 ceph-mon[102754]: pgmap v654: 89 pgs: 89 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-06-14T22:58:27.732 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:27.732 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:27.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:27.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:27.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:27.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:27.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:27.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:27.979 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:27 smithi137 ceph-mon[90634]: pgmap v654: 89 pgs: 89 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-06-14T22:58:27.979 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:58:27.979 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:28.030 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:28.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:28.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:28.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:28.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:28.246 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:28.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:28.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:28.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:28.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:28.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:28.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:28.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:28.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:28.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:28.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:28.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:28.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:28.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:29.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:29.100 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:29.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:29.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:29.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:29.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:29.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:29.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:29.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:29.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:29.510 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:29.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:29.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:58:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T22:58:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:58:29.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:29.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:29.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:29.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:29.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:29.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:29.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:29 smithi086 ceph-mon[97913]: pgmap v655: 89 pgs: 89 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-06-14T22:58:29.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:29 smithi086 ceph-mon[102754]: pgmap v655: 89 pgs: 89 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-06-14T22:58:29.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:29.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:29.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:29.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:29.955 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:29 smithi137 ceph-mon[90634]: pgmap v655: 89 pgs: 89 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-06-14T22:58:30.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T22:58:30.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:30.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:30.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:30.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:30.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:30.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:30.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-14T22:58:30.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:30.381 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:30.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:30.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:30.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:30.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:30.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:30.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:30.705 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T22:58:30.739 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:30.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:30.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:30.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:30.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:30.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:31.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:31.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:31.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:31.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:31.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:31.248 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:31.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:31.303 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:31.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:31.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:31.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:31.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:31.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:31.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:31.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:31.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:31.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:31.709 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:31.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.7 primary-affinity to 0.39621 (8259662) 2024-06-14T22:58:31.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:31.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:31.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:31.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:31.975 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:31.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:31.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:31 smithi137 ceph-mon[90634]: pgmap v656: 89 pgs: 89 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-06-14T22:58:31.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2005651086' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3962101288436226}]: dispatch 2024-06-14T22:58:32.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:32.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:31 smithi086 ceph-mon[97913]: pgmap v656: 89 pgs: 89 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-06-14T22:58:32.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2005651086' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3962101288436226}]: dispatch 2024-06-14T22:58:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:31 smithi086 ceph-mon[102754]: pgmap v656: 89 pgs: 89 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-06-14T22:58:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2005651086' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3962101288436226}]: dispatch 2024-06-14T22:58:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:32.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:32.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:32.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:32.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:32.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:32.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:32.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:32.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:32.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:32.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:32.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:32.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:32.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:32.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:32.720 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:32.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:32.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:32.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:32.907 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2005651086' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3962101288436226}]': finished 2024-06-14T22:58:32.907 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:32 smithi137 ceph-mon[90634]: osdmap e167: 8 total, 8 up, 8 in 2024-06-14T22:58:32.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:32.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:32.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:32.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:33.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2005651086' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3962101288436226}]': finished 2024-06-14T22:58:33.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:32 smithi086 ceph-mon[102754]: osdmap e167: 8 total, 8 up, 8 in 2024-06-14T22:58:33.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2005651086' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.3962101288436226}]': finished 2024-06-14T22:58:33.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:32 smithi086 ceph-mon[97913]: osdmap e167: 8 total, 8 up, 8 in 2024-06-14T22:58:33.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:33.144 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:33.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:33.214 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T22:58:33.214 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:58:33.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:33.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:33.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:33.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:33.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:33.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:33.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:33.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:33.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:33.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:33.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:33.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:33.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:33.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:33.960 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:33.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:33.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:33 smithi137 ceph-mon[90634]: pgmap v658: 89 pgs: 89 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 841 op/s 2024-06-14T22:58:33.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:33 smithi137 ceph-mon[90634]: osdmap e168: 8 total, 8 up, 8 in 2024-06-14T22:58:34.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-14T22:58:34.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:34.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:33 smithi086 ceph-mon[102754]: pgmap v658: 89 pgs: 89 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 841 op/s 2024-06-14T22:58:34.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:33 smithi086 ceph-mon[102754]: osdmap e168: 8 total, 8 up, 8 in 2024-06-14T22:58:34.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:33 smithi086 ceph-mon[97913]: pgmap v658: 89 pgs: 89 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 841 op/s 2024-06-14T22:58:34.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:33 smithi086 ceph-mon[97913]: osdmap e168: 8 total, 8 up, 8 in 2024-06-14T22:58:34.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:34.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:34.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:34.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:34.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:34.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:34.398 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:34.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:34.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:34.531 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:34.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:34.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:34.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:34.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:34.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:34.809 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:34.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:34.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:34.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:34.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:35.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:35.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:35.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:35.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:58:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:58:34] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.43.0" 2024-06-14T22:58:35.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:35.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:35.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:35.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:35.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:35.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:35.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:35.476 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:35.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:35.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:35.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:35.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:35.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:35.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:35.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:35.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:35.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:35.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:35.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:36.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:36.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:36.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:35 smithi086 ceph-mon[102754]: pgmap v660: 89 pgs: 89 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-06-14T22:58:36.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:35 smithi086 ceph-mon[97913]: pgmap v660: 89 pgs: 89 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-06-14T22:58:36.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:36.170 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:36.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:36.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:35 smithi137 ceph-mon[90634]: pgmap v660: 89 pgs: 89 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-06-14T22:58:36.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:58:36.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:58:36.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:36.294 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:58:36.294 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:58:36.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:36.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:36.329 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:58:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:58:36.366 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:58:36.366 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:58:36.401 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:58:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:58:36.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:36.438 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:58:36.439 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:58:36.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:36.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:36.471 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:58:36.471 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:58:36.501 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:58:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:58:36.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:36.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:36.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:36.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:36.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:36.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:36.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:36.860 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:36.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:36.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:36.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:36.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:37.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3400520682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:37.029 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3400520682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:37.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:37.133 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:37.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:37.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:37.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3400520682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:37.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:37.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:37.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:37.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:37.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:37.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:37.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:37.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:37.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:37.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:37.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:37.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:37.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:37.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:37.949 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:37.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:37.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:38.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:38.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:37 smithi086 ceph-mon[102754]: pgmap v661: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-06-14T22:58:38.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:37 smithi086 ceph-mon[97913]: pgmap v661: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-06-14T22:58:38.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:38.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:38.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:38.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:38.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:38.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:38.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:37 smithi137 ceph-mon[90634]: pgmap v661: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-06-14T22:58:38.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:38.396 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:38.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:38.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:38.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:38.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:38.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:38.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:38.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:38.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:38.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:38.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:38.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:38.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:38.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:38.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:38.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:39.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:39.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:39.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:39.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:39.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:39.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:39.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:39.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:39.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:58:39.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:39.492 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:39.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:39.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:39.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:39.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:39.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:39.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:39.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:39.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:39.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:40.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:39 smithi086 ceph-mon[97913]: pgmap v662: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 593 op/s 2024-06-14T22:58:40.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:39 smithi086 ceph-mon[102754]: pgmap v662: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 593 op/s 2024-06-14T22:58:40.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:40.045 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:39 smithi137 ceph-mon[90634]: pgmap v662: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 593 op/s 2024-06-14T22:58:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:40.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:40.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:40.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:40.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:40.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:40.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:207529 9.8 9:1f48baec:::benchmark_data_smithi086_152062_object75047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:58:40.346330+0000", 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.004861186, 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048955489999999999, 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:58:40.352 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 207529 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:40.346330+0000", 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:40.346330+0000", 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:58:40.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:40.346328+0000", 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:40.346337+0000", 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0459999999999994e-06 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:40.346364+0000", 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6325999999999998e-05 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:40.346375+0000", 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1137e-05 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:40.346421+0000", 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6594000000000001e-05 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:40.346560+0000", 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013846 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:58:40.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,5", 2024-06-14T22:58:40.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:40.346684+0000", 2024-06-14T22:58:40.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012458599999999999 2024-06-14T22:58:40.356 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:58:40.356 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:58:40.356 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:58:40.356 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:58:40.356 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-06-14T22:58:40.356 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 1 2024-06-14T22:58:40.356 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:58:40.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:40.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:40.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:40.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:40.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:40.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:40.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:40.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:40.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:40.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:40.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:40.783 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T22:58:40.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:40.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:40.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:41.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:41.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1585625508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:41.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:40 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:41.023 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1585625508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:41.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:40 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:41.056 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1585625508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:41.056 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:40 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:58:41.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-14T22:58:41.075 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:41.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:41.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:41.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:41.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:41.284 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:41.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:41.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:41.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:41.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:41.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:41.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:41.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:41.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:41.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:41.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:41.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:41.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:41.915 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T22:58:41.915 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T22:58:41.915 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-06-14T22:58:41.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:58:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:41.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:41.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:42.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:42.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:41 smithi086 ceph-mon[102754]: pgmap v663: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 798 op/s 2024-06-14T22:58:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:41 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:41 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:58:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:41 smithi086 ceph-mon[102754]: osdmap e169: 8 total, 8 up, 8 in 2024-06-14T22:58:42.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:41 smithi086 ceph-mon[97913]: pgmap v663: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 798 op/s 2024-06-14T22:58:42.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:41 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:42.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:41 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:58:42.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:41 smithi086 ceph-mon[97913]: osdmap e169: 8 total, 8 up, 8 in 2024-06-14T22:58:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:42.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:42.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:42.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:42.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:42.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:42.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:41 smithi137 ceph-mon[90634]: pgmap v663: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 798 op/s 2024-06-14T22:58:42.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:41 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:42.242 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:41 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:58:42.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:41 smithi137 ceph-mon[90634]: osdmap e169: 8 total, 8 up, 8 in 2024-06-14T22:58:42.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:42.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:42.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:42.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:42.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:42.504 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:42.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:42.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:42.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:42.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:42.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:42.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:42.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:42.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:42.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:42.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:42.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:43.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:43.022 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:43.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:43.033 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:43.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:43.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:43.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:43.213 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:43.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:43.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:43.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:43.362 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:43.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:43.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:43.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:43.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:43.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:43.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:43.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:43.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:43.798 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:43.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:43.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:43.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:43.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:43.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:43.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:43 smithi086 ceph-mon[97913]: pgmap v665: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 832 op/s 2024-06-14T22:58:43.962 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:43 smithi086 ceph-mon[102754]: pgmap v665: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 832 op/s 2024-06-14T22:58:44.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:44.054 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:58:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:58:44.080 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:44.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:44.093 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:58:44.094 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:58:44.128 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:58:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:58:44.165 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:58:44.165 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:58:44.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:44.197 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:58:44.198 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:58:44.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:44.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:44.231 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:58:44.231 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:58:44.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:43 smithi137 ceph-mon[90634]: pgmap v665: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 832 op/s 2024-06-14T22:58:44.261 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:58:44.261 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:58:44.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T22:58:44.297 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:58:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:58:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:44.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:44.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:44.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:44.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:44.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:44.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:44.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:44.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:44.658 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:44.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:44.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:44.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:44.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:44.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:44.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:44.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:45.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:45.073 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:45.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:45.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/980101912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:45.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:58:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:58:44] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T22:58:45.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/980101912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:45.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:45.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:45.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:45.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/980101912' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:45.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:45.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:45.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:45.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:45.481 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:45.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:45.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:45.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:45.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:45.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:45.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:45.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:45.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:45.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:45.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:45.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:45.963 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:45 smithi086 ceph-mon[97913]: pgmap v666: 89 pgs: 89 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 856 op/s 2024-06-14T22:58:45.964 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:45 smithi086 ceph-mon[102754]: pgmap v666: 89 pgs: 89 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 856 op/s 2024-06-14T22:58:45.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:46.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:46.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:46.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:46.166 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:46.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:46.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:45 smithi137 ceph-mon[90634]: pgmap v666: 89 pgs: 89 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 856 op/s 2024-06-14T22:58:46.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:46.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:46.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:46.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:46.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:46.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:46.551 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:46.588 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:46.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:46.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:46.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:46.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:46.786 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:58:46.786 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:58:46.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:46.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:46.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:46.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:47.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:47.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:47.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:47.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:47.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:47.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:47.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:47.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:47.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:47.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:47.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:47.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:47.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:47.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:47.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:47.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:47.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:47.821 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:47.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:47.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:47.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:47.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:47.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:48.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:58:48.044 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:47 smithi086 ceph-mon[102754]: pgmap v667: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-06-14T22:58:48.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:47 smithi086 ceph-mon[97913]: pgmap v667: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-06-14T22:58:48.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:48.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:48.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:48.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:48.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:47 smithi137 ceph-mon[90634]: pgmap v667: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-06-14T22:58:48.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:48.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:48.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:48.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:48.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:48.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:48.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:48.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:48.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:48.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:48.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-14T22:58:48.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:48.827 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:48.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:48.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:48.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:48.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:49.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:49.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:49.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:48 smithi086 ceph-mon[97913]: pgmap v668: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-06-14T22:58:49.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:48 smithi086 ceph-mon[102754]: pgmap v668: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-06-14T22:58:49.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:49.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:49.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:49.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:49.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:48 smithi137 ceph-mon[90634]: pgmap v668: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-06-14T22:58:49.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:49.305 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:49.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:49.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:49.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:49.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:49.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:49.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:49.567 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:49.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:49.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:49.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:49.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:49.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:49.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:49.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:49.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:49.981 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:49.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:50.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:50.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:50.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:50.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:50.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:50.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:50.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:50.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:50.398 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:50.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:50.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:50.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:50.539 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:50.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:50.629 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:58:50.634 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:58:50.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:50.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:50.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:50.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:50.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:50.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:50.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:50.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:51.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:51.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:51.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:51.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:51.220 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:51.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:51.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:51.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:51.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:51.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:51.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:51.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:51.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:51.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:51.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:51.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:51.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:51 smithi137 ceph-mon[90634]: from='client.55345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:58:51.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:51 smithi137 ceph-mon[90634]: pgmap v669: 89 pgs: 89 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-06-14T22:58:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:51.764 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:51.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:51.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:51 smithi086 ceph-mon[97913]: from='client.55345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:58:51.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:51 smithi086 ceph-mon[97913]: pgmap v669: 89 pgs: 89 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-06-14T22:58:51.813 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:51 smithi086 ceph-mon[102754]: from='client.55345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:58:51.814 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:51 smithi086 ceph-mon[102754]: pgmap v669: 89 pgs: 89 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-06-14T22:58:51.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:51.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:51.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:51.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:52.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:52.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:52.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:52.176 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-06-14T22:58:48.638174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3513980894,"num_objects":53635,"num_object_clones":0,"num_object_copies":160905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53635,"num_whiteouts":0,"num_read":487,"num_read_kb":913,"num_write":211649,"num_write_kb":8481102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142714,"ondisk_log_size":142714,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14622436,"kb_used_data":10450720,"kb_used_omap":29,"kb_used_meta":4171490,"kb_avail":735338780,"statfs":{"total":767960285184,"available":752986910720,"internally_reserved":0,"allocated":10701537280,"data_stored":10694590438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":4271606437},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-580780032,"num_objects":-8862,"num_object_clones":0,"num_object_copies":-26586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8862,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003398"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":98,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750312+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T22:58:32.750312+0000","last_peered":"2024-06-14T22:58:32.750312+0000","last_clean":"2024-06-14T22:58:32.750312+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T22:58:32.750312+0000","last_undegraded":"2024-06-14T22:58:32.750312+0000","last_fullsized":"2024-06-14T22:58:32.750312+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":132,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750539+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T22:58:32.750539+0000","last_peered":"2024-06-14T22:58:32.750539+0000","last_clean":"2024-06-14T22:58:32.750539+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T22:58:32.750539+0000","last_undegraded":"2024-06-14T22:58:32.750539+0000","last_fullsized":"2024-06-14T22:58:32.750539+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":129,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748911+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T22:58:32.748911+0000","last_peered":"2024-06-14T22:58:32.748911+0000","last_clean":"2024-06-14T22:58:32.748911+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T22:58:32.748911+0000","last_undegraded":"2024-06-14T22:58:32.748911+0000","last_fullsized":"2024-06-14T22:58:32.748911+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":119,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748686+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T22:58:32.748686+0000","last_peered":"2024-06-14T22:58:32.748686+0000","last_clean":"2024-06-14T22:58:32.748686+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T22:58:32.748686+0000","last_undegraded":"2024-06-14T22:58:32.748686+0000","last_fullsized":"2024-06-14T22:58:32.748686+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":120,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749182+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T22:58:32.749182+0000","last_peered":"2024-06-14T22:58:32.749182+0000","last_clean":"2024-06-14T22:58:32.749182+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T22:58:32.749182+0000","last_undegraded":"2024-06-14T22:58:32.749182+0000","last_fullsized":"2024-06-14T22:58:32.749182+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.751745+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T22:58:32.751745+0000","last_peered":"2024-06-14T22:58:32.751745+0000","last_clean":"2024-06-14T22:58:32.751745+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T22:58:32.751745+0000","last_undegraded":"2024-06-14T22:58:32.751745+0000","last_fullsized":"2024-06-14T22:58:32.751745+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":121,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.753146+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T22:58:32.753146+0000","last_peered":"2024-06-14T22:58:32.753146+0000","last_clean":"2024-06-14T22:58:32.753146+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T22:58:32.753146+0000","last_undegraded":"2024-06-14T22:58:32.753146+0000","last_fullsized":"2024-06-14T22:58:32.753146+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":138,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.762113+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T22:58:32.762113+0000","last_peered":"2024-06-14T22:58:32.762113+0000","last_clean":"2024-06-14T22:58:32.762113+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T22:58:32.762113+0000","last_undegraded":"2024-06-14T22:58:32.762113+0000","last_fullsized":"2024-06-14T22:58:32.762113+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"169'13745","reported_seq":13784,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:46.546437+0000","last_change":"2024-06-14T22:55:38.730745+0000","last_active":"2024-06-14T22:58:46.546437+0000","last_peered":"2024-06-14T22:58:46.546437+0000","last_clean":"2024-06-14T22:58:46.546437+0000","last_became_active":"2024-06-14T22:55:38.730320+0000","last_became_peered":"2024-06-14T22:55:38.730320+0000","last_unstale":"2024-06-14T22:58:46.546437+0000","last_undegraded":"2024-06-14T22:58:46.546437+0000","last_fullsized":"2024-06-14T22:58:46.546437+0000","mapping_epoch":140,"log_start":"147'3700","ondisk_log_start":"147'3700","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:29.533721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212664320,"num_objects":3245,"num_object_clones":0,"num_object_copies":9735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13745,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":119,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.753371+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T22:58:32.753371+0000","last_peered":"2024-06-14T22:58:32.753371+0000","last_clean":"2024-06-14T22:58:32.753371+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T22:58:32.753371+0000","last_undegraded":"2024-06-14T22:58:32.753371+0000","last_fullsized":"2024-06-14T22:58:32.753371+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":112,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748741+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:58:32.748741+0000","last_peered":"2024-06-14T22:58:32.748741+0000","last_clean":"2024-06-14T22:58:32.748741+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:58:32.748741+0000","last_undegraded":"2024-06-14T22:58:32.748741+0000","last_fullsized":"2024-06-14T22:58:32.748741+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"169'12972","reported_seq":13011,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:44.693089+0000","last_change":"2024-06-14T22:55:38.730175+0000","last_active":"2024-06-14T22:58:44.693089+0000","last_peered":"2024-06-14T22:58:44.693089+0000","last_clean":"2024-06-14T22:58:44.693089+0000","last_became_active":"2024-06-14T22:55:38.729339+0000","last_became_peered":"2024-06-14T22:55:38.729339+0000","last_unstale":"2024-06-14T22:58:44.693089+0000","last_undegraded":"2024-06-14T22:58:44.693089+0000","last_fullsized":"2024-06-14T22:58:44.693089+0000","mapping_epoch":140,"log_start":"150'5600","ondisk_log_start":"150'5600","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7372,"log_dups_size":0,"ondisk_log_size":7372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:40:50.307712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220987392,"num_objects":3372,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12972,"num_write_kb":523008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":120,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.753557+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T22:58:32.753557+0000","last_peered":"2024-06-14T22:58:32.753557+0000","last_clean":"2024-06-14T22:58:32.753557+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T22:58:32.753557+0000","last_undegraded":"2024-06-14T22:58:32.753557+0000","last_fullsized":"2024-06-14T22:58:32.753557+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":146,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.751853+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T22:58:32.751853+0000","last_peered":"2024-06-14T22:58:32.751853+0000","last_clean":"2024-06-14T22:58:32.751853+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T22:58:32.751853+0000","last_undegraded":"2024-06-14T22:58:32.751853+0000","last_fullsized":"2024-06-14T22:58:32.751853+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"169'13108","reported_seq":13145,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:43.979006+0000","last_change":"2024-06-14T22:55:38.730856+0000","last_active":"2024-06-14T22:58:43.979006+0000","last_peered":"2024-06-14T22:58:43.979006+0000","last_clean":"2024-06-14T22:58:43.979006+0000","last_became_active":"2024-06-14T22:55:38.730542+0000","last_became_peered":"2024-06-14T22:55:38.730542+0000","last_unstale":"2024-06-14T22:58:43.979006+0000","last_undegraded":"2024-06-14T22:58:43.979006+0000","last_fullsized":"2024-06-14T22:58:43.979006+0000","mapping_epoch":140,"log_start":"147'3400","ondisk_log_start":"147'3400","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9708,"log_dups_size":0,"ondisk_log_size":9708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:16:51.828869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218759168,"num_objects":3338,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13108,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":47,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748873+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T22:58:32.748873+0000","last_peered":"2024-06-14T22:58:32.748873+0000","last_clean":"2024-06-14T22:58:32.748873+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T22:58:32.748873+0000","last_undegraded":"2024-06-14T22:58:32.748873+0000","last_fullsized":"2024-06-14T22:58:32.748873+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":129,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.762107+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T22:58:32.762107+0000","last_peered":"2024-06-14T22:58:32.762107+0000","last_clean":"2024-06-14T22:58:32.762107+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T22:58:32.762107+0000","last_undegraded":"2024-06-14T22:58:32.762107+0000","last_fullsized":"2024-06-14T22:58:32.762107+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"169'13153","reported_seq":13190,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:44.039035+0000","last_change":"2024-06-14T22:55:39.130045+0000","last_active":"2024-06-14T22:58:44.039035+0000","last_peered":"2024-06-14T22:58:44.039035+0000","last_clean":"2024-06-14T22:58:44.039035+0000","last_became_active":"2024-06-14T22:55:39.129733+0000","last_became_peered":"2024-06-14T22:55:39.129733+0000","last_unstale":"2024-06-14T22:58:44.039035+0000","last_undegraded":"2024-06-14T22:58:44.039035+0000","last_fullsized":"2024-06-14T22:58:44.039035+0000","mapping_epoch":140,"log_start":"147'3400","ondisk_log_start":"147'3400","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9753,"log_dups_size":0,"ondisk_log_size":9753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:34:14.279180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225116160,"num_objects":3435,"num_object_clones":0,"num_object_copies":10305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":130,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749012+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T22:58:32.749012+0000","last_peered":"2024-06-14T22:58:32.749012+0000","last_clean":"2024-06-14T22:58:32.749012+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T22:58:32.749012+0000","last_undegraded":"2024-06-14T22:58:32.749012+0000","last_fullsized":"2024-06-14T22:58:32.749012+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":121,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749206+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T22:58:32.749206+0000","last_peered":"2024-06-14T22:58:32.749206+0000","last_clean":"2024-06-14T22:58:32.749206+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T22:58:32.749206+0000","last_undegraded":"2024-06-14T22:58:32.749206+0000","last_fullsized":"2024-06-14T22:58:32.749206+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"169'13081","reported_seq":13127,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:43.631189+0000","last_change":"2024-06-14T22:58:09.809570+0000","last_active":"2024-06-14T22:58:43.631189+0000","last_peered":"2024-06-14T22:58:43.631189+0000","last_clean":"2024-06-14T22:58:43.631189+0000","last_became_active":"2024-06-14T22:58:09.753927+0000","last_became_peered":"2024-06-14T22:58:09.753927+0000","last_unstale":"2024-06-14T22:58:43.631189+0000","last_undegraded":"2024-06-14T22:58:43.631189+0000","last_fullsized":"2024-06-14T22:58:43.631189+0000","mapping_epoch":163,"log_start":"147'3500","ondisk_log_start":"147'3500","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9581,"log_dups_size":0,"ondisk_log_size":9581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:46:54.799901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219611136,"num_objects":3351,"num_object_clones":0,"num_object_copies":10053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13081,"num_write_kb":525824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":37,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748895+0000","last_change":"2024-06-14T22:58:09.752985+0000","last_active":"2024-06-14T22:58:32.748895+0000","last_peered":"2024-06-14T22:58:32.748895+0000","last_clean":"2024-06-14T22:58:32.748895+0000","last_became_active":"2024-06-14T22:58:09.752630+0000","last_became_peered":"2024-06-14T22:58:09.752630+0000","last_unstale":"2024-06-14T22:58:32.748895+0000","last_undegraded":"2024-06-14T22:58:32.748895+0000","last_fullsized":"2024-06-14T22:58:32.748895+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:22.320986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":129,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.752131+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T22:58:32.752131+0000","last_peered":"2024-06-14T22:58:32.752131+0000","last_clean":"2024-06-14T22:58:32.752131+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T22:58:32.752131+0000","last_undegraded":"2024-06-14T22:58:32.752131+0000","last_fullsized":"2024-06-14T22:58:32.752131+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"169'13393","reported_seq":13432,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:44.667850+0000","last_change":"2024-06-14T22:55:39.129787+0000","last_active":"2024-06-14T22:58:44.667850+0000","last_peered":"2024-06-14T22:58:44.667850+0000","last_clean":"2024-06-14T22:58:44.667850+0000","last_became_active":"2024-06-14T22:55:39.129528+0000","last_became_peered":"2024-06-14T22:55:39.129528+0000","last_unstale":"2024-06-14T22:58:44.667850+0000","last_undegraded":"2024-06-14T22:58:44.667850+0000","last_fullsized":"2024-06-14T22:58:44.667850+0000","mapping_epoch":140,"log_start":"150'6000","ondisk_log_start":"150'6000","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:24:05.362187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218431488,"num_objects":3333,"num_object_clones":0,"num_object_copies":9999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13393,"num_write_kb":535232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":130,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.762215+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T22:58:32.762215+0000","last_peered":"2024-06-14T22:58:32.762215+0000","last_clean":"2024-06-14T22:58:32.762215+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T22:58:32.762215+0000","last_undegraded":"2024-06-14T22:58:32.762215+0000","last_fullsized":"2024-06-14T22:58:32.762215+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":139,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.752204+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T22:58:32.752204+0000","last_peered":"2024-06-14T22:58:32.752204+0000","last_clean":"2024-06-14T22:58:32.752204+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T22:58:32.752204+0000","last_undegraded":"2024-06-14T22:58:32.752204+0000","last_fullsized":"2024-06-14T22:58:32.752204+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"169'13102","reported_seq":13164,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:43.865137+0000","last_change":"2024-06-14T22:58:32.802664+0000","last_active":"2024-06-14T22:58:43.865137+0000","last_peered":"2024-06-14T22:58:43.865137+0000","last_clean":"2024-06-14T22:58:43.865137+0000","last_became_active":"2024-06-14T22:58:32.753751+0000","last_became_peered":"2024-06-14T22:58:32.753751+0000","last_unstale":"2024-06-14T22:58:43.865137+0000","last_undegraded":"2024-06-14T22:58:43.865137+0000","last_fullsized":"2024-06-14T22:58:43.865137+0000","mapping_epoch":167,"log_start":"150'5330","ondisk_log_start":"150'5330","created":140,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7772,"log_dups_size":0,"ondisk_log_size":7772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:17:54.717972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221642752,"num_objects":3382,"num_object_clones":0,"num_object_copies":10146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13102,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":130,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.751980+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T22:58:32.751980+0000","last_peered":"2024-06-14T22:58:32.751980+0000","last_clean":"2024-06-14T22:58:32.751980+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T22:58:32.751980+0000","last_undegraded":"2024-06-14T22:58:32.751980+0000","last_fullsized":"2024-06-14T22:58:32.751980+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":129,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.752394+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T22:58:32.752394+0000","last_peered":"2024-06-14T22:58:32.752394+0000","last_clean":"2024-06-14T22:58:32.752394+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T22:58:32.752394+0000","last_undegraded":"2024-06-14T22:58:32.752394+0000","last_fullsized":"2024-06-14T22:58:32.752394+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"169'13301","reported_seq":13348,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:44.699985+0000","last_change":"2024-06-14T22:58:09.794739+0000","last_active":"2024-06-14T22:58:44.699985+0000","last_peered":"2024-06-14T22:58:44.699985+0000","last_clean":"2024-06-14T22:58:44.699985+0000","last_became_active":"2024-06-14T22:58:09.755744+0000","last_became_peered":"2024-06-14T22:58:09.755744+0000","last_unstale":"2024-06-14T22:58:44.699985+0000","last_undegraded":"2024-06-14T22:58:44.699985+0000","last_fullsized":"2024-06-14T22:58:44.699985+0000","mapping_epoch":163,"log_start":"150'5921","ondisk_log_start":"150'5921","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7380,"log_dups_size":0,"ondisk_log_size":7380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:51:51.355892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225378304,"num_objects":3439,"num_object_clones":0,"num_object_copies":10317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13301,"num_write_kb":535680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":123,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.753059+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T22:58:32.753059+0000","last_peered":"2024-06-14T22:58:32.753059+0000","last_clean":"2024-06-14T22:58:32.753059+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T22:58:32.753059+0000","last_undegraded":"2024-06-14T22:58:32.753059+0000","last_fullsized":"2024-06-14T22:58:32.753059+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":136,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750552+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T22:58:32.750552+0000","last_peered":"2024-06-14T22:58:32.750552+0000","last_clean":"2024-06-14T22:58:32.750552+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T22:58:32.750552+0000","last_undegraded":"2024-06-14T22:58:32.750552+0000","last_fullsized":"2024-06-14T22:58:32.750552+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"169'13085","reported_seq":13131,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:46.566432+0000","last_change":"2024-06-14T22:58:09.784892+0000","last_active":"2024-06-14T22:58:46.566432+0000","last_peered":"2024-06-14T22:58:46.566432+0000","last_clean":"2024-06-14T22:58:46.566432+0000","last_became_active":"2024-06-14T22:58:09.752020+0000","last_became_peered":"2024-06-14T22:58:09.752020+0000","last_unstale":"2024-06-14T22:58:46.566432+0000","last_undegraded":"2024-06-14T22:58:46.566432+0000","last_fullsized":"2024-06-14T22:58:46.566432+0000","mapping_epoch":163,"log_start":"147'3400","ondisk_log_start":"147'3400","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:53.007432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212008960,"num_objects":3235,"num_object_clones":0,"num_object_copies":9705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13085,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":435,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748978+0000","last_change":"2024-06-14T22:55:53.665022+0000","last_active":"2024-06-14T22:58:32.748978+0000","last_peered":"2024-06-14T22:58:32.748978+0000","last_clean":"2024-06-14T22:58:32.748978+0000","last_became_active":"2024-06-14T22:55:53.664025+0000","last_became_peered":"2024-06-14T22:55:53.664025+0000","last_unstale":"2024-06-14T22:58:32.748978+0000","last_undegraded":"2024-06-14T22:58:32.748978+0000","last_fullsized":"2024-06-14T22:58:32.748978+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.106147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":134,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749057+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T22:58:32.749057+0000","last_peered":"2024-06-14T22:58:32.749057+0000","last_clean":"2024-06-14T22:58:32.749057+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T22:58:32.749057+0000","last_undegraded":"2024-06-14T22:58:32.749057+0000","last_fullsized":"2024-06-14T22:58:32.749057+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":73,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748855+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T22:58:32.748855+0000","last_peered":"2024-06-14T22:58:32.748855+0000","last_clean":"2024-06-14T22:58:32.748855+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T22:58:32.748855+0000","last_undegraded":"2024-06-14T22:58:32.748855+0000","last_fullsized":"2024-06-14T22:58:32.748855+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":112,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.762400+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:58:32.762400+0000","last_peered":"2024-06-14T22:58:32.762400+0000","last_clean":"2024-06-14T22:58:32.762400+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:58:32.762400+0000","last_undegraded":"2024-06-14T22:58:32.762400+0000","last_fullsized":"2024-06-14T22:58:32.762400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"169'13468","reported_seq":13507,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:46.555448+0000","last_change":"2024-06-14T22:55:38.974930+0000","last_active":"2024-06-14T22:58:46.555448+0000","last_peered":"2024-06-14T22:58:46.555448+0000","last_clean":"2024-06-14T22:58:46.555448+0000","last_became_active":"2024-06-14T22:55:38.974625+0000","last_became_peered":"2024-06-14T22:55:38.974625+0000","last_unstale":"2024-06-14T22:58:46.555448+0000","last_undegraded":"2024-06-14T22:58:46.555448+0000","last_fullsized":"2024-06-14T22:58:46.555448+0000","mapping_epoch":140,"log_start":"147'3400","ondisk_log_start":"147'3400","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:44:40.336597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213254144,"num_objects":3254,"num_object_clones":0,"num_object_copies":9762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13468,"num_write_kb":535104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":126,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749938+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T22:58:32.749938+0000","last_peered":"2024-06-14T22:58:32.749938+0000","last_clean":"2024-06-14T22:58:32.749938+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T22:58:32.749938+0000","last_undegraded":"2024-06-14T22:58:32.749938+0000","last_fullsized":"2024-06-14T22:58:32.749938+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":120,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748756+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T22:58:32.748756+0000","last_peered":"2024-06-14T22:58:32.748756+0000","last_clean":"2024-06-14T22:58:32.748756+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T22:58:32.748756+0000","last_undegraded":"2024-06-14T22:58:32.748756+0000","last_fullsized":"2024-06-14T22:58:32.748756+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":134,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749405+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T22:58:32.749405+0000","last_peered":"2024-06-14T22:58:32.749405+0000","last_clean":"2024-06-14T22:58:32.749405+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T22:58:32.749405+0000","last_undegraded":"2024-06-14T22:58:32.749405+0000","last_fullsized":"2024-06-14T22:58:32.749405+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"169'13237","reported_seq":13271,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:44.073068+0000","last_change":"2024-06-14T22:55:38.731601+0000","last_active":"2024-06-14T22:58:44.073068+0000","last_peered":"2024-06-14T22:58:44.073068+0000","last_clean":"2024-06-14T22:58:44.073068+0000","last_became_active":"2024-06-14T22:55:38.731200+0000","last_became_peered":"2024-06-14T22:55:38.731200+0000","last_unstale":"2024-06-14T22:58:44.073068+0000","last_undegraded":"2024-06-14T22:58:44.073068+0000","last_fullsized":"2024-06-14T22:58:44.073068+0000","mapping_epoch":140,"log_start":"147'3500","ondisk_log_start":"147'3500","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9737,"log_dups_size":0,"ondisk_log_size":9737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:53:02.395452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223674368,"num_objects":3413,"num_object_clones":0,"num_object_copies":10239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13237,"num_write_kb":532800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":121,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.804555+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T22:58:32.804555+0000","last_peered":"2024-06-14T22:58:32.804555+0000","last_clean":"2024-06-14T22:58:32.804555+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T22:58:32.804555+0000","last_undegraded":"2024-06-14T22:58:32.804555+0000","last_fullsized":"2024-06-14T22:58:32.804555+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":136,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750170+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T22:58:32.750170+0000","last_peered":"2024-06-14T22:58:32.750170+0000","last_clean":"2024-06-14T22:58:32.750170+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T22:58:32.750170+0000","last_undegraded":"2024-06-14T22:58:32.750170+0000","last_fullsized":"2024-06-14T22:58:32.750170+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":121,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750340+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T22:58:32.750340+0000","last_peered":"2024-06-14T22:58:32.750340+0000","last_clean":"2024-06-14T22:58:32.750340+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T22:58:32.750340+0000","last_undegraded":"2024-06-14T22:58:32.750340+0000","last_fullsized":"2024-06-14T22:58:32.750340+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"169'13188","reported_seq":13227,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:46.559010+0000","last_change":"2024-06-14T22:55:38.729565+0000","last_active":"2024-06-14T22:58:46.559010+0000","last_peered":"2024-06-14T22:58:46.559010+0000","last_clean":"2024-06-14T22:58:46.559010+0000","last_became_active":"2024-06-14T22:55:38.729284+0000","last_became_peered":"2024-06-14T22:55:38.729284+0000","last_unstale":"2024-06-14T22:58:46.559010+0000","last_undegraded":"2024-06-14T22:58:46.559010+0000","last_fullsized":"2024-06-14T22:58:46.559010+0000","mapping_epoch":140,"log_start":"146'3100","ondisk_log_start":"146'3100","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:15.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214695936,"num_objects":3276,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13188,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":134,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748950+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T22:58:32.748950+0000","last_peered":"2024-06-14T22:58:32.748950+0000","last_clean":"2024-06-14T22:58:32.748950+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T22:58:32.748950+0000","last_undegraded":"2024-06-14T22:58:32.748950+0000","last_fullsized":"2024-06-14T22:58:32.748950+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":130,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.752153+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T22:58:32.752153+0000","last_peered":"2024-06-14T22:58:32.752153+0000","last_clean":"2024-06-14T22:58:32.752153+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T22:58:32.752153+0000","last_undegraded":"2024-06-14T22:58:32.752153+0000","last_fullsized":"2024-06-14T22:58:32.752153+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":151,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750205+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T22:58:32.750205+0000","last_peered":"2024-06-14T22:58:32.750205+0000","last_clean":"2024-06-14T22:58:32.750205+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T22:58:32.750205+0000","last_undegraded":"2024-06-14T22:58:32.750205+0000","last_fullsized":"2024-06-14T22:58:32.750205+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"169'13144","reported_seq":13181,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:44.024737+0000","last_change":"2024-06-14T22:55:38.729989+0000","last_active":"2024-06-14T22:58:44.024737+0000","last_peered":"2024-06-14T22:58:44.024737+0000","last_clean":"2024-06-14T22:58:44.024737+0000","last_became_active":"2024-06-14T22:55:38.729695+0000","last_became_peered":"2024-06-14T22:55:38.729695+0000","last_unstale":"2024-06-14T22:58:44.024737+0000","last_undegraded":"2024-06-14T22:58:44.024737+0000","last_fullsized":"2024-06-14T22:58:44.024737+0000","mapping_epoch":140,"log_start":"147'3400","ondisk_log_start":"147'3400","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9744,"log_dups_size":0,"ondisk_log_size":9744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:44:38.218727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221380608,"num_objects":3378,"num_object_clones":0,"num_object_copies":10134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13144,"num_write_kb":528704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":147,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749081+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T22:58:32.749081+0000","last_peered":"2024-06-14T22:58:32.749081+0000","last_clean":"2024-06-14T22:58:32.749081+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T22:58:32.749081+0000","last_undegraded":"2024-06-14T22:58:32.749081+0000","last_fullsized":"2024-06-14T22:58:32.749081+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":126,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750365+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T22:58:32.750365+0000","last_peered":"2024-06-14T22:58:32.750365+0000","last_clean":"2024-06-14T22:58:32.750365+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T22:58:32.750365+0000","last_undegraded":"2024-06-14T22:58:32.750365+0000","last_fullsized":"2024-06-14T22:58:32.750365+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":121,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750393+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T22:58:32.750393+0000","last_peered":"2024-06-14T22:58:32.750393+0000","last_clean":"2024-06-14T22:58:32.750393+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T22:58:32.750393+0000","last_undegraded":"2024-06-14T22:58:32.750393+0000","last_fullsized":"2024-06-14T22:58:32.750393+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"169'13097","reported_seq":13143,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:43.635515+0000","last_change":"2024-06-14T22:55:53.672630+0000","last_active":"2024-06-14T22:58:43.635515+0000","last_peered":"2024-06-14T22:58:43.635515+0000","last_clean":"2024-06-14T22:58:43.635515+0000","last_became_active":"2024-06-14T22:55:53.663693+0000","last_became_peered":"2024-06-14T22:55:53.663693+0000","last_unstale":"2024-06-14T22:58:43.635515+0000","last_undegraded":"2024-06-14T22:58:43.635515+0000","last_fullsized":"2024-06-14T22:58:43.635515+0000","mapping_epoch":145,"log_start":"149'4700","ondisk_log_start":"149'4700","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":8397,"log_dups_size":0,"ondisk_log_size":8397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:20:34.289185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225902592,"num_objects":3447,"num_object_clones":0,"num_object_copies":10341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13097,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":136,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750227+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T22:58:32.750227+0000","last_peered":"2024-06-14T22:58:32.750227+0000","last_clean":"2024-06-14T22:58:32.750227+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T22:58:32.750227+0000","last_undegraded":"2024-06-14T22:58:32.750227+0000","last_fullsized":"2024-06-14T22:58:32.750227+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":134,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750977+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T22:58:32.750977+0000","last_peered":"2024-06-14T22:58:32.750977+0000","last_clean":"2024-06-14T22:58:32.750977+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T22:58:32.750977+0000","last_undegraded":"2024-06-14T22:58:32.750977+0000","last_fullsized":"2024-06-14T22:58:32.750977+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":106,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749084+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T22:58:32.749084+0000","last_peered":"2024-06-14T22:58:32.749084+0000","last_clean":"2024-06-14T22:58:32.749084+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T22:58:32.749084+0000","last_undegraded":"2024-06-14T22:58:32.749084+0000","last_fullsized":"2024-06-14T22:58:32.749084+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"169'13254","reported_seq":13301,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:44.697876+0000","last_change":"2024-06-14T22:55:53.668144+0000","last_active":"2024-06-14T22:58:44.697876+0000","last_peered":"2024-06-14T22:58:44.697876+0000","last_clean":"2024-06-14T22:58:44.697876+0000","last_became_active":"2024-06-14T22:55:53.663690+0000","last_became_peered":"2024-06-14T22:55:53.663690+0000","last_unstale":"2024-06-14T22:58:44.697876+0000","last_undegraded":"2024-06-14T22:58:44.697876+0000","last_fullsized":"2024-06-14T22:58:44.697876+0000","mapping_epoch":145,"log_start":"150'5900","ondisk_log_start":"150'5900","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7354,"log_dups_size":0,"ondisk_log_size":7354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:52:03.686394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217645080,"num_objects":3322,"num_object_clones":0,"num_object_copies":9966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13254,"num_write_kb":530369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":28,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749442+0000","last_change":"2024-06-14T22:58:09.752729+0000","last_active":"2024-06-14T22:58:32.749442+0000","last_peered":"2024-06-14T22:58:32.749442+0000","last_clean":"2024-06-14T22:58:32.749442+0000","last_became_active":"2024-06-14T22:58:09.752393+0000","last_became_peered":"2024-06-14T22:58:09.752393+0000","last_unstale":"2024-06-14T22:58:32.749442+0000","last_undegraded":"2024-06-14T22:58:32.749442+0000","last_fullsized":"2024-06-14T22:58:32.749442+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:21:58.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":120,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.750710+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T22:58:32.750710+0000","last_peered":"2024-06-14T22:58:32.750710+0000","last_clean":"2024-06-14T22:58:32.750710+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T22:58:32.750710+0000","last_undegraded":"2024-06-14T22:58:32.750710+0000","last_fullsized":"2024-06-14T22:58:32.750710+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":114,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.754276+0000","last_change":"2024-06-14T22:58:32.754079+0000","last_active":"2024-06-14T22:58:32.754276+0000","last_peered":"2024-06-14T22:58:32.754276+0000","last_clean":"2024-06-14T22:58:32.754276+0000","last_became_active":"2024-06-14T22:58:32.753814+0000","last_became_peered":"2024-06-14T22:58:32.753814+0000","last_unstale":"2024-06-14T22:58:32.754276+0000","last_undegraded":"2024-06-14T22:58:32.754276+0000","last_fullsized":"2024-06-14T22:58:32.754276+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:37:07.414370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"169'13021","reported_seq":13068,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:43.596302+0000","last_change":"2024-06-14T22:58:09.804712+0000","last_active":"2024-06-14T22:58:43.596302+0000","last_peered":"2024-06-14T22:58:43.596302+0000","last_clean":"2024-06-14T22:58:43.596302+0000","last_became_active":"2024-06-14T22:58:09.757262+0000","last_became_peered":"2024-06-14T22:58:09.757262+0000","last_unstale":"2024-06-14T22:58:43.596302+0000","last_undegraded":"2024-06-14T22:58:43.596302+0000","last_fullsized":"2024-06-14T22:58:43.596302+0000","mapping_epoch":163,"log_start":"149'4600","ondisk_log_start":"149'4600","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:28.952637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222232576,"num_objects":3391,"num_object_clones":0,"num_object_copies":10173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13021,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":124,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749055+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T22:58:32.749055+0000","last_peered":"2024-06-14T22:58:32.749055+0000","last_clean":"2024-06-14T22:58:32.749055+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T22:58:32.749055+0000","last_undegraded":"2024-06-14T22:58:32.749055+0000","last_fullsized":"2024-06-14T22:58:32.749055+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":134,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.762575+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T22:58:32.762575+0000","last_peered":"2024-06-14T22:58:32.762575+0000","last_clean":"2024-06-14T22:58:32.762575+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T22:58:32.762575+0000","last_undegraded":"2024-06-14T22:58:32.762575+0000","last_fullsized":"2024-06-14T22:58:32.762575+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":112,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.751759+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:58:32.751759+0000","last_peered":"2024-06-14T22:58:32.751759+0000","last_clean":"2024-06-14T22:58:32.751759+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:58:32.751759+0000","last_undegraded":"2024-06-14T22:58:32.751759+0000","last_fullsized":"2024-06-14T22:58:32.751759+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":129,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.752166+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T22:58:32.752166+0000","last_peered":"2024-06-14T22:58:32.752166+0000","last_clean":"2024-06-14T22:58:32.752166+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T22:58:32.752166+0000","last_undegraded":"2024-06-14T22:58:32.752166+0000","last_fullsized":"2024-06-14T22:58:32.752166+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":135,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.752216+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:58:32.752216+0000","last_peered":"2024-06-14T22:58:32.752216+0000","last_clean":"2024-06-14T22:58:32.752216+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:58:32.752216+0000","last_undegraded":"2024-06-14T22:58:32.752216+0000","last_fullsized":"2024-06-14T22:58:32.752216+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":136,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:40.787236+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T22:58:40.787236+0000","last_peered":"2024-06-14T22:58:40.787236+0000","last_clean":"2024-06-14T22:58:40.787236+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T22:58:40.787236+0000","last_undegraded":"2024-06-14T22:58:40.787236+0000","last_fullsized":"2024-06-14T22:58:40.787236+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":47,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.752966+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T22:58:32.752966+0000","last_peered":"2024-06-14T22:58:32.752966+0000","last_clean":"2024-06-14T22:58:32.752966+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T22:58:32.752966+0000","last_undegraded":"2024-06-14T22:58:32.752966+0000","last_fullsized":"2024-06-14T22:58:32.752966+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":130,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749028+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T22:58:32.749028+0000","last_peered":"2024-06-14T22:58:32.749028+0000","last_clean":"2024-06-14T22:58:32.749028+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T22:58:32.749028+0000","last_undegraded":"2024-06-14T22:58:32.749028+0000","last_fullsized":"2024-06-14T22:58:32.749028+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":73,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.762062+0000","last_change":"2024-06-14T22:58:09.753103+0000","last_active":"2024-06-14T22:58:32.762062+0000","last_peered":"2024-06-14T22:58:32.762062+0000","last_clean":"2024-06-14T22:58:32.762062+0000","last_became_active":"2024-06-14T22:58:09.752761+0000","last_became_peered":"2024-06-14T22:58:09.752761+0000","last_unstale":"2024-06-14T22:58:32.762062+0000","last_undegraded":"2024-06-14T22:58:32.762062+0000","last_fullsized":"2024-06-14T22:58:32.762062+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:20:20.619331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":119,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749348+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T22:58:32.749348+0000","last_peered":"2024-06-14T22:58:32.749348+0000","last_clean":"2024-06-14T22:58:32.749348+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T22:58:32.749348+0000","last_undegraded":"2024-06-14T22:58:32.749348+0000","last_fullsized":"2024-06-14T22:58:32.749348+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":122,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-06-14T22:58:40.781281+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T22:58:40.781281+0000","last_peered":"2024-06-14T22:58:40.781281+0000","last_clean":"2024-06-14T22:58:40.781281+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T22:58:40.781281+0000","last_undegraded":"2024-06-14T22:58:40.781281+0000","last_fullsized":"2024-06-14T22:58:40.781281+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":129,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.762524+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T22:58:32.762524+0000","last_peered":"2024-06-14T22:58:32.762524+0000","last_clean":"2024-06-14T22:58:32.762524+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T22:58:32.762524+0000","last_undegraded":"2024-06-14T22:58:32.762524+0000","last_fullsized":"2024-06-14T22:58:32.762524+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":116,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748860+0000","last_change":"2024-06-14T22:58:09.785074+0000","last_active":"2024-06-14T22:58:32.748860+0000","last_peered":"2024-06-14T22:58:32.748860+0000","last_clean":"2024-06-14T22:58:32.748860+0000","last_became_active":"2024-06-14T22:58:09.784309+0000","last_became_peered":"2024-06-14T22:58:09.784309+0000","last_unstale":"2024-06-14T22:58:32.748860+0000","last_undegraded":"2024-06-14T22:58:32.748860+0000","last_fullsized":"2024-06-14T22:58:32.748860+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:23.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":121,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749364+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T22:58:32.749364+0000","last_peered":"2024-06-14T22:58:32.749364+0000","last_clean":"2024-06-14T22:58:32.749364+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T22:58:32.749364+0000","last_undegraded":"2024-06-14T22:58:32.749364+0000","last_fullsized":"2024-06-14T22:58:32.749364+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":135,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749890+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T22:58:32.749890+0000","last_peered":"2024-06-14T22:58:32.749890+0000","last_clean":"2024-06-14T22:58:32.749890+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T22:58:32.749890+0000","last_undegraded":"2024-06-14T22:58:32.749890+0000","last_fullsized":"2024-06-14T22:58:32.749890+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":124,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.804813+0000","last_change":"2024-06-14T22:58:09.756040+0000","last_active":"2024-06-14T22:58:32.804813+0000","last_peered":"2024-06-14T22:58:32.804813+0000","last_clean":"2024-06-14T22:58:32.804813+0000","last_became_active":"2024-06-14T22:58:09.755659+0000","last_became_peered":"2024-06-14T22:58:09.755659+0000","last_unstale":"2024-06-14T22:58:32.804813+0000","last_undegraded":"2024-06-14T22:58:32.804813+0000","last_fullsized":"2024-06-14T22:58:32.804813+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:40.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":148,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.751983+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T22:58:32.751983+0000","last_peered":"2024-06-14T22:58:32.751983+0000","last_clean":"2024-06-14T22:58:32.751983+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T22:58:32.751983+0000","last_undegraded":"2024-06-14T22:58:32.751983+0000","last_fullsized":"2024-06-14T22:58:32.751983+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":141,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.804758+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T22:58:32.804758+0000","last_peered":"2024-06-14T22:58:32.804758+0000","last_clean":"2024-06-14T22:58:32.804758+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T22:58:32.804758+0000","last_undegraded":"2024-06-14T22:58:32.804758+0000","last_fullsized":"2024-06-14T22:58:32.804758+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":108,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749015+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T22:58:32.749015+0000","last_peered":"2024-06-14T22:58:32.749015+0000","last_clean":"2024-06-14T22:58:32.749015+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T22:58:32.749015+0000","last_undegraded":"2024-06-14T22:58:32.749015+0000","last_fullsized":"2024-06-14T22:58:32.749015+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":146,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.752070+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T22:58:32.752070+0000","last_peered":"2024-06-14T22:58:32.752070+0000","last_clean":"2024-06-14T22:58:32.752070+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T22:58:32.752070+0000","last_undegraded":"2024-06-14T22:58:32.752070+0000","last_fullsized":"2024-06-14T22:58:32.752070+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":132,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.748796+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T22:58:32.748796+0000","last_peered":"2024-06-14T22:58:32.748796+0000","last_clean":"2024-06-14T22:58:32.748796+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T22:58:32.748796+0000","last_undegraded":"2024-06-14T22:58:32.748796+0000","last_fullsized":"2024-06-14T22:58:32.748796+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":146,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.752154+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T22:58:32.752154+0000","last_peered":"2024-06-14T22:58:32.752154+0000","last_clean":"2024-06-14T22:58:32.752154+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T22:58:32.752154+0000","last_undegraded":"2024-06-14T22:58:32.752154+0000","last_fullsized":"2024-06-14T22:58:32.752154+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":115,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749041+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:58:32.749041+0000","last_peered":"2024-06-14T22:58:32.749041+0000","last_clean":"2024-06-14T22:58:32.749041+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:58:32.749041+0000","last_undegraded":"2024-06-14T22:58:32.749041+0000","last_fullsized":"2024-06-14T22:58:32.749041+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":131,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.804451+0000","last_change":"2024-06-14T22:58:09.752281+0000","last_active":"2024-06-14T22:58:32.804451+0000","last_peered":"2024-06-14T22:58:32.804451+0000","last_clean":"2024-06-14T22:58:32.804451+0000","last_became_active":"2024-06-14T22:58:09.751952+0000","last_became_peered":"2024-06-14T22:58:09.751952+0000","last_unstale":"2024-06-14T22:58:32.804451+0000","last_undegraded":"2024-06-14T22:58:32.804451+0000","last_fullsized":"2024-06-14T22:58:32.804451+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:39:19.836734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":28,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.762484+0000","last_change":"2024-06-14T22:58:09.755622+0000","last_active":"2024-06-14T22:58:32.762484+0000","last_peered":"2024-06-14T22:58:32.762484+0000","last_clean":"2024-06-14T22:58:32.762484+0000","last_became_active":"2024-06-14T22:58:09.755383+0000","last_became_peered":"2024-06-14T22:58:09.755383+0000","last_unstale":"2024-06-14T22:58:32.762484+0000","last_undegraded":"2024-06-14T22:58:32.762484+0000","last_fullsized":"2024-06-14T22:58:32.762484+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:13.884346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":129,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.751701+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T22:58:32.751701+0000","last_peered":"2024-06-14T22:58:32.751701+0000","last_clean":"2024-06-14T22:58:32.751701+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T22:58:32.751701+0000","last_undegraded":"2024-06-14T22:58:32.751701+0000","last_fullsized":"2024-06-14T22:58:32.751701+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":124,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-14T22:58:32.749391+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T22:58:32.749391+0000","last_peered":"2024-06-14T22:58:32.749391+0000","last_clean":"2024-06-14T22:58:32.749391+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T22:58:32.749391+0000","last_undegraded":"2024-06-14T22:58:32.749391+0000","last_fullsized":"2024-06-14T22:58:32.749391+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3513384984,"num_objects":53611,"num_object_clones":0,"num_object_copies":160833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211349,"num_write_kb":8478657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10593382400,"data_stored":10593370184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142498,"ondisk_log_size":142498,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234296,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2860500,"kb_used_data":1953356,"kb_used_omap":1,"kb_used_meta":907134,"kb_avail":90884652,"statfs":{"total":95995035648,"available":93065883648,"internally_reserved":0,"allocated":2000236544,"data_stored":1999347337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":928905560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":0.81200000000000006,"15min":1.099},"min":{"1min":0.38,"5min":0.249,"15min":0.20200000000000001},"max":{"1min":3.2160000000000002,"5min":4.0330000000000004,"15min":9.3919999999999995},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.353,"5min":0.77900000000000003,"15min":1.23},"min":{"1min":0.27000000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.0220000000000002,"5min":5.0220000000000002,"15min":10.416},"last":3.9049999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:58:41 2024","interfaces":[{"interface":"back","average":{"1min":2.02,"5min":2.5910000000000002,"15min":1.8100000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.28100000000000003},"max":{"1min":8.5340000000000007,"5min":83.295000000000002,"15min":83.295000000000002},"last":0.67300000000000004},{"interface":"front","average":{"1min":2.0840000000000001,"5min":2.5739999999999998,"15min":1.8799999999999999},"min":{"1min":0.42899999999999999,"5min":0.25800000000000001,"15min":0.223},"max":{"1min":6.8620000000000001,"5min":82.972999999999999,"15min":82.972999999999999},"last":3.4039999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:58:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":0.94099999999999995,"15min":0.70099999999999996},"min":{"1min":0.27200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.9269999999999996,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":3.3730000000000002},{"interface":"front","average":{"1min":1.909,"5min":1.143,"15min":0.76200000000000001},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.9779999999999998,"5min":8.1150000000000002,"15min":8.1150000000000002},"last":3.4380000000000002}]},{"osd":3,"last update":"Fri Jun 14 22:58:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.6240000000000001,"15min":0.89900000000000002},"min":{"1min":0.41299999999999998,"5min":0.308,"15min":0.308},"max":{"1min":12.992000000000001,"5min":55.683999999999997,"15min":55.683999999999997},"last":1.022},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.631,"15min":0.94299999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":9.5559999999999992,"5min":55.533999999999999,"15min":55.533999999999999},"last":1.2290000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.70999999999999996,"15min":0.63800000000000001},"min":{"1min":0.27300000000000002,"5min":0.245,"15min":0.22900000000000001},"max":{"1min":6.7610000000000001,"5min":6.7610000000000001,"15min":6.7610000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.647,"5min":0.83099999999999996,"15min":0.68999999999999995},"min":{"1min":0.30199999999999999,"5min":0.26800000000000002,"15min":0.25800000000000001},"max":{"1min":16.879000000000001,"5min":16.879000000000001,"15min":16.879000000000001},"last":0.97299999999999998}]},{"osd":5,"last update":"Fri Jun 14 22:57:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":0.78800000000000003,"15min":0.68799999999999994},"min":{"1min":0.29499999999999998,"5min":0.26100000000000001,"15min":0.22},"max":{"1min":3.383,"5min":3.9140000000000001,"15min":3.9860000000000002},"last":3.464},{"interface":"front","average":{"1min":1.3300000000000001,"5min":0.84599999999999997,"15min":0.81299999999999994},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.28699999999999998},"max":{"1min":6.8529999999999998,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":0.41899999999999998}]},{"osd":6,"last update":"Fri Jun 14 22:57:58 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.80800000000000005,"15min":0.746},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28100000000000003},"max":{"1min":4.1340000000000003,"5min":5.806,"15min":5.806},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":0.82199999999999995,"15min":0.73899999999999999},"min":{"1min":0.34100000000000003,"5min":0.30299999999999999,"15min":0.29899999999999999},"max":{"1min":12.951000000000001,"5min":12.951000000000001,"15min":12.951000000000001},"last":1.002}]}]},{"osd":6,"up_from":46,"seq":197568495932,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1248964,"kb_used_data":659128,"kb_used_omap":1,"kb_used_meta":589822,"kb_avail":92496188,"statfs":{"total":95995035648,"available":94716096512,"internally_reserved":0,"allocated":674947072,"data_stored":674077852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":603978072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:12 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":0.86199999999999999,"15min":1.2370000000000001},"min":{"1min":0.23699999999999999,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":9.9350000000000005,"5min":9.9350000000000005,"15min":31.358000000000001},"last":2.2610000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.76700000000000002,"15min":1.345},"min":{"1min":0.27100000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.1949999999999998,"5min":6.0730000000000004,"15min":29.399000000000001},"last":0.37}]},{"osd":1,"last update":"Fri Jun 14 22:58:37 2024","interfaces":[{"interface":"back","average":{"1min":1.9470000000000001,"5min":1.093,"15min":1.2709999999999999},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":9.9559999999999995},"last":1.228},{"interface":"front","average":{"1min":1.857,"5min":1.04,"15min":1.2450000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.247},"max":{"1min":6.7519999999999998,"5min":6.7519999999999998,"15min":9.5510000000000002},"last":4.2599999999999998}]},{"osd":2,"last update":"Fri Jun 14 22:58:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":0.89000000000000001,"15min":0.67900000000000005},"min":{"1min":0.29799999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.431,"5min":0.91200000000000003,"15min":0.70799999999999996},"min":{"1min":0.33700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.6959999999999997,"5min":6.6959999999999997,"15min":6.6959999999999997},"last":0.41999999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:58:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1499999999999999,"5min":1.371,"15min":0.79200000000000004},"min":{"1min":0.41999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":9.8970000000000002,"5min":29.152000000000001,"15min":29.152000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.3520000000000001,"15min":0.83399999999999996},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.9699999999999998,"5min":25.864000000000001,"15min":25.864000000000001},"last":2.3929999999999998}]},{"osd":4,"last update":"Fri Jun 14 22:58:20 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":0.66700000000000004,"15min":0.80600000000000005},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":6.774,"5min":6.774,"15min":47.698},"last":0.307},{"interface":"front","average":{"1min":1.0800000000000001,"5min":0.63400000000000001,"15min":0.77000000000000002},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.23899999999999999},"max":{"1min":9.9610000000000003,"5min":9.9610000000000003,"15min":48.197000000000003},"last":0.47099999999999997}]},{"osd":5,"last update":"Fri Jun 14 22:58:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":0.73799999999999999,"15min":0.72299999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.245},"max":{"1min":5.8289999999999997,"5min":5.8289999999999997,"15min":8.6500000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":0.76500000000000001,"15min":0.77400000000000002},"min":{"1min":0.41699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":6.7949999999999999,"5min":6.7949999999999999,"15min":8.6349999999999998},"last":0.48699999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":0.755,"15min":0.77500000000000002},"min":{"1min":0.32500000000000001,"5min":0.314,"15min":0.28799999999999998},"max":{"1min":5.9039999999999999,"5min":5.9039999999999999,"15min":8.6950000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.73799999999999999,"15min":0.79500000000000004},"min":{"1min":0.36699999999999999,"5min":0.33500000000000002,"15min":0.28899999999999998},"max":{"1min":3.496,"5min":3.496,"15min":8.673},"last":0.57299999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757571,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2855032,"kb_used_data":1959792,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":90890120,"statfs":{"total":95995035648,"available":93071482880,"internally_reserved":0,"allocated":2006827008,"data_stored":2005961274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":0.88700000000000001,"15min":1.2050000000000001},"min":{"1min":0.27400000000000002,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":3.5910000000000002,"5min":4.3470000000000004,"15min":17.766999999999999},"last":1.734},{"interface":"front","average":{"1min":0.997,"5min":0.86299999999999999,"15min":1.214},"min":{"1min":0.29599999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.9430000000000001,"5min":4.3659999999999997,"15min":17.806000000000001},"last":1.825}]},{"osd":1,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":0.93999999999999995,"15min":1.264},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.23300000000000001},"max":{"1min":5.6429999999999998,"5min":5.6429999999999998,"15min":7.694},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.0880000000000001,"15min":1.181},"min":{"1min":0.26400000000000001,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":4.4939999999999998,"5min":4.4939999999999998,"15min":5.3259999999999996},"last":0.77700000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":0.95299999999999996,"15min":0.68200000000000005},"min":{"1min":0.39200000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.5579999999999998,"5min":5.9379999999999997,"15min":5.9379999999999997},"last":1.601},{"interface":"front","average":{"1min":1.4379999999999999,"5min":0.90800000000000003,"15min":0.68200000000000005},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.6050000000000004,"5min":4.6050000000000004,"15min":4.6050000000000004},"last":0.29199999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:58:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.4930000000000001,"15min":0.84999999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.8519999999999999,"5min":45.744,"15min":45.744},"last":1.129},{"interface":"front","average":{"1min":1.8049999999999999,"5min":1.6699999999999999,"15min":0.98199999999999998},"min":{"1min":0.33600000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.1779999999999999,"5min":45.838000000000001,"15min":45.838000000000001},"last":1.105}]},{"osd":4,"last update":"Fri Jun 14 22:58:04 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.58799999999999997,"15min":0.57499999999999996},"min":{"1min":0.29599999999999999,"5min":0.26200000000000001,"15min":0.214},"max":{"1min":2.9359999999999999,"5min":3.3039999999999998,"15min":4.9790000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.623,"15min":0.65600000000000003},"min":{"1min":0.28299999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.1059999999999999,"5min":3.1059999999999999,"15min":6.0149999999999997},"last":0.39200000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:58:33 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":0.73499999999999999,"15min":0.746},"min":{"1min":0.35999999999999999,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":6.931,"5min":6.931,"15min":9.6020000000000003},"last":0.432},{"interface":"front","average":{"1min":1.069,"5min":0.68799999999999994,"15min":0.72999999999999998},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":4.452,"5min":4.452,"15min":12.339},"last":0.42099999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.81299999999999994,"15min":0.72799999999999998},"min":{"1min":0.34200000000000003,"5min":0.32600000000000001,"15min":0.27900000000000003},"max":{"1min":4.8689999999999998,"5min":5.2629999999999999,"15min":5.2629999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.069,"5min":0.86699999999999999,"15min":0.70999999999999996},"min":{"1min":0.38400000000000001,"5min":0.29999999999999999,"15min":0.25600000000000001},"max":{"1min":2.972,"5min":10.638999999999999,"15min":10.638999999999999},"last":0.40999999999999998}]}]},{"osd":4,"up_from":30,"seq":128849019207,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1254660,"kb_used_data":668924,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92490492,"statfs":{"total":95995035648,"available":94710263808,"internally_reserved":0,"allocated":684978176,"data_stored":684104450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":0.70399999999999996,"15min":1.018},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.5389999999999997,"5min":4.5389999999999997,"15min":7.5650000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.66100000000000003,"15min":1.165},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":2.552,"5min":3.718,"15min":9.1470000000000002},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":0.94299999999999995,"15min":1.1279999999999999},"min":{"1min":0.26300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":8.4960000000000004,"5min":8.4960000000000004,"15min":8.4960000000000004},"last":1.637},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.0429999999999999,"15min":1.2749999999999999},"min":{"1min":0.30099999999999999,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":5.4160000000000004,"5min":6.5549999999999997,"15min":6.5549999999999997},"last":4.8529999999999998}]},{"osd":2,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":0.754,"15min":0.628},"min":{"1min":0.29399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":8.9250000000000007,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.78600000000000003,"15min":0.64500000000000002},"min":{"1min":0.32500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.456,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:58:16 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.0049999999999999,"15min":0.69799999999999995},"min":{"1min":0.35599999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.617,"5min":6.617,"15min":6.617},"last":1.8240000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":0.97699999999999998,"15min":0.72099999999999997},"min":{"1min":0.36499999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.6989999999999998,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":1.627}]},{"osd":5,"last update":"Fri Jun 14 22:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":0.74399999999999999,"15min":0.71799999999999997},"min":{"1min":0.29399999999999998,"5min":0.24199999999999999,"15min":0.21199999999999999},"max":{"1min":5.2279999999999998,"5min":5.2279999999999998,"15min":5.7030000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.5329999999999999,"5min":0.75900000000000001,"15min":0.71099999999999997},"min":{"1min":0.29399999999999998,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":6.3700000000000001,"5min":6.3700000000000001,"15min":6.3700000000000001},"last":0.33000000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:58:25 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.63700000000000001,"15min":0.63400000000000001},"min":{"1min":0.30599999999999999,"5min":0.30199999999999999,"15min":0.254},"max":{"1min":5.1210000000000004,"5min":5.1210000000000004,"15min":5.1210000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":0.68999999999999995,"15min":0.623},"min":{"1min":0.33000000000000002,"5min":0.28899999999999998,"15min":0.23400000000000001},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.34799999999999998}]},{"osd":7,"last update":"Fri Jun 14 22:57:48 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.65600000000000003,"15min":0.65000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.246},"max":{"1min":3.0950000000000002,"5min":3.0950000000000002,"15min":5.0389999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.173,"5min":0.86399999999999999,"15min":0.72599999999999998},"min":{"1min":0.372,"5min":0.308,"15min":0.29399999999999998},"max":{"1min":4.4029999999999996,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":0.58299999999999996}]}]},{"osd":3,"up_from":138,"seq":592705486906,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1935028,"kb_used_data":1738732,"kb_used_omap":4,"kb_used_meta":196283,"kb_avail":91810124,"statfs":{"total":95995035648,"available":94013566976,"internally_reserved":0,"allocated":1780461568,"data_stored":1779599679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":200994270},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:09 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":0.79700000000000004,"15min":0.48099999999999998},"min":{"1min":0.19,"5min":0.184,"15min":0.184},"max":{"1min":4.8070000000000004,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":5.585},{"interface":"front","average":{"1min":1.177,"5min":0.67600000000000005,"15min":0.46400000000000002},"min":{"1min":0.23599999999999999,"5min":0.185,"15min":0.185},"max":{"1min":3.7229999999999999,"5min":3.7229999999999999,"15min":3.7229999999999999},"last":1.349}]},{"osd":1,"last update":"Fri Jun 14 22:58:09 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":0.90900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34399999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.1520000000000001},"last":2.3239999999999998},{"interface":"front","average":{"1min":1.653,"5min":0.878,"15min":0.55500000000000005},"min":{"1min":0.32200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.1280000000000001,"5min":4.1280000000000001,"15min":4.1280000000000001},"last":1.3300000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:58:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":0.77500000000000002,"15min":0.53200000000000003},"min":{"1min":0.249,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.2910000000000004,"5min":5.2910000000000004,"15min":5.2910000000000004},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.71599999999999997,"15min":0.47599999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.1709999999999998,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.505}]},{"osd":4,"last update":"Fri Jun 14 22:58:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.86499999999999999,"15min":0.70899999999999996},"min":{"1min":0.28100000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.2509999999999999,"5min":5.7960000000000003,"15min":5.7960000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.47,"5min":0.95399999999999996,"15min":0.73799999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.952,"5min":5.2720000000000002,"15min":5.2720000000000002},"last":5.6660000000000004}]},{"osd":5,"last update":"Fri Jun 14 22:58:09 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.04,"15min":0.77200000000000002},"min":{"1min":0.45500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.919,"5min":5.3559999999999999,"15min":5.3559999999999999},"last":5.6200000000000001},{"interface":"front","average":{"1min":1.6479999999999999,"5min":0.98599999999999999,"15min":0.73599999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.1829999999999998,"5min":4.1829999999999998,"15min":4.1829999999999998},"last":1.3100000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:58:09 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":0.92900000000000005,"15min":0.73899999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":4.0449999999999999,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.85099999999999998,"15min":0.69099999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":3.202,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.71299999999999997}]},{"osd":7,"last update":"Fri Jun 14 22:58:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.0960000000000001,"15min":0.79900000000000004},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.2210000000000001,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":3.423},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.099,"15min":0.78800000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":4.5419999999999998,"5min":5.3300000000000001,"15min":5.3300000000000001},"last":5.6449999999999996}]}]},{"osd":2,"up_from":133,"seq":571230650433,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762612,"kb_used_data":653292,"kb_used_omap":6,"kb_used_meta":109305,"kb_avail":92982540,"statfs":{"total":95995035648,"available":95214120960,"internally_reserved":0,"allocated":668971008,"data_stored":668109333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":111928821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.63400000000000001,"15min":0.45200000000000001},"min":{"1min":0.19700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.3820000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.442},{"interface":"front","average":{"1min":1.3280000000000001,"5min":0.65500000000000003,"15min":0.45900000000000002},"min":{"1min":0.19700000000000001,"5min":0.182,"15min":0.182},"max":{"1min":4.8019999999999996,"5min":4.8019999999999996,"15min":4.8019999999999996},"last":0.71699999999999997}]},{"osd":1,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.1419999999999999,"15min":0.63400000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.8200000000000003,"5min":21.920000000000002,"15min":21.920000000000002},"last":3.016},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.111,"15min":0.61099999999999999},"min":{"1min":0.27200000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.4620000000000002,"5min":21.573,"15min":21.573},"last":0.41499999999999998}]},{"osd":3,"last update":"Fri Jun 14 22:58:13 2024","interfaces":[{"interface":"back","average":{"1min":1.903,"5min":1.0249999999999999,"15min":0.58399999999999996},"min":{"1min":0.315,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.1189999999999998,"5min":21.66,"15min":21.66},"last":1.071},{"interface":"front","average":{"1min":1.117,"5min":1.0549999999999999,"15min":0.61299999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.0449999999999999,"5min":22.341000000000001,"15min":22.341000000000001},"last":0.36199999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":0.83299999999999996,"15min":0.66700000000000004},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":6.7030000000000003,"5min":6.7030000000000003,"15min":6.7030000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.034,"5min":0.77900000000000003,"15min":0.64500000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.8890000000000002,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":3.052}]},{"osd":5,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":0.86799999999999999,"15min":0.66200000000000003},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.0800000000000001,"5min":5.0800000000000001,"15min":5.0800000000000001},"last":0.75},{"interface":"front","average":{"1min":1.399,"5min":0.80600000000000005,"15min":0.65600000000000003},"min":{"1min":0.38100000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.0449999999999999,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":0.69099999999999995}]},{"osd":6,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":0.70899999999999996,"15min":0.61199999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":5.8579999999999997,"5min":5.8579999999999997,"15min":5.8579999999999997},"last":3.0760000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.83499999999999996,"15min":0.66300000000000003},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.3759999999999999,"5min":5.1829999999999998,"15min":5.1829999999999998},"last":0.54500000000000004}]},{"osd":7,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":0.94599999999999995,"15min":0.71499999999999997},"min":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.226,"5min":6.226,"15min":6.226},"last":3.0990000000000002},{"interface":"front","average":{"1min":1.3049999999999999,"5min":0.80400000000000005,"15min":0.65800000000000003},"min":{"1min":0.41499999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":7.6260000000000003,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":0.52100000000000002}]}]},{"osd":1,"up_from":116,"seq":498216206450,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2366644,"kb_used_data":1970348,"kb_used_omap":5,"kb_used_meta":396154,"kb_avail":91378508,"statfs":{"total":95995035648,"available":93571592192,"internally_reserved":0,"allocated":2017636352,"data_stored":2016774397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":405661976},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":0.999,"15min":1.98},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":6.1870000000000003,"5min":8.2560000000000002,"15min":9.0559999999999992},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.254,"5min":0.879,"15min":1.5069999999999999},"min":{"1min":0.20399999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":4.5330000000000004,"5min":5.0220000000000002,"15min":10.426},"last":0.65100000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.792,"5min":1.0720000000000001,"15min":0.58099999999999996},"min":{"1min":0.22600000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":6.1539999999999999,"5min":8.2880000000000003,"15min":8.2880000000000003},"last":1.9299999999999999},{"interface":"front","average":{"1min":1.113,"5min":0.81799999999999995,"15min":0.496},"min":{"1min":0.28199999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":0.67200000000000004}]},{"osd":3,"last update":"Fri Jun 14 22:58:10 2024","interfaces":[{"interface":"back","average":{"1min":2.3839999999999999,"5min":1.431,"15min":0.69699999999999995},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.2089999999999996,"5min":21.422000000000001,"15min":21.422000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.3240000000000001,"15min":0.68700000000000006},"min":{"1min":0.307,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":5.2039999999999997,"5min":21.016999999999999,"15min":21.016999999999999},"last":0.73599999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.161,"15min":1.7649999999999999},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.28399999999999997},"max":{"1min":6.6360000000000001,"5min":7.673,"15min":8.8599999999999994},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.109,"5min":1.327,"15min":1.79},"min":{"1min":0.29399999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.4500000000000002,"5min":8.3620000000000001,"15min":11.161},"last":0.78700000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":1.147,"15min":1.6719999999999999},"min":{"1min":0.39500000000000002,"5min":0.32800000000000001,"15min":0.30599999999999999},"max":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":9.7539999999999996},"last":2.6160000000000001},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.24,"15min":1.823},"min":{"1min":0.499,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":6.5730000000000004,"5min":7.7439999999999998,"15min":8.125},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.042,"15min":1.7},"min":{"1min":0.373,"5min":0.35799999999999998,"15min":0.32100000000000001},"max":{"1min":4.9459999999999997,"5min":5.3849999999999998,"15min":9.7279999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.633,"5min":1.153,"15min":1.7869999999999999},"min":{"1min":0.39100000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.8899999999999997,"5min":7.7060000000000004,"15min":8.1479999999999997},"last":0.57199999999999995}]},{"osd":7,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":2.7189999999999999,"5min":1.419,"15min":1.899},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.33700000000000002},"max":{"1min":6.218,"5min":8.3369999999999997,"15min":9.0960000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.8999999999999999,"5min":1.1779999999999999,"15min":1.581},"min":{"1min":0.41499999999999998,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":6.3529999999999998,"5min":6.3529999999999998,"15min":9.5449999999999999},"last":0.70199999999999996}]}]},{"osd":0,"up_from":91,"seq":390842024105,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1338996,"kb_used_data":847148,"kb_used_omap":5,"kb_used_meta":491834,"kb_avail":92406156,"statfs":{"total":95995035648,"available":94623903744,"internally_reserved":0,"allocated":867479552,"data_stored":866616116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":503638106},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:58:34 2024","interfaces":[{"interface":"back","average":{"1min":2.327,"5min":1.2749999999999999,"15min":1.609},"min":{"1min":0.19900000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":15.547000000000001,"5min":16.305,"15min":16.305},"last":0.44400000000000001},{"interface":"front","average":{"1min":2.016,"5min":1.3879999999999999,"15min":2.0640000000000001},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.9889999999999999,"5min":16.184999999999999,"15min":16.184999999999999},"last":0.39100000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:57:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":0.79900000000000004,"15min":0.505},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.7679999999999998,"5min":16.757000000000001,"15min":16.757000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.4830000000000001,"5min":0.85399999999999998,"15min":0.55000000000000004},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":4.7350000000000003,"5min":16.055,"15min":16.055},"last":0.246}]},{"osd":3,"last update":"Fri Jun 14 22:58:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1080000000000001,"5min":1.169,"15min":0.63200000000000001},"min":{"1min":0.33500000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":7.04,"5min":16.873999999999999,"15min":16.873999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.1779999999999999,"15min":0.63200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":12.308999999999999,"5min":16.085999999999999,"15min":16.085999999999999},"last":0.23400000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:58:19 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.0169999999999999,"15min":1.498},"min":{"1min":0.44,"5min":0.308,"15min":0.27600000000000002},"max":{"1min":5.7050000000000001,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.307},{"interface":"front","average":{"1min":2.0409999999999999,"5min":1.117,"15min":1.3700000000000001},"min":{"1min":0.39200000000000002,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":6.9210000000000003,"5min":15.976000000000001,"15min":15.976000000000001},"last":0.42399999999999999}]},{"osd":5,"last update":"Fri Jun 14 22:58:19 2024","interfaces":[{"interface":"back","average":{"1min":2.1400000000000001,"5min":1.3069999999999999,"15min":1.4710000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29299999999999998},"max":{"1min":6.9640000000000004,"5min":16.108000000000001,"15min":16.108000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.173,"15min":1.4930000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.3890000000000002,"5min":16.006,"15min":16.006},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:58:19 2024","interfaces":[{"interface":"back","average":{"1min":1.984,"5min":1.143,"15min":1.4390000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.016,"5min":16.152000000000001,"15min":16.152000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.206,"5min":1.0429999999999999,"15min":1.478},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.29399999999999998},"max":{"1min":4.4210000000000003,"5min":16.227,"15min":16.227},"last":0.32700000000000001}]},{"osd":7,"last update":"Fri Jun 14 22:58:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.1779999999999999,"15min":1.5309999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.33100000000000002},"max":{"1min":4.4509999999999996,"5min":16.199999999999999,"15min":16.199999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":2.1899999999999999,"5min":1.2010000000000001,"15min":1.641},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.314},"max":{"1min":12.361000000000001,"5min":16.259,"15min":16.259},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":852230144,"data_stored":852230144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1985417216,"data_stored":1985413144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":662241280,"data_stored":662241280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1762328576,"data_stored":1762328576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":673054720,"data_stored":673054720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1996951552,"data_stored":1996947480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":663617536,"data_stored":663617536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1997541376,"data_stored":1997537304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:58:52.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-06-14T22:58:52.200 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:52.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:52.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:52.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:52.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:52.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:52.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:52.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:52.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:52.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:52.734 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:52.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:52.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:52.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:52.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:52.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:52.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:53.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:53.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:53.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:53.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:53.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:53.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:53.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:53.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:53.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:53.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:53.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:53.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:53.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:53.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:53.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:53.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:53.705 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T22:58:53.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:53.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:53.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:53.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:53.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:53.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:53 smithi137 ceph-mon[90634]: pgmap v670: 89 pgs: 89 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 842 op/s 2024-06-14T22:58:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4063989407' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:58:54.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:54.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:53 smithi086 ceph-mon[97913]: pgmap v670: 89 pgs: 89 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 842 op/s 2024-06-14T22:58:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4063989407' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:58:54.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:53 smithi086 ceph-mon[102754]: pgmap v670: 89 pgs: 89 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 842 op/s 2024-06-14T22:58:54.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4063989407' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:58:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:54.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:54.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:54.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:54.243 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:54.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:54.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:54.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:54.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:54.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:54.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:54.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:54.559 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:54.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:54.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:54.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:54.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:54.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:54.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:54.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:54 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:54.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4063989407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:58:54.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:54 smithi086 ceph-mon[97913]: osdmap e170: 8 total, 8 up, 8 in 2024-06-14T22:58:54.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:54 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:54.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4063989407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:58:54.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:54 smithi086 ceph-mon[102754]: osdmap e170: 8 total, 8 up, 8 in 2024-06-14T22:58:54.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:54.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:54.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:54 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:58:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:54 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4063989407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:58:54.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:54 smithi137 ceph-mon[90634]: osdmap e170: 8 total, 8 up, 8 in 2024-06-14T22:58:55.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:55.077 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:55.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:55.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:55.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:55.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:55.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:55.334 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:58:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:58:54] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T22:58:55.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:55.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:55.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:55.488 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:55.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:55.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:55.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:55.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:55.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:55.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:55.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:55.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:55.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:55.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:55.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:55 smithi137 ceph-mon[90634]: pgmap v672: 89 pgs: 89 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 896 op/s 2024-06-14T22:58:56.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:56.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:56.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:56.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:56.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:55 smithi086 ceph-mon[97913]: pgmap v672: 89 pgs: 89 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 896 op/s 2024-06-14T22:58:56.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:55 smithi086 ceph-mon[102754]: pgmap v672: 89 pgs: 89 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 896 op/s 2024-06-14T22:58:56.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:56.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:56.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:58:56.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:56.315 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:56.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:56.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:56.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:56.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:56.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:56.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T22:58:56.603 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:56.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:56.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:56.736 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:56.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:56.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:56.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:56.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:57.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:57.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:57.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:57.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:57.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:57.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:57.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:57.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:57.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:57.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:57.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:57.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:57.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:57.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:57.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:57.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:57.822 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:58:57.822 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:58:57.822 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:58:57.822 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:58:57.822 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:58:57.822 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:58:57.823 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:58:57.823 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:58:57.823 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:58:57.823 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:58:57.823 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:58:57.823 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:58:57.823 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:58:57.823 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:58:57.823 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:58:57.824 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:58:57.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:57.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:57 smithi086 ceph-mon[97913]: pgmap v673: 89 pgs: 89 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 896 op/s 2024-06-14T22:58:57.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1051008798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:57.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:57.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:57 smithi086 ceph-mon[102754]: pgmap v673: 89 pgs: 89 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 896 op/s 2024-06-14T22:58:57.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1051008798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:57.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:57.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:57.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:57.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:57.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:57 smithi137 ceph-mon[90634]: pgmap v673: 89 pgs: 89 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 896 op/s 2024-06-14T22:58:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1051008798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:58:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:58:57.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:57.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:58.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:58.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:58.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:58.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:58.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:58.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:58.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:58:58.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:58.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:58.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:58:58.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:58.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:58.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:58.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:58:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:58.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:58.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:58:58.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:58.822 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:58.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:58.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:58.960 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:58.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:59.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:58:59.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:59.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:58:59.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:58:59.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:59.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:58:59.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:58:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:59.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:59.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:58:59.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:58:59.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:59.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:58:59.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:58:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:59.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:58:59.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:58:59.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:59.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:59.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:59.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:59.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:58:59.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:58:59.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:58:59.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:58:59.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:58:59 smithi137 ceph-mon[90634]: pgmap v674: 89 pgs: 89 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 896 op/s 2024-06-14T22:59:00.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:00.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:00.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:00.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:58:59 smithi086 ceph-mon[102754]: pgmap v674: 89 pgs: 89 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 896 op/s 2024-06-14T22:59:00.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:58:59 smithi086 ceph-mon[97913]: pgmap v674: 89 pgs: 89 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 896 op/s 2024-06-14T22:59:00.149 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:00.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:00.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:00.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:00.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:00.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:00.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:00.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:00.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:00.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:00.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:00.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:00.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:00.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:59:00.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:00.769 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:00.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:00.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:00.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:00.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:01.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:01.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:01.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:01.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:01.195 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:01.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:01.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:01.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:01.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:01.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:01.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:01.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:01.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:01.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:01.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:01.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:01.734 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:01.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:01.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:01.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:01.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:01.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:02.006 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:02.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:02.076 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:01 smithi086 ceph-mon[102754]: pgmap v675: 89 pgs: 89 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 888 op/s 2024-06-14T22:59:02.076 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:01 smithi086 ceph-mon[97913]: pgmap v675: 89 pgs: 89 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 888 op/s 2024-06-14T22:59:02.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:02.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:02.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:02.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:02.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:01 smithi137 ceph-mon[90634]: pgmap v675: 89 pgs: 89 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 888 op/s 2024-06-14T22:59:02.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:02.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:02.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:02.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:02.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:02.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:02.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:02.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:02.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:02.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:02.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:02.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:02.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:02.840 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:02.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:02.916 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4106498913' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T22:59:02.916 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4106498913' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T22:59:02.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:02.979 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:02.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:59:02.989 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T22:59:02.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T22:59:02.990 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T22:59:02.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.990 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:20569 8.1 8:8a52555f:::benchmark_data_smithi086_122122_object20568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e113)", 2024-06-14T22:59:02.990 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:08.128996+0000", 2024-06-14T22:59:02.990 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 594.78311880900003, 2024-06-14T22:59:02.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99832512699999998, 2024-06-14T22:59:02.990 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:02.990 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T22:59:02.990 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:02.991 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:02.991 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:02.991 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 20569 2024-06-14T22:59:02.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.991 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:02.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:02.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.128996+0000", 2024-06-14T22:59:02.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.128996+0000", 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.128994+0000", 2024-06-14T22:59:02.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T22:59:02.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:02.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129024+0000", 2024-06-14T22:59:02.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9388000000000001e-05 2024-06-14T22:59:02.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:02.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129046+0000", 2024-06-14T22:59:02.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2385e-05 2024-06-14T22:59:02.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:02.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129058+0000", 2024-06-14T22:59:02.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1654000000000001e-05 2024-06-14T22:59:02.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129111+0000", 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3245000000000003e-05 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:08.129122+0000", 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0903e-05 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:09.124962+0000", 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99583976600000002 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:09.125103+0000", 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014192999999999999 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:09.127246+0000", 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021425060000000002 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:09.127321+0000", 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5135999999999993e-05 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:02.997 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55937 8.1 8:8b1d3009:::benchmark_data_smithi086_122122_object55936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e117)", 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:25.469855+0000", 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 577.44226030100003, 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1369695980000001, 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:02.998 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:02.999 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:02.999 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55937 2024-06-14T22:59:02.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:02.999 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:02.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:02.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:02.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469855+0000", 2024-06-14T22:59:02.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:02.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469855+0000", 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469853+0000", 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:59:03.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469880+0000", 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6359e-05 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469890+0000", 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0591000000000001e-05 2024-06-14T22:59:03.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469900+0000", 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3640000000000005e-06 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.470053+0000", 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015309699999999999 2024-06-14T22:59:03.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.470062+0000", 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1870000000000003e-06 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.476164+0000", 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006101843 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.476176+0000", 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1759e-05 2024-06-14T22:59:03.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.587460+0000", 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1112849090000001 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.587708+0000", 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000247545 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.587896+0000", 2024-06-14T22:59:03.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000187896 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.594356+0000", 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064597040000000001 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.594397+0000", 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0932e-05 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.604097+0000", 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.009700195 2024-06-14T22:59:03.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.604523+0000", 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00042586799999999998 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.606824+0000", 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023015380000000001 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:55950 8.1 8:8ad1e56a:::benchmark_data_smithi086_122122_object55949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e117)", 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:49:25.469906+0000", 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 577.44220920999999, 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.138639081, 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55950 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469906+0000", 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469906+0000", 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469905+0000", 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:03.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469938+0000", 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3105000000000002e-05 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469945+0000", 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.289e-06 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.469951+0000", 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8760000000000002e-06 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.470110+0000", 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015907599999999999 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting_for_map not empty", 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.470117+0000", 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6850000000000004e-06 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.476497+0000", 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063794699999999999 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:25.476506+0000", 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2760000000000006e-06 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.591937+0000", 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1154308989999999 2024-06-14T22:59:03.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.592083+0000", 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014589200000000001 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.592211+0000", 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012861799999999999 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.594817+0000", 2024-06-14T22:59:03.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026060699999999998 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.594940+0000", 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012278300000000001 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.606883+0000", 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011943339000000001 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.607027+0000", 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014385299999999999 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:49:26.608545+0000", 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015173859999999999 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83971 9.9 9:99f5752d:::benchmark_data_smithi086_152062_object83970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.088245+0000", 2024-06-14T22:59:03.014 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.82387018899999, 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.169417085, 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83971 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088245+0000", 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088245+0000", 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088241+0000", 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999967 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088265+0000", 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4185999999999998e-05 2024-06-14T22:59:03.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088276+0000", 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0818999999999999e-05 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088282+0000", 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2559999999999999e-06 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088318+0000", 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5987e-05 2024-06-14T22:59:03.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088428+0000", 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109157 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088618+0000", 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000190936 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.095042+0000", 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064230329999999999 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105426+0000", 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010384167999999999 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257565+0000", 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15213905999999999 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257619+0000", 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4005999999999998e-05 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257662+0000", 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2976000000000003e-05 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83974 9.9 9:99e21f48:::benchmark_data_smithi086_152062_object83973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.020 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.089191+0000", 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.82292338600001, 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.16863312799999999, 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83974 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089191+0000", 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089191+0000", 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089191+0000", 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089220+0000", 2024-06-14T22:59:03.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9252999999999999e-05 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089230+0000", 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8130000000000007e-06 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089236+0000", 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.268e-06 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.091590+0000", 2024-06-14T22:59:03.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023543259999999999 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.091768+0000", 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017736499999999999 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.091883+0000", 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011573099999999999 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.095048+0000", 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031641579999999998 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105462+0000", 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010414526 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257737+0000", 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.152274889 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257791+0000", 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4196000000000001e-05 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257825+0000", 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3441000000000001e-05 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83983 9.b 9:d50ba4e2:::benchmark_data_smithi086_152062_object83982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.026 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.097967+0000", 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.814147706, 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.159703397, 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83983 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.097967+0000", 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.097967+0000", 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.097966+0000", 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.097987+0000", 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1386000000000002e-05 2024-06-14T22:59:03.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098000+0000", 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2683999999999999e-05 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098006+0000", 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4840000000000001e-06 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.099132+0000", 2024-06-14T22:59:03.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011255939999999999 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.099231+0000", 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9570999999999999e-05 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.099318+0000", 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6347999999999994e-05 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105456+0000", 2024-06-14T22:59:03.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061385839999999999 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105470+0000", 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4134999999999999e-05 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257539+0000", 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.152069074 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257635+0000", 2024-06-14T22:59:03.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5579000000000001e-05 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257671+0000", 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5451000000000003e-05 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83985 9.b 9:d28a972b:::benchmark_data_smithi086_152062_object83984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.098330+0000", 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.813785142, 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15945879099999999, 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.032 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83985 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098330+0000", 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098330+0000", 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098329+0000", 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098346+0000", 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6656000000000001e-05 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098351+0000", 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098355+0000", 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5190000000000001e-06 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102075+0000", 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037196130000000001 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102215+0000", 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014068300000000001 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102357+0000", 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014141199999999999 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105599+0000", 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0032428119999999999 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108779+0000", 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031796620000000002 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257712+0000", 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14893310400000001 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257759+0000", 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6999999999999997e-05 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257788+0000", 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9207999999999999e-05 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83989 9.9 9:9deded04:::benchmark_data_smithi086_152062_object83988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.102222+0000", 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80989238199999, 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15581730699999999, 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.038 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83989 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102222+0000", 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102222+0000", 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102221+0000", 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102245+0000", 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4485e-05 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102259+0000", 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4005999999999999e-05 2024-06-14T22:59:03.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102268+0000", 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2279999999999998e-06 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102421+0000", 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015389099999999999 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102550+0000", 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000128279 2024-06-14T22:59:03.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102676+0000", 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000125964 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105594+0000", 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002918258 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257705+0000", 2024-06-14T22:59:03.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15211116299999999 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257913+0000", 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000208216 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257990+0000", 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6217999999999994e-05 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258040+0000", 2024-06-14T22:59:03.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0238000000000002e-05 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83994 9.9 9:92cd671a:::benchmark_data_smithi086_152062_object83993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108378+0000", 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80373645, 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14977987000000001, 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.044 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83994 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108378+0000", 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108378+0000", 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108377+0000", 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108399+0000", 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2325999999999999e-05 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108411+0000", 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1615e-05 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108417+0000", 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0340000000000002e-06 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108456+0000", 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8645e-05 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108609+0000", 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015293699999999999 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:03.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108775+0000", 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016661099999999999 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257869+0000", 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14909354699999999 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257878+0000", 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7390000000000005e-06 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258082+0000", 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000204184 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258125+0000", 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2861999999999997e-05 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258158+0000", 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3636000000000001e-05 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.049 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83995 9.9 9:91dc56da:::benchmark_data_smithi086_152062_object83994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108431+0000", 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80368427600001, 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15774317700000001, 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83995 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.050 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108431+0000", 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108431+0000", 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108430+0000", 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:03.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108449+0000", 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8578000000000001e-05 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108455+0000", 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3119999999999999e-06 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108460+0000", 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6999999999999999e-06 2024-06-14T22:59:03.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254637+0000", 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.146177636 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254830+0000", 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000192404 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254981+0000", 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015106399999999999 2024-06-14T22:59:03.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.262344+0000", 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073629960000000001 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.262359+0000", 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4986e-05 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.266048+0000", 2024-06-14T22:59:03.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036887959999999998 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.266134+0000", 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6101999999999997e-05 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.266174+0000", 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0074000000000003e-05 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83971 9.9 e150/140 ondisk, result = 0)", 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108484+0000", 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80363066000001, 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.149187404, 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108484+0000", 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108484+0000", 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108484+0000", 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108489+0000", 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.575e-06 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108502+0000", 2024-06-14T22:59:03.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2952999999999999e-05 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108506+0000", 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0640000000000004e-06 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257550+0000", 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14904347300000001 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257558+0000", 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9370000000000004e-06 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257672+0000", 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114052 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83974 9.9 e150/140 ondisk, result = 0)", 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108513+0000", 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.803601888, 2024-06-14T22:59:03.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.149319274, 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108513+0000", 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108513+0000", 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108513+0000", 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108514+0000", 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7129999999999999e-06 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108518+0000", 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1350000000000001e-06 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108520+0000", 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2809999999999998e-06 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257726+0000", 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14920634999999999 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257732+0000", 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7590000000000003e-06 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257832+0000", 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100278 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83989 9.9 e150/140 ondisk, result = 0)", 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108621+0000", 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80349331799999, 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14942707399999999, 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108621+0000", 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108621+0000", 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108621+0000", 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108625+0000", 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3640000000000001e-06 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108634+0000", 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108639+0000", 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2469999999999999e-06 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257903+0000", 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14926384300000001 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257909+0000", 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.913e-06 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258049+0000", 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000139808 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83997 9.b 9:dc8147a4:::benchmark_data_smithi086_152062_object83996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108697+0000", 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80341773200001, 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15510373, 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83997 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108697+0000", 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108697+0000", 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108696+0000", 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108716+0000", 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9347e-05 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108726+0000", 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0271999999999999e-05 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108734+0000", 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7020000000000001e-06 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108882+0000", 2024-06-14T22:59:03.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014828399999999999 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109035+0000", 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015296 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251864+0000", 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14282935599999999 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257801+0000", 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059364630000000003 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257872+0000", 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0970999999999995e-05 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.263662+0000", 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057900950000000003 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.263762+0000", 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100052 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.263801+0000", 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.909e-05 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83989 9.9 e150/140 ondisk, result = 0)", 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108796+0000", 2024-06-14T22:59:03.072 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80331901599999, 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14891169800000001, 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108796+0000", 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108796+0000", 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108795+0000", 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108828+0000", 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2316e-05 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108837+0000", 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.217e-06 2024-06-14T22:59:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108843+0000", 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8839999999999999e-06 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257693+0000", 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14885061399999999 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257700+0000", 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5259999999999999e-06 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257707+0000", 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7069999999999992e-06 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83998 9.b 9:dbbacc67:::benchmark_data_smithi086_152062_object83997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108876+0000", 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80323868299999, 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15535863599999999, 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83998 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108876+0000", 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108876+0000", 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108875+0000", 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108895+0000", 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9137e-05 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108904+0000", 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7950000000000002e-06 2024-06-14T22:59:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108910+0000", 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6359999999999999e-06 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254780+0000", 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.145869586 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254884+0000", 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104069 2024-06-14T22:59:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.255046+0000", 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016279700000000001 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257805+0000", 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027586429999999999 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258788+0000", 2024-06-14T22:59:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0009826940000000001 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.263865+0000", 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0050769819999999998 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.264197+0000", 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000332434 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.264235+0000", 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7564999999999997e-05 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83999 9.9 9:9a69a847:::benchmark_data_smithi086_152062_object83998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.109179+0000", 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80293587200001, 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15805195, 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83999 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109179+0000", 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109179+0000", 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109178+0000", 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109200+0000", 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1909e-05 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251700+0000", 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14249978799999999 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251713+0000", 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3613e-05 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258720+0000", 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070072509999999999 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.259074+0000", 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000353815 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.259462+0000", 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00038739500000000001 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.262363+0000", 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029017909999999999 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.266703+0000", 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043395769999999998 2024-06-14T22:59:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.267058+0000", 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035485300000000003 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.267188+0000", 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013003099999999999 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.267231+0000", 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2994999999999997e-05 2024-06-14T22:59:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83983 9.b e150/140 ondisk, result = 0)", 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.110471+0000", 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 153.80164388899999, 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.147209854, 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.110471+0000", 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.110471+0000", 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.110470+0000", 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.110475+0000", 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9019999999999998e-06 2024-06-14T22:59:03.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251660+0000", 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14118467000000001 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251668+0000", 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5520000000000005e-06 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257523+0000", 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058546839999999998 2024-06-14T22:59:03.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257533+0000", 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0125e-05 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257681+0000", 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147687 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.091 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:189328 9.b 9:ded1c998:::benchmark_data_smithi086_152062_object56846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:58:18.751601+0000", 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.160514032000002, 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14561811199999999, 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 189328 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751601+0000", 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751601+0000", 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751599+0000", 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751606+0000", 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.297e-06 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751620+0000", 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4623e-05 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751628+0000", 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0949999999999996e-06 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751707+0000", 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8498000000000006e-05 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751886+0000", 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017868199999999999 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751997+0000", 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011175699999999999 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.757120+0000", 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005122817 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.757780+0000", 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00065953400000000003 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897069+0000", 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13928911799999999 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897155+0000", 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6499e-05 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897219+0000", 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3600000000000001e-05 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:189340 9.b 9:d0a0718a:::benchmark_data_smithi086_152062_object56858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:58:18.765759+0000", 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.146355481000001, 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13161884700000001, 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:03.097 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 189340 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765759+0000", 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765759+0000", 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:03.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765758+0000", 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765764+0000", 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6969999999999998e-06 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765775+0000", 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1484000000000001e-05 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765782+0000", 2024-06-14T22:59:03.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.905e-06 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765830+0000", 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7967999999999999e-05 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765967+0000", 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013704799999999999 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.766092+0000", 2024-06-14T22:59:03.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012515099999999999 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.774208+0000", 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0081152389999999998 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.774256+0000", 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8254000000000003e-05 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897253+0000", 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.122996893 2024-06-14T22:59:03.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897342+0000", 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9734000000000005e-05 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897378+0000", 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5734e-05 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:03.102 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:59:03.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:03.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:03.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:03.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4106498913' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T22:59:03.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:03.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:03.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:03.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:03.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:03.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:03.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:03.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:03.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:03.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:03.674 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 9 pgp_num to 16 2024-06-14T22:59:03.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:03.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:03.776 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T22:59:03.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:03.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:03.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:03.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:03.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:03.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:03.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:04.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:04.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:03 smithi086 ceph-mon[97913]: pgmap v676: 89 pgs: 89 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-06-14T22:59:04.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4106498913' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T22:59:04.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:03 smithi086 ceph-mon[97913]: osdmap e171: 8 total, 8 up, 8 in 2024-06-14T22:59:04.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1709146914' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:04.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:03 smithi086 ceph-mon[102754]: pgmap v676: 89 pgs: 89 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-06-14T22:59:04.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4106498913' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T22:59:04.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:03 smithi086 ceph-mon[102754]: osdmap e171: 8 total, 8 up, 8 in 2024-06-14T22:59:04.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1709146914' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:04.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:04.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:04.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:04.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:03 smithi137 ceph-mon[90634]: pgmap v676: 89 pgs: 89 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-06-14T22:59:04.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4106498913' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T22:59:04.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:03 smithi137 ceph-mon[90634]: osdmap e171: 8 total, 8 up, 8 in 2024-06-14T22:59:04.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1709146914' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:04.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:04.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:04.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:04.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:04.522 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:04.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:04.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:04.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:04.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:04.755 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:04.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:04.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:04.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:04.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:04 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:59:04.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:04 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:59:04.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:04 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1709146914' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:59:04.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:04 smithi086 ceph-mon[97913]: osdmap e172: 8 total, 8 up, 8 in 2024-06-14T22:59:04.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:04 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:59:04.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:04 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:59:04.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:04 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1709146914' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:59:04.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:04 smithi086 ceph-mon[102754]: osdmap e172: 8 total, 8 up, 8 in 2024-06-14T22:59:04.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:04.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:04.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:05.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:05.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:05.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:05.165 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:59:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:59:04] "GET /metrics HTTP/1.1" 200 36546 "" "Prometheus/2.43.0" 2024-06-14T22:59:05.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:05.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:05.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:05.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:04 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:59:05.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:04 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:59:05.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:04 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1709146914' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:59:05.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:04 smithi137 ceph-mon[90634]: osdmap e172: 8 total, 8 up, 8 in 2024-06-14T22:59:05.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:05.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:05.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:05.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:05.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:05.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:05.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:05.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:05.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:05.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:05.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:05.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:05.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:05.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:05.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:06.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:06.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:06.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:05 smithi086 ceph-mon[102754]: pgmap v679: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 821 op/s 2024-06-14T22:59:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:05 smithi086 ceph-mon[97913]: pgmap v679: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 821 op/s 2024-06-14T22:59:06.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:06.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:06.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:06.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:05 smithi137 ceph-mon[90634]: pgmap v679: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 821 op/s 2024-06-14T22:59:06.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:06.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:06.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:06.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:06.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:06.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:06.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:06.571 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:06.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:06.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:06.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:06.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:06.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:06.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:06.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:06.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:06.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:06.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:06.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:07.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:07.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:07.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:07.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:07.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:07.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:07.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-14T22:59:07.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:07.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:07.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:07.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:07.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:07.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:07.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:07.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:07.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T22:59:07.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:07.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:07.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:07.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:07.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:07.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:08.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:08.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:07 smithi086 ceph-mon[97913]: pgmap v680: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:59:08.079 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:07 smithi086 ceph-mon[102754]: pgmap v680: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:59:08.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:08.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:08.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:08.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:08.194 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:08.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:08.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:08.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:07 smithi137 ceph-mon[90634]: pgmap v680: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-06-14T22:59:08.276 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:08.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:08.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:08.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:08.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:08.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:08.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:08.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:08.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:08.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:08.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:08.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:08.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:08.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:08.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:08.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:09.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:09.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:09.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:09.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:09.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:09.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:09.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:09.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:09.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:09.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:09.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:09.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:09.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:09.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:09.581 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:59:09.581 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:59:09.590 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:59:09.590 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:59:09.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:09.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:09.621 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:59:09.621 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:59:09.657 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:59:09.657 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:59:09.688 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:59:09.688 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:59:09.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:09.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:09.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:09.732 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:59:09.732 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:59:09.763 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:59:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:59:09.806 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:59:09.807 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:59:09.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:09.856 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:09.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:09.910 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:09 smithi086 ceph-mon[102754]: pgmap v681: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 677 op/s 2024-06-14T22:59:09.910 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2005279465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:09.910 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:09 smithi086 ceph-mon[97913]: pgmap v681: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 677 op/s 2024-06-14T22:59:09.910 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2005279465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:09.931 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T22:59:09.931 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T22:59:09.932 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-06-14T22:59:09.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:59:09.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:09.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:09.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:10.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:10.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:10.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:10.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:10.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:09 smithi137 ceph-mon[90634]: pgmap v681: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 677 op/s 2024-06-14T22:59:10.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2005279465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:10.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:10.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:10.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:10.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:10.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:10.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:10.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:10.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:10.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:10.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:10.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:10.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:10.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:10.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:10.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:10.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:10.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:11.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:11.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:11.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:11.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:11.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:11.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:11.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:11.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:11.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:11.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:11.508 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:11.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:11.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:11.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:11.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:11.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:11.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:11.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:11.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:11.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:11.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:12.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:12.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:12.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:12.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:11 smithi086 ceph-mon[97913]: pgmap v682: 89 pgs: 89 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 965 op/s 2024-06-14T22:59:12.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:11 smithi086 ceph-mon[102754]: pgmap v682: 89 pgs: 89 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 965 op/s 2024-06-14T22:59:12.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:12.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:12.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:12.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:11 smithi137 ceph-mon[90634]: pgmap v682: 89 pgs: 89 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 965 op/s 2024-06-14T22:59:12.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:12.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:12.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:12.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:12.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:12.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:12.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:12.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:12.581 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:12.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:12.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:12.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:12.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:12.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:12.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:12.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:13.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:13.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:13.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:13.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:13.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:13.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:13.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:13.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:13.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:13.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:13.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:13.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:13.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:13.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:13.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:13.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:13.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:13 smithi086 ceph-mon[102754]: pgmap v683: 89 pgs: 89 active+clean; 1.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 895 op/s 2024-06-14T22:59:13.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:13.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:13.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:13 smithi086 ceph-mon[97913]: pgmap v683: 89 pgs: 89 active+clean; 1.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 895 op/s 2024-06-14T22:59:13.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:13.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:13.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:13.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:13 smithi137 ceph-mon[90634]: pgmap v683: 89 pgs: 89 active+clean; 1.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 895 op/s 2024-06-14T22:59:13.767 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:59:13.767 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:59:13.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:13.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:13.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:13.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:14.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:14.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:14.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:14.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:14.254 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:14.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:14.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:14.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:14.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:14.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:14.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:14.536 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:14.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:14.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2545421293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:59:14.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2545421293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:59:14.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:14.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:14.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:14.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2545421293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T22:59:14.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:59:14.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-14T22:59:14.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:14.834 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T22:59:14.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:14.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:14.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:15.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:15.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:15.056 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:15.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:59:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:59:14] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T22:59:15.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:15.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:15.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:15.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:15.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:15.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:15.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:15.437 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:15.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:15.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:15.569 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:15.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:15.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:15.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:15.734 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:59:15.734 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2545421293' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:59:15.735 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[102754]: osdmap e173: 8 total, 8 up, 8 in 2024-06-14T22:59:15.735 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:59:15.735 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[102754]: pgmap v685: 89 pgs: 89 active+clean; 1.8 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-06-14T22:59:15.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:59:15.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2545421293' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:59:15.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[97913]: osdmap e173: 8 total, 8 up, 8 in 2024-06-14T22:59:15.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:59:15.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:15 smithi086 ceph-mon[97913]: pgmap v685: 89 pgs: 89 active+clean; 1.8 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-06-14T22:59:15.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:15 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:59:15.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2545421293' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T22:59:15.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:15 smithi137 ceph-mon[90634]: osdmap e173: 8 total, 8 up, 8 in 2024-06-14T22:59:15.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:15 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T22:59:15.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:15 smithi137 ceph-mon[90634]: pgmap v685: 89 pgs: 89 active+clean; 1.8 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-06-14T22:59:15.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:15.847 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:15.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:15.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:15.986 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:15.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:16.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:16.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:16.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:16.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:16.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:16.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:16.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:16.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:16.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:16.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:16.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:16.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:16.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:16.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:16.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:16.783 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:16.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:16.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:16.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:16.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:16.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:17.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:17.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:17.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:17.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:17.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:17.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:17.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:17.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:17.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:17.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:17.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:17.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:17.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:17.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:17.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T22:59:17.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:17.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:17.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:17.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:17.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:17.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:17.833 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:17 smithi137 ceph-mon[90634]: pgmap v686: 89 pgs: 89 active+clean; 1.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 883 op/s 2024-06-14T22:59:17.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:17.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:17.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:18.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:18.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:18.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:18.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:17 smithi086 ceph-mon[102754]: pgmap v686: 89 pgs: 89 active+clean; 1.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 883 op/s 2024-06-14T22:59:18.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:17 smithi086 ceph-mon[97913]: pgmap v686: 89 pgs: 89 active+clean; 1.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 883 op/s 2024-06-14T22:59:18.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:18.175 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:18.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:18.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:18.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:18.329 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:18.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:18.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:18.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:18.467 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:18.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:18.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:18.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:18.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:18.736 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:18.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:18.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:18.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:18.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:18.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:19.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:19.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:19.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:19.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:19.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:19.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:19.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:19.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:19.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:19.434 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:59:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:59:19.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:19.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:19.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:19.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:19.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:19.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:19.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:19.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:19.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:19.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:19.910 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:19 smithi137 ceph-mon[90634]: pgmap v687: 89 pgs: 89 active+clean; 1.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 883 op/s 2024-06-14T22:59:19.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:19.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:19.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:20.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:19 smithi086 ceph-mon[97913]: pgmap v687: 89 pgs: 89 active+clean; 1.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 883 op/s 2024-06-14T22:59:20.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:19 smithi086 ceph-mon[102754]: pgmap v687: 89 pgs: 89 active+clean; 1.7 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 883 op/s 2024-06-14T22:59:20.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:20.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:20.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:20.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:20.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:20.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:20.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:20.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:20.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:20.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:20.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:20.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:20.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:20.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:20.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:20.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:20.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:20.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:20.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:20.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:20 smithi137 ceph-mon[90634]: from='client.35805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:59:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:20.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:21.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:21.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:20 smithi086 ceph-mon[97913]: from='client.35805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:59:21.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:21.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:21.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:20 smithi086 ceph-mon[102754]: from='client.35805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:59:21.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:21.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:21.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:21.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:21.133 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":687,"stamp":"2024-06-14T22:59:18.649890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1774852062,"num_objects":27098,"num_object_clones":0,"num_object_copies":81294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27098,"num_whiteouts":0,"num_read":487,"num_read_kb":913,"num_write":238186,"num_write_kb":8481102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143251,"ondisk_log_size":143251,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9560364,"kb_used_data":5352612,"kb_used_omap":29,"kb_used_meta":4207586,"kb_avail":740400852,"statfs":{"total":767960285184,"available":758170472448,"internally_reserved":0,"allocated":5481074688,"data_stored":5474023808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":4308568741},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-579010560,"num_objects":-8835,"num_object_clones":0,"num_object_copies":-26505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8835,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003691"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":102,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.777307+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T22:59:03.777307+0000","last_peered":"2024-06-14T22:59:03.777307+0000","last_clean":"2024-06-14T22:59:03.777307+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T22:59:03.777307+0000","last_undegraded":"2024-06-14T22:59:03.777307+0000","last_fullsized":"2024-06-14T22:59:03.777307+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":135,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.767734+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T22:59:02.767734+0000","last_peered":"2024-06-14T22:59:02.767734+0000","last_clean":"2024-06-14T22:59:02.767734+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T22:59:02.767734+0000","last_undegraded":"2024-06-14T22:59:02.767734+0000","last_fullsized":"2024-06-14T22:59:02.767734+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":132,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.769121+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T22:59:02.769121+0000","last_peered":"2024-06-14T22:59:02.769121+0000","last_clean":"2024-06-14T22:59:02.769121+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T22:59:02.769121+0000","last_undegraded":"2024-06-14T22:59:02.769121+0000","last_fullsized":"2024-06-14T22:59:02.769121+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":124,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.790202+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T22:59:03.790202+0000","last_peered":"2024-06-14T22:59:03.790202+0000","last_clean":"2024-06-14T22:59:03.790202+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T22:59:03.790202+0000","last_undegraded":"2024-06-14T22:59:03.790202+0000","last_fullsized":"2024-06-14T22:59:03.790202+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":124,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.790486+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T22:59:03.790486+0000","last_peered":"2024-06-14T22:59:03.790486+0000","last_clean":"2024-06-14T22:59:03.790486+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T22:59:03.790486+0000","last_undegraded":"2024-06-14T22:59:03.790486+0000","last_fullsized":"2024-06-14T22:59:03.790486+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":39,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.771302+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T22:59:02.771302+0000","last_peered":"2024-06-14T22:59:02.771302+0000","last_clean":"2024-06-14T22:59:02.771302+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T22:59:02.771302+0000","last_undegraded":"2024-06-14T22:59:02.771302+0000","last_fullsized":"2024-06-14T22:59:02.771302+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":126,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.773804+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T22:59:03.773804+0000","last_peered":"2024-06-14T22:59:03.773804+0000","last_clean":"2024-06-14T22:59:03.773804+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T22:59:03.773804+0000","last_undegraded":"2024-06-14T22:59:03.773804+0000","last_fullsized":"2024-06-14T22:59:03.773804+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":141,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.779468+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T22:59:02.779468+0000","last_peered":"2024-06-14T22:59:02.779468+0000","last_clean":"2024-06-14T22:59:02.779468+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T22:59:02.779468+0000","last_undegraded":"2024-06-14T22:59:02.779468+0000","last_fullsized":"2024-06-14T22:59:02.779468+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"173'15362","reported_seq":15403,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-06-14T22:59:16.610460+0000","last_change":"2024-06-14T22:55:38.730745+0000","last_active":"2024-06-14T22:59:16.610460+0000","last_peered":"2024-06-14T22:59:16.610460+0000","last_clean":"2024-06-14T22:59:16.610460+0000","last_became_active":"2024-06-14T22:55:38.730320+0000","last_became_peered":"2024-06-14T22:55:38.730320+0000","last_unstale":"2024-06-14T22:59:16.610460+0000","last_undegraded":"2024-06-14T22:59:16.610460+0000","last_fullsized":"2024-06-14T22:59:16.610460+0000","mapping_epoch":140,"log_start":"149'5300","ondisk_log_start":"149'5300","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:29.533721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15362,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":122,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.768037+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T22:59:02.768037+0000","last_peered":"2024-06-14T22:59:02.768037+0000","last_clean":"2024-06-14T22:59:02.768037+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T22:59:02.768037+0000","last_undegraded":"2024-06-14T22:59:02.768037+0000","last_fullsized":"2024-06-14T22:59:02.768037+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":115,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.768730+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:59:02.768730+0000","last_peered":"2024-06-14T22:59:02.768730+0000","last_clean":"2024-06-14T22:59:02.768730+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:59:02.768730+0000","last_undegraded":"2024-06-14T22:59:02.768730+0000","last_fullsized":"2024-06-14T22:59:02.768730+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"173'14668","reported_seq":14711,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-06-14T22:59:14.827802+0000","last_change":"2024-06-14T22:55:38.730175+0000","last_active":"2024-06-14T22:59:14.827802+0000","last_peered":"2024-06-14T22:59:14.827802+0000","last_clean":"2024-06-14T22:59:14.827802+0000","last_became_active":"2024-06-14T22:55:38.729339+0000","last_became_peered":"2024-06-14T22:55:38.729339+0000","last_unstale":"2024-06-14T22:59:14.827802+0000","last_undegraded":"2024-06-14T22:59:14.827802+0000","last_fullsized":"2024-06-14T22:59:14.827802+0000","mapping_epoch":140,"log_start":"153'7300","ondisk_log_start":"153'7300","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7368,"log_dups_size":0,"ondisk_log_size":7368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:40:50.307712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109838336,"num_objects":1676,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14668,"num_write_kb":523008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":123,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.773746+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T22:59:02.773746+0000","last_peered":"2024-06-14T22:59:02.773746+0000","last_clean":"2024-06-14T22:59:02.773746+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T22:59:02.773746+0000","last_undegraded":"2024-06-14T22:59:02.773746+0000","last_fullsized":"2024-06-14T22:59:02.773746+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":149,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.771162+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T22:59:02.771162+0000","last_peered":"2024-06-14T22:59:02.771162+0000","last_clean":"2024-06-14T22:59:02.771162+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T22:59:02.771162+0000","last_undegraded":"2024-06-14T22:59:02.771162+0000","last_fullsized":"2024-06-14T22:59:02.771162+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"172'14677","reported_seq":14716,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:14.141642+0000","last_change":"2024-06-14T22:55:38.730856+0000","last_active":"2024-06-14T22:59:14.141642+0000","last_peered":"2024-06-14T22:59:14.141642+0000","last_clean":"2024-06-14T22:59:14.141642+0000","last_became_active":"2024-06-14T22:55:38.730542+0000","last_became_peered":"2024-06-14T22:55:38.730542+0000","last_unstale":"2024-06-14T22:59:14.141642+0000","last_undegraded":"2024-06-14T22:59:14.141642+0000","last_fullsized":"2024-06-14T22:59:14.141642+0000","mapping_epoch":140,"log_start":"149'4900","ondisk_log_start":"149'4900","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:16:51.828869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115933184,"num_objects":1769,"num_object_clones":0,"num_object_copies":5307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14677,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":50,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.769053+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T22:59:02.769053+0000","last_peered":"2024-06-14T22:59:02.769053+0000","last_clean":"2024-06-14T22:59:02.769053+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T22:59:02.769053+0000","last_undegraded":"2024-06-14T22:59:02.769053+0000","last_fullsized":"2024-06-14T22:59:02.769053+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":132,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.779431+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T22:59:02.779431+0000","last_peered":"2024-06-14T22:59:02.779431+0000","last_clean":"2024-06-14T22:59:02.779431+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T22:59:02.779431+0000","last_undegraded":"2024-06-14T22:59:02.779431+0000","last_fullsized":"2024-06-14T22:59:02.779431+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"172'14881","reported_seq":14920,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:14.132106+0000","last_change":"2024-06-14T22:55:39.130045+0000","last_active":"2024-06-14T22:59:14.132106+0000","last_peered":"2024-06-14T22:59:14.132106+0000","last_clean":"2024-06-14T22:59:14.132106+0000","last_became_active":"2024-06-14T22:55:39.129733+0000","last_became_peered":"2024-06-14T22:55:39.129733+0000","last_unstale":"2024-06-14T22:59:14.132106+0000","last_undegraded":"2024-06-14T22:59:14.132106+0000","last_fullsized":"2024-06-14T22:59:14.132106+0000","mapping_epoch":140,"log_start":"150'5200","ondisk_log_start":"150'5200","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9681,"log_dups_size":0,"ondisk_log_size":9681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:34:14.279180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111869952,"num_objects":1707,"num_object_clones":0,"num_object_copies":5121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14881,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.768970+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T22:59:02.768970+0000","last_peered":"2024-06-14T22:59:02.768970+0000","last_clean":"2024-06-14T22:59:02.768970+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T22:59:02.768970+0000","last_undegraded":"2024-06-14T22:59:02.768970+0000","last_fullsized":"2024-06-14T22:59:02.768970+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":125,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.791355+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T22:59:03.791355+0000","last_peered":"2024-06-14T22:59:03.791355+0000","last_clean":"2024-06-14T22:59:03.791355+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T22:59:03.791355+0000","last_undegraded":"2024-06-14T22:59:03.791355+0000","last_fullsized":"2024-06-14T22:59:03.791355+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"172'14720","reported_seq":14769,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:13.716101+0000","last_change":"2024-06-14T22:58:09.809570+0000","last_active":"2024-06-14T22:59:13.716101+0000","last_peered":"2024-06-14T22:59:13.716101+0000","last_clean":"2024-06-14T22:59:13.716101+0000","last_became_active":"2024-06-14T22:58:09.753927+0000","last_became_peered":"2024-06-14T22:58:09.753927+0000","last_unstale":"2024-06-14T22:59:13.716101+0000","last_undegraded":"2024-06-14T22:59:13.716101+0000","last_fullsized":"2024-06-14T22:59:13.716101+0000","mapping_epoch":163,"log_start":"149'4700","ondisk_log_start":"149'4700","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:46:54.799901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14720,"num_write_kb":525824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":41,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.791059+0000","last_change":"2024-06-14T22:58:09.752985+0000","last_active":"2024-06-14T22:59:03.791059+0000","last_peered":"2024-06-14T22:59:03.791059+0000","last_clean":"2024-06-14T22:59:03.791059+0000","last_became_active":"2024-06-14T22:58:09.752630+0000","last_became_peered":"2024-06-14T22:58:09.752630+0000","last_unstale":"2024-06-14T22:59:03.791059+0000","last_undegraded":"2024-06-14T22:59:03.791059+0000","last_fullsized":"2024-06-14T22:59:03.791059+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:22.320986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":132,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.775562+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T22:59:02.775562+0000","last_peered":"2024-06-14T22:59:02.775562+0000","last_clean":"2024-06-14T22:59:02.775562+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T22:59:02.775562+0000","last_undegraded":"2024-06-14T22:59:02.775562+0000","last_fullsized":"2024-06-14T22:59:02.775562+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"173'15052","reported_seq":15094,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-06-14T22:59:14.766399+0000","last_change":"2024-06-14T22:55:39.129787+0000","last_active":"2024-06-14T22:59:14.766399+0000","last_peered":"2024-06-14T22:59:14.766399+0000","last_clean":"2024-06-14T22:59:14.766399+0000","last_became_active":"2024-06-14T22:55:39.129528+0000","last_became_peered":"2024-06-14T22:55:39.129528+0000","last_unstale":"2024-06-14T22:59:14.766399+0000","last_undegraded":"2024-06-14T22:59:14.766399+0000","last_fullsized":"2024-06-14T22:59:14.766399+0000","mapping_epoch":140,"log_start":"153'7700","ondisk_log_start":"153'7700","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7352,"log_dups_size":0,"ondisk_log_size":7352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:24:05.362187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109707264,"num_objects":1674,"num_object_clones":0,"num_object_copies":5022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15052,"num_write_kb":535232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.779557+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T22:59:02.779557+0000","last_peered":"2024-06-14T22:59:02.779557+0000","last_clean":"2024-06-14T22:59:02.779557+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T22:59:02.779557+0000","last_undegraded":"2024-06-14T22:59:02.779557+0000","last_fullsized":"2024-06-14T22:59:02.779557+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":142,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.771818+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T22:59:02.771818+0000","last_peered":"2024-06-14T22:59:02.771818+0000","last_clean":"2024-06-14T22:59:02.771818+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T22:59:02.771818+0000","last_undegraded":"2024-06-14T22:59:02.771818+0000","last_fullsized":"2024-06-14T22:59:02.771818+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"172'14746","reported_seq":14811,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:13.881769+0000","last_change":"2024-06-14T22:58:32.802664+0000","last_active":"2024-06-14T22:59:13.881769+0000","last_peered":"2024-06-14T22:59:13.881769+0000","last_clean":"2024-06-14T22:59:13.881769+0000","last_became_active":"2024-06-14T22:58:32.753751+0000","last_became_peered":"2024-06-14T22:58:32.753751+0000","last_unstale":"2024-06-14T22:59:13.881769+0000","last_undegraded":"2024-06-14T22:59:13.881769+0000","last_fullsized":"2024-06-14T22:59:13.881769+0000","mapping_epoch":167,"log_start":"151'7030","ondisk_log_start":"151'7030","created":140,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:17:54.717972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113901568,"num_objects":1738,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14746,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.771237+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T22:59:02.771237+0000","last_peered":"2024-06-14T22:59:02.771237+0000","last_clean":"2024-06-14T22:59:02.771237+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T22:59:02.771237+0000","last_undegraded":"2024-06-14T22:59:02.771237+0000","last_fullsized":"2024-06-14T22:59:02.771237+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":132,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.774569+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T22:59:02.774569+0000","last_peered":"2024-06-14T22:59:02.774569+0000","last_clean":"2024-06-14T22:59:02.774569+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T22:59:02.774569+0000","last_undegraded":"2024-06-14T22:59:02.774569+0000","last_fullsized":"2024-06-14T22:59:02.774569+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"173'14986","reported_seq":15037,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-06-14T22:59:14.796485+0000","last_change":"2024-06-14T22:58:09.794739+0000","last_active":"2024-06-14T22:59:14.796485+0000","last_peered":"2024-06-14T22:59:14.796485+0000","last_clean":"2024-06-14T22:59:14.796485+0000","last_became_active":"2024-06-14T22:58:09.755744+0000","last_became_peered":"2024-06-14T22:58:09.755744+0000","last_unstale":"2024-06-14T22:59:14.796485+0000","last_undegraded":"2024-06-14T22:59:14.796485+0000","last_fullsized":"2024-06-14T22:59:14.796485+0000","mapping_epoch":163,"log_start":"153'7621","ondisk_log_start":"153'7621","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7365,"log_dups_size":0,"ondisk_log_size":7365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:51:51.355892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114950144,"num_objects":1754,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14986,"num_write_kb":535680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":127,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.780633+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T22:59:03.780633+0000","last_peered":"2024-06-14T22:59:03.780633+0000","last_clean":"2024-06-14T22:59:03.780633+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T22:59:03.780633+0000","last_undegraded":"2024-06-14T22:59:03.780633+0000","last_fullsized":"2024-06-14T22:59:03.780633+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":140,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.775316+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T22:59:03.775316+0000","last_peered":"2024-06-14T22:59:03.775316+0000","last_clean":"2024-06-14T22:59:03.775316+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T22:59:03.775316+0000","last_undegraded":"2024-06-14T22:59:03.775316+0000","last_fullsized":"2024-06-14T22:59:03.775316+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"173'14780","reported_seq":14830,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-06-14T22:59:16.556553+0000","last_change":"2024-06-14T22:58:09.784892+0000","last_active":"2024-06-14T22:59:16.556553+0000","last_peered":"2024-06-14T22:59:16.556553+0000","last_clean":"2024-06-14T22:59:16.556553+0000","last_became_active":"2024-06-14T22:58:09.752020+0000","last_became_peered":"2024-06-14T22:58:09.752020+0000","last_unstale":"2024-06-14T22:59:16.556553+0000","last_undegraded":"2024-06-14T22:59:16.556553+0000","last_fullsized":"2024-06-14T22:59:16.556553+0000","mapping_epoch":163,"log_start":"149'4700","ondisk_log_start":"149'4700","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:53.007432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100925440,"num_objects":1540,"num_object_clones":0,"num_object_copies":4620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14780,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":439,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.801517+0000","last_change":"2024-06-14T22:55:53.665022+0000","last_active":"2024-06-14T22:59:03.801517+0000","last_peered":"2024-06-14T22:59:03.801517+0000","last_clean":"2024-06-14T22:59:03.801517+0000","last_became_active":"2024-06-14T22:55:53.664025+0000","last_became_peered":"2024-06-14T22:55:53.664025+0000","last_unstale":"2024-06-14T22:59:03.801517+0000","last_undegraded":"2024-06-14T22:59:03.801517+0000","last_fullsized":"2024-06-14T22:59:03.801517+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.106147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.768951+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T22:59:02.768951+0000","last_peered":"2024-06-14T22:59:02.768951+0000","last_clean":"2024-06-14T22:59:02.768951+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T22:59:02.768951+0000","last_undegraded":"2024-06-14T22:59:02.768951+0000","last_fullsized":"2024-06-14T22:59:02.768951+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":76,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.768742+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T22:59:02.768742+0000","last_peered":"2024-06-14T22:59:02.768742+0000","last_clean":"2024-06-14T22:59:02.768742+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T22:59:02.768742+0000","last_undegraded":"2024-06-14T22:59:02.768742+0000","last_fullsized":"2024-06-14T22:59:02.768742+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":115,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.779721+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:59:02.779721+0000","last_peered":"2024-06-14T22:59:02.779721+0000","last_clean":"2024-06-14T22:59:02.779721+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:59:02.779721+0000","last_undegraded":"2024-06-14T22:59:02.779721+0000","last_fullsized":"2024-06-14T22:59:02.779721+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"173'15108","reported_seq":15151,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-06-14T22:59:16.599530+0000","last_change":"2024-06-14T22:55:38.974930+0000","last_active":"2024-06-14T22:59:16.599530+0000","last_peered":"2024-06-14T22:59:16.599530+0000","last_clean":"2024-06-14T22:59:16.599530+0000","last_became_active":"2024-06-14T22:55:38.974625+0000","last_became_peered":"2024-06-14T22:55:38.974625+0000","last_unstale":"2024-06-14T22:59:16.599530+0000","last_undegraded":"2024-06-14T22:59:16.599530+0000","last_fullsized":"2024-06-14T22:59:16.599530+0000","mapping_epoch":140,"log_start":"149'5100","ondisk_log_start":"149'5100","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:44:40.336597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105775104,"num_objects":1614,"num_object_clones":0,"num_object_copies":4842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15108,"num_write_kb":535104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":130,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.777838+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T22:59:03.777838+0000","last_peered":"2024-06-14T22:59:03.777838+0000","last_clean":"2024-06-14T22:59:03.777838+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T22:59:03.777838+0000","last_undegraded":"2024-06-14T22:59:03.777838+0000","last_fullsized":"2024-06-14T22:59:03.777838+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":124,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.794870+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T22:59:03.794870+0000","last_peered":"2024-06-14T22:59:03.794870+0000","last_clean":"2024-06-14T22:59:03.794870+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T22:59:03.794870+0000","last_undegraded":"2024-06-14T22:59:03.794870+0000","last_fullsized":"2024-06-14T22:59:03.794870+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":138,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.790600+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T22:59:03.790600+0000","last_peered":"2024-06-14T22:59:03.790600+0000","last_clean":"2024-06-14T22:59:03.790600+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T22:59:03.790600+0000","last_undegraded":"2024-06-14T22:59:03.790600+0000","last_fullsized":"2024-06-14T22:59:03.790600+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"172'14889","reported_seq":14929,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:14.123421+0000","last_change":"2024-06-14T22:55:38.731601+0000","last_active":"2024-06-14T22:59:14.123421+0000","last_peered":"2024-06-14T22:59:14.123421+0000","last_clean":"2024-06-14T22:59:14.123421+0000","last_became_active":"2024-06-14T22:55:38.731200+0000","last_became_peered":"2024-06-14T22:55:38.731200+0000","last_unstale":"2024-06-14T22:59:14.123421+0000","last_undegraded":"2024-06-14T22:59:14.123421+0000","last_fullsized":"2024-06-14T22:59:14.123421+0000","mapping_epoch":140,"log_start":"149'5200","ondisk_log_start":"149'5200","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:53:02.395452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115408896,"num_objects":1761,"num_object_clones":0,"num_object_copies":5283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14889,"num_write_kb":532800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":125,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.792605+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T22:59:03.792605+0000","last_peered":"2024-06-14T22:59:03.792605+0000","last_clean":"2024-06-14T22:59:03.792605+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T22:59:03.792605+0000","last_undegraded":"2024-06-14T22:59:03.792605+0000","last_fullsized":"2024-06-14T22:59:03.792605+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":140,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.778017+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T22:59:03.778017+0000","last_peered":"2024-06-14T22:59:03.778017+0000","last_clean":"2024-06-14T22:59:03.778017+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T22:59:03.778017+0000","last_undegraded":"2024-06-14T22:59:03.778017+0000","last_fullsized":"2024-06-14T22:59:03.778017+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":125,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.775077+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T22:59:03.775077+0000","last_peered":"2024-06-14T22:59:03.775077+0000","last_clean":"2024-06-14T22:59:03.775077+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T22:59:03.775077+0000","last_undegraded":"2024-06-14T22:59:03.775077+0000","last_fullsized":"2024-06-14T22:59:03.775077+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"173'14838","reported_seq":14881,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-06-14T22:59:16.609508+0000","last_change":"2024-06-14T22:55:38.729565+0000","last_active":"2024-06-14T22:59:16.609508+0000","last_peered":"2024-06-14T22:59:16.609508+0000","last_clean":"2024-06-14T22:59:16.609508+0000","last_became_active":"2024-06-14T22:55:38.729284+0000","last_became_peered":"2024-06-14T22:55:38.729284+0000","last_unstale":"2024-06-14T22:59:16.609508+0000","last_undegraded":"2024-06-14T22:59:16.609508+0000","last_fullsized":"2024-06-14T22:59:16.609508+0000","mapping_epoch":140,"log_start":"149'4800","ondisk_log_start":"149'4800","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:15.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":4878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14838,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.768755+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T22:59:02.768755+0000","last_peered":"2024-06-14T22:59:02.768755+0000","last_clean":"2024-06-14T22:59:02.768755+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T22:59:02.768755+0000","last_undegraded":"2024-06-14T22:59:02.768755+0000","last_fullsized":"2024-06-14T22:59:02.768755+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.771555+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T22:59:02.771555+0000","last_peered":"2024-06-14T22:59:02.771555+0000","last_clean":"2024-06-14T22:59:02.771555+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T22:59:02.771555+0000","last_undegraded":"2024-06-14T22:59:02.771555+0000","last_fullsized":"2024-06-14T22:59:02.771555+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":155,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.777075+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T22:59:03.777075+0000","last_peered":"2024-06-14T22:59:03.777075+0000","last_clean":"2024-06-14T22:59:03.777075+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T22:59:03.777075+0000","last_undegraded":"2024-06-14T22:59:03.777075+0000","last_fullsized":"2024-06-14T22:59:03.777075+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"172'14811","reported_seq":14851,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:14.051919+0000","last_change":"2024-06-14T22:55:38.729989+0000","last_active":"2024-06-14T22:59:14.051919+0000","last_peered":"2024-06-14T22:59:14.051919+0000","last_clean":"2024-06-14T22:59:14.051919+0000","last_became_active":"2024-06-14T22:55:38.729695+0000","last_became_peered":"2024-06-14T22:55:38.729695+0000","last_unstale":"2024-06-14T22:59:14.051919+0000","last_undegraded":"2024-06-14T22:59:14.051919+0000","last_fullsized":"2024-06-14T22:59:14.051919+0000","mapping_epoch":140,"log_start":"149'5100","ondisk_log_start":"149'5100","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9711,"log_dups_size":0,"ondisk_log_size":9711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:44:38.218727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112132096,"num_objects":1711,"num_object_clones":0,"num_object_copies":5133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14811,"num_write_kb":528704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":150,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.769195+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T22:59:02.769195+0000","last_peered":"2024-06-14T22:59:02.769195+0000","last_clean":"2024-06-14T22:59:02.769195+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T22:59:02.769195+0000","last_undegraded":"2024-06-14T22:59:02.769195+0000","last_fullsized":"2024-06-14T22:59:02.769195+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":130,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.777245+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T22:59:03.777245+0000","last_peered":"2024-06-14T22:59:03.777245+0000","last_clean":"2024-06-14T22:59:03.777245+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T22:59:03.777245+0000","last_undegraded":"2024-06-14T22:59:03.777245+0000","last_fullsized":"2024-06-14T22:59:03.777245+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":125,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.776660+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T22:59:03.776660+0000","last_peered":"2024-06-14T22:59:03.776660+0000","last_clean":"2024-06-14T22:59:03.776660+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T22:59:03.776660+0000","last_undegraded":"2024-06-14T22:59:03.776660+0000","last_fullsized":"2024-06-14T22:59:03.776660+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"172'14822","reported_seq":14871,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:13.678967+0000","last_change":"2024-06-14T22:55:53.672630+0000","last_active":"2024-06-14T22:59:13.678967+0000","last_peered":"2024-06-14T22:59:13.678967+0000","last_clean":"2024-06-14T22:59:13.678967+0000","last_became_active":"2024-06-14T22:55:53.663693+0000","last_became_peered":"2024-06-14T22:55:53.663693+0000","last_unstale":"2024-06-14T22:59:13.678967+0000","last_undegraded":"2024-06-14T22:59:13.678967+0000","last_fullsized":"2024-06-14T22:59:13.678967+0000","mapping_epoch":145,"log_start":"151'6400","ondisk_log_start":"151'6400","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":8422,"log_dups_size":0,"ondisk_log_size":8422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:20:34.289185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112852992,"num_objects":1722,"num_object_clones":0,"num_object_copies":5166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14822,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":140,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.777144+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T22:59:03.777144+0000","last_peered":"2024-06-14T22:59:03.777144+0000","last_clean":"2024-06-14T22:59:03.777144+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T22:59:03.777144+0000","last_undegraded":"2024-06-14T22:59:03.777144+0000","last_fullsized":"2024-06-14T22:59:03.777144+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.768121+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T22:59:02.768121+0000","last_peered":"2024-06-14T22:59:02.768121+0000","last_clean":"2024-06-14T22:59:02.768121+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T22:59:02.768121+0000","last_undegraded":"2024-06-14T22:59:02.768121+0000","last_fullsized":"2024-06-14T22:59:02.768121+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":110,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.791206+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T22:59:03.791206+0000","last_peered":"2024-06-14T22:59:03.791206+0000","last_clean":"2024-06-14T22:59:03.791206+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T22:59:03.791206+0000","last_undegraded":"2024-06-14T22:59:03.791206+0000","last_fullsized":"2024-06-14T22:59:03.791206+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"173'14873","reported_seq":14924,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-06-14T22:59:14.804050+0000","last_change":"2024-06-14T22:55:53.668144+0000","last_active":"2024-06-14T22:59:14.804050+0000","last_peered":"2024-06-14T22:59:14.804050+0000","last_clean":"2024-06-14T22:59:14.804050+0000","last_became_active":"2024-06-14T22:55:53.663690+0000","last_became_peered":"2024-06-14T22:55:53.663690+0000","last_unstale":"2024-06-14T22:59:14.804050+0000","last_undegraded":"2024-06-14T22:59:14.804050+0000","last_fullsized":"2024-06-14T22:59:14.804050+0000","mapping_epoch":145,"log_start":"153'7500","ondisk_log_start":"153'7500","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7373,"log_dups_size":0,"ondisk_log_size":7373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:52:03.686394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111542296,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14873,"num_write_kb":530369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":32,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.790823+0000","last_change":"2024-06-14T22:58:09.752729+0000","last_active":"2024-06-14T22:59:03.790823+0000","last_peered":"2024-06-14T22:59:03.790823+0000","last_clean":"2024-06-14T22:59:03.790823+0000","last_became_active":"2024-06-14T22:58:09.752393+0000","last_became_peered":"2024-06-14T22:58:09.752393+0000","last_unstale":"2024-06-14T22:59:03.790823+0000","last_undegraded":"2024-06-14T22:59:03.790823+0000","last_fullsized":"2024-06-14T22:59:03.790823+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:21:58.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":124,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.776451+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T22:59:03.776451+0000","last_peered":"2024-06-14T22:59:03.776451+0000","last_clean":"2024-06-14T22:59:03.776451+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T22:59:03.776451+0000","last_undegraded":"2024-06-14T22:59:03.776451+0000","last_fullsized":"2024-06-14T22:59:03.776451+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":117,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.767470+0000","last_change":"2024-06-14T22:58:32.754079+0000","last_active":"2024-06-14T22:59:02.767470+0000","last_peered":"2024-06-14T22:59:02.767470+0000","last_clean":"2024-06-14T22:59:02.767470+0000","last_became_active":"2024-06-14T22:58:32.753814+0000","last_became_peered":"2024-06-14T22:58:32.753814+0000","last_unstale":"2024-06-14T22:59:02.767470+0000","last_undegraded":"2024-06-14T22:59:02.767470+0000","last_fullsized":"2024-06-14T22:59:02.767470+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:37:07.414370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"172'14673","reported_seq":14723,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:13.640625+0000","last_change":"2024-06-14T22:58:09.804712+0000","last_active":"2024-06-14T22:59:13.640625+0000","last_peered":"2024-06-14T22:59:13.640625+0000","last_clean":"2024-06-14T22:59:13.640625+0000","last_became_active":"2024-06-14T22:58:09.757262+0000","last_became_peered":"2024-06-14T22:58:09.757262+0000","last_unstale":"2024-06-14T22:59:13.640625+0000","last_undegraded":"2024-06-14T22:59:13.640625+0000","last_fullsized":"2024-06-14T22:59:13.640625+0000","mapping_epoch":163,"log_start":"151'6300","ondisk_log_start":"151'6300","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":8373,"log_dups_size":0,"ondisk_log_size":8373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:28.952637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113967104,"num_objects":1739,"num_object_clones":0,"num_object_copies":5217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14673,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":128,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.790422+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T22:59:03.790422+0000","last_peered":"2024-06-14T22:59:03.790422+0000","last_clean":"2024-06-14T22:59:03.790422+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T22:59:03.790422+0000","last_undegraded":"2024-06-14T22:59:03.790422+0000","last_fullsized":"2024-06-14T22:59:03.790422+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":137,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.779747+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T22:59:02.779747+0000","last_peered":"2024-06-14T22:59:02.779747+0000","last_clean":"2024-06-14T22:59:02.779747+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T22:59:02.779747+0000","last_undegraded":"2024-06-14T22:59:02.779747+0000","last_fullsized":"2024-06-14T22:59:02.779747+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":115,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.771038+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:59:02.771038+0000","last_peered":"2024-06-14T22:59:02.771038+0000","last_clean":"2024-06-14T22:59:02.771038+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:59:02.771038+0000","last_undegraded":"2024-06-14T22:59:02.771038+0000","last_fullsized":"2024-06-14T22:59:02.771038+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":132,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.771491+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T22:59:02.771491+0000","last_peered":"2024-06-14T22:59:02.771491+0000","last_clean":"2024-06-14T22:59:02.771491+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T22:59:02.771491+0000","last_undegraded":"2024-06-14T22:59:02.771491+0000","last_fullsized":"2024-06-14T22:59:02.771491+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":138,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.775627+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:59:02.775627+0000","last_peered":"2024-06-14T22:59:02.775627+0000","last_clean":"2024-06-14T22:59:02.775627+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:59:02.775627+0000","last_undegraded":"2024-06-14T22:59:02.775627+0000","last_fullsized":"2024-06-14T22:59:02.775627+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":140,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.776117+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T22:59:03.776117+0000","last_peered":"2024-06-14T22:59:03.776117+0000","last_clean":"2024-06-14T22:59:03.776117+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T22:59:03.776117+0000","last_undegraded":"2024-06-14T22:59:03.776117+0000","last_fullsized":"2024-06-14T22:59:03.776117+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":51,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.773045+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T22:59:03.773045+0000","last_peered":"2024-06-14T22:59:03.773045+0000","last_clean":"2024-06-14T22:59:03.773045+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T22:59:03.773045+0000","last_undegraded":"2024-06-14T22:59:03.773045+0000","last_fullsized":"2024-06-14T22:59:03.773045+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":133,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.768973+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T22:59:02.768973+0000","last_peered":"2024-06-14T22:59:02.768973+0000","last_clean":"2024-06-14T22:59:02.768973+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T22:59:02.768973+0000","last_undegraded":"2024-06-14T22:59:02.768973+0000","last_fullsized":"2024-06-14T22:59:02.768973+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":76,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.779401+0000","last_change":"2024-06-14T22:58:09.753103+0000","last_active":"2024-06-14T22:59:02.779401+0000","last_peered":"2024-06-14T22:59:02.779401+0000","last_clean":"2024-06-14T22:59:02.779401+0000","last_became_active":"2024-06-14T22:58:09.752761+0000","last_became_peered":"2024-06-14T22:58:09.752761+0000","last_unstale":"2024-06-14T22:59:02.779401+0000","last_undegraded":"2024-06-14T22:59:02.779401+0000","last_fullsized":"2024-06-14T22:59:02.779401+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:20:20.619331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":123,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.790755+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T22:59:03.790755+0000","last_peered":"2024-06-14T22:59:03.790755+0000","last_clean":"2024-06-14T22:59:03.790755+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T22:59:03.790755+0000","last_undegraded":"2024-06-14T22:59:03.790755+0000","last_fullsized":"2024-06-14T22:59:03.790755+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":126,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.774961+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T22:59:03.774961+0000","last_peered":"2024-06-14T22:59:03.774961+0000","last_clean":"2024-06-14T22:59:03.774961+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T22:59:03.774961+0000","last_undegraded":"2024-06-14T22:59:03.774961+0000","last_fullsized":"2024-06-14T22:59:03.774961+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":132,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.779712+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T22:59:02.779712+0000","last_peered":"2024-06-14T22:59:02.779712+0000","last_clean":"2024-06-14T22:59:02.779712+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T22:59:02.779712+0000","last_undegraded":"2024-06-14T22:59:02.779712+0000","last_fullsized":"2024-06-14T22:59:02.779712+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":120,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.801158+0000","last_change":"2024-06-14T22:58:09.785074+0000","last_active":"2024-06-14T22:59:03.801158+0000","last_peered":"2024-06-14T22:59:03.801158+0000","last_clean":"2024-06-14T22:59:03.801158+0000","last_became_active":"2024-06-14T22:58:09.784309+0000","last_became_peered":"2024-06-14T22:58:09.784309+0000","last_unstale":"2024-06-14T22:59:03.801158+0000","last_undegraded":"2024-06-14T22:59:03.801158+0000","last_fullsized":"2024-06-14T22:59:03.801158+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:23.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":126,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.790543+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T22:59:03.790543+0000","last_peered":"2024-06-14T22:59:03.790543+0000","last_clean":"2024-06-14T22:59:03.790543+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T22:59:03.790543+0000","last_undegraded":"2024-06-14T22:59:03.790543+0000","last_fullsized":"2024-06-14T22:59:03.790543+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":140,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:04.818913+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T22:59:04.818913+0000","last_peered":"2024-06-14T22:59:04.818913+0000","last_clean":"2024-06-14T22:59:04.818913+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T22:59:04.818913+0000","last_undegraded":"2024-06-14T22:59:04.818913+0000","last_fullsized":"2024-06-14T22:59:04.818913+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":128,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.793031+0000","last_change":"2024-06-14T22:58:09.756040+0000","last_active":"2024-06-14T22:59:03.793031+0000","last_peered":"2024-06-14T22:59:03.793031+0000","last_clean":"2024-06-14T22:59:03.793031+0000","last_became_active":"2024-06-14T22:58:09.755659+0000","last_became_peered":"2024-06-14T22:58:09.755659+0000","last_unstale":"2024-06-14T22:59:03.793031+0000","last_undegraded":"2024-06-14T22:59:03.793031+0000","last_fullsized":"2024-06-14T22:59:03.793031+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:40.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":151,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.772040+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T22:59:02.772040+0000","last_peered":"2024-06-14T22:59:02.772040+0000","last_clean":"2024-06-14T22:59:02.772040+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T22:59:02.772040+0000","last_undegraded":"2024-06-14T22:59:02.772040+0000","last_fullsized":"2024-06-14T22:59:02.772040+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":145,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.792892+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T22:59:03.792892+0000","last_peered":"2024-06-14T22:59:03.792892+0000","last_clean":"2024-06-14T22:59:03.792892+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T22:59:03.792892+0000","last_undegraded":"2024-06-14T22:59:03.792892+0000","last_fullsized":"2024-06-14T22:59:03.792892+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":112,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.791136+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T22:59:03.791136+0000","last_peered":"2024-06-14T22:59:03.791136+0000","last_clean":"2024-06-14T22:59:03.791136+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T22:59:03.791136+0000","last_undegraded":"2024-06-14T22:59:03.791136+0000","last_fullsized":"2024-06-14T22:59:03.791136+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":149,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.771431+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T22:59:02.771431+0000","last_peered":"2024-06-14T22:59:02.771431+0000","last_clean":"2024-06-14T22:59:02.771431+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T22:59:02.771431+0000","last_undegraded":"2024-06-14T22:59:02.771431+0000","last_fullsized":"2024-06-14T22:59:02.771431+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":135,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.768888+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T22:59:02.768888+0000","last_peered":"2024-06-14T22:59:02.768888+0000","last_clean":"2024-06-14T22:59:02.768888+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T22:59:02.768888+0000","last_undegraded":"2024-06-14T22:59:02.768888+0000","last_fullsized":"2024-06-14T22:59:02.768888+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":149,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.771956+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T22:59:02.771956+0000","last_peered":"2024-06-14T22:59:02.771956+0000","last_clean":"2024-06-14T22:59:02.771956+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T22:59:02.771956+0000","last_undegraded":"2024-06-14T22:59:02.771956+0000","last_fullsized":"2024-06-14T22:59:02.771956+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":118,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.769258+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:59:02.769258+0000","last_peered":"2024-06-14T22:59:02.769258+0000","last_clean":"2024-06-14T22:59:02.769258+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:59:02.769258+0000","last_undegraded":"2024-06-14T22:59:02.769258+0000","last_fullsized":"2024-06-14T22:59:02.769258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":135,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.792543+0000","last_change":"2024-06-14T22:58:09.752281+0000","last_active":"2024-06-14T22:59:03.792543+0000","last_peered":"2024-06-14T22:59:03.792543+0000","last_clean":"2024-06-14T22:59:03.792543+0000","last_became_active":"2024-06-14T22:58:09.751952+0000","last_became_peered":"2024-06-14T22:58:09.751952+0000","last_unstale":"2024-06-14T22:59:03.792543+0000","last_undegraded":"2024-06-14T22:59:03.792543+0000","last_fullsized":"2024-06-14T22:59:03.792543+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:39:19.836734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":31,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.779806+0000","last_change":"2024-06-14T22:58:09.755622+0000","last_active":"2024-06-14T22:59:02.779806+0000","last_peered":"2024-06-14T22:59:02.779806+0000","last_clean":"2024-06-14T22:59:02.779806+0000","last_became_active":"2024-06-14T22:58:09.755383+0000","last_became_peered":"2024-06-14T22:58:09.755383+0000","last_unstale":"2024-06-14T22:59:02.779806+0000","last_undegraded":"2024-06-14T22:59:02.779806+0000","last_fullsized":"2024-06-14T22:59:02.779806+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:13.884346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":132,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-06-14T22:59:02.770938+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T22:59:02.770938+0000","last_peered":"2024-06-14T22:59:02.770938+0000","last_clean":"2024-06-14T22:59:02.770938+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T22:59:02.770938+0000","last_undegraded":"2024-06-14T22:59:02.770938+0000","last_fullsized":"2024-06-14T22:59:02.770938+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":128,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-06-14T22:59:03.790690+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T22:59:03.790690+0000","last_peered":"2024-06-14T22:59:03.790690+0000","last_clean":"2024-06-14T22:59:03.790690+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T22:59:03.790690+0000","last_undegraded":"2024-06-14T22:59:03.790690+0000","last_fullsized":"2024-06-14T22:59:03.790690+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1774256152,"num_objects":27074,"num_object_clones":0,"num_object_copies":81222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27074,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237886,"num_write_kb":8478657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5374095360,"data_stored":5374083144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143035,"ondisk_log_size":143035,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234302,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1943032,"kb_used_data":1031792,"kb_used_omap":1,"kb_used_meta":911230,"kb_avail":91802120,"statfs":{"total":95995035648,"available":94005370880,"internally_reserved":0,"allocated":1056555008,"data_stored":1055654279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":933099864},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.897,"5min":1.0940000000000001,"15min":1.1479999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":6.9539999999999997,"5min":6.9539999999999997,"15min":9.3919999999999995},"last":0.221},{"interface":"front","average":{"1min":1.9319999999999999,"5min":1.0580000000000001,"15min":1.286},"min":{"1min":0.313,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.2130000000000001,"5min":5.2130000000000001,"15min":10.416},"last":0.878}]},{"osd":1,"last update":"Fri Jun 14 22:58:41 2024","interfaces":[{"interface":"back","average":{"1min":2.02,"5min":2.5910000000000002,"15min":1.8100000000000001},"min":{"1min":0.35799999999999998,"5min":0.28899999999999998,"15min":0.28100000000000003},"max":{"1min":8.5340000000000007,"5min":83.295000000000002,"15min":83.295000000000002},"last":0.442},{"interface":"front","average":{"1min":2.0840000000000001,"5min":2.5739999999999998,"15min":1.8799999999999999},"min":{"1min":0.42899999999999999,"5min":0.25800000000000001,"15min":0.223},"max":{"1min":6.8620000000000001,"5min":82.972999999999999,"15min":82.972999999999999},"last":0.25900000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:58:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":0.94099999999999995,"15min":0.70099999999999996},"min":{"1min":0.27200000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.9269999999999996,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.909,"5min":1.143,"15min":0.76200000000000001},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.9779999999999998,"5min":8.1150000000000002,"15min":8.1150000000000002},"last":2.306}]},{"osd":3,"last update":"Fri Jun 14 22:59:09 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":1.8740000000000001,"15min":0.98199999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.4930000000000003,"5min":55.683999999999997,"15min":55.683999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.907,"5min":1.8919999999999999,"15min":1.03},"min":{"1min":0.32100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.7629999999999999,"5min":55.533999999999999,"15min":55.533999999999999},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Jun 14 22:58:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":0.79400000000000004,"15min":0.67900000000000005},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.22900000000000001},"max":{"1min":5.141,"5min":6.7610000000000001,"15min":6.7610000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":0.96099999999999997,"15min":0.71699999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":4.9729999999999999,"5min":16.879000000000001,"15min":16.879000000000001},"last":0.34399999999999997}]},{"osd":5,"last update":"Fri Jun 14 22:58:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0190000000000001,"5min":1.0820000000000001,"15min":0.78400000000000003},"min":{"1min":0.29899999999999999,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":6.9000000000000004,"5min":6.9000000000000004,"15min":6.9000000000000004},"last":2.355},{"interface":"front","average":{"1min":1.397,"5min":0.998,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":5.2649999999999997,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":0.64400000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:58:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":0.93300000000000005,"15min":0.80000000000000004},"min":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.28100000000000003},"max":{"1min":5.5,"5min":5.5,"15min":5.806},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.89800000000000002,"15min":0.75700000000000001},"min":{"1min":0.374,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.8099999999999996,"5min":12.951000000000001,"15min":12.951000000000001},"last":0.51600000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495938,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":936612,"kb_used_data":342684,"kb_used_omap":1,"kb_used_meta":593918,"kb_avail":92808540,"statfs":{"total":95995035648,"available":95035944960,"internally_reserved":0,"allocated":350908416,"data_stored":350027674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608172376},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.0620000000000001,"15min":1.284},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":5.6859999999999999,"5min":9.9350000000000005,"15min":31.358000000000001},"last":1.9099999999999999},{"interface":"front","average":{"1min":1.222,"5min":0.90800000000000003,"15min":1.3420000000000001},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.9989999999999997,"5min":6.0730000000000004,"15min":29.399000000000001},"last":2.1280000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:58:37 2024","interfaces":[{"interface":"back","average":{"1min":1.9470000000000001,"5min":1.093,"15min":1.2709999999999999},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":9.9559999999999995},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.857,"5min":1.04,"15min":1.2450000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.247},"max":{"1min":6.7519999999999998,"5min":6.7519999999999998,"15min":9.5510000000000002},"last":2.1120000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:58:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":0.89000000000000001,"15min":0.67900000000000005},"min":{"1min":0.29799999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":2.544},{"interface":"front","average":{"1min":1.431,"5min":0.91200000000000003,"15min":0.70799999999999996},"min":{"1min":0.33700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.6959999999999997,"5min":6.6959999999999997,"15min":6.6959999999999997},"last":1.8109999999999999}]},{"osd":3,"last update":"Fri Jun 14 22:59:12 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.575,"15min":0.85999999999999999},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.5449999999999999,"5min":29.152000000000001,"15min":29.152000000000001},"last":1.669},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.579,"15min":0.90900000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":6.3040000000000003,"5min":25.864000000000001,"15min":25.864000000000001},"last":1.776}]},{"osd":4,"last update":"Fri Jun 14 22:58:20 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":0.66700000000000004,"15min":0.80600000000000005},"min":{"1min":0.24299999999999999,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":6.774,"5min":6.774,"15min":47.698},"last":1.748},{"interface":"front","average":{"1min":1.0800000000000001,"5min":0.63400000000000001,"15min":0.77000000000000002},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.23899999999999999},"max":{"1min":9.9610000000000003,"5min":9.9610000000000003,"15min":48.197000000000003},"last":2.3530000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:58:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":0.73799999999999999,"15min":0.72299999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.245},"max":{"1min":5.8289999999999997,"5min":5.8289999999999997,"15min":8.6500000000000004},"last":1.655},{"interface":"front","average":{"1min":1.3660000000000001,"5min":0.76500000000000001,"15min":0.77400000000000002},"min":{"1min":0.41699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":6.7949999999999999,"5min":6.7949999999999999,"15min":8.6349999999999998},"last":4.6139999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:58:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":0.89100000000000001,"15min":0.79600000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":5.6790000000000003,"5min":5.9039999999999999,"15min":8.6950000000000003},"last":1.948},{"interface":"front","average":{"1min":1.478,"5min":0.88300000000000001,"15min":0.82499999999999996},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.28899999999999998},"max":{"1min":6.8620000000000001,"5min":6.8620000000000001,"15min":8.673},"last":2.004}]}]},{"osd":5,"up_from":38,"seq":163208757577,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1911068,"kb_used_data":1011732,"kb_used_omap":1,"kb_used_meta":899326,"kb_avail":91834084,"statfs":{"total":95995035648,"available":94038102016,"internally_reserved":0,"allocated":1036013568,"data_stored":1035136312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920910168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":0.88700000000000001,"15min":1.2050000000000001},"min":{"1min":0.27400000000000002,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":3.5910000000000002,"5min":4.3470000000000004,"15min":17.766999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.86299999999999999,"15min":1.214},"min":{"1min":0.29599999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.9430000000000001,"5min":4.3659999999999997,"15min":17.806000000000001},"last":0.372}]},{"osd":1,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":0.93999999999999995,"15min":1.264},"min":{"1min":0.33000000000000002,"5min":0.24199999999999999,"15min":0.23300000000000001},"max":{"1min":5.6429999999999998,"5min":5.6429999999999998,"15min":7.694},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.0880000000000001,"15min":1.181},"min":{"1min":0.26400000000000001,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":4.4939999999999998,"5min":4.4939999999999998,"15min":5.3259999999999996},"last":1.103}]},{"osd":2,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":0.95299999999999996,"15min":0.68200000000000005},"min":{"1min":0.39200000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.5579999999999998,"5min":5.9379999999999997,"15min":5.9379999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.4379999999999999,"5min":0.90800000000000003,"15min":0.68200000000000005},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.6050000000000004,"5min":4.6050000000000004,"15min":4.6050000000000004},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":1.7709999999999999,"15min":0.94299999999999995},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":6.6029999999999998,"5min":45.744,"15min":45.744},"last":0.40100000000000002},{"interface":"front","average":{"1min":2.0129999999999999,"5min":1.9450000000000001,"15min":1.0740000000000001},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.5679999999999996,"5min":45.838000000000001,"15min":45.838000000000001},"last":0.46500000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:59:07 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.72399999999999998,"15min":0.60199999999999998},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.214},"max":{"1min":5.5880000000000001,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.307},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.70499999999999996,"15min":0.64200000000000002},"min":{"1min":0.308,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.6800000000000002,"5min":3.6800000000000002,"15min":5.8109999999999999},"last":0.42399999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:58:33 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":0.73499999999999999,"15min":0.746},"min":{"1min":0.35999999999999999,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":6.931,"5min":6.931,"15min":9.6020000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.069,"5min":0.68799999999999994,"15min":0.72999999999999998},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":4.452,"5min":4.452,"15min":12.339},"last":0.438}]},{"osd":7,"last update":"Fri Jun 14 22:58:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":0.95599999999999996,"15min":0.77700000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.3129999999999997,"5min":5.3129999999999997,"15min":5.3129999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.76,"5min":0.98599999999999999,"15min":0.76700000000000002},"min":{"1min":0.40400000000000003,"5min":0.29999999999999999,"15min":0.25600000000000001},"max":{"1min":5.6189999999999998,"5min":5.6189999999999998,"15min":10.638999999999999},"last":0.41199999999999998}]}]},{"osd":4,"up_from":30,"seq":128849019213,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":935080,"kb_used_data":345248,"kb_used_omap":1,"kb_used_meta":589822,"kb_avail":92810072,"statfs":{"total":95995035648,"available":95037513728,"internally_reserved":0,"allocated":353533952,"data_stored":352648704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":603978072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":0.70399999999999996,"15min":1.018},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.5389999999999997,"5min":4.5389999999999997,"15min":7.5650000000000004},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.66100000000000003,"15min":1.165},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":2.552,"5min":3.718,"15min":9.1470000000000002},"last":0.35799999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":0.94299999999999995,"15min":1.1279999999999999},"min":{"1min":0.26300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":8.4960000000000004,"5min":8.4960000000000004,"15min":8.4960000000000004},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.0429999999999999,"15min":1.2749999999999999},"min":{"1min":0.30099999999999999,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":5.4160000000000004,"5min":6.5549999999999997,"15min":6.5549999999999997},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":0.754,"15min":0.628},"min":{"1min":0.29399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":8.9250000000000007,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":0.432},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.78600000000000003,"15min":0.64500000000000002},"min":{"1min":0.32500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.456,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":0.45100000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:58:16 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.0049999999999999,"15min":0.69799999999999995},"min":{"1min":0.35599999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.617,"5min":6.617,"15min":6.617},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.4430000000000001,"5min":0.97699999999999998,"15min":0.72099999999999997},"min":{"1min":0.36499999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.6989999999999998,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":0.41299999999999998}]},{"osd":5,"last update":"Fri Jun 14 22:59:02 2024","interfaces":[{"interface":"back","average":{"1min":8.4789999999999992,"5min":2.3170000000000002,"15min":1.2090000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.21199999999999999},"max":{"1min":139.13200000000001,"5min":139.13200000000001,"15min":139.13200000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":9.1820000000000004,"5min":2.4780000000000002,"15min":1.2529999999999999},"min":{"1min":0.28000000000000003,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":138.86000000000001,"5min":138.86000000000001,"15min":138.86000000000001},"last":1.7789999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:58:25 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.63700000000000001,"15min":0.63400000000000001},"min":{"1min":0.30599999999999999,"5min":0.30199999999999999,"15min":0.254},"max":{"1min":5.1210000000000004,"5min":5.1210000000000004,"15min":5.1210000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.3089999999999999,"5min":0.68999999999999995,"15min":0.623},"min":{"1min":0.33000000000000002,"5min":0.28899999999999998,"15min":0.23400000000000001},"max":{"1min":8.8819999999999997,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.48799999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:58:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":0.85999999999999999,"15min":0.69299999999999995},"min":{"1min":0.377,"5min":0.30299999999999999,"15min":0.30099999999999999},"max":{"1min":8.9489999999999998,"5min":8.9489999999999998,"15min":8.9489999999999998},"last":3.4529999999999998},{"interface":"front","average":{"1min":1.4790000000000001,"5min":0.94199999999999995,"15min":0.76400000000000001},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.29399999999999998},"max":{"1min":6.4939999999999998,"5min":6.4939999999999998,"15min":8.4730000000000008},"last":3.4399999999999999}]}]},{"osd":3,"up_from":138,"seq":592705486912,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1095576,"kb_used_data":899280,"kb_used_omap":4,"kb_used_meta":196283,"kb_avail":92649576,"statfs":{"total":95995035648,"available":94873165824,"internally_reserved":0,"allocated":920862720,"data_stored":919989309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":200994270},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.022,"15min":0.55600000000000005},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.3360000000000003,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":6.3360000000000003},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.80000000000000004,"15min":0.505},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.5950000000000002,"5min":3.7229999999999999,"15min":3.7229999999999999},"last":3.5950000000000002}]},{"osd":1,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":2.5499999999999998,"5min":1.3480000000000001,"15min":0.68500000000000005},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":24.097999999999999,"5min":24.097999999999999,"15min":24.097999999999999},"last":3.508},{"interface":"front","average":{"1min":2.4500000000000002,"5min":1.2889999999999999,"15min":0.69199999999999995},"min":{"1min":0.313,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":24.132999999999999,"5min":24.132999999999999,"15min":24.132999999999999},"last":3.6789999999999998}]},{"osd":2,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":0.93300000000000005,"15min":0.58499999999999996},"min":{"1min":0.251,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.9900000000000002,"5min":5.2910000000000004,"15min":5.2910000000000004},"last":3.556},{"interface":"front","average":{"1min":1.222,"5min":0.88900000000000001,"15min":0.53400000000000003},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.9900000000000002,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.247}]},{"osd":4,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.0169999999999999,"15min":0.75900000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.1189999999999998,"5min":5.7960000000000003,"15min":5.7960000000000003},"last":2.226},{"interface":"front","average":{"1min":1.393,"5min":1.1060000000000001,"15min":0.78800000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"last":0.27300000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.282,"15min":0.85199999999999998},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.9359999999999999,"5min":5.9359999999999999,"15min":5.9359999999999999},"last":3.5310000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.1599999999999999,"15min":0.79400000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.4260000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":3.657}]},{"osd":6,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.0349999999999999,"15min":0.77400000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.6309999999999998,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":3.6309999999999998},{"interface":"front","average":{"1min":1.502,"5min":1.0289999999999999,"15min":0.751},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":5.0439999999999996,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":2.2629999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.9730000000000001,"5min":1.3600000000000001,"15min":0.88700000000000001},"min":{"1min":0.38500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.7240000000000002,"5min":7.7240000000000002,"15min":7.7240000000000002},"last":2.2829999999999999},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.325,"15min":0.86299999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.8300000000000001,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.32400000000000001}]}]},{"osd":2,"up_from":133,"seq":571230650439,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440360,"kb_used_data":331036,"kb_used_omap":6,"kb_used_meta":109305,"kb_avail":93304792,"statfs":{"total":95995035648,"available":95544107008,"internally_reserved":0,"allocated":338980864,"data_stored":338103713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":111928821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.63400000000000001,"15min":0.45200000000000001},"min":{"1min":0.19700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.3820000000000001,"5min":3.9750000000000001,"15min":3.9750000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":0.65500000000000003,"15min":0.45900000000000002},"min":{"1min":0.19700000000000001,"5min":0.182,"15min":0.182},"max":{"1min":4.8019999999999996,"5min":4.8019999999999996,"15min":4.8019999999999996},"last":0.38400000000000001}]},{"osd":1,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.1419999999999999,"15min":0.63400000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.8200000000000003,"5min":21.920000000000002,"15min":21.920000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.111,"15min":0.61099999999999999},"min":{"1min":0.27200000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.4620000000000002,"5min":21.573,"15min":21.573},"last":0.94099999999999995}]},{"osd":3,"last update":"Fri Jun 14 22:58:13 2024","interfaces":[{"interface":"back","average":{"1min":1.903,"5min":1.0249999999999999,"15min":0.58399999999999996},"min":{"1min":0.315,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.1189999999999998,"5min":21.66,"15min":21.66},"last":2.6749999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.0549999999999999,"15min":0.61299999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.0449999999999999,"5min":22.341000000000001,"15min":22.341000000000001},"last":2.7360000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":0.83299999999999996,"15min":0.66700000000000004},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":6.7030000000000003,"5min":6.7030000000000003,"15min":6.7030000000000003},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.034,"5min":0.77900000000000003,"15min":0.64500000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.8890000000000002,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":0.86799999999999999,"15min":0.66200000000000003},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.0800000000000001,"5min":5.0800000000000001,"15min":5.0800000000000001},"last":1.645},{"interface":"front","average":{"1min":1.399,"5min":0.80600000000000005,"15min":0.65600000000000003},"min":{"1min":0.38100000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.0449999999999999,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":6.1790000000000003}]},{"osd":6,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":0.70899999999999996,"15min":0.61199999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":5.8579999999999997,"5min":5.8579999999999997,"15min":5.8579999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.83499999999999996,"15min":0.66300000000000003},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.3759999999999999,"5min":5.1829999999999998,"15min":5.1829999999999998},"last":0.36299999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:58:38 2024","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":0.94599999999999995,"15min":0.71499999999999997},"min":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.226,"5min":6.226,"15min":6.226},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":0.80400000000000005,"15min":0.65800000000000003},"min":{"1min":0.41499999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":7.6260000000000003,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":4.7080000000000002}]}]},{"osd":1,"up_from":116,"seq":498216206456,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382952,"kb_used_data":976860,"kb_used_omap":5,"kb_used_meta":406010,"kb_avail":92362200,"statfs":{"total":95995035648,"available":94578892800,"internally_reserved":0,"allocated":1000304640,"data_stored":999427209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":415754520},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":0.999,"15min":1.98},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":6.1870000000000003,"5min":8.2560000000000002,"15min":9.0559999999999992},"last":2.3029999999999999},{"interface":"front","average":{"1min":1.254,"5min":0.879,"15min":1.5069999999999999},"min":{"1min":0.20399999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":4.5330000000000004,"5min":5.0220000000000002,"15min":10.426},"last":2.3639999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.792,"5min":1.0720000000000001,"15min":0.58099999999999996},"min":{"1min":0.22600000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":6.1539999999999999,"5min":8.2880000000000003,"15min":8.2880000000000003},"last":2.2269999999999999},{"interface":"front","average":{"1min":1.113,"5min":0.81799999999999995,"15min":0.496},"min":{"1min":0.28199999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":2.427}]},{"osd":3,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.6739999999999999,"15min":0.77800000000000002},"min":{"1min":0.27400000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.8479999999999999,"5min":21.422000000000001,"15min":21.422000000000001},"last":1.8460000000000001},{"interface":"front","average":{"1min":2.5670000000000002,"5min":1.764,"15min":0.83399999999999996},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":9.4990000000000006,"5min":21.016999999999999,"15min":21.016999999999999},"last":2.4580000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.161,"15min":1.7649999999999999},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.28399999999999997},"max":{"1min":6.6360000000000001,"5min":7.673,"15min":8.8599999999999994},"last":1.177},{"interface":"front","average":{"1min":2.109,"5min":1.327,"15min":1.79},"min":{"1min":0.29399999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.4500000000000002,"5min":8.3620000000000001,"15min":11.161},"last":2.1890000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":1.147,"15min":1.6719999999999999},"min":{"1min":0.39500000000000002,"5min":0.32800000000000001,"15min":0.30599999999999999},"max":{"1min":5.6870000000000003,"5min":5.6870000000000003,"15min":9.7539999999999996},"last":2.387},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.24,"15min":1.823},"min":{"1min":0.499,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":6.5730000000000004,"5min":7.7439999999999998,"15min":8.125},"last":1.246}]},{"osd":6,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.042,"15min":1.7},"min":{"1min":0.373,"5min":0.35799999999999998,"15min":0.32100000000000001},"max":{"1min":4.9459999999999997,"5min":5.3849999999999998,"15min":9.7279999999999998},"last":2.3260000000000001},{"interface":"front","average":{"1min":1.633,"5min":1.153,"15min":1.7869999999999999},"min":{"1min":0.39100000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.8899999999999997,"5min":7.7060000000000004,"15min":8.1479999999999997},"last":1.224}]},{"osd":7,"last update":"Fri Jun 14 22:58:39 2024","interfaces":[{"interface":"back","average":{"1min":2.7189999999999999,"5min":1.419,"15min":1.899},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.33700000000000002},"max":{"1min":6.218,"5min":8.3369999999999997,"15min":9.0960000000000001},"last":2.1560000000000001},{"interface":"front","average":{"1min":1.8999999999999999,"5min":1.1779999999999999,"15min":1.581},"min":{"1min":0.41499999999999998,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":6.3529999999999998,"5min":6.3529999999999998,"15min":9.5449999999999999},"last":2.407}]}]},{"osd":0,"up_from":91,"seq":390842024111,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":915684,"kb_used_data":413980,"kb_used_omap":5,"kb_used_meta":501690,"kb_avail":92829468,"statfs":{"total":95995035648,"available":95057375232,"internally_reserved":0,"allocated":423915520,"data_stored":423036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":513730650},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:58:34 2024","interfaces":[{"interface":"back","average":{"1min":2.327,"5min":1.2749999999999999,"15min":1.609},"min":{"1min":0.19900000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":15.547000000000001,"5min":16.305,"15min":16.305},"last":0.47299999999999998},{"interface":"front","average":{"1min":2.016,"5min":1.3879999999999999,"15min":2.0640000000000001},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.9889999999999999,"5min":16.184999999999999,"15min":16.184999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Jun 14 22:58:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":0.94199999999999995,"15min":0.55200000000000005},"min":{"1min":0.216,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.5069999999999997,"5min":16.757000000000001,"15min":16.757000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.127,"15min":0.64100000000000001},"min":{"1min":0.246,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.641,"5min":16.055,"15min":16.055},"last":0.25}]},{"osd":3,"last update":"Fri Jun 14 22:58:12 2024","interfaces":[{"interface":"back","average":{"1min":2.1080000000000001,"5min":1.169,"15min":0.63200000000000001},"min":{"1min":0.33500000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":7.04,"5min":16.873999999999999,"15min":16.873999999999999},"last":2.3919999999999999},{"interface":"front","average":{"1min":2.4289999999999998,"5min":1.1779999999999999,"15min":0.63200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":12.308999999999999,"5min":16.085999999999999,"15min":16.085999999999999},"last":1.347}]},{"osd":4,"last update":"Fri Jun 14 22:58:19 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.0169999999999999,"15min":1.498},"min":{"1min":0.44,"5min":0.308,"15min":0.27600000000000002},"max":{"1min":5.7050000000000001,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":2.0409999999999999,"5min":1.117,"15min":1.3700000000000001},"min":{"1min":0.39200000000000002,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":6.9210000000000003,"5min":15.976000000000001,"15min":15.976000000000001},"last":0.55700000000000005}]},{"osd":5,"last update":"Fri Jun 14 22:58:19 2024","interfaces":[{"interface":"back","average":{"1min":2.1400000000000001,"5min":1.3069999999999999,"15min":1.4710000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29299999999999998},"max":{"1min":6.9640000000000004,"5min":16.108000000000001,"15min":16.108000000000001},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.173,"15min":1.4930000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.3890000000000002,"5min":16.006,"15min":16.006},"last":0.40500000000000003}]},{"osd":6,"last update":"Fri Jun 14 22:58:19 2024","interfaces":[{"interface":"back","average":{"1min":1.984,"5min":1.143,"15min":1.4390000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.016,"5min":16.152000000000001,"15min":16.152000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.206,"5min":1.0429999999999999,"15min":1.478},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.29399999999999998},"max":{"1min":4.4210000000000003,"5min":16.227,"15min":16.227},"last":0.36599999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:58:19 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.1779999999999999,"15min":1.5309999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.33100000000000002},"max":{"1min":4.4509999999999996,"5min":16.199999999999999,"15min":16.199999999999999},"last":0.501},{"interface":"front","average":{"1min":2.1899999999999999,"5min":1.2010000000000001,"15min":1.641},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.314},"max":{"1min":12.361000000000001,"5min":16.259,"15min":16.259},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":419954688,"data_stored":419954688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":997920768,"data_stored":997916696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331284480,"data_stored":331284480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":894042112,"data_stored":894042112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":341114880,"data_stored":341114880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1021186048,"data_stored":1021181976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":340852736,"data_stored":340852736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1027739648,"data_stored":1027735576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:59:21.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-06-14T22:59:21.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:21.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:21.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:21.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:21.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:21.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:21.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:21.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:21.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:21.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:21.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:21 smithi137 ceph-mon[90634]: pgmap v688: 89 pgs: 89 active+clean; 1.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 854 op/s 2024-06-14T22:59:21.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:21.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:21.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:21.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:21.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:22.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:22.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:21 smithi086 ceph-mon[97913]: pgmap v688: 89 pgs: 89 active+clean; 1.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 854 op/s 2024-06-14T22:59:22.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:21 smithi086 ceph-mon[102754]: pgmap v688: 89 pgs: 89 active+clean; 1.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 854 op/s 2024-06-14T22:59:22.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:22.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:22.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:22.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:22.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:22.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:22.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:22.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:22.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:22.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:22.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:22.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:22.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:22.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:22.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:22.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:22.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:22.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:22.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:22.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:22.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:23.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:23.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:23.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:23.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:23.234 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:23.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:23.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:23.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:23.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:23.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:23.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:23.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:23.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:23.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:23.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:23.635 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:23.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:23.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:23.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:23.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:23.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:23.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:23.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:23.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:23.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:23 smithi137 ceph-mon[90634]: pgmap v689: 89 pgs: 89 active+clean; 1.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 847 op/s 2024-06-14T22:59:24.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:24.039 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:24.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:24.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:24.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:23 smithi086 ceph-mon[97913]: pgmap v689: 89 pgs: 89 active+clean; 1.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 847 op/s 2024-06-14T22:59:24.062 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:23 smithi086 ceph-mon[102754]: pgmap v689: 89 pgs: 89 active+clean; 1.4 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 847 op/s 2024-06-14T22:59:24.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:24.176 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:24.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:24.226 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:59:24.226 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:59:24.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:24.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:24.328 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:24.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:24.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:24.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:24.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:24.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:24.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:24.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:24.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:24.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:24.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:24.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:24.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:24.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:24.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:24.953 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:59:24.953 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:59:24.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:24.997 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:59:24.997 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:59:25.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:25.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:25.034 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:59:25.034 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:59:25.067 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:59:25.067 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:59:25.067 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:59:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:59:24] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T22:59:25.103 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:59:25.103 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:59:25.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:25.137 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:59:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:59:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:25.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:25.174 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:59:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:59:25.205 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:59:25.206 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:59:25.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:25.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:25.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:25.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:25.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:25.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:25.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:25.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:25.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:25.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:25.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:25.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:25.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-14T22:59:25.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:25 smithi086 ceph-mon[97913]: pgmap v690: 89 pgs: 89 active+clean; 1.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 841 op/s 2024-06-14T22:59:25.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3272776665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:25.761 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:25 smithi086 ceph-mon[102754]: pgmap v690: 89 pgs: 89 active+clean; 1.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 841 op/s 2024-06-14T22:59:25.761 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3272776665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:25.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:25.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:25.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:25.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:25.958 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:25.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:25.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:25 smithi137 ceph-mon[90634]: pgmap v690: 89 pgs: 89 active+clean; 1.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 841 op/s 2024-06-14T22:59:25.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3272776665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:26.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:26.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:26.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:26.194 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:26.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:26.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:26.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:26.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:26.413 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:26.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:26.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:26.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:26.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:26.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:26.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:26.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:26.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:26.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:26.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:26.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:26.966 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:26.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:27.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:27.101 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:27.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:27.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:27.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:27.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:27.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:27.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:27.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:27.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:27.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:27.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:27.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:27.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:27.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:27.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:27.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:27.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:27.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:27.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:27.949 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:27.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:27.983 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:27 smithi086 ceph-mon[102754]: pgmap v691: 89 pgs: 89 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 898 op/s 2024-06-14T22:59:27.984 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:27.984 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:27.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:27 smithi086 ceph-mon[97913]: pgmap v691: 89 pgs: 89 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 898 op/s 2024-06-14T22:59:27.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:27.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:27.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:27 smithi137 ceph-mon[90634]: pgmap v691: 89 pgs: 89 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 898 op/s 2024-06-14T22:59:27.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:27.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:28.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:28.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:28.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:28.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:28.220 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:28.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:28.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:28.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:28.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:28.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:28.503 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:28.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:28.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:28.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:28.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:28.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:28.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:28.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:28.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:59:28.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:28.896 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:28.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:28.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:29.027 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:29.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:29.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:29.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:29.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:29.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:29.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:29.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:29.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:29.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:29.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:29.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:29.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:29.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:29.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:29.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:29.677 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:29.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:29 smithi086 ceph-mon[97913]: pgmap v692: 89 pgs: 89 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-06-14T22:59:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:29.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:29.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:29.882 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:29.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:29.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:29.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:29 smithi137 ceph-mon[90634]: pgmap v692: 89 pgs: 89 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-06-14T22:59:30.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:30.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:30.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:29 smithi086 ceph-mon[102754]: pgmap v692: 89 pgs: 89 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-06-14T22:59:30.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:30.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:30.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:30.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:30.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:30.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:30.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:30.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:30.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:30.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:30.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:30.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:30.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:30.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T22:59:30.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:30.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:30.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:30.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:30.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2408408723' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:30.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:30 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:30.998 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:30.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2408408723' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:30 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:31.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2408408723' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:31.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:30 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:31.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:31.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:31.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:31.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:31.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:31.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:31.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:31.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:31.400 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:31.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:31.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:31.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:31.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:31.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:31.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:31.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:31.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:31.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:31.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:31.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:31 smithi086 ceph-mon[102754]: pgmap v693: 89 pgs: 89 active+clean; 955 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 982 op/s 2024-06-14T22:59:31.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:31 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:59:31.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:31 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:59:31.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:31 smithi086 ceph-mon[102754]: osdmap e174: 8 total, 8 up, 8 in 2024-06-14T22:59:31.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:31 smithi086 ceph-mon[97913]: pgmap v693: 89 pgs: 89 active+clean; 955 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 982 op/s 2024-06-14T22:59:31.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:31 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:59:31.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:31 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:59:31.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:31 smithi086 ceph-mon[97913]: osdmap e174: 8 total, 8 up, 8 in 2024-06-14T22:59:31.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:31.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:31.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:31.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:31 smithi137 ceph-mon[90634]: pgmap v693: 89 pgs: 89 active+clean; 955 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 982 op/s 2024-06-14T22:59:31.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:31 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:59:31.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:31 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T22:59:31.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:31 smithi137 ceph-mon[90634]: osdmap e174: 8 total, 8 up, 8 in 2024-06-14T22:59:32.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:32.084 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:32.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:32.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:32.220 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:32.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:32.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:32.357 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:32.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:32.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:32.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:32.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:32.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:32.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:32.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:32.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:32.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:32.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:32.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:32.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:33.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:33.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:33.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:33.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:33.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:33.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:33.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:33.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:33.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:33.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:33.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:33.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:33.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:33.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:33.622 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:33.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:33.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:33.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:33.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:33.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:33.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:33.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:33.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:33.947 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:59:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T22:59:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T22:59:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T22:59:33.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98108 8.a 8:580c610c:::benchmark_data_smithi086_122122_object98107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.663895+0000", 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 567.27934512000002, 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.007937861, 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98108 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663895+0000", 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663895+0000", 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663893+0000", 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-06-14T22:59:33.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663938+0000", 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4446000000000001e-05 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663958+0000", 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.984e-05 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:33.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.663967+0000", 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6819999999999998e-06 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.664096+0000", 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000128489 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.667900+0000", 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038044369999999999 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.662435+0000", 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.994535222 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.662563+0000", 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012799 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.662679+0000", 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011518099999999999 2024-06-14T22:59:33.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.669323+0000", 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066442560000000003 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671664+0000", 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002341278 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671736+0000", 2024-06-14T22:59:33.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1549000000000002e-05 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671787+0000", 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1718999999999997e-05 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671833+0000", 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6004999999999997e-05 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98111 8.a 8:52d8109b:::benchmark_data_smithi086_122122_object98110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.666047+0000", 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 567.27719334899996, 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0059651590000001, 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98111 2024-06-14T22:59:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666047+0000", 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666047+0000", 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666045+0000", 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666073+0000", 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7528000000000001e-05 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666087+0000", 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4144999999999999e-05 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.666103+0000", 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6232999999999999e-05 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.668274+0000", 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002170834 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.668286+0000", 2024-06-14T22:59:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1773999999999999e-05 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.665038+0000", 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99675228999999999 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.665152+0000", 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114057 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:59:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.665289+0000", 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136279 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671689+0000", 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006400579 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671888+0000", 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000198803 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671934+0000", 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5759e-05 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671974+0000", 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0763000000000003e-05 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672012+0000", 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7978000000000001e-05 2024-06-14T22:59:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98134 8.a 8:5734d724:::benchmark_data_smithi086_122122_object98133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.691770+0000", 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 567.25147066199997, 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98507571699999996, 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98134 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691770+0000", 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691770+0000", 2024-06-14T22:59:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691768+0000", 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691802+0000", 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3120999999999997e-05 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691817+0000", 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5747e-05 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691824+0000", 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7830000000000001e-06 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691873+0000", 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9345000000000003e-05 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.691884+0000", 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0575e-05 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.667450+0000", 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97556580900000001 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.667567+0000", 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011715 2024-06-14T22:59:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.667680+0000", 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011291899999999999 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.671898+0000", 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042183409999999996 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672064+0000", 2024-06-14T22:59:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000166167 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676751+0000", 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046863879999999997 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676805+0000", 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4749e-05 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676846+0000", 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0166000000000001e-05 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98150 8.a 8:58ff2b8c:::benchmark_data_smithi086_122122_object98149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.706092+0000", 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 567.23714896199999, 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97089046899999998, 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98150 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706092+0000", 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706092+0000", 2024-06-14T22:59:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706090+0000", 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706136+0000", 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5948999999999997e-05 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706194+0000", 2024-06-14T22:59:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7394e-05 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706212+0000", 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8150000000000001e-05 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706306+0000", 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4178000000000005e-05 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.706317+0000", 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.135e-05 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.669367+0000", 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96304950300000003 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.669484+0000", 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000117681 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.669627+0000", 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000142724 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672073+0000", 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024456830000000001 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672097+0000", 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4059000000000001e-05 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676894+0000", 2024-06-14T22:59:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004797 2024-06-14T22:59:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676954+0000", 2024-06-14T22:59:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0337999999999997e-05 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.676982+0000", 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7861e-05 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:98157 8.a 8:5815c7bb:::benchmark_data_smithi086_122122_object98156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:06.713784+0000", 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 567.22945664999997, 2024-06-14T22:59:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96892899099999996, 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 98157 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713784+0000", 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713784+0000", 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713783+0000", 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713807+0000", 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4148999999999999e-05 2024-06-14T22:59:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713825+0000", 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8187999999999999e-05 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713834+0000", 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6710000000000005e-06 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713891+0000", 2024-06-14T22:59:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7528000000000002e-05 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:06.713906+0000", 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4440000000000001e-05 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672124+0000", 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95821818800000003 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672245+0000", 2024-06-14T22:59:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000121508 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,5", 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.672386+0000", 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000141218 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.679864+0000", 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074774819999999997 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.682556+0000", 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026924309999999999 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.682587+0000", 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0913e-05 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.682653+0000", 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5442000000000001e-05 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:07.682713+0000", 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0180000000000003e-05 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:129201 8.3 8:c7ab2e14:::benchmark_data_smithi086_122122_object8612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:43.140054+0000", 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 530.80318697999996, 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97529428799999995, 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 129201 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140054+0000", 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140054+0000", 2024-06-14T22:59:33.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140052+0000", 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140060+0000", 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3979999999999999e-06 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140079+0000", 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8173e-05 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140087+0000", 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9200000000000004e-06 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140149+0000", 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2477000000000005e-05 2024-06-14T22:59:33.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140161+0000", 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1712999999999999e-05 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.108330+0000", 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96816938500000005 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.108456+0000", 2024-06-14T22:59:33.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012608199999999999 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.109046+0000", 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00058971700000000002 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115204+0000", 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061580799999999998 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115254+0000", 2024-06-14T22:59:33.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9706999999999997e-05 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115265+0000", 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0859999999999999e-05 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115307+0000", 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2867999999999997e-05 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:33.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115348+0000", 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0429999999999997e-05 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:129202 8.3 8:cccec3f8:::benchmark_data_smithi086_122122_object8613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:43.140692+0000", 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 530.80254902499996, 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98892060800000003, 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:33.990 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 129202 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140692+0000", 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140692+0000", 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140691+0000", 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140696+0000", 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5260000000000001e-06 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140704+0000", 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9330000000000001e-06 2024-06-14T22:59:33.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140710+0000", 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0170000000000002e-06 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140783+0000", 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3145999999999994e-05 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.140806+0000", 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3059000000000001e-05 2024-06-14T22:59:33.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115379+0000", 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97457263599999999 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115465+0000", 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5741999999999993e-05 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.115543+0000", 2024-06-14T22:59:33.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8623999999999998e-05 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.123140+0000", 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0075970279999999996 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.123181+0000", 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0568000000000002e-05 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129524+0000", 2024-06-14T22:59:33.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063430120000000003 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129572+0000", 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8378999999999999e-05 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129612+0000", 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9752999999999997e-05 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.34776.0:129204 8.3 8:c65a5a5e:::benchmark_data_smithi086_122122_object8615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:50:43.142040+0000", 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 530.80120078100003, 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98773785999999997, 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.34776", 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 129204 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:33.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142040+0000", 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142040+0000", 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142039+0000", 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:59:33.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142046+0000", 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.781e-06 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142062+0000", 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6713000000000001e-05 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.142070+0000", 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2050000000000002e-06 2024-06-14T22:59:33.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.163576+0000", 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021505791 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:43.163587+0000", 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0869999999999999e-05 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.117985+0000", 2024-06-14T22:59:34.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95439831399999997 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.118089+0000", 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103247 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.118203+0000", 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011443 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.126190+0000", 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079872650000000003 2024-06-14T22:59:34.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129473+0000", 2024-06-14T22:59:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0032825979999999999 2024-06-14T22:59:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:33 smithi086 ceph-mon[97913]: pgmap v695: 89 pgs: 89 active+clean; 890 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-06-14T22:59:34.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:33 smithi086 ceph-mon[102754]: pgmap v695: 89 pgs: 89 active+clean; 890 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-06-14T22:59:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129702+0000", 2024-06-14T22:59:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00022937899999999999 2024-06-14T22:59:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129745+0000", 2024-06-14T22:59:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2685000000000002e-05 2024-06-14T22:59:34.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:50:44.129778+0000", 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2648999999999999e-05 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24715 9.d 9:b7390049:::benchmark_data_smithi086_152062_object24714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.661292+0000", 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 221.281949024, 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.018053425, 2024-06-14T22:59:34.004 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24715 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661292+0000", 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661292+0000", 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661290+0000", 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661315+0000", 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5228999999999998e-05 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661325+0000", 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8689999999999999e-06 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661332+0000", 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8990000000000004e-06 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661364+0000", 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661375+0000", 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0645999999999999e-05 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.665805+0000", 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.004429807 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.666631+0000", 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082573 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.666788+0000", 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000157484 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673421+0000", 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066326279999999998 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673465+0000", 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4218999999999998e-05 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679261+0000", 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057959120000000003 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679308+0000", 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6998e-05 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679345+0000", 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7234999999999998e-05 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24723 9.d 9:b2f904ff:::benchmark_data_smithi086_152062_object24722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.664488+0000", 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 221.278752247, 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0150036360000001, 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24723 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664488+0000", 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664488+0000", 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664487+0000", 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664523+0000", 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5485000000000003e-05 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664538+0000", 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5412000000000001e-05 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664545+0000", 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.781e-06 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664598+0000", 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2976000000000002e-05 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664607+0000", 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6290000000000005e-06 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668290+0000", 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0036830910000001 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668426+0000", 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013543800000000001 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668573+0000", 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147498 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673521+0000", 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049472129999999998 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.676833+0000", 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0033120250000000001 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679379+0000", 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025462699999999998 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679445+0000", 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6476999999999994e-05 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679492+0000", 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6640000000000001e-05 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24729 9.d 9:b6335b33:::benchmark_data_smithi086_152062_object24728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.667823+0000", 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 221.27541727400001, 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0118812850000001, 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24729 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667823+0000", 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667823+0000", 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667822+0000", 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667847+0000", 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4666000000000001e-05 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667864+0000", 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7611000000000001e-05 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667878+0000", 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3182e-05 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668333+0000", 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000454931 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668342+0000", 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1630000000000002e-06 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.669852+0000", 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.00151035 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.670010+0000", 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000158257 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.670183+0000", 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017219199999999999 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.677648+0000", 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007465546 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679516+0000", 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001868429 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679566+0000", 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9395999999999999e-05 2024-06-14T22:59:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679670+0000", 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010412900000000001 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679705+0000", 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4591999999999997e-05 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24730 9.d 9:be481d2c:::benchmark_data_smithi086_152062_object24729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.668425+0000", 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 221.27481539799999, 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0114436330000001, 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24730 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668447+0000", 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2699000000000001e-05 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668454+0000", 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6880000000000002e-06 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668459+0000", 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4940000000000003e-06 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668500+0000", 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0577000000000001e-05 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668508+0000", 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5229999999999998e-06 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.671828+0000", 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.003320011 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.671965+0000", 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000137081 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.672100+0000", 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013549800000000001 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.677685+0000", 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055849710000000002 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679521+0000", 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001835761 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679784+0000", 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000263465 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679825+0000", 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0980999999999999e-05 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679869+0000", 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3461000000000003e-05 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24738 9.d 9:b3d2a961:::benchmark_data_smithi086_152062_object24737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.671698+0000", 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 221.271542281, 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.009193183, 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24738 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671698+0000", 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671698+0000", 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671697+0000", 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671720+0000", 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3345000000000001e-05 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671734+0000", 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3669000000000001e-05 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671742+0000", 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7470000000000005e-06 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671822+0000", 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.051e-05 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671834+0000", 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1847e-05 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673548+0000", 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.001713718 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673671+0000", 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012297600000000001 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673796+0000", 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012489900000000001 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679254+0000", 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054580119999999999 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679525+0000", 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00027125800000000002 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680809+0000", 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012839290000000001 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680856+0000", 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7094e-05 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680891+0000", 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5438999999999997e-05 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249750 9.2 9:4ea8c052:::benchmark_data_smithi086_152062_object117268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.641870+0000", 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.3013708299999998, 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.071249919999999994, 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249750 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641870+0000", 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641870+0000", 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641869+0000", 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641874+0000", 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.605e-06 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641889+0000", 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5305999999999999e-05 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641896+0000", 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7360000000000004e-06 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.680196+0000", 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038300010000000002 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.680308+0000", 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112149 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T22:59:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.680416+0000", 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108577 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693896+0000", 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013479804999999999 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693903+0000", 2024-06-14T22:59:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4049999999999997e-06 2024-06-14T22:59:34.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:34.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:33 smithi137 ceph-mon[90634]: pgmap v695: 89 pgs: 89 active+clean; 890 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 867 op/s 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.713033+0000", 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019130945999999999 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.713081+0000", 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7457999999999999e-05 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.713120+0000", 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8748999999999998e-05 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249768 9.2 9:4907852a:::benchmark_data_smithi086_152062_object117286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.667370+0000", 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.2758704590000001, 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.057787167, 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249768 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667370+0000", 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667370+0000", 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667369+0000", 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667374+0000", 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5079999999999996e-06 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667384+0000", 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0035e-05 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667391+0000", 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3339999999999998e-06 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.686242+0000", 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018851422 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.686343+0000", 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100976 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.686444+0000", 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100659 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693937+0000", 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074934650000000004 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.699813+0000", 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058754219999999999 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.724891+0000", 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025078644000000001 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.725125+0000", 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00023403900000000001 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.725157+0000", 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1955999999999997e-05 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249775 9.2 9:4b4108a1:::benchmark_data_smithi086_152062_object117293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.674316+0000", 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.2689246909999996, 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.079108740999999996, 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249775 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.050 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674316+0000", 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674316+0000", 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674315+0000", 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674321+0000", 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2870000000000001e-06 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674334+0000", 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3183999999999999e-05 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674341+0000", 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7009999999999999e-06 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.690107+0000", 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015766227000000001 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.690222+0000", 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115213 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.690337+0000", 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114992 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.697790+0000", 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074525579999999998 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.705669+0000", 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0078787340000000001 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753341+0000", 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047672842999999999 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753388+0000", 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7018999999999998e-05 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753425+0000", 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6276000000000003e-05 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249790 9.2 9:429f8418:::benchmark_data_smithi086_152062_object117308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.692884+0000", 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.2503568600000001, 2024-06-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060712204999999998, 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249790 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692884+0000", 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692884+0000", 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692882+0000", 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692888+0000", 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7150000000000003e-06 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692898+0000", 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0008e-05 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692905+0000", 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6200000000000001e-06 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693956+0000", 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010514299999999999 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.694076+0000", 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012010099999999999 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.694192+0000", 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115711 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.699378+0000", 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0051855069999999998 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.705966+0000", 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0065888329999999997 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753529+0000", 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047562184 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753564+0000", 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5240000000000001e-05 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753596+0000", 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2150999999999998e-05 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249820 9.2 9:4639dddc:::benchmark_data_smithi086_152062_object117338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.729017+0000", 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.2142236049999999, 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.049999134000000001, 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.061 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249820 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.729017+0000", 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.729017+0000", 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.729015+0000", 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.729022+0000", 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1400000000000002e-06 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.752612+0000", 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023589868 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.752632+0000", 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9986000000000001e-05 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.761847+0000", 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092154179999999995 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.762087+0000", 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00023995099999999999 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.762197+0000", 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109374 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.773409+0000", 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011212583999999999 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.774990+0000", 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00158102 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.778663+0000", 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036723699999999999 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.778788+0000", 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012529500000000001 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.779016+0000", 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000228058 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:254239 9.2 9:4c339b56:::benchmark_data_smithi086_152062_object121757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:33.777110+0000", 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.16613045100000001, 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.054250944000000002, 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 254239 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777110+0000", 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777110+0000", 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777108+0000", 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777119+0000", 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0147e-05 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777135+0000", 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6742999999999999e-05 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777145+0000", 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.499e-06 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.783108+0000", 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059630259999999997 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.783213+0000", 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104886 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.783322+0000", 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108861 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.798666+0000", 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015344750000000001 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.805636+0000", 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069697200000000004 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.831285+0000", 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025648908000000002 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.831325+0000", 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0095999999999997e-05 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.831361+0000", 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6087999999999997e-05 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:254245 9.2 9:4e3ab7c5:::benchmark_data_smithi086_152062_object121763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:33.779286+0000", 2024-06-14T22:59:34.072 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.163954444, 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.053949624000000002, 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 254245 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.779286+0000", 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.779286+0000", 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.779286+0000", 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.779288+0000", 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.486e-06 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.779294+0000", 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9039999999999997e-06 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.779299+0000", 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.437e-06 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.787399+0000", 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0081004939999999998 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.787518+0000", 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118812 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.787635+0000", 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000117537 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.798671+0000", 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011035458999999999 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.805660+0000", 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069896159999999997 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.833148+0000", 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027487812 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.833195+0000", 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7144000000000001e-05 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T22:59:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.833236+0000", 2024-06-14T22:59:34.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0373000000000001e-05 2024-06-14T22:59:34.078 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.078 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.078 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.078 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T22:59:34.078 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T22:59:34.078 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:59:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:34.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:34.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:34.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:34.222 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:34.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:34.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:34.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:34.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:34.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:34.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:34.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:34.630 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:34.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:34.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:34.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:34.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:34.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:34.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:34.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:34.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:35.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:35.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:35.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:35.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:35.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:35.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:35.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:35.334 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:35.335 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:59:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:59:34] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.43.0" 2024-06-14T22:59:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:35.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:35.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:35.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:35.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:35.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:35.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:35.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:35.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:35.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:35.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:35.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:35.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:35.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:35.899 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:35.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:36.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:36.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:36.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:35 smithi086 ceph-mon[102754]: pgmap v696: 89 pgs: 89 active+clean; 750 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-06-14T22:59:36.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:35 smithi086 ceph-mon[97913]: pgmap v696: 89 pgs: 89 active+clean; 750 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-06-14T22:59:36.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:36.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:36.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:36.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:35 smithi137 ceph-mon[90634]: pgmap v696: 89 pgs: 89 active+clean; 750 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-06-14T22:59:36.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:36.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:36.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:36.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T22:59:36.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:36.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:36.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:36.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:36.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:36.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:36.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:36.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:36.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:36.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:36.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:36.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:36.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:37.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:37.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:37.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T22:59:37.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:37.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:37.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:37.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:37.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:37.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:37.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:37.475 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:37.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:37.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:37.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:37.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:37.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:37.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:37.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 9 pgp_num to 16 2024-06-14T22:59:37.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:37.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:37.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:38.039 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:38.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:38.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:37 smithi086 ceph-mon[97913]: pgmap v697: 89 pgs: 89 active+clean; 681 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-06-14T22:59:38.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1091192521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T22:59:38.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:37 smithi086 ceph-mon[102754]: pgmap v697: 89 pgs: 89 active+clean; 681 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-06-14T22:59:38.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1091192521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T22:59:38.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:38.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:38.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:38.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:38.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:37 smithi137 ceph-mon[90634]: pgmap v697: 89 pgs: 89 active+clean; 681 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-06-14T22:59:38.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1091192521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T22:59:38.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:38.334 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:38.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:38.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:38.467 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:38.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:38.569 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:38.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:38.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:38.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:59:38.662 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:59:38.697 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:59:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:59:38.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:38.731 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:59:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:59:38.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:38.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:38.778 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:59:38.778 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:59:38.812 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:59:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:59:38.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:38.862 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:59:38.862 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:59:38.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:38.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:38.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:38.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:59:38.888 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:59:38.922 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:59:38.922 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:59:38.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:39.019 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:39.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:39.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:38 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1091192521' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T22:59:39.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:38 smithi086 ceph-mon[97913]: osdmap e175: 8 total, 8 up, 8 in 2024-06-14T22:59:39.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:38 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2992819441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:39.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:38 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1091192521' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T22:59:39.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:38 smithi086 ceph-mon[102754]: osdmap e175: 8 total, 8 up, 8 in 2024-06-14T22:59:39.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:38 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2992819441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:39.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:39.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:39.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:39.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:38 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1091192521' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T22:59:39.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:38 smithi137 ceph-mon[90634]: osdmap e175: 8 total, 8 up, 8 in 2024-06-14T22:59:39.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:38 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2992819441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:39.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:39.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:39.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:39.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:39.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:39.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:39.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:39.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:39.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:39.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:39.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:39.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:39.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:39.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:39.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:39.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:39.964 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:39.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:39.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:40.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:40.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:39 smithi086 ceph-mon[97913]: pgmap v699: 89 pgs: 89 active+clean; 609 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 691 op/s 2024-06-14T22:59:40.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:39 smithi086 ceph-mon[102754]: pgmap v699: 89 pgs: 89 active+clean; 609 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 691 op/s 2024-06-14T22:59:40.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:40.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:40.145 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:39 smithi137 ceph-mon[90634]: pgmap v699: 89 pgs: 89 active+clean; 609 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 691 op/s 2024-06-14T22:59:40.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:40.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:40.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:40.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:40.368 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:40.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:40.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:40.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:40.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:40.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:40.643 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:40.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:40.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:40.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:40.773 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:40.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:40.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:40.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:40.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:41.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:41.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:41.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:41.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:41.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:41.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:41.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:41.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:41.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:41.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:41.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:41.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:41.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:41.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:41.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:41.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:41.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:41.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:41.748 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:41.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:41.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:41.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:41.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:41.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:42.015 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:42.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:42.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:42.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:42.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:41 smithi086 ceph-mon[102754]: pgmap v700: 89 pgs: 89 active+clean; 405 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 888 op/s 2024-06-14T22:59:42.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:41 smithi086 ceph-mon[97913]: pgmap v700: 89 pgs: 89 active+clean; 405 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 888 op/s 2024-06-14T22:59:42.060 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:42.103 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:41 smithi137 ceph-mon[90634]: pgmap v700: 89 pgs: 89 active+clean; 405 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 888 op/s 2024-06-14T22:59:42.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:42.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:42.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:42.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:42.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:42.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:42.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:42.434 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:42.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:42.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:42.569 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:42.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:42.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:42.709 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:42.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:42.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:42.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:42.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:42.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:42.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:42.948 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:42.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:42.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:42.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:43.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:43.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:43.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:43.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:43.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:43.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:43.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:43.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:43.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:43.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:43.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:43.537 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:43.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:43.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:43.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:43.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:43.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:43.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:43.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:43.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:43.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:43.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:44.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T22:59:44.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:44.074 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:43 smithi086 ceph-mon[102754]: pgmap v701: 89 pgs: 89 active+clean; 335 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 886 op/s 2024-06-14T22:59:44.078 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:43 smithi086 ceph-mon[97913]: pgmap v701: 89 pgs: 89 active+clean; 335 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 886 op/s 2024-06-14T22:59:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:44.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:44.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:44.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:44.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:44.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:44.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:43 smithi137 ceph-mon[90634]: pgmap v701: 89 pgs: 89 active+clean; 335 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 886 op/s 2024-06-14T22:59:44.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:44.368 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T22:59:44.368 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T22:59:44.368 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-14T22:59:44.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T22:59:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:44.370 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:44.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:44.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:44.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:44.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:44.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:44.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:44.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:44.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:44.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:44.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:44.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:44.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:44.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:45.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:45.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:45.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:45.044 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:59:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:59:44] "GET /metrics HTTP/1.1" 200 36534 "" "Prometheus/2.43.0" 2024-06-14T22:59:45.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:45.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:45.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:45.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:45.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:45.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:45.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:45.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:45.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:45.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[102754]: pgmap v702: 89 pgs: 89 active+clean; 203 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 874 op/s 2024-06-14T22:59:45.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:59:45.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:59:45.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:45.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:45.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:45.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[97913]: pgmap v702: 89 pgs: 89 active+clean; 203 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 874 op/s 2024-06-14T22:59:45.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:59:45.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:59:45.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:45.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:45.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:45.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:45.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:45.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:45.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:45 smithi137 ceph-mon[90634]: pgmap v702: 89 pgs: 89 active+clean; 203 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 874 op/s 2024-06-14T22:59:45.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T22:59:45.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T22:59:45.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:45.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:45.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:45.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:45.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:45.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:45.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:45.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:45.860 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:45.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:45.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:45.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:45.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:46.078 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T22:59:46.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:46.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:46.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:46.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:46.248 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 132479 objects 2024-06-14T22:59:46.248 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :150.976 2024-06-14T22:59:46.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:46.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:46.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/927257747' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:59:46.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:46 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:59:46.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/927257747' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:59:46.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:46 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:59:46.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:46.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:46.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:46.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:46.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/927257747' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:59:46.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:46 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T22:59:46.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:46.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:46.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:46.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:46.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:46.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:46.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:46.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:46.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:46.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:46.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:46.983 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T22:59:46.983 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T22:59:46.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:46.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:47.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:47.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:47.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:47.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:47.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:47.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:47.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:47 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:59:47.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:47 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:59:47.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:47 smithi086 ceph-mon[102754]: osdmap e176: 8 total, 8 up, 8 in 2024-06-14T22:59:47.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:47 smithi086 ceph-mon[102754]: pgmap v704: 89 pgs: 89 active+clean; 136 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 851 op/s 2024-06-14T22:59:47.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:47 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:59:47.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:47 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:59:47.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:47 smithi086 ceph-mon[97913]: osdmap e176: 8 total, 8 up, 8 in 2024-06-14T22:59:47.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:47 smithi086 ceph-mon[97913]: pgmap v704: 89 pgs: 89 active+clean; 136 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 851 op/s 2024-06-14T22:59:47.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:47.362 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:47.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:47.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:47.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:47.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:47.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:47 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T22:59:47.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:47 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T22:59:47.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:47 smithi137 ceph-mon[90634]: osdmap e176: 8 total, 8 up, 8 in 2024-06-14T22:59:47.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:47 smithi137 ceph-mon[90634]: pgmap v704: 89 pgs: 89 active+clean; 136 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 851 op/s 2024-06-14T22:59:47.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:47.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:47.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:47.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:47.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:47.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:47.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:47.831 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:47.831 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:47.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:47.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:47.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:47.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:47.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:47.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:47.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:48.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:48.108 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:48.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:48.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:48.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:48.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:48.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:48.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-14T22:59:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:48.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:48.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:48.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:48.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:48.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:48.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:48.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:48.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:48.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:48.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:48.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:48.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:48.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:49.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:49.027 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:49.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:49.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:49.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:49.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:49.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:49.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:49.274 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:49.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:49.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:49.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:49.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:49.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T22:59:49.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:49.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:49.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:49.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:49.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:49.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:49.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:49.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:49.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:49.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:49 smithi086 ceph-mon[97913]: pgmap v705: 89 pgs: 89 active+clean; 76 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 853 op/s 2024-06-14T22:59:49.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:49 smithi086 ceph-mon[102754]: pgmap v705: 89 pgs: 89 active+clean; 76 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 853 op/s 2024-06-14T22:59:49.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:49.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:49.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:49.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:49 smithi137 ceph-mon[90634]: pgmap v705: 89 pgs: 89 active+clean; 76 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 853 op/s 2024-06-14T22:59:50.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:50.029 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:50.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:50.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:50.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:50.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:50.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:50.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:50.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:50.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:50.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:50.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:50.471 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:50.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:50.537 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:50.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:50.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:50.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:50.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:50.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:50.787 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:50.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:50.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:50.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:50.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:51.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:51.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:51.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:51.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:51.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:51.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:51.203 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T22:59:51.203 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T22:59:51.221 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T22:59:51.221 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T22:59:51.237 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T22:59:51.237 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T22:59:51.253 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T22:59:51.254 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T22:59:51.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:51.270 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T22:59:51.270 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T22:59:51.287 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T22:59:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T22:59:51.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:51.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:51.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:51.304 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T22:59:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T22:59:51.320 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T22:59:51.321 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T22:59:51.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:51.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:51.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:51.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:51.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:51.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:51.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:51.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:51.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:51.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:51.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:51.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:51.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:51.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:51.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:51.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:51 smithi086 ceph-mon[97913]: pgmap v706: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 647 op/s 2024-06-14T22:59:51.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1082693237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:51.993 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:51 smithi086 ceph-mon[102754]: pgmap v706: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 647 op/s 2024-06-14T22:59:51.993 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1082693237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:51.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:51 smithi137 ceph-mon[90634]: pgmap v706: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 647 op/s 2024-06-14T22:59:51.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1082693237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T22:59:52.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:52.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:52.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:52.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:52.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:52.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:52.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:52.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:52.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:52.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:52.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:52.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:52.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:52.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:52.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:52.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:52.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:52.658 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:52.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:52.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:52.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:52.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:52.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:52.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:53.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:53.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:53.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:53.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:53.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:53.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:53.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:53.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:53.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:53.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:53.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:53.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:53.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:53.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:53.519 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:53.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:53.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:53.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:53.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:53.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:53.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:53.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:53 smithi086 ceph-mon[97913]: pgmap v707: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 535 op/s 2024-06-14T22:59:53.784 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:53 smithi086 ceph-mon[102754]: pgmap v707: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 535 op/s 2024-06-14T22:59:53.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:53.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:53.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:53.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:53.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:53.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:53.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:53 smithi137 ceph-mon[90634]: pgmap v707: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 535 op/s 2024-06-14T22:59:54.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:54.039 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:54.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:54.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:54.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:54.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:54.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:54.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:54.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:54.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:54.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T22:59:54.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:54.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:54.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:54.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:54.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:54.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:54.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:54.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:54.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:54.727 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T22:59:54.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:54.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:54.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:54.849 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T22:59:54.850 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T22:59:54.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:54.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:54.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:54.963 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2036388360' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:54.964 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2036388360' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:54.964 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 22:59:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:22:59:54] "GET /metrics HTTP/1.1" 200 36534 "" "Prometheus/2.43.0" 2024-06-14T22:59:54.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:54 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2036388360' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T22:59:55.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:55.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:55.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:55.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:55.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:55.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:55.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:55.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:55.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:55.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:55.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:55.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:55.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:55.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:55.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:55.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:55.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:55.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:55.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:55.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:55.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:55.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:55.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:55.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:55.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:55.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:55 smithi137 ceph-mon[90634]: pgmap v708: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 323 op/s 2024-06-14T22:59:55.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:55 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:59:55.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:55 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T22:59:55.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2036388360' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:59:55.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:55 smithi137 ceph-mon[90634]: osdmap e177: 8 total, 8 up, 8 in 2024-06-14T22:59:55.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:55 smithi137 ceph-mon[90634]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:59:56.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:56.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:56.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:56.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[97913]: pgmap v708: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 323 op/s 2024-06-14T22:59:56.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:59:56.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T22:59:56.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2036388360' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:59:56.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[97913]: osdmap e177: 8 total, 8 up, 8 in 2024-06-14T22:59:56.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[97913]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:59:56.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[102754]: pgmap v708: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 323 op/s 2024-06-14T22:59:56.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T22:59:56.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T22:59:56.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2036388360' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T22:59:56.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[102754]: osdmap e177: 8 total, 8 up, 8 in 2024-06-14T22:59:56.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:55 smithi086 ceph-mon[102754]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T22:59:56.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:56.174 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:56.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:56.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:56.245 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-06-14T22:59:54.725241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":487,"num_read_kb":913,"num_write":265260,"num_write_kb":8481102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143225,"ondisk_log_size":143225,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4524296,"kb_used_data":18164,"kb_used_omap":29,"kb_used_meta":4506018,"kb_avail":745436920,"statfs":{"total":767960285184,"available":763327406080,"internally_reserved":0,"allocated":18599936,"data_stored":11449326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":4614163109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-141819928,"num_objects":-2165,"num_object_clones":0,"num_object_copies":-6495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2165,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.654435"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":105,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771984+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T22:59:37.771984+0000","last_peered":"2024-06-14T22:59:37.771984+0000","last_clean":"2024-06-14T22:59:37.771984+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T22:59:37.771984+0000","last_undegraded":"2024-06-14T22:59:37.771984+0000","last_fullsized":"2024-06-14T22:59:37.771984+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":139,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.776427+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T22:59:37.776427+0000","last_peered":"2024-06-14T22:59:37.776427+0000","last_clean":"2024-06-14T22:59:37.776427+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T22:59:37.776427+0000","last_undegraded":"2024-06-14T22:59:37.776427+0000","last_fullsized":"2024-06-14T22:59:37.776427+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.772868+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T22:59:37.772868+0000","last_peered":"2024-06-14T22:59:37.772868+0000","last_clean":"2024-06-14T22:59:37.772868+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T22:59:37.772868+0000","last_undegraded":"2024-06-14T22:59:37.772868+0000","last_fullsized":"2024-06-14T22:59:37.772868+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":127,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.773831+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T22:59:37.773831+0000","last_peered":"2024-06-14T22:59:37.773831+0000","last_clean":"2024-06-14T22:59:37.773831+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T22:59:37.773831+0000","last_undegraded":"2024-06-14T22:59:37.773831+0000","last_fullsized":"2024-06-14T22:59:37.773831+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":128,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774301+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T22:59:37.774301+0000","last_peered":"2024-06-14T22:59:37.774301+0000","last_clean":"2024-06-14T22:59:37.774301+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T22:59:37.774301+0000","last_undegraded":"2024-06-14T22:59:37.774301+0000","last_fullsized":"2024-06-14T22:59:37.774301+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770521+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T22:59:37.770521+0000","last_peered":"2024-06-14T22:59:37.770521+0000","last_clean":"2024-06-14T22:59:37.770521+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T22:59:37.770521+0000","last_undegraded":"2024-06-14T22:59:37.770521+0000","last_fullsized":"2024-06-14T22:59:37.770521+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":129,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.779703+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T22:59:37.779703+0000","last_peered":"2024-06-14T22:59:37.779703+0000","last_clean":"2024-06-14T22:59:37.779703+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T22:59:37.779703+0000","last_undegraded":"2024-06-14T22:59:37.779703+0000","last_fullsized":"2024-06-14T22:59:37.779703+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":145,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.778171+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T22:59:37.778171+0000","last_peered":"2024-06-14T22:59:37.778171+0000","last_clean":"2024-06-14T22:59:37.778171+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T22:59:37.778171+0000","last_undegraded":"2024-06-14T22:59:37.778171+0000","last_fullsized":"2024-06-14T22:59:37.778171+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"176'16990","reported_seq":17037,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.208817+0000","last_change":"2024-06-14T22:55:38.730745+0000","last_active":"2024-06-14T22:59:46.208817+0000","last_peered":"2024-06-14T22:59:46.208817+0000","last_clean":"2024-06-14T22:59:46.208817+0000","last_became_active":"2024-06-14T22:55:38.730320+0000","last_became_peered":"2024-06-14T22:55:38.730320+0000","last_unstale":"2024-06-14T22:59:46.208817+0000","last_undegraded":"2024-06-14T22:59:46.208817+0000","last_fullsized":"2024-06-14T22:59:46.208817+0000","mapping_epoch":140,"log_start":"151'6900","ondisk_log_start":"151'6900","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:29.533721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16990,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":126,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.776302+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T22:59:37.776302+0000","last_peered":"2024-06-14T22:59:37.776302+0000","last_clean":"2024-06-14T22:59:37.776302+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T22:59:37.776302+0000","last_undegraded":"2024-06-14T22:59:37.776302+0000","last_fullsized":"2024-06-14T22:59:37.776302+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":119,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770689+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T22:59:37.770689+0000","last_peered":"2024-06-14T22:59:37.770689+0000","last_clean":"2024-06-14T22:59:37.770689+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T22:59:37.770689+0000","last_undegraded":"2024-06-14T22:59:37.770689+0000","last_fullsized":"2024-06-14T22:59:37.770689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"176'16344","reported_seq":16391,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.216328+0000","last_change":"2024-06-14T22:55:38.730175+0000","last_active":"2024-06-14T22:59:46.216328+0000","last_peered":"2024-06-14T22:59:46.216328+0000","last_clean":"2024-06-14T22:59:46.216328+0000","last_became_active":"2024-06-14T22:55:38.729339+0000","last_became_peered":"2024-06-14T22:55:38.729339+0000","last_unstale":"2024-06-14T22:59:46.216328+0000","last_undegraded":"2024-06-14T22:59:46.216328+0000","last_fullsized":"2024-06-14T22:59:46.216328+0000","mapping_epoch":140,"log_start":"157'9000","ondisk_log_start":"157'9000","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7344,"log_dups_size":0,"ondisk_log_size":7344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:40:50.307712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16344,"num_write_kb":523008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":127,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.776254+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T22:59:37.776254+0000","last_peered":"2024-06-14T22:59:37.776254+0000","last_clean":"2024-06-14T22:59:37.776254+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T22:59:37.776254+0000","last_undegraded":"2024-06-14T22:59:37.776254+0000","last_fullsized":"2024-06-14T22:59:37.776254+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":153,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771071+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T22:59:37.771071+0000","last_peered":"2024-06-14T22:59:37.771071+0000","last_clean":"2024-06-14T22:59:37.771071+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T22:59:37.771071+0000","last_undegraded":"2024-06-14T22:59:37.771071+0000","last_fullsized":"2024-06-14T22:59:37.771071+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"176'16446","reported_seq":16490,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.215434+0000","last_change":"2024-06-14T22:55:38.730856+0000","last_active":"2024-06-14T22:59:46.215434+0000","last_peered":"2024-06-14T22:59:46.215434+0000","last_clean":"2024-06-14T22:59:46.215434+0000","last_became_active":"2024-06-14T22:55:38.730542+0000","last_became_peered":"2024-06-14T22:55:38.730542+0000","last_unstale":"2024-06-14T22:59:46.215434+0000","last_undegraded":"2024-06-14T22:59:46.215434+0000","last_fullsized":"2024-06-14T22:59:46.215434+0000","mapping_epoch":140,"log_start":"151'6700","ondisk_log_start":"151'6700","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:16:51.828869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16446,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":54,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.772668+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T22:59:37.772668+0000","last_peered":"2024-06-14T22:59:37.772668+0000","last_clean":"2024-06-14T22:59:37.772668+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T22:59:37.772668+0000","last_undegraded":"2024-06-14T22:59:37.772668+0000","last_fullsized":"2024-06-14T22:59:37.772668+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.777959+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T22:59:37.777959+0000","last_peered":"2024-06-14T22:59:37.777959+0000","last_clean":"2024-06-14T22:59:37.777959+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T22:59:37.777959+0000","last_undegraded":"2024-06-14T22:59:37.777959+0000","last_fullsized":"2024-06-14T22:59:37.777959+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"176'16588","reported_seq":16632,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.247651+0000","last_change":"2024-06-14T22:55:39.130045+0000","last_active":"2024-06-14T22:59:46.247651+0000","last_peered":"2024-06-14T22:59:46.247651+0000","last_clean":"2024-06-14T22:59:46.247651+0000","last_became_active":"2024-06-14T22:55:39.129733+0000","last_became_peered":"2024-06-14T22:55:39.129733+0000","last_unstale":"2024-06-14T22:59:46.247651+0000","last_undegraded":"2024-06-14T22:59:46.247651+0000","last_fullsized":"2024-06-14T22:59:46.247651+0000","mapping_epoch":140,"log_start":"151'6900","ondisk_log_start":"151'6900","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9688,"log_dups_size":0,"ondisk_log_size":9688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:34:14.279180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16588,"num_write_kb":530816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770865+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T22:59:37.770865+0000","last_peered":"2024-06-14T22:59:37.770865+0000","last_clean":"2024-06-14T22:59:37.770865+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T22:59:37.770865+0000","last_undegraded":"2024-06-14T22:59:37.770865+0000","last_fullsized":"2024-06-14T22:59:37.770865+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":128,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.778205+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T22:59:37.778205+0000","last_peered":"2024-06-14T22:59:37.778205+0000","last_clean":"2024-06-14T22:59:37.778205+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T22:59:37.778205+0000","last_undegraded":"2024-06-14T22:59:37.778205+0000","last_fullsized":"2024-06-14T22:59:37.778205+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"176'16432","reported_seq":16485,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.175187+0000","last_change":"2024-06-14T22:58:09.809570+0000","last_active":"2024-06-14T22:59:46.175187+0000","last_peered":"2024-06-14T22:59:46.175187+0000","last_clean":"2024-06-14T22:59:46.175187+0000","last_became_active":"2024-06-14T22:58:09.753927+0000","last_became_peered":"2024-06-14T22:58:09.753927+0000","last_unstale":"2024-06-14T22:59:46.175187+0000","last_undegraded":"2024-06-14T22:59:46.175187+0000","last_fullsized":"2024-06-14T22:59:46.175187+0000","mapping_epoch":163,"log_start":"151'6400","ondisk_log_start":"151'6400","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:46:54.799901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16432,"num_write_kb":525824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":44,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.777903+0000","last_change":"2024-06-14T22:58:09.752985+0000","last_active":"2024-06-14T22:59:37.777903+0000","last_peered":"2024-06-14T22:59:37.777903+0000","last_clean":"2024-06-14T22:59:37.777903+0000","last_became_active":"2024-06-14T22:58:09.752630+0000","last_became_peered":"2024-06-14T22:58:09.752630+0000","last_unstale":"2024-06-14T22:59:37.777903+0000","last_undegraded":"2024-06-14T22:59:37.777903+0000","last_fullsized":"2024-06-14T22:59:37.777903+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:22.320986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771659+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T22:59:37.771659+0000","last_peered":"2024-06-14T22:59:37.771659+0000","last_clean":"2024-06-14T22:59:37.771659+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T22:59:37.771659+0000","last_undegraded":"2024-06-14T22:59:37.771659+0000","last_fullsized":"2024-06-14T22:59:37.771659+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"176'16726","reported_seq":16773,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.234051+0000","last_change":"2024-06-14T22:55:39.129787+0000","last_active":"2024-06-14T22:59:46.234051+0000","last_peered":"2024-06-14T22:59:46.234051+0000","last_clean":"2024-06-14T22:59:46.234051+0000","last_became_active":"2024-06-14T22:55:39.129528+0000","last_became_peered":"2024-06-14T22:55:39.129528+0000","last_unstale":"2024-06-14T22:59:46.234051+0000","last_undegraded":"2024-06-14T22:59:46.234051+0000","last_fullsized":"2024-06-14T22:59:46.234051+0000","mapping_epoch":140,"log_start":"158'9400","ondisk_log_start":"158'9400","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7326,"log_dups_size":0,"ondisk_log_size":7326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:24:05.362187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16726,"num_write_kb":535232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.778174+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T22:59:37.778174+0000","last_peered":"2024-06-14T22:59:37.778174+0000","last_clean":"2024-06-14T22:59:37.778174+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T22:59:37.778174+0000","last_undegraded":"2024-06-14T22:59:37.778174+0000","last_fullsized":"2024-06-14T22:59:37.778174+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":146,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771573+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T22:59:37.771573+0000","last_peered":"2024-06-14T22:59:37.771573+0000","last_clean":"2024-06-14T22:59:37.771573+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T22:59:37.771573+0000","last_undegraded":"2024-06-14T22:59:37.771573+0000","last_fullsized":"2024-06-14T22:59:37.771573+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"176'16484","reported_seq":16553,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.206805+0000","last_change":"2024-06-14T22:58:32.802664+0000","last_active":"2024-06-14T22:59:46.206805+0000","last_peered":"2024-06-14T22:59:46.206805+0000","last_clean":"2024-06-14T22:59:46.206805+0000","last_became_active":"2024-06-14T22:58:32.753751+0000","last_became_peered":"2024-06-14T22:58:32.753751+0000","last_unstale":"2024-06-14T22:59:46.206805+0000","last_undegraded":"2024-06-14T22:59:46.206805+0000","last_fullsized":"2024-06-14T22:59:46.206805+0000","mapping_epoch":167,"log_start":"156'8730","ondisk_log_start":"156'8730","created":140,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7754,"log_dups_size":0,"ondisk_log_size":7754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:17:54.717972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16484,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771142+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T22:59:37.771142+0000","last_peered":"2024-06-14T22:59:37.771142+0000","last_clean":"2024-06-14T22:59:37.771142+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T22:59:37.771142+0000","last_undegraded":"2024-06-14T22:59:37.771142+0000","last_fullsized":"2024-06-14T22:59:37.771142+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.773340+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T22:59:37.773340+0000","last_peered":"2024-06-14T22:59:37.773340+0000","last_clean":"2024-06-14T22:59:37.773340+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T22:59:37.773340+0000","last_undegraded":"2024-06-14T22:59:37.773340+0000","last_fullsized":"2024-06-14T22:59:37.773340+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"176'16740","reported_seq":16794,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.194794+0000","last_change":"2024-06-14T22:58:09.794739+0000","last_active":"2024-06-14T22:59:46.194794+0000","last_peered":"2024-06-14T22:59:46.194794+0000","last_clean":"2024-06-14T22:59:46.194794+0000","last_became_active":"2024-06-14T22:58:09.755744+0000","last_became_peered":"2024-06-14T22:58:09.755744+0000","last_unstale":"2024-06-14T22:59:46.194794+0000","last_undegraded":"2024-06-14T22:59:46.194794+0000","last_fullsized":"2024-06-14T22:59:46.194794+0000","mapping_epoch":163,"log_start":"158'9421","ondisk_log_start":"158'9421","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7319,"log_dups_size":0,"ondisk_log_size":7319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:51:51.355892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16740,"num_write_kb":535680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":130,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.779448+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T22:59:37.779448+0000","last_peered":"2024-06-14T22:59:37.779448+0000","last_clean":"2024-06-14T22:59:37.779448+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T22:59:37.779448+0000","last_undegraded":"2024-06-14T22:59:37.779448+0000","last_fullsized":"2024-06-14T22:59:37.779448+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":143,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771290+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T22:59:37.771290+0000","last_peered":"2024-06-14T22:59:37.771290+0000","last_clean":"2024-06-14T22:59:37.771290+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T22:59:37.771290+0000","last_undegraded":"2024-06-14T22:59:37.771290+0000","last_fullsized":"2024-06-14T22:59:37.771290+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"176'16320","reported_seq":16373,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.201408+0000","last_change":"2024-06-14T22:58:09.784892+0000","last_active":"2024-06-14T22:59:46.201408+0000","last_peered":"2024-06-14T22:59:46.201408+0000","last_clean":"2024-06-14T22:59:46.201408+0000","last_became_active":"2024-06-14T22:58:09.752020+0000","last_became_peered":"2024-06-14T22:58:09.752020+0000","last_unstale":"2024-06-14T22:59:46.201408+0000","last_undegraded":"2024-06-14T22:59:46.201408+0000","last_fullsized":"2024-06-14T22:59:46.201408+0000","mapping_epoch":163,"log_start":"151'6300","ondisk_log_start":"151'6300","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:53.007432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16320,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":442,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774532+0000","last_change":"2024-06-14T22:55:53.665022+0000","last_active":"2024-06-14T22:59:37.774532+0000","last_peered":"2024-06-14T22:59:37.774532+0000","last_clean":"2024-06-14T22:59:37.774532+0000","last_became_active":"2024-06-14T22:55:53.664025+0000","last_became_peered":"2024-06-14T22:55:53.664025+0000","last_unstale":"2024-06-14T22:59:37.774532+0000","last_undegraded":"2024-06-14T22:59:37.774532+0000","last_fullsized":"2024-06-14T22:59:37.774532+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.106147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":141,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.776102+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T22:59:37.776102+0000","last_peered":"2024-06-14T22:59:37.776102+0000","last_clean":"2024-06-14T22:59:37.776102+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T22:59:37.776102+0000","last_undegraded":"2024-06-14T22:59:37.776102+0000","last_fullsized":"2024-06-14T22:59:37.776102+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":80,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770831+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T22:59:37.770831+0000","last_peered":"2024-06-14T22:59:37.770831+0000","last_clean":"2024-06-14T22:59:37.770831+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T22:59:37.770831+0000","last_undegraded":"2024-06-14T22:59:37.770831+0000","last_fullsized":"2024-06-14T22:59:37.770831+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":119,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.778628+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T22:59:37.778628+0000","last_peered":"2024-06-14T22:59:37.778628+0000","last_clean":"2024-06-14T22:59:37.778628+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T22:59:37.778628+0000","last_undegraded":"2024-06-14T22:59:37.778628+0000","last_fullsized":"2024-06-14T22:59:37.778628+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"176'16722","reported_seq":16768,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.176317+0000","last_change":"2024-06-14T22:55:38.974930+0000","last_active":"2024-06-14T22:59:46.176317+0000","last_peered":"2024-06-14T22:59:46.176317+0000","last_clean":"2024-06-14T22:59:46.176317+0000","last_became_active":"2024-06-14T22:55:38.974625+0000","last_became_peered":"2024-06-14T22:55:38.974625+0000","last_unstale":"2024-06-14T22:59:46.176317+0000","last_undegraded":"2024-06-14T22:59:46.176317+0000","last_fullsized":"2024-06-14T22:59:46.176317+0000","mapping_epoch":140,"log_start":"151'6700","ondisk_log_start":"151'6700","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:44:40.336597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16722,"num_write_kb":535104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":133,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771018+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T22:59:37.771018+0000","last_peered":"2024-06-14T22:59:37.771018+0000","last_clean":"2024-06-14T22:59:37.771018+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T22:59:37.771018+0000","last_undegraded":"2024-06-14T22:59:37.771018+0000","last_fullsized":"2024-06-14T22:59:37.771018+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":128,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774010+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T22:59:37.774010+0000","last_peered":"2024-06-14T22:59:37.774010+0000","last_clean":"2024-06-14T22:59:37.774010+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T22:59:37.774010+0000","last_undegraded":"2024-06-14T22:59:37.774010+0000","last_fullsized":"2024-06-14T22:59:37.774010+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":141,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774346+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T22:59:37.774346+0000","last_peered":"2024-06-14T22:59:37.774346+0000","last_clean":"2024-06-14T22:59:37.774346+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T22:59:37.774346+0000","last_undegraded":"2024-06-14T22:59:37.774346+0000","last_fullsized":"2024-06-14T22:59:37.774346+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"176'16650","reported_seq":16694,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.199996+0000","last_change":"2024-06-14T22:55:38.731601+0000","last_active":"2024-06-14T22:59:46.199996+0000","last_peered":"2024-06-14T22:59:46.199996+0000","last_clean":"2024-06-14T22:59:46.199996+0000","last_became_active":"2024-06-14T22:55:38.731200+0000","last_became_peered":"2024-06-14T22:55:38.731200+0000","last_unstale":"2024-06-14T22:59:46.199996+0000","last_undegraded":"2024-06-14T22:59:46.199996+0000","last_fullsized":"2024-06-14T22:59:46.199996+0000","mapping_epoch":140,"log_start":"151'6900","ondisk_log_start":"151'6900","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9750,"log_dups_size":0,"ondisk_log_size":9750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:53:02.395452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16650,"num_write_kb":532800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:45.203773+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T22:59:45.203773+0000","last_peered":"2024-06-14T22:59:45.203773+0000","last_clean":"2024-06-14T22:59:45.203773+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T22:59:45.203773+0000","last_undegraded":"2024-06-14T22:59:45.203773+0000","last_fullsized":"2024-06-14T22:59:45.203773+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":144,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771194+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T22:59:37.771194+0000","last_peered":"2024-06-14T22:59:37.771194+0000","last_clean":"2024-06-14T22:59:37.771194+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T22:59:37.771194+0000","last_undegraded":"2024-06-14T22:59:37.771194+0000","last_fullsized":"2024-06-14T22:59:37.771194+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":128,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771058+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T22:59:37.771058+0000","last_peered":"2024-06-14T22:59:37.771058+0000","last_clean":"2024-06-14T22:59:37.771058+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T22:59:37.771058+0000","last_undegraded":"2024-06-14T22:59:37.771058+0000","last_fullsized":"2024-06-14T22:59:37.771058+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"176'16464","reported_seq":16510,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.227409+0000","last_change":"2024-06-14T22:55:38.729565+0000","last_active":"2024-06-14T22:59:46.227409+0000","last_peered":"2024-06-14T22:59:46.227409+0000","last_clean":"2024-06-14T22:59:46.227409+0000","last_became_active":"2024-06-14T22:55:38.729284+0000","last_became_peered":"2024-06-14T22:55:38.729284+0000","last_unstale":"2024-06-14T22:59:46.227409+0000","last_undegraded":"2024-06-14T22:59:46.227409+0000","last_fullsized":"2024-06-14T22:59:46.227409+0000","mapping_epoch":140,"log_start":"151'6400","ondisk_log_start":"151'6400","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:15.355340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16464,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":141,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770898+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T22:59:37.770898+0000","last_peered":"2024-06-14T22:59:37.770898+0000","last_clean":"2024-06-14T22:59:37.770898+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T22:59:37.770898+0000","last_undegraded":"2024-06-14T22:59:37.770898+0000","last_fullsized":"2024-06-14T22:59:37.770898+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771574+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T22:59:37.771574+0000","last_peered":"2024-06-14T22:59:37.771574+0000","last_clean":"2024-06-14T22:59:37.771574+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T22:59:37.771574+0000","last_undegraded":"2024-06-14T22:59:37.771574+0000","last_fullsized":"2024-06-14T22:59:37.771574+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":159,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.079775+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T22:59:46.079775+0000","last_peered":"2024-06-14T22:59:46.079775+0000","last_clean":"2024-06-14T22:59:46.079775+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T22:59:46.079775+0000","last_undegraded":"2024-06-14T22:59:46.079775+0000","last_fullsized":"2024-06-14T22:59:46.079775+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"176'16522","reported_seq":16566,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.228486+0000","last_change":"2024-06-14T22:55:38.729989+0000","last_active":"2024-06-14T22:59:46.228486+0000","last_peered":"2024-06-14T22:59:46.228486+0000","last_clean":"2024-06-14T22:59:46.228486+0000","last_became_active":"2024-06-14T22:55:38.729695+0000","last_became_peered":"2024-06-14T22:55:38.729695+0000","last_unstale":"2024-06-14T22:59:46.228486+0000","last_undegraded":"2024-06-14T22:59:46.228486+0000","last_fullsized":"2024-06-14T22:59:46.228486+0000","mapping_epoch":140,"log_start":"151'6800","ondisk_log_start":"151'6800","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:44:38.218727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16522,"num_write_kb":528704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":154,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.773070+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T22:59:37.773070+0000","last_peered":"2024-06-14T22:59:37.773070+0000","last_clean":"2024-06-14T22:59:37.773070+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T22:59:37.773070+0000","last_undegraded":"2024-06-14T22:59:37.773070+0000","last_fullsized":"2024-06-14T22:59:37.773070+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":134,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.079921+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T22:59:46.079921+0000","last_peered":"2024-06-14T22:59:46.079921+0000","last_clean":"2024-06-14T22:59:46.079921+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T22:59:46.079921+0000","last_undegraded":"2024-06-14T22:59:46.079921+0000","last_fullsized":"2024-06-14T22:59:46.079921+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":129,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.080405+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T22:59:46.080405+0000","last_peered":"2024-06-14T22:59:46.080405+0000","last_clean":"2024-06-14T22:59:46.080405+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T22:59:46.080405+0000","last_undegraded":"2024-06-14T22:59:46.080405+0000","last_fullsized":"2024-06-14T22:59:46.080405+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"176'16544","reported_seq":16597,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.232554+0000","last_change":"2024-06-14T22:55:53.672630+0000","last_active":"2024-06-14T22:59:46.232554+0000","last_peered":"2024-06-14T22:59:46.232554+0000","last_clean":"2024-06-14T22:59:46.232554+0000","last_became_active":"2024-06-14T22:55:53.663693+0000","last_became_peered":"2024-06-14T22:55:53.663693+0000","last_unstale":"2024-06-14T22:59:46.232554+0000","last_undegraded":"2024-06-14T22:59:46.232554+0000","last_fullsized":"2024-06-14T22:59:46.232554+0000","mapping_epoch":145,"log_start":"155'8200","ondisk_log_start":"155'8200","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:20:34.289185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16544,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":144,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.079849+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T22:59:46.079849+0000","last_peered":"2024-06-14T22:59:46.079849+0000","last_clean":"2024-06-14T22:59:46.079849+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T22:59:46.079849+0000","last_undegraded":"2024-06-14T22:59:46.079849+0000","last_fullsized":"2024-06-14T22:59:46.079849+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":141,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.778393+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T22:59:37.778393+0000","last_peered":"2024-06-14T22:59:37.778393+0000","last_clean":"2024-06-14T22:59:37.778393+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T22:59:37.778393+0000","last_undegraded":"2024-06-14T22:59:37.778393+0000","last_fullsized":"2024-06-14T22:59:37.778393+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":114,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.778073+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T22:59:37.778073+0000","last_peered":"2024-06-14T22:59:37.778073+0000","last_clean":"2024-06-14T22:59:37.778073+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T22:59:37.778073+0000","last_undegraded":"2024-06-14T22:59:37.778073+0000","last_fullsized":"2024-06-14T22:59:37.778073+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"176'16576","reported_seq":16630,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.233728+0000","last_change":"2024-06-14T22:55:53.668144+0000","last_active":"2024-06-14T22:59:46.233728+0000","last_peered":"2024-06-14T22:59:46.233728+0000","last_clean":"2024-06-14T22:59:46.233728+0000","last_became_active":"2024-06-14T22:55:53.663690+0000","last_became_peered":"2024-06-14T22:55:53.663690+0000","last_unstale":"2024-06-14T22:59:46.233728+0000","last_undegraded":"2024-06-14T22:59:46.233728+0000","last_fullsized":"2024-06-14T22:59:46.233728+0000","mapping_epoch":145,"log_start":"157'9200","ondisk_log_start":"157'9200","created":140,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:52:03.686394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16576,"num_write_kb":530369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":35,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774184+0000","last_change":"2024-06-14T22:58:09.752729+0000","last_active":"2024-06-14T22:59:37.774184+0000","last_peered":"2024-06-14T22:59:37.774184+0000","last_clean":"2024-06-14T22:59:37.774184+0000","last_became_active":"2024-06-14T22:58:09.752393+0000","last_became_peered":"2024-06-14T22:58:09.752393+0000","last_unstale":"2024-06-14T22:59:37.774184+0000","last_undegraded":"2024-06-14T22:59:37.774184+0000","last_fullsized":"2024-06-14T22:59:37.774184+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:21:58.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":127,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.775329+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T22:59:37.775329+0000","last_peered":"2024-06-14T22:59:37.775329+0000","last_clean":"2024-06-14T22:59:37.775329+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T22:59:37.775329+0000","last_undegraded":"2024-06-14T22:59:37.775329+0000","last_fullsized":"2024-06-14T22:59:37.775329+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":121,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.776036+0000","last_change":"2024-06-14T22:58:32.754079+0000","last_active":"2024-06-14T22:59:37.776036+0000","last_peered":"2024-06-14T22:59:37.776036+0000","last_clean":"2024-06-14T22:59:37.776036+0000","last_became_active":"2024-06-14T22:58:32.753814+0000","last_became_peered":"2024-06-14T22:58:32.753814+0000","last_unstale":"2024-06-14T22:59:37.776036+0000","last_undegraded":"2024-06-14T22:59:37.776036+0000","last_fullsized":"2024-06-14T22:59:37.776036+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:37:07.414370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"176'16412","reported_seq":16466,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.191801+0000","last_change":"2024-06-14T22:58:09.804712+0000","last_active":"2024-06-14T22:59:46.191801+0000","last_peered":"2024-06-14T22:59:46.191801+0000","last_clean":"2024-06-14T22:59:46.191801+0000","last_became_active":"2024-06-14T22:58:09.757262+0000","last_became_peered":"2024-06-14T22:58:09.757262+0000","last_unstale":"2024-06-14T22:59:46.191801+0000","last_undegraded":"2024-06-14T22:59:46.191801+0000","last_fullsized":"2024-06-14T22:59:46.191801+0000","mapping_epoch":163,"log_start":"154'8000","ondisk_log_start":"154'8000","created":140,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:55:37.708585+0000","last_clean_scrub_stamp":"2024-06-14T22:55:37.708585+0000","objects_scrubbed":0,"log_size":8412,"log_dups_size":0,"ondisk_log_size":8412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:28.952637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16412,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":131,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774074+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T22:59:37.774074+0000","last_peered":"2024-06-14T22:59:37.774074+0000","last_clean":"2024-06-14T22:59:37.774074+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T22:59:37.774074+0000","last_undegraded":"2024-06-14T22:59:37.774074+0000","last_fullsized":"2024-06-14T22:59:37.774074+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":141,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.778833+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T22:59:37.778833+0000","last_peered":"2024-06-14T22:59:37.778833+0000","last_clean":"2024-06-14T22:59:37.778833+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T22:59:37.778833+0000","last_undegraded":"2024-06-14T22:59:37.778833+0000","last_fullsized":"2024-06-14T22:59:37.778833+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":119,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770937+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T22:59:37.770937+0000","last_peered":"2024-06-14T22:59:37.770937+0000","last_clean":"2024-06-14T22:59:37.770937+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T22:59:37.770937+0000","last_undegraded":"2024-06-14T22:59:37.770937+0000","last_fullsized":"2024-06-14T22:59:37.770937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771502+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T22:59:37.771502+0000","last_peered":"2024-06-14T22:59:37.771502+0000","last_clean":"2024-06-14T22:59:37.771502+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T22:59:37.771502+0000","last_undegraded":"2024-06-14T22:59:37.771502+0000","last_fullsized":"2024-06-14T22:59:37.771502+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":142,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771812+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T22:59:37.771812+0000","last_peered":"2024-06-14T22:59:37.771812+0000","last_clean":"2024-06-14T22:59:37.771812+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T22:59:37.771812+0000","last_undegraded":"2024-06-14T22:59:37.771812+0000","last_fullsized":"2024-06-14T22:59:37.771812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":143,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771261+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T22:59:37.771261+0000","last_peered":"2024-06-14T22:59:37.771261+0000","last_clean":"2024-06-14T22:59:37.771261+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T22:59:37.771261+0000","last_undegraded":"2024-06-14T22:59:37.771261+0000","last_fullsized":"2024-06-14T22:59:37.771261+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":54,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.777899+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T22:59:37.777899+0000","last_peered":"2024-06-14T22:59:37.777899+0000","last_clean":"2024-06-14T22:59:37.777899+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T22:59:37.777899+0000","last_undegraded":"2024-06-14T22:59:37.777899+0000","last_fullsized":"2024-06-14T22:59:37.777899+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":137,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771397+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T22:59:37.771397+0000","last_peered":"2024-06-14T22:59:37.771397+0000","last_clean":"2024-06-14T22:59:37.771397+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T22:59:37.771397+0000","last_undegraded":"2024-06-14T22:59:37.771397+0000","last_fullsized":"2024-06-14T22:59:37.771397+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":80,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.777885+0000","last_change":"2024-06-14T22:58:09.753103+0000","last_active":"2024-06-14T22:59:37.777885+0000","last_peered":"2024-06-14T22:59:37.777885+0000","last_clean":"2024-06-14T22:59:37.777885+0000","last_became_active":"2024-06-14T22:58:09.752761+0000","last_became_peered":"2024-06-14T22:58:09.752761+0000","last_unstale":"2024-06-14T22:59:37.777885+0000","last_undegraded":"2024-06-14T22:59:37.777885+0000","last_fullsized":"2024-06-14T22:59:37.777885+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:20:20.619331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":127,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774095+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T22:59:37.774095+0000","last_peered":"2024-06-14T22:59:37.774095+0000","last_clean":"2024-06-14T22:59:37.774095+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T22:59:37.774095+0000","last_undegraded":"2024-06-14T22:59:37.774095+0000","last_fullsized":"2024-06-14T22:59:37.774095+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":129,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.779214+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T22:59:37.779214+0000","last_peered":"2024-06-14T22:59:37.779214+0000","last_clean":"2024-06-14T22:59:37.779214+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T22:59:37.779214+0000","last_undegraded":"2024-06-14T22:59:37.779214+0000","last_fullsized":"2024-06-14T22:59:37.779214+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.778634+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T22:59:37.778634+0000","last_peered":"2024-06-14T22:59:37.778634+0000","last_clean":"2024-06-14T22:59:37.778634+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T22:59:37.778634+0000","last_undegraded":"2024-06-14T22:59:37.778634+0000","last_fullsized":"2024-06-14T22:59:37.778634+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":123,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774116+0000","last_change":"2024-06-14T22:58:09.785074+0000","last_active":"2024-06-14T22:59:37.774116+0000","last_peered":"2024-06-14T22:59:37.774116+0000","last_clean":"2024-06-14T22:59:37.774116+0000","last_became_active":"2024-06-14T22:58:09.784309+0000","last_became_peered":"2024-06-14T22:58:09.784309+0000","last_unstale":"2024-06-14T22:59:37.774116+0000","last_undegraded":"2024-06-14T22:59:37.774116+0000","last_fullsized":"2024-06-14T22:59:37.774116+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:23.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":129,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774392+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T22:59:37.774392+0000","last_peered":"2024-06-14T22:59:37.774392+0000","last_clean":"2024-06-14T22:59:37.774392+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T22:59:37.774392+0000","last_undegraded":"2024-06-14T22:59:37.774392+0000","last_fullsized":"2024-06-14T22:59:37.774392+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":143,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771792+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T22:59:37.771792+0000","last_peered":"2024-06-14T22:59:37.771792+0000","last_clean":"2024-06-14T22:59:37.771792+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T22:59:37.771792+0000","last_undegraded":"2024-06-14T22:59:37.771792+0000","last_fullsized":"2024-06-14T22:59:37.771792+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":131,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.776157+0000","last_change":"2024-06-14T22:58:09.756040+0000","last_active":"2024-06-14T22:59:37.776157+0000","last_peered":"2024-06-14T22:59:37.776157+0000","last_clean":"2024-06-14T22:59:37.776157+0000","last_became_active":"2024-06-14T22:58:09.755659+0000","last_became_peered":"2024-06-14T22:58:09.755659+0000","last_unstale":"2024-06-14T22:59:37.776157+0000","last_undegraded":"2024-06-14T22:59:37.776157+0000","last_fullsized":"2024-06-14T22:59:37.776157+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:40.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":155,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770866+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T22:59:37.770866+0000","last_peered":"2024-06-14T22:59:37.770866+0000","last_clean":"2024-06-14T22:59:37.770866+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T22:59:37.770866+0000","last_undegraded":"2024-06-14T22:59:37.770866+0000","last_fullsized":"2024-06-14T22:59:37.770866+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":148,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.775846+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T22:59:37.775846+0000","last_peered":"2024-06-14T22:59:37.775846+0000","last_clean":"2024-06-14T22:59:37.775846+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T22:59:37.775846+0000","last_undegraded":"2024-06-14T22:59:37.775846+0000","last_fullsized":"2024-06-14T22:59:37.775846+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":116,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-06-14T22:59:46.075555+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T22:59:46.075555+0000","last_peered":"2024-06-14T22:59:46.075555+0000","last_clean":"2024-06-14T22:59:46.075555+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T22:59:46.075555+0000","last_undegraded":"2024-06-14T22:59:46.075555+0000","last_fullsized":"2024-06-14T22:59:46.075555+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":153,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.771438+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T22:59:37.771438+0000","last_peered":"2024-06-14T22:59:37.771438+0000","last_clean":"2024-06-14T22:59:37.771438+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T22:59:37.771438+0000","last_undegraded":"2024-06-14T22:59:37.771438+0000","last_fullsized":"2024-06-14T22:59:37.771438+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":139,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770740+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T22:59:37.770740+0000","last_peered":"2024-06-14T22:59:37.770740+0000","last_clean":"2024-06-14T22:59:37.770740+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T22:59:37.770740+0000","last_undegraded":"2024-06-14T22:59:37.770740+0000","last_fullsized":"2024-06-14T22:59:37.770740+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":153,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770780+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T22:59:37.770780+0000","last_peered":"2024-06-14T22:59:37.770780+0000","last_clean":"2024-06-14T22:59:37.770780+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T22:59:37.770780+0000","last_undegraded":"2024-06-14T22:59:37.770780+0000","last_fullsized":"2024-06-14T22:59:37.770780+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":122,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.772960+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T22:59:37.772960+0000","last_peered":"2024-06-14T22:59:37.772960+0000","last_clean":"2024-06-14T22:59:37.772960+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T22:59:37.772960+0000","last_undegraded":"2024-06-14T22:59:37.772960+0000","last_fullsized":"2024-06-14T22:59:37.772960+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":138,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.775985+0000","last_change":"2024-06-14T22:58:09.752281+0000","last_active":"2024-06-14T22:59:37.775985+0000","last_peered":"2024-06-14T22:59:37.775985+0000","last_clean":"2024-06-14T22:59:37.775985+0000","last_became_active":"2024-06-14T22:58:09.751952+0000","last_became_peered":"2024-06-14T22:58:09.751952+0000","last_unstale":"2024-06-14T22:59:37.775985+0000","last_undegraded":"2024-06-14T22:59:37.775985+0000","last_fullsized":"2024-06-14T22:59:37.775985+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:39:19.836734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":35,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.778731+0000","last_change":"2024-06-14T22:58:09.755622+0000","last_active":"2024-06-14T22:59:37.778731+0000","last_peered":"2024-06-14T22:59:37.778731+0000","last_clean":"2024-06-14T22:59:37.778731+0000","last_became_active":"2024-06-14T22:58:09.755383+0000","last_became_peered":"2024-06-14T22:58:09.755383+0000","last_unstale":"2024-06-14T22:59:37.778731+0000","last_undegraded":"2024-06-14T22:59:37.778731+0000","last_fullsized":"2024-06-14T22:59:37.778731+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:13.884346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":136,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.770446+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T22:59:37.770446+0000","last_peered":"2024-06-14T22:59:37.770446+0000","last_clean":"2024-06-14T22:59:37.770446+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T22:59:37.770446+0000","last_undegraded":"2024-06-14T22:59:37.770446+0000","last_fullsized":"2024-06-14T22:59:37.770446+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":132,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-06-14T22:59:37.774089+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T22:59:37.774089+0000","last_peered":"2024-06-14T22:59:37.774089+0000","last_clean":"2024-06-14T22:59:37.774089+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T22:59:37.774089+0000","last_undegraded":"2024-06-14T22:59:37.774089+0000","last_fullsized":"2024-06-14T22:59:37.774089+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":264960,"num_write_kb":8478657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143009,"ondisk_log_size":143009,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":486,"num_read_kb":912,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234310,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905700,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":903038,"kb_avail":92839452,"statfs":{"total":95995035648,"available":95067598848,"internally_reserved":0,"allocated":2715648,"data_stored":1803515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924711256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.897,"5min":1.0940000000000001,"15min":1.1479999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":6.9539999999999997,"5min":6.9539999999999997,"15min":9.3919999999999995},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.9319999999999999,"5min":1.0580000000000001,"15min":1.286},"min":{"1min":0.313,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.2130000000000001,"5min":5.2130000000000001,"15min":10.416},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:59:43 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":2.8380000000000001,"15min":1.833},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28100000000000003},"max":{"1min":5.7249999999999996,"5min":83.295000000000002,"15min":83.295000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":2.0449999999999999,"5min":2.8730000000000002,"15min":1.9159999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.223},"max":{"1min":6.3390000000000004,"5min":82.972999999999999,"15min":82.972999999999999},"last":0.51400000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:59:43 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.1080000000000001,"15min":0.75700000000000001},"min":{"1min":0.27100000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.9269999999999996,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.443},{"interface":"front","average":{"1min":1.369,"5min":1.3029999999999999,"15min":0.81499999999999995},"min":{"1min":0.31900000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":4.9000000000000004,"5min":8.1150000000000002,"15min":8.1150000000000002},"last":0.55900000000000005}]},{"osd":3,"last update":"Fri Jun 14 22:59:09 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":1.8740000000000001,"15min":0.98199999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.4930000000000003,"5min":55.683999999999997,"15min":55.683999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.907,"5min":1.8919999999999999,"15min":1.03},"min":{"1min":0.32100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.7629999999999999,"5min":55.533999999999999,"15min":55.533999999999999},"last":0.57899999999999996}]},{"osd":4,"last update":"Fri Jun 14 22:58:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":0.79400000000000004,"15min":0.67900000000000005},"min":{"1min":0.26200000000000001,"5min":0.245,"15min":0.22900000000000001},"max":{"1min":5.141,"5min":6.7610000000000001,"15min":6.7610000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.1259999999999999,"5min":0.96099999999999997,"15min":0.71699999999999997},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":4.9729999999999999,"5min":16.879000000000001,"15min":16.879000000000001},"last":0.53100000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:58:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0190000000000001,"5min":1.0820000000000001,"15min":0.78400000000000003},"min":{"1min":0.29899999999999999,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":6.9000000000000004,"5min":6.9000000000000004,"15min":6.9000000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.397,"5min":0.998,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":5.2649999999999997,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:58:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":0.93300000000000005,"15min":0.80000000000000004},"min":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.28100000000000003},"max":{"1min":5.5,"5min":5.5,"15min":5.806},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.89800000000000002,"15min":0.75700000000000001},"min":{"1min":0.374,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.8099999999999996,"5min":12.951000000000001,"15min":12.951000000000001},"last":0.502}]}]},{"osd":6,"up_from":46,"seq":197568495946,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600664,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":598014,"kb_avail":93144488,"statfs":{"total":95995035648,"available":95379955712,"internally_reserved":0,"allocated":2699264,"data_stored":1803046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612366680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.0620000000000001,"15min":1.284},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":5.6859999999999999,"5min":9.9350000000000005,"15min":31.358000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.222,"5min":0.90800000000000003,"15min":1.3420000000000001},"min":{"1min":0.27200000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.9989999999999997,"5min":6.0730000000000004,"15min":29.399000000000001},"last":0.55300000000000005}]},{"osd":1,"last update":"Fri Jun 14 22:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2330000000000001,"15min":1.2669999999999999},"min":{"1min":0.26400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.1280000000000001,"5min":6.6680000000000001,"15min":9.9559999999999995},"last":0.434},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.28,"15min":1.2769999999999999},"min":{"1min":0.30099999999999999,"5min":0.25,"15min":0.247},"max":{"1min":6.5510000000000002,"5min":6.7519999999999998,"15min":9.5510000000000002},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:59:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.91100000000000003,"15min":0.68600000000000005},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.544,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0089999999999999,"15min":0.74099999999999999},"min":{"1min":0.33400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.5030000000000001,"5min":6.6959999999999997,"15min":6.6959999999999997},"last":0.38}]},{"osd":3,"last update":"Fri Jun 14 22:59:12 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.575,"15min":0.85999999999999999},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.5449999999999999,"5min":29.152000000000001,"15min":29.152000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.579,"15min":0.90900000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":6.3040000000000003,"5min":25.864000000000001,"15min":25.864000000000001},"last":0.51800000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.71999999999999997,"15min":0.83099999999999996},"min":{"1min":0.307,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":2.7050000000000001,"5min":6.774,"15min":47.698},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.0569999999999999,"5min":0.746,"15min":0.79400000000000004},"min":{"1min":0.317,"5min":0.26800000000000002,"15min":0.23899999999999999},"max":{"1min":5.7930000000000001,"5min":9.9610000000000003,"15min":48.197000000000003},"last":0.50900000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:59:21 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.85699999999999998,"15min":0.753},"min":{"1min":0.373,"5min":0.29699999999999999,"15min":0.245},"max":{"1min":5.0750000000000002,"5min":5.8289999999999997,"15min":8.6500000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":0.95299999999999996,"15min":0.82799999999999996},"min":{"1min":0.38800000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.3769999999999998,"5min":7.3769999999999998,"15min":8.6349999999999998},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Jun 14 22:58:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":0.89100000000000001,"15min":0.79600000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":5.6790000000000003,"5min":5.9039999999999999,"15min":8.6950000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.478,"5min":0.88300000000000001,"15min":0.82499999999999996},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.28899999999999998},"max":{"1min":6.8620000000000001,"5min":6.8620000000000001,"15min":8.673},"last":0.63800000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757585,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893196,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":891134,"kb_avail":92851956,"statfs":{"total":95995035648,"available":95080402944,"internally_reserved":0,"allocated":2097152,"data_stored":1208492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912521560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.038,"15min":1.224},"min":{"1min":0.24099999999999999,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":4.5819999999999999,"5min":4.5819999999999999,"15min":17.766999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.0069999999999999,"15min":1.2210000000000001},"min":{"1min":0.28599999999999998,"5min":0.221,"15min":0.221},"max":{"1min":3.6600000000000001,"5min":4.3659999999999997,"15min":17.806000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Jun 14 22:59:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.105,"15min":1.26},"min":{"1min":0.27100000000000002,"5min":0.24199999999999999,"15min":0.23300000000000001},"max":{"1min":4.5250000000000004,"5min":5.6429999999999998,"15min":7.694},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.643,"5min":1.3169999999999999,"15min":1.2110000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":7.0700000000000003,"5min":7.0700000000000003,"15min":7.0700000000000003},"last":0.39900000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:59:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.05,"15min":0.71399999999999997},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.77,"5min":5.9379999999999997,"15min":5.9379999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.98099999999999998,"15min":0.70699999999999996},"min":{"1min":0.28100000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.6150000000000002,"5min":4.6050000000000004,"15min":4.6050000000000004},"last":0.504}]},{"osd":3,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":1.7709999999999999,"15min":0.94299999999999995},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":6.6029999999999998,"5min":45.744,"15min":45.744},"last":0.63600000000000001},{"interface":"front","average":{"1min":2.0129999999999999,"5min":1.9450000000000001,"15min":1.0740000000000001},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.5679999999999996,"5min":45.838000000000001,"15min":45.838000000000001},"last":0.58799999999999997}]},{"osd":4,"last update":"Fri Jun 14 22:59:07 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.72399999999999998,"15min":0.60199999999999998},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.214},"max":{"1min":5.5880000000000001,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.70499999999999996,"15min":0.64200000000000002},"min":{"1min":0.308,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.6800000000000002,"5min":3.6800000000000002,"15min":5.8109999999999999},"last":0.44700000000000001}]},{"osd":6,"last update":"Fri Jun 14 22:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.82599999999999996,"15min":0.76500000000000001},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":3.6320000000000001,"5min":6.931,"15min":9.6020000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.76100000000000001,"15min":0.73199999999999998},"min":{"1min":0.30099999999999999,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":4.3040000000000003,"5min":4.452,"15min":12.339},"last":0.69099999999999995}]},{"osd":7,"last update":"Fri Jun 14 22:58:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":0.95599999999999996,"15min":0.77700000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.3129999999999997,"5min":5.3129999999999997,"15min":5.3129999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.76,"5min":0.98599999999999999,"15min":0.76700000000000002},"min":{"1min":0.40400000000000003,"5min":0.29999999999999999,"15min":0.25600000000000001},"max":{"1min":5.6189999999999998,"5min":5.6189999999999998,"15min":10.638999999999999},"last":0.59799999999999998}]}]},{"osd":4,"up_from":30,"seq":128849019221,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896856,"kb_used_data":2064,"kb_used_omap":1,"kb_used_meta":894782,"kb_avail":92848296,"statfs":{"total":95995035648,"available":95076655104,"internally_reserved":0,"allocated":2113536,"data_stored":1212812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916257112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:23 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.79600000000000004,"15min":1.0229999999999999},"min":{"1min":0.26600000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.3580000000000001,"5min":4.5389999999999997,"15min":7.5650000000000004},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.996,"5min":0.75800000000000001,"15min":1.135},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":3.5800000000000001,"5min":3.718,"15min":9.1470000000000002},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:59:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.1830000000000001,"15min":1.1659999999999999},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.4589999999999996,"5min":8.4960000000000004,"15min":8.4960000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.2390000000000001,"15min":1.2809999999999999},"min":{"1min":0.26600000000000001,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":4.8529999999999998,"5min":6.5549999999999997,"15min":6.5549999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Jun 14 22:59:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.80600000000000005,"15min":0.64600000000000002},"min":{"1min":0.27200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4809999999999999,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.091,"5min":0.89000000000000001,"15min":0.68000000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.48,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Jun 14 22:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.113,"15min":0.73399999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.637,"5min":6.617,"15min":6.617},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.1439999999999999,"15min":0.77700000000000002},"min":{"1min":0.34399999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.6589999999999998,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Jun 14 22:59:02 2024","interfaces":[{"interface":"back","average":{"1min":8.4789999999999992,"5min":2.3170000000000002,"15min":1.2090000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.21199999999999999},"max":{"1min":139.13200000000001,"5min":139.13200000000001,"15min":139.13200000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":9.1820000000000004,"5min":2.4780000000000002,"15min":1.2529999999999999},"min":{"1min":0.28000000000000003,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":138.86000000000001,"5min":138.86000000000001,"15min":138.86000000000001},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Jun 14 22:59:28 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":0.745,"15min":0.64500000000000002},"min":{"1min":0.314,"5min":0.30199999999999999,"15min":0.254},"max":{"1min":3.6139999999999999,"5min":5.1210000000000004,"15min":5.1210000000000004},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.72599999999999998,"15min":0.61399999999999999},"min":{"1min":0.34799999999999998,"5min":0.28899999999999998,"15min":0.23400000000000001},"max":{"1min":3.3650000000000002,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.53300000000000003}]},{"osd":7,"last update":"Fri Jun 14 22:59:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":0.99299999999999999,"15min":0.71999999999999997},"min":{"1min":0.38700000000000001,"5min":0.30299999999999999,"15min":0.30099999999999999},"max":{"1min":3.9689999999999999,"5min":8.9489999999999998,"15min":8.9489999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.2070000000000001,"15min":0.83299999999999996},"min":{"1min":0.35599999999999998,"5min":0.308,"15min":0.29399999999999998},"max":{"1min":7.25,"5min":7.25,"15min":8.4730000000000008},"last":0.745}]}]},{"osd":3,"up_from":138,"seq":592705486920,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":220492,"kb_used_data":2048,"kb_used_omap":4,"kb_used_meta":218427,"kb_avail":93524660,"statfs":{"total":95995035648,"available":95769251840,"internally_reserved":0,"allocated":2097152,"data_stored":1208265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":223669726},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.022,"15min":0.55600000000000005},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.3360000000000003,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.80000000000000004,"15min":0.505},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.5950000000000002,"5min":3.7229999999999999,"15min":3.7229999999999999},"last":0.36799999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":2.5499999999999998,"5min":1.3480000000000001,"15min":0.68500000000000005},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":24.097999999999999,"5min":24.097999999999999,"15min":24.097999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":2.4500000000000002,"5min":1.2889999999999999,"15min":0.69199999999999995},"min":{"1min":0.313,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":24.132999999999999,"5min":24.132999999999999,"15min":24.132999999999999},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":0.93300000000000005,"15min":0.58499999999999996},"min":{"1min":0.251,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.9900000000000002,"5min":5.2910000000000004,"15min":5.2910000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":1.222,"5min":0.88900000000000001,"15min":0.53400000000000003},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.9900000000000002,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.315}]},{"osd":4,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.0169999999999999,"15min":0.75900000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.1189999999999998,"5min":5.7960000000000003,"15min":5.7960000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.393,"5min":1.1060000000000001,"15min":0.78800000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.6660000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.282,"15min":0.85199999999999998},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.9359999999999999,"5min":5.9359999999999999,"15min":5.9359999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.1599999999999999,"15min":0.79400000000000004},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":5.4260000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.0349999999999999,"15min":0.77400000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.6309999999999998,"5min":4.0449999999999999,"15min":4.0449999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.502,"5min":1.0289999999999999,"15min":0.751},"min":{"1min":0.32100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":5.0439999999999996,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.55200000000000005}]},{"osd":7,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.9730000000000001,"5min":1.3600000000000001,"15min":0.88700000000000001},"min":{"1min":0.38500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.7240000000000002,"5min":7.7240000000000002,"15min":7.7240000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.325,"15min":0.86299999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.8300000000000001,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.622}]}]},{"osd":2,"up_from":133,"seq":571230650446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99080,"kb_used_data":2048,"kb_used_omap":6,"kb_used_meta":97017,"kb_avail":93646072,"statfs":{"total":95995035648,"available":95893577728,"internally_reserved":0,"allocated":2097152,"data_stored":1208479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":99345909},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.75900000000000001,"15min":0.49299999999999999},"min":{"1min":0.20399999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":7.476,"5min":7.476,"15min":7.476},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.6919999999999999,"5min":0.92100000000000004,"15min":0.54700000000000004},"min":{"1min":0.27300000000000002,"5min":0.182,"15min":0.182},"max":{"1min":9.0869999999999997,"5min":9.0869999999999997,"15min":9.0869999999999997},"last":0.316}]},{"osd":1,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.3759999999999999,"15min":0.71199999999999997},"min":{"1min":0.22900000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.843,"5min":21.920000000000002,"15min":21.920000000000002},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.3400000000000001,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":13.054,"5min":21.573,"15min":21.573},"last":0.39200000000000002}]},{"osd":3,"last update":"Fri Jun 14 22:59:16 2024","interfaces":[{"interface":"back","average":{"1min":1.8540000000000001,"5min":1.323,"15min":0.68300000000000005},"min":{"1min":0.29199999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.1689999999999996,"5min":21.66,"15min":21.66},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.2869999999999999,"15min":0.69099999999999995},"min":{"1min":0.255,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.8600000000000003,"5min":22.341000000000001,"15min":22.341000000000001},"last":0.375}]},{"osd":4,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.0629999999999999,"15min":0.74299999999999999},"min":{"1min":0.45400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":7.8019999999999996,"5min":7.8019999999999996,"15min":7.8019999999999996},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.8069999999999999,"5min":1.0249999999999999,"15min":0.72699999999999998},"min":{"1min":0.30099999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.7839999999999998,"5min":7.7839999999999998,"15min":7.7839999999999998},"last":0.63700000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.0289999999999999,"15min":0.71599999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.218,"5min":5.218,"15min":5.218},"last":0.626},{"interface":"front","average":{"1min":1.544,"5min":0.999,"15min":0.71999999999999997},"min":{"1min":0.42999999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.1790000000000003,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":0.57099999999999995}]},{"osd":6,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":0.877,"15min":0.66800000000000004},"min":{"1min":0.36399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.0099999999999998,"5min":5.8579999999999997,"15min":5.8579999999999997},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.95399999999999996,"15min":0.70199999999999996},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.298,"5min":5.1829999999999998,"15min":5.1829999999999998},"last":0.58199999999999996}]},{"osd":7,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.1679999999999999,"15min":0.78900000000000003},"min":{"1min":0.39900000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":3.9750000000000001,"5min":6.226,"15min":6.226},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.294,"5min":0.94599999999999995,"15min":0.70599999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.7080000000000002,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":0.54000000000000004}]}]},{"osd":1,"up_from":116,"seq":498216206463,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412168,"kb_used_data":2044,"kb_used_omap":5,"kb_used_meta":410106,"kb_avail":93332984,"statfs":{"total":95995035648,"available":95572975616,"internally_reserved":0,"allocated":2093056,"data_stored":1208175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":419948824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.3220000000000001,"15min":1.9339999999999999},"min":{"1min":0.29199999999999998,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":11.257999999999999,"5min":11.257999999999999,"15min":11.257999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.069,"15min":1.4650000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":4.5090000000000003,"5min":5.0220000000000002,"15min":10.426},"last":0.61399999999999999}]},{"osd":2,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":2.1909999999999998,"5min":1.4430000000000001,"15min":0.70499999999999996},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.3759999999999994,"5min":9.3759999999999994,"15min":9.3759999999999994},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.4100000000000001,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Jun 14 22:59:13 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.6739999999999999,"15min":0.77800000000000002},"min":{"1min":0.27400000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.8479999999999999,"5min":21.422000000000001,"15min":21.422000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":2.5670000000000002,"5min":1.764,"15min":0.83399999999999996},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":9.4990000000000006,"5min":21.016999999999999,"15min":21.016999999999999},"last":0.63700000000000001}]},{"osd":4,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3180000000000001,"15min":1.704},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.4990000000000001,"5min":7.673,"15min":8.8599999999999994},"last":0.79800000000000004},{"interface":"front","average":{"1min":2.1099999999999999,"5min":1.635,"15min":1.79},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":9.4169999999999998,"5min":9.4169999999999998,"15min":11.161},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":1.4890000000000001,"15min":1.696},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":10.103,"5min":10.103,"15min":10.103},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.151,"5min":1.3540000000000001,"15min":1.7430000000000001},"min":{"1min":0.32400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.5920000000000001,"5min":7.7439999999999998,"15min":8.125},"last":0.74099999999999999}]},{"osd":6,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.2490000000000001,"15min":1.677},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":9.7279999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.6970000000000001,"5min":1.371,"15min":1.742},"min":{"1min":0.38200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":8.4209999999999994,"5min":8.4209999999999994,"15min":8.4209999999999994},"last":0.83099999999999996}]},{"osd":7,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":1.772,"15min":1.8999999999999999},"min":{"1min":0.46999999999999997,"5min":0.35199999999999998,"15min":0.33700000000000002},"max":{"1min":9.4429999999999996,"5min":9.4429999999999996,"15min":9.4429999999999996},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.383,"15min":1.5720000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.36499999999999999},"max":{"1min":4.3899999999999997,"5min":6.3529999999999998,"15min":9.5449999999999999},"last":0.84399999999999997}]}]},{"osd":0,"up_from":91,"seq":390842024118,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":496140,"kb_used_data":2624,"kb_used_omap":5,"kb_used_meta":493498,"kb_avail":93249012,"statfs":{"total":95995035648,"available":95486988288,"internally_reserved":0,"allocated":2686976,"data_stored":1796542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":505342042},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.496,"15min":1.603},"min":{"1min":0.186,"5min":0.186,"15min":0.18099999999999999},"max":{"1min":5.3419999999999996,"5min":16.305,"15min":16.305},"last":0.435},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.5980000000000001,"15min":1.992},"min":{"1min":0.25,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.6820000000000004,"5min":16.184999999999999,"15min":16.184999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:59:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.157,"15min":0.624},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.326000000000001,"5min":16.757000000000001,"15min":16.757000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.3720000000000001,"15min":0.72199999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":11.519,"5min":16.055,"15min":16.055},"last":0.44600000000000001}]},{"osd":3,"last update":"Fri Jun 14 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.365,"15min":0.69699999999999995},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.3550000000000004,"5min":16.873999999999999,"15min":16.873999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.254,"5min":1.357,"15min":0.69099999999999995},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.21,"5min":16.085999999999999,"15min":16.085999999999999},"last":0.64800000000000002}]},{"osd":4,"last update":"Fri Jun 14 22:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.121,"15min":1.45},"min":{"1min":0.307,"5min":0.307,"15min":0.27600000000000002},"max":{"1min":4.1790000000000003,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.042,"5min":1.202,"15min":1.363},"min":{"1min":0.33200000000000002,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":4.1959999999999997,"5min":15.976000000000001,"15min":15.976000000000001},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Jun 14 22:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.875,"5min":1.5620000000000001,"15min":1.5169999999999999},"min":{"1min":0.49199999999999999,"5min":0.30399999999999999,"15min":0.29299999999999998},"max":{"1min":4.242,"5min":16.108000000000001,"15min":16.108000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.3859999999999999,"15min":1.5209999999999999},"min":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.149,"5min":16.006,"15min":16.006},"last":0.54900000000000004}]},{"osd":6,"last update":"Fri Jun 14 22:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2869999999999999,"15min":1.4470000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.2690000000000001,"5min":16.152000000000001,"15min":16.152000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.379,"5min":1.1910000000000001,"15min":1.4690000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.29399999999999998},"max":{"1min":4.298,"5min":16.227,"15min":16.227},"last":0.59499999999999997}]},{"osd":7,"last update":"Fri Jun 14 22:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3089999999999999,"15min":1.508},"min":{"1min":0.40999999999999998,"5min":0.35899999999999999,"15min":0.33100000000000002},"max":{"1min":4.3319999999999999,"5min":16.199999999999999,"15min":16.199999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.353,"15min":1.6040000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.314},"max":{"1min":5.1660000000000004,"5min":16.259,"15min":16.259},"last":0.626}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T22:59:56.250 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-14T22:59:56.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-06-14T22:59:56.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:56.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:56.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:56.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:56.437 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:56.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:56.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:56.559 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:56.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:56.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:56.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:56.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:56.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:56.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:56.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:56.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:56.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:56.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:57.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:57.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:57.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:57.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:57.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:57.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:57.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:57.207 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T22:59:57.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:57.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:57.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:57.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:57.434 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:57.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:57.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:57.560 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:57.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:57.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:57.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:57.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:57.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:57.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:57 smithi086 ceph-mon[97913]: pgmap v710: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 216 op/s 2024-06-14T22:59:57.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:57.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:57.794 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:57 smithi086 ceph-mon[102754]: pgmap v710: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 216 op/s 2024-06-14T22:59:57.794 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:57.795 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:57.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:57.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:57.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:57.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:57.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:57.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:57 smithi137 ceph-mon[90634]: pgmap v710: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 216 op/s 2024-06-14T22:59:57.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T22:59:57.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T22:59:58.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:58.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:58.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:58.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:58.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:58.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:58.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:58.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:58.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:58.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:58.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:58.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:58.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T22:59:58.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:58.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T22:59:58.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:58.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:58.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:58.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T22:59:58.821 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:58.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T22:59:58.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:58.939 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T22:59:58.940 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T22:59:58.940 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T22:59:58.940 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T22:59:58.940 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T22:59:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:58.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:59.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T22:59:59.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:59.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T22:59:59.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T22:59:59.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:59.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:59.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T22:59:59.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T22:59:59.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:59.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:59.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:59.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T22:59:59.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:59.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T22:59:59.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T22:59:59.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:59.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:59.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T22:59:59.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-14T22:59:59.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T22:59:59.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T22:59:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:59.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T22:59:59.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:59.821 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:59.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:59.907 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 22:59:59 smithi086 ceph-mon[97913]: pgmap v711: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 120 op/s 2024-06-14T22:59:59.908 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 22:59:59 smithi086 ceph-mon[102754]: pgmap v711: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 120 op/s 2024-06-14T22:59:59.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T22:59:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T22:59:59.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T22:59:59.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T22:59:59.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 22:59:59 smithi137 ceph-mon[90634]: pgmap v711: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 120 op/s 2024-06-14T23:00:00.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:00.073 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:00.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:00.147 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:00:00.147 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:00:00.154 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:00.159 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:00:00.159 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:00:00.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:00.177 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:00:00.178 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:00:00.198 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:00:00.199 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:00:00.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:00.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:00.204 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:00.216 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:00:00.216 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:00:00.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:00:00.233 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:00:00.250 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:00:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:00:00.266 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:00:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:00:00.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:00.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:00.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:00.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:00.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:00.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:00.551 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:00.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:00.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:00.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:00.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:00.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:00.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:00.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:00.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:00.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:00.953 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:00.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:00.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:00 smithi137 ceph-mon[90634]: overall HEALTH_OK 2024-06-14T23:00:00.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/818843856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:01.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:01.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:01.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:01.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:01.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:00 smithi086 ceph-mon[97913]: overall HEALTH_OK 2024-06-14T23:00:01.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/818843856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:01.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:00 smithi086 ceph-mon[102754]: overall HEALTH_OK 2024-06-14T23:00:01.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/818843856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:01.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:01.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:01.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:01.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:01.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:01.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:01.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:01.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:01.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:01.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:01.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:01.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:01.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:01.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:01.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:01.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:01.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:01.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:01 smithi086 ceph-mon[97913]: pgmap v712: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:01.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:01 smithi086 ceph-mon[102754]: pgmap v712: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:01.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:01.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:01.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:01.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:01.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:01.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:01 smithi137 ceph-mon[90634]: pgmap v712: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:02.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:02.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:02.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:02.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:02.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:02.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:02.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:02.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:02.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:02.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:02.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:02.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:02.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:02.548 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:02.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:02.591 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:02.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:02.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:02.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:02.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:02.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:02.838 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:02.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:02.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:02.960 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:02.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:02.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:03.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:03.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:03.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:03.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:03.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:03.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:00:03.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:03.360 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:03.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:03.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:03.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:03.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:03.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:03.613 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:03.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:03.708 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:00:03.709 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:00:03.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:03.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:03.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:03.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:03.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:03.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:03.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:03.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:03 smithi137 ceph-mon[90634]: pgmap v713: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:03.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:03.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:04.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:03 smithi086 ceph-mon[97913]: pgmap v713: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:04.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:03 smithi086 ceph-mon[102754]: pgmap v713: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:04.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:04.122 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:04.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:04.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:04.248 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:04.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:04.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:04.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:04.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:04.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:04.498 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:04.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:04.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:04.625 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:04.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:04.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-14T23:00:04.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:04.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:04.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:04.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:04.873 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:04.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:04.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:04.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:05.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:05.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:05.123 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:05.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:05.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:05.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:05.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:05.335 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:00:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:00:04] "GET /metrics HTTP/1.1" 200 36494 "" "Prometheus/2.43.0" 2024-06-14T23:00:05.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:05.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:05.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:05.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:05.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:05.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:05.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:05.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:05.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:05.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:05.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:05.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:05.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:05.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:05.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:05 smithi086 ceph-mon[97913]: pgmap v714: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:05.931 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:05 smithi086 ceph-mon[102754]: pgmap v714: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:05.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:05 smithi137 ceph-mon[90634]: pgmap v714: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:06.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:06.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:06.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:06.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:06.160 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:06.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:06.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:06.268 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:06.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:06.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:06.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:06.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:06.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:06.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:06.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:06.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:06.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:06.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:06.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:06.749 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 1 pg_num to 11 2024-06-14T23:00:06.762 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:06.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:06.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:06.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:06.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:06.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:07.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:07.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:07.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3675068639' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-14T23:00:07.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3675068639' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-14T23:00:07.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:07.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:07.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3675068639' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-14T23:00:07.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:07.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:07.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:07.403 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:07.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:07.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:07.526 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:07.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:07.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:07.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:07.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:07.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:07.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:07.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:07.815 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-06-14T23:00:07.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-06-14T23:00:07.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:07.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:07.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:08.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:08.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:08.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:08.077 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:07 smithi086 ceph-mon[102754]: pgmap v715: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:08.077 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3675068639' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-06-14T23:00:08.077 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:07 smithi086 ceph-mon[102754]: osdmap e178: 8 total, 8 up, 8 in 2024-06-14T23:00:08.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:07 smithi086 ceph-mon[97913]: pgmap v715: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:08.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3675068639' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-06-14T23:00:08.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:07 smithi086 ceph-mon[97913]: osdmap e178: 8 total, 8 up, 8 in 2024-06-14T23:00:08.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:08.152 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:08.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:08.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:08.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:07 smithi137 ceph-mon[90634]: pgmap v715: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3675068639' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-06-14T23:00:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:07 smithi137 ceph-mon[90634]: osdmap e178: 8 total, 8 up, 8 in 2024-06-14T23:00:08.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:08.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:08.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:08.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:08.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:08.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:08.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:08.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:08.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:08.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:08.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:08.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:08.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:08.762 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:00:08.780 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:08.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:08.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:08.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:08.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:09.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2928219841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:09.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:08 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:09.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:08 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-06-14T23:00:09.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2928219841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:09.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:08 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:09.004 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:08 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-06-14T23:00:09.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:09.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:09.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:09.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:09.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:09.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:09.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2928219841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:09.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:08 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:09.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:08 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-06-14T23:00:09.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:09.273 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:09.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:09.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:09.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:09.396 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:09.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:09.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:09.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:09.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:09.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:09.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:09.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:09.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:09.779 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:09.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:09.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:09.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:09.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:10.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:10.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:10.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:10.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[97913]: pgmap v717: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:10.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:10.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:00:10.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-06-14T23:00:10.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[97913]: osdmap e179: 8 total, 8 up, 8 in 2024-06-14T23:00:10.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[102754]: pgmap v717: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:10.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:10.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:00:10.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-06-14T23:00:10.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:09 smithi086 ceph-mon[102754]: osdmap e179: 8 total, 8 up, 8 in 2024-06-14T23:00:10.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:10.145 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:09 smithi137 ceph-mon[90634]: pgmap v717: 89 pgs: 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:10.145 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:09 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:10.145 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:09 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:00:10.145 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:09 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-06-14T23:00:10.145 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:09 smithi137 ceph-mon[90634]: osdmap e179: 8 total, 8 up, 8 in 2024-06-14T23:00:10.152 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:10.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:10.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:10.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:10.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:10.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:10.398 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:10.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:10.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:10.525 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:10.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:10.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:10.632 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:10.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:10.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:10.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:10.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:10.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:10.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:10.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:10.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:10.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:11.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:11.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:11.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:11.036 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:10 smithi086 ceph-mon[97913]: osdmap e180: 8 total, 8 up, 8 in 2024-06-14T23:00:11.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:10 smithi086 ceph-mon[102754]: osdmap e180: 8 total, 8 up, 8 in 2024-06-14T23:00:11.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:11.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:11.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:11.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:10 smithi137 ceph-mon[90634]: osdmap e180: 8 total, 8 up, 8 in 2024-06-14T23:00:11.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:11.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:11.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:11.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:11.395 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:00:11.395 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:00:11.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:11.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:11.412 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:00:11.412 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:00:11.428 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:00:11.428 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:00:11.444 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:00:11.444 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:00:11.461 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:00:11.461 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:00:11.477 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:00:11.478 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:00:11.494 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:00:11.494 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:00:11.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:11.511 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:00:11.511 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:00:11.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:11.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:11.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:11.650 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:11.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:11.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:11.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:11.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:11.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:11.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:11.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:12.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:12.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:12.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:12.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:12.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:11 smithi086 ceph-mon[102754]: pgmap v720: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:12.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3687108327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:12.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:11 smithi086 ceph-mon[97913]: pgmap v720: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:12.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3687108327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:12.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:12.153 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:12.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:12.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:11 smithi137 ceph-mon[90634]: pgmap v720: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:12.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3687108327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:12.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:12.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:12.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:12.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:00:12.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:12.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:12.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:12.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:12.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:12.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:12.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:12.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:12.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:12.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:12.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:12.815 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:00:12.816 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:00:12.816 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-06-14T23:00:12.816 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:00:12.816 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:00:12.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-06-14T23:00:12.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:12.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:12.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:13.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:13.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:13.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:13.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:13.088 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:00:13.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:13.145 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:13.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:13.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:13.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:13.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:13.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:13.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:13.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:13.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:13.493 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:13.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:13 smithi137 ceph-mon[90634]: pgmap v721: 99 pgs: 99 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:13.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-06-14T23:00:13.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:13.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:13.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:13.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:13.626 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:13.626 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:13.626 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:13 smithi086 ceph-mon[102754]: pgmap v721: 99 pgs: 99 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:13.626 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-06-14T23:00:13.626 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:13.626 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:13.626 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:13 smithi086 ceph-mon[97913]: pgmap v721: 99 pgs: 99 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:13.626 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-06-14T23:00:13.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:13.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:13.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:13.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:13.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:13.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:13.889 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:13.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:13.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:14.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:14.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:14.032 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:14.115 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:14.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:14.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:14.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:14.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:14.261 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:14.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:14.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:14.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:14.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:14.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:14.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:14.508 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:14 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-06-14T23:00:14.509 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:14 smithi086 ceph-mon[102754]: osdmap e181: 8 total, 8 up, 8 in 2024-06-14T23:00:14.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:14 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-06-14T23:00:14.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:14 smithi086 ceph-mon[97913]: osdmap e181: 8 total, 8 up, 8 in 2024-06-14T23:00:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:14.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:14.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:14.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:14.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:14.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:14.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:14.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:14 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-06-14T23:00:14.741 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:14 smithi137 ceph-mon[90634]: osdmap e181: 8 total, 8 up, 8 in 2024-06-14T23:00:14.755 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:14.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:14.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:14.879 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:14.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:14.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:14.970 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:00:14.970 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-14T23:00:14.970 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:00:14.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:15.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:15.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:15.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:15.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:15.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:15.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:15.196 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:00:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:00:14] "GET /metrics HTTP/1.1" 200 36498 "" "Prometheus/2.43.0" 2024-06-14T23:00:15.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:15.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:15.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:15.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:15.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:15.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:00:15.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:15.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:15.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:15.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:15 smithi086 ceph-mon[97913]: osdmap e182: 8 total, 8 up, 8 in 2024-06-14T23:00:15.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:15 smithi086 ceph-mon[97913]: pgmap v724: 99 pgs: 2 peering, 97 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:15.547 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:15 smithi086 ceph-mon[102754]: osdmap e182: 8 total, 8 up, 8 in 2024-06-14T23:00:15.547 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:15 smithi086 ceph-mon[102754]: pgmap v724: 99 pgs: 2 peering, 97 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:15.579 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:15 smithi137 ceph-mon[90634]: osdmap e182: 8 total, 8 up, 8 in 2024-06-14T23:00:15.579 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:15 smithi137 ceph-mon[90634]: pgmap v724: 99 pgs: 2 peering, 97 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:15.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:15.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:15.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:15.745 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:15.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:15.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:15.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:15.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:15.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:15.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:15.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:16.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:16.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:16.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:16.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:16.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:16.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:16.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:16.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:16.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_4' removed 2024-06-14T23:00:16.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:16.489 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:16 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:00:16.489 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:16 smithi086 ceph-mon[97913]: osdmap e183: 8 total, 8 up, 8 in 2024-06-14T23:00:16.489 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2867469794' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:00:16.490 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:16 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:00:16.490 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:16 smithi086 ceph-mon[102754]: osdmap e183: 8 total, 8 up, 8 in 2024-06-14T23:00:16.490 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:16 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2867469794' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:00:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:16.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:16.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:16.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:16.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:16 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:00:16.743 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:16 smithi137 ceph-mon[90634]: osdmap e183: 8 total, 8 up, 8 in 2024-06-14T23:00:16.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:16 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2867469794' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:00:16.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:16.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:16.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:16.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:16.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:16.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:17.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:17.023 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:17.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:17.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:17.146 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:17.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:17.246 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T23:00:17.247 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T23:00:17.248 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-06-14T23:00:17.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-06-14T23:00:17.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:17.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:17.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:17.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:17.393 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:00:17.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:17.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:17.488 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:00:17.488 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:00:17.488 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-06-14T23:00:17.488 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2867469794' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[97913]: osdmap e184: 8 total, 8 up, 8 in 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2164734750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[97913]: pgmap v727: 83 pgs: 2 peering, 81 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2164734750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[97913]: osdmap e185: 8 total, 8 up, 8 in 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2867469794' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[102754]: osdmap e184: 8 total, 8 up, 8 in 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2164734750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:00:17.498 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[102754]: pgmap v727: 83 pgs: 2 peering, 81 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:17.499 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:17.499 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2164734750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:00:17.499 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:17 smithi086 ceph-mon[102754]: osdmap e185: 8 total, 8 up, 8 in 2024-06-14T23:00:17.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:17.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:17.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:17.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:17.644 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:17.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:17.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2867469794' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:00:17.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:17 smithi137 ceph-mon[90634]: osdmap e184: 8 total, 8 up, 8 in 2024-06-14T23:00:17.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2164734750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:00:17.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:17 smithi137 ceph-mon[90634]: pgmap v727: 83 pgs: 2 peering, 81 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:17.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:17 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:17.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2164734750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:00:17.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:17 smithi137 ceph-mon[90634]: osdmap e185: 8 total, 8 up, 8 in 2024-06-14T23:00:17.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:17.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:17.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:17.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:17.893 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:17.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:17.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:18.018 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:18.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:18.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:18.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:18.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:18.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:18.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:18.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:18.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:18.394 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:18.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:18.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-14T23:00:18.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:18.503 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:18.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:18.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:18.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:18.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:18.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:18.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:18.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:18.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:18.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:18.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:18.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:18.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:19.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:19.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:19.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:19.145 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:19.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:19.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:19.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:19.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:19.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:19.396 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:19.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:19.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:19.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:19.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:19.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:19.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:19.744 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:19.744 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:19.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:19.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:19.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:19.899 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:19.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:19.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:19 smithi137 ceph-mon[90634]: pgmap v729: 83 pgs: 3 peering, 80 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:20.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:20.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:20.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:20.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:19 smithi086 ceph-mon[97913]: pgmap v729: 83 pgs: 3 peering, 80 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:20.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:19 smithi086 ceph-mon[102754]: pgmap v729: 83 pgs: 3 peering, 80 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:20.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:20.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:20.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:20.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:20.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:20.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:20.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:20.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:20.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:20.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:20.526 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:20.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:20.604 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:00:20.604 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:00:20.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:20.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:20.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:20.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:20.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:20.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:20.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:20.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:20.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:20.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:20.922 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:20 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-06-14T23:00:20.922 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:20 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-06-14T23:00:21.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:00:21.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:21.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:21.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:21.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:21.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:21.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:21.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:20 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-06-14T23:00:21.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:21.279 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:21.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:21.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:21.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:21.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:21.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:21.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:21.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:21.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:21.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:21.718 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:00:21.718 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:00:21.735 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:00:21.735 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:00:21.755 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:00:21.755 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:00:21.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:21.772 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:00:21.772 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:00:21.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:21.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:21.790 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:00:21.791 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:00:21.807 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:00:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:00:21.823 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:00:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:00:21.840 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:00:21.840 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:00:21.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:21.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:21.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:22.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:22.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:22.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:21 smithi086 ceph-mon[97913]: pgmap v730: 83 pgs: 1 peering, 82 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T23:00:22.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2637562665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:22.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:21 smithi086 ceph-mon[102754]: pgmap v730: 83 pgs: 1 peering, 82 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T23:00:22.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2637562665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:22.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:22.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:22.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:22.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:21 smithi137 ceph-mon[90634]: pgmap v730: 83 pgs: 1 peering, 82 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-14T23:00:22.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2637562665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:22.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:22.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:22.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:22.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:22.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:22.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:22.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:22.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:22.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:22.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:22.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:22.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:22.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:22.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:22.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:22.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:22.909 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:22.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:23.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:23.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:23.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:23.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:23.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:23.152 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:00:23.152 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:00:23.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:23.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:23.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:23.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:23.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:23.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:23.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:23.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:23.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:23.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:23.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:23.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:23.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:23.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:23.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:23.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:23.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:23.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:23.910 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:23.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:24.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:24.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:24.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:24.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:23 smithi086 ceph-mon[102754]: pgmap v731: 83 pgs: 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-06-14T23:00:24.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:23 smithi086 ceph-mon[97913]: pgmap v731: 83 pgs: 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-06-14T23:00:24.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:24.161 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:24.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:24.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:23 smithi137 ceph-mon[90634]: pgmap v731: 83 pgs: 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-06-14T23:00:24.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:24.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:24.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:24.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:24.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:24.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:24.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:24.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:24.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:24.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:24.658 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:24.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:24.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:24.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:24.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:24.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:24.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:24.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:24.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:25.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:25.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:25.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:25.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:00:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:00:24] "GET /metrics HTTP/1.1" 200 36498 "" "Prometheus/2.43.0" 2024-06-14T23:00:25.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-14T23:00:25.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:25.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:25.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:25.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:25.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:25.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:25.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:25.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:25.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:25.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:25.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:25.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:25.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:25.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:25.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:25.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:25.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:25.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:25.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:25.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:25.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:26.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:26.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:26.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:26.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:25 smithi086 ceph-mon[102754]: pgmap v732: 83 pgs: 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-06-14T23:00:26.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:25 smithi086 ceph-mon[97913]: pgmap v732: 83 pgs: 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-06-14T23:00:26.164 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:26.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:26.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:26.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:25 smithi137 ceph-mon[90634]: pgmap v732: 83 pgs: 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-06-14T23:00:26.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:26.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:00:26.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:26.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:26.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:26.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:26.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:26.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:26.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:26.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:26.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:26.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:26.790 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_5' created 2024-06-14T23:00:26.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:26.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:26.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:26.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:26.940 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:26.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:26.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:27.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:27.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:27.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:27.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2213838119' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-14T23:00:27.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:26 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-14T23:00:27.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2213838119' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-14T23:00:27.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:26 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-14T23:00:27.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:27.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:27.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:27.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2213838119' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-14T23:00:27.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:26 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-14T23:00:27.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:27.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:27.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:27.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:27.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:27.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:27.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:27.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-06-14T23:00:27.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:27.545 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:27.569 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:27.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:27.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:27.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:27.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:27.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:27.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:27.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:27.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:27.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:27.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:28.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:28.071 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:28.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:28.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:27 smithi086 ceph-mon[102754]: pgmap v733: 83 pgs: 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T23:00:28.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:27 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-06-14T23:00:28.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:27 smithi086 ceph-mon[102754]: osdmap e186: 8 total, 8 up, 8 in 2024-06-14T23:00:28.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:28.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:27 smithi086 ceph-mon[97913]: pgmap v733: 83 pgs: 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T23:00:28.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:27 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-06-14T23:00:28.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:27 smithi086 ceph-mon[97913]: osdmap e186: 8 total, 8 up, 8 in 2024-06-14T23:00:28.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:28.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:28.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:28.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:27 smithi137 ceph-mon[90634]: pgmap v733: 83 pgs: 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-14T23:00:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:27 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-06-14T23:00:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:27 smithi137 ceph-mon[90634]: osdmap e186: 8 total, 8 up, 8 in 2024-06-14T23:00:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:28.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:28.313 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:28.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:28.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:28.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:28.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:28.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:28.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:28.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:28.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:28.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:28.704 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:28.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:28.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:28.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:28.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:28.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:28.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:29.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:29.079 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:29.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:29.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:28 smithi086 ceph-mon[97913]: osdmap e187: 8 total, 8 up, 8 in 2024-06-14T23:00:29.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:28 smithi086 ceph-mon[97913]: osdmap e188: 8 total, 8 up, 8 in 2024-06-14T23:00:29.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:28 smithi086 ceph-mon[102754]: osdmap e187: 8 total, 8 up, 8 in 2024-06-14T23:00:29.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:28 smithi086 ceph-mon[102754]: osdmap e188: 8 total, 8 up, 8 in 2024-06-14T23:00:29.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:29.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:29.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:29.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:28 smithi137 ceph-mon[90634]: osdmap e187: 8 total, 8 up, 8 in 2024-06-14T23:00:29.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:28 smithi137 ceph-mon[90634]: osdmap e188: 8 total, 8 up, 8 in 2024-06-14T23:00:29.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:29.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:29.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:29.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:29.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:29.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:29.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:29.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:29.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:29.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:29.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:29.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:29.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:29.817 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:00:29.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:29 smithi086 ceph-mon[97913]: pgmap v736: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:29.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:29 smithi086 ceph-mon[97913]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:00:29.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:29 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1789625787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:00:29.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:29 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:00:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:29 smithi086 ceph-mon[102754]: pgmap v736: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:29 smithi086 ceph-mon[102754]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:00:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:29 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1789625787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:00:29.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:29 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:00:29.834 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:29.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:29.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:29.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:29.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:30.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:30.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:30.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:30.198 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:00:30.198 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:00:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:30.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:30.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:29 smithi137 ceph-mon[90634]: pgmap v736: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:29 smithi137 ceph-mon[90634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:00:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:29 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1789625787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:00:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:29 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:00:30.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:30.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:30.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:30.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:30.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:30.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:30.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:30.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:30.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:30.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:30.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:30.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:30.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:30.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:30.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:30.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:30.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:31.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:00:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[97913]: osdmap e189: 8 total, 8 up, 8 in 2024-06-14T23:00:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[97913]: 1.0 scrub starts 2024-06-14T23:00:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[97913]: 1.0 scrub ok 2024-06-14T23:00:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[97913]: 1.3 scrub starts 2024-06-14T23:00:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[97913]: 1.3 scrub ok 2024-06-14T23:00:31.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:31.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:00:31.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[102754]: osdmap e189: 8 total, 8 up, 8 in 2024-06-14T23:00:31.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[102754]: 1.0 scrub starts 2024-06-14T23:00:31.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[102754]: 1.0 scrub ok 2024-06-14T23:00:31.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[102754]: 1.3 scrub starts 2024-06-14T23:00:31.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:30 smithi086 ceph-mon[102754]: 1.3 scrub ok 2024-06-14T23:00:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:31.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:31.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:31.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:31.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:31.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:31.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:31.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:30 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:30 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:00:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:30 smithi137 ceph-mon[90634]: osdmap e189: 8 total, 8 up, 8 in 2024-06-14T23:00:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:30 smithi137 ceph-mon[90634]: 1.0 scrub starts 2024-06-14T23:00:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:30 smithi137 ceph-mon[90634]: 1.0 scrub ok 2024-06-14T23:00:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:30 smithi137 ceph-mon[90634]: 1.3 scrub starts 2024-06-14T23:00:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:30 smithi137 ceph-mon[90634]: 1.3 scrub ok 2024-06-14T23:00:31.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:31.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-14T23:00:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:31.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:31.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:31.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:31.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:31.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:31.586 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:00:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:31.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:31.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:31.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:31.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:31.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:31.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:31.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:31.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:31.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:31.956 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:31.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:31.965 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:31 smithi137 ceph-mon[90634]: 1.9 scrub starts 2024-06-14T23:00:31.965 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:31 smithi137 ceph-mon[90634]: 1.9 scrub ok 2024-06-14T23:00:31.965 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:31 smithi137 ceph-mon[90634]: pgmap v739: 99 pgs: 2 unknown, 97 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:31.965 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:31 smithi137 ceph-mon[90634]: 1.6 scrub starts 2024-06-14T23:00:31.966 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:31 smithi137 ceph-mon[90634]: 1.6 scrub ok 2024-06-14T23:00:31.966 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:31 smithi137 ceph-mon[90634]: 1.2 scrub starts 2024-06-14T23:00:31.966 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:31 smithi137 ceph-mon[90634]: 1.2 scrub ok 2024-06-14T23:00:32.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:32.078 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:32.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[102754]: 1.9 scrub starts 2024-06-14T23:00:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[102754]: 1.9 scrub ok 2024-06-14T23:00:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[102754]: pgmap v739: 99 pgs: 2 unknown, 97 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[102754]: 1.6 scrub starts 2024-06-14T23:00:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[102754]: 1.6 scrub ok 2024-06-14T23:00:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[102754]: 1.2 scrub starts 2024-06-14T23:00:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[102754]: 1.2 scrub ok 2024-06-14T23:00:32.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[97913]: 1.9 scrub starts 2024-06-14T23:00:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[97913]: 1.9 scrub ok 2024-06-14T23:00:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[97913]: pgmap v739: 99 pgs: 2 unknown, 97 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[97913]: 1.6 scrub starts 2024-06-14T23:00:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[97913]: 1.6 scrub ok 2024-06-14T23:00:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[97913]: 1.2 scrub starts 2024-06-14T23:00:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:31 smithi086 ceph-mon[97913]: 1.2 scrub ok 2024-06-14T23:00:32.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:32.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:32.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:32.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:32.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:32.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:32.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:32.447 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:32.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:32.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:32.569 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:32.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:32.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:32.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:32.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:32.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:32.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:32.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:32.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:32.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:32.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:32.989 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:32 smithi086 ceph-mon[102754]: 1.a scrub starts 2024-06-14T23:00:32.989 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:32 smithi086 ceph-mon[102754]: 1.a scrub ok 2024-06-14T23:00:32.990 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:32 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:00:32.990 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:32 smithi086 ceph-mon[102754]: 1.5 scrub ok 2024-06-14T23:00:32.990 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:32 smithi086 ceph-mon[97913]: 1.a scrub starts 2024-06-14T23:00:32.990 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:32 smithi086 ceph-mon[97913]: 1.a scrub ok 2024-06-14T23:00:32.990 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:32 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:00:32.990 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:32 smithi086 ceph-mon[97913]: 1.5 scrub ok 2024-06-14T23:00:32.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:32 smithi137 ceph-mon[90634]: 1.a scrub starts 2024-06-14T23:00:32.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:32 smithi137 ceph-mon[90634]: 1.a scrub ok 2024-06-14T23:00:32.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:32 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:00:32.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:32 smithi137 ceph-mon[90634]: 1.5 scrub ok 2024-06-14T23:00:33.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:33.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:33.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:33.130 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:00:33.131 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:00:33.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:33.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:33.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:33.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:33.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:33.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:33.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:33.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:33.430 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:33.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:33.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:33.552 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:33.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:33.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:33.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:33.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-14T23:00:33.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:33.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:33.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:33.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:33.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:33.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:33.913 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:00:33.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:33.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:33.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[97913]: pgmap v740: 99 pgs: 99 active+clean; 477 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[97913]: 1.4 scrub starts 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[97913]: 1.4 scrub ok 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[97913]: 1.1 scrub starts 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[97913]: 1.1 scrub ok 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[102754]: pgmap v740: 99 pgs: 99 active+clean; 477 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[102754]: 1.4 scrub starts 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[102754]: 1.4 scrub ok 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[102754]: 1.1 scrub starts 2024-06-14T23:00:33.979 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:33 smithi086 ceph-mon[102754]: 1.1 scrub ok 2024-06-14T23:00:33.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:33 smithi137 ceph-mon[90634]: pgmap v740: 99 pgs: 99 active+clean; 477 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:33.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:33 smithi137 ceph-mon[90634]: 1.4 scrub starts 2024-06-14T23:00:33.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:33 smithi137 ceph-mon[90634]: 1.4 scrub ok 2024-06-14T23:00:33.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:33 smithi137 ceph-mon[90634]: 1.1 scrub starts 2024-06-14T23:00:33.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:33 smithi137 ceph-mon[90634]: 1.1 scrub ok 2024-06-14T23:00:34.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:34.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:34.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:34.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:34.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:34.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:34.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:34.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:34.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:34.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:34.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:34.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:34.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:34.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:34.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:34.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:34.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:34.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:34.684 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:00:34.684 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:00:34.684 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:00:34.684 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:00:34.700 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:00:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:00:34.716 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:00:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:00:34.733 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:00:34.733 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:00:34.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:00:34.750 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:00:34.750 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:00:34.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:34.766 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:00:34.767 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:00:34.782 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:00:34.783 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:00:34.783 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:34.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:34.836 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-06-14T23:00:34.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:34.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:34.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:35.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:35.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:35.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:35.067 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:34 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:00:35.067 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:34 smithi137 ceph-mon[90634]: 1.8 scrub ok 2024-06-14T23:00:35.067 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/145800818' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:00:35.067 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:34 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:00:35.067 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3680919071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:35.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:00:34] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T23:00:35.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:00:35.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[97913]: 1.8 scrub ok 2024-06-14T23:00:35.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/145800818' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:00:35.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:00:35.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3680919071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:00:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[102754]: 1.8 scrub ok 2024-06-14T23:00:35.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/145800818' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:00:35.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:00:35.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3680919071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:35.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:35.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:35.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:35.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:35.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:35.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:35.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:35.399 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:35.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:35.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:35.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:35.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:35.552 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:00:35.552 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:00:35.552 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:00:35.552 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:00:35.553 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:00:35.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:35.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:35.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:35.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:35.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:35.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:35.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:35.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:35.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:35.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:36.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:36.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:36.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:35 smithi086 ceph-mon[102754]: pgmap v741: 99 pgs: 99 active+clean; 739 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:36.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:35 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:00:36.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:35 smithi086 ceph-mon[102754]: osdmap e190: 8 total, 8 up, 8 in 2024-06-14T23:00:36.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:35 smithi086 ceph-mon[97913]: pgmap v741: 99 pgs: 99 active+clean; 739 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:36.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:35 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:00:36.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:35 smithi086 ceph-mon[97913]: osdmap e190: 8 total, 8 up, 8 in 2024-06-14T23:00:36.157 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:35 smithi137 ceph-mon[90634]: pgmap v741: 99 pgs: 99 active+clean; 739 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:36.158 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:35 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:00:36.158 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:35 smithi137 ceph-mon[90634]: osdmap e190: 8 total, 8 up, 8 in 2024-06-14T23:00:36.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:36.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-14T23:00:36.197 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:36.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:36.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:36.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:36.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:36.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:36.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:36.434 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:00:36.443 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:36.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:36.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:36.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:36.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:36.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:36.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:36.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:36.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:36.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:36.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:36.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:36.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:36.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:37.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:37.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:37.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:37.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:36 smithi086 ceph-mon[97913]: pgmap v743: 99 pgs: 99 active+clean; 634 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:37.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:36 smithi086 ceph-mon[97913]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:00:37.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:36 smithi086 ceph-mon[102754]: pgmap v743: 99 pgs: 99 active+clean; 634 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:37.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:36 smithi086 ceph-mon[102754]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:00:37.160 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-06-14T23:00:37.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:37.161 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-06-14T23:00:37.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:00:37.164 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T23:00:37.164 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T23:00:37.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:37.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:37.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:37.208 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T23:00:37.208 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T23:00:37.208 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_189745 2024-06-14T23:00:37.209 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:00:37.209 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T23:00:37.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:36 smithi137 ceph-mon[90634]: pgmap v743: 99 pgs: 99 active+clean; 634 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:37.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:36 smithi137 ceph-mon[90634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:00:37.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:37.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:37.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:37.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:37.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:37.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:37.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:00:37.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:37.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:37.692 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:37.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:37.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:37.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:37.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:37.889 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:37.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:37.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:38.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:38.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:38.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:38.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:38.209 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 3848 3832 239.487 239.5 0.00313185 0.00416129 2024-06-14T23:00:38.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:38.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:38.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:38.333 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:00:38.333 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:00:38.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:00:38.333 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:00:38.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.333 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.34776.0:176620 8.2 8:4b8894ff:::benchmark_data_smithi086_122122_object56031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-06-14T23:00:38.333 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:51:43.418851+0000", 2024-06-14T23:00:38.333 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.90894628299998, 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.100082461, 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.34776", 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 176620 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.418851+0000", 2024-06-14T23:00:38.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.418851+0000", 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.418849+0000", 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.418856+0000", 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.72e-06 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.418867+0000", 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1432e-05 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.418873+0000", 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5389999999999998e-06 2024-06-14T23:00:38.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.418928+0000", 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5430000000000003e-05 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.419038+0000", 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110121 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:00:38.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.419145+0000", 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010665100000000001 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423260+0000", 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004115039 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518792+0000", 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095532052000000006 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518816+0000", 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4445000000000001e-05 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518889+0000", 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2946000000000003e-05 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518933+0000", 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3866999999999999e-05 2024-06-14T23:00:38.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.34776.0:176619 8.e e124/109 8:7fe203b2:::benchmark_data_smithi086_122122_object56030:head v 124'11142, mlcod=124'11139)", 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:51:43.423050+0000", 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.90474674699999, 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.096292561999999998, 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423050+0000", 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423050+0000", 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423049+0000", 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423057+0000", 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7330000000000003e-06 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423068+0000", 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0926e-05 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423074+0000", 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4190000000000001e-06 2024-06-14T23:00:38.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423118+0000", 2024-06-14T23:00:38.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4898999999999998e-05 2024-06-14T23:00:38.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423138+0000", 2024-06-14T23:00:38.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9389999999999999e-05 2024-06-14T23:00:38.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.519287+0000", 2024-06-14T23:00:38.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.096149525 2024-06-14T23:00:38.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.519343+0000", 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5461000000000003e-05 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.34776.0:176622 8.2 8:4517f244:::benchmark_data_smithi086_122122_object56033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-06-14T23:00:38.345 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:51:43.423333+0000", 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.904464233, 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.11007919300000001, 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.34776", 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 176622 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423333+0000", 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423333+0000", 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423331+0000", 2024-06-14T23:00:38.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:00:38.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423338+0000", 2024-06-14T23:00:38.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0140000000000001e-06 2024-06-14T23:00:38.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423350+0000", 2024-06-14T23:00:38.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1829e-05 2024-06-14T23:00:38.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423355+0000", 2024-06-14T23:00:38.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7599999999999999e-06 2024-06-14T23:00:38.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423397+0000", 2024-06-14T23:00:38.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1925999999999998e-05 2024-06-14T23:00:38.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.514704+0000", 2024-06-14T23:00:38.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.091306490000000004 2024-06-14T23:00:38.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.514813+0000", 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109567 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.523357+0000", 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0085432500000000005 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.525176+0000", 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001819621 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.531735+0000", 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065589189999999999 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.533377+0000", 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001642178 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.533412+0000", 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4489000000000001e-05 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.34776.0:176620 8.2 e124/109 ondisk, result = 0)", 2024-06-14T23:00:38.353 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:51:43.423497+0000", 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.90429958699997, 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095439657999999997, 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423497+0000", 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423497+0000", 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423497+0000", 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.423500+0000", 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3210000000000001e-06 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.513711+0000", 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.090211213999999998 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.513720+0000", 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6009999999999995e-06 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518808+0000", 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005087902 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518812+0000", 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3800000000000004e-06 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518937+0000", 2024-06-14T23:00:38.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012467700000000001 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.34776.0:176620 8.2 e124/109 ondisk, result = 0)", 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:51:43.432296+0000", 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.89550125899996, 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.086497615, 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:00:38.358 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.432296+0000", 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.432296+0000", 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.432294+0000", 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.432304+0000", 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9170000000000001e-06 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.513283+0000", 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.080978573999999998 2024-06-14T23:00:38.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.513294+0000", 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0903999999999999e-05 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518779+0000", 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054855889999999999 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518786+0000", 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4300000000000003e-06 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:43.518793+0000", 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5610000000000001e-06 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.34776.0:178575 8.2 8:463130a3:::benchmark_data_smithi086_122122_object57986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:51:45.982888+0000", 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.34490911399996, 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.090406789000000001, 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.34776", 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 178575 2024-06-14T23:00:38.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.982888+0000", 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.982888+0000", 2024-06-14T23:00:38.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.982886+0000", 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.982893+0000", 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8220000000000001e-06 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.982910+0000", 2024-06-14T23:00:38.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6677999999999999e-05 2024-06-14T23:00:38.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.982920+0000", 2024-06-14T23:00:38.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0722e-05 2024-06-14T23:00:38.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.982982+0000", 2024-06-14T23:00:38.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1320000000000002e-05 2024-06-14T23:00:38.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.983076+0000", 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4166000000000006e-05 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.983143+0000", 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6921000000000005e-05 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.991084+0000", 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079414050000000003 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.000797+0000", 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097125920000000008 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.073200+0000", 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.072403228 2024-06-14T23:00:38.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.073250+0000", 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.003e-05 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.073295+0000", 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4505000000000002e-05 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.370 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.34776.0:178578 8.2 8:4ff5f054:::benchmark_data_smithi086_122122_object57989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:51:45.984926+0000", 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.34287082200001, 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.097774649000000005, 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.34776", 2024-06-14T23:00:38.371 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 178578 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.984926+0000", 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.984926+0000", 2024-06-14T23:00:38.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.984924+0000", 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.984930+0000", 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3539999999999999e-06 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.984942+0000", 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2364999999999999e-05 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.984948+0000", 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0100000000000001e-06 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.986723+0000", 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001775109 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.986822+0000", 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8468000000000007e-05 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.986927+0000", 2024-06-14T23:00:38.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010492000000000001 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995118+0000", 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081915979999999996 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.000834+0000", 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057153300000000002 2024-06-14T23:00:38.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.082642+0000", 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081808634000000005 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.082674+0000", 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2147000000000002e-05 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.082701+0000", 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6410999999999998e-05 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.34776.0:178581 8.2 8:4b1ea322:::benchmark_data_smithi086_122122_object57992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:51:45.995188+0000", 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.33260914799996, 2024-06-14T23:00:38.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.088126196000000004, 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.34776", 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/1416240303", 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 178581 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995188+0000", 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995188+0000", 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995186+0000", 2024-06-14T23:00:38.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995192+0000", 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1789999999999996e-06 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995210+0000", 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.751e-05 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995217+0000", 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.131e-06 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995271+0000", 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3893000000000001e-05 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995435+0000", 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000164408 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.995592+0000", 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000157324 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:45.999061+0000", 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034681379999999999 2024-06-14T23:00:38.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.000860+0000", 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017995999999999999 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.083259+0000", 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.082398785000000002 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.083288+0000", 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8846000000000001e-05 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:51:46.083314+0000", 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6166e-05 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.385 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:24732 9.e 9:7eaedcc6:::benchmark_data_smithi086_152062_object24731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:55:52.672402+0000", 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 285.655394715, 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.000701877, 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 24732 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672402+0000", 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672402+0000", 2024-06-14T23:00:38.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672400+0000", 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672569+0000", 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000168104 2024-06-14T23:00:38.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672584+0000", 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5814000000000001e-05 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672592+0000", 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4630000000000004e-06 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672659+0000", 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7403000000000005e-05 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672668+0000", 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5459999999999992e-06 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.665640+0000", 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99297187200000003 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.666460+0000", 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082062499999999996 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:00:38.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.666642+0000", 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018165999999999999 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672317+0000", 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056752010000000004 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672900+0000", 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058328000000000004 2024-06-14T23:00:38.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672946+0000", 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5955999999999998e-05 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.673063+0000", 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116419 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.673104+0000", 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1393999999999997e-05 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:24769 9.e 9:7ad8e74e:::benchmark_data_smithi086_152062_object24768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:55:52.688040+0000", 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 285.639756487, 2024-06-14T23:00:38.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98628556000000001, 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 24769 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688040+0000", 2024-06-14T23:00:38.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688040+0000", 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688032+0000", 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999914 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688071+0000", 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9511999999999997e-05 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688090+0000", 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8975e-05 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688099+0000", 2024-06-14T23:00:38.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7900000000000005e-06 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688164+0000", 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4812000000000002e-05 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688174+0000", 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7259999999999997e-06 2024-06-14T23:00:38.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.668016+0000", 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97984286799999998 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.668178+0000", 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000161087 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.668379+0000", 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000201475 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672327+0000", 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039483599999999997 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.673146+0000", 2024-06-14T23:00:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00081893899999999997 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674250+0000", 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011034510000000001 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674285+0000", 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5465e-05 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674326+0000", 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0779999999999999e-05 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:24787 9.e 9:7afcc34a:::benchmark_data_smithi086_152062_object24786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:55:52.701009+0000", 2024-06-14T23:00:38.402 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 285.62678844800001, 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97348232199999996, 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 24787 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701009+0000", 2024-06-14T23:00:38.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701009+0000", 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701007+0000", 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:00:38.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701112+0000", 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104862 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701129+0000", 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6694e-05 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701134+0000", 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7769999999999999e-06 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701188+0000", 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3649000000000001e-05 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:00:38.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701196+0000", 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7729999999999999e-06 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.669515+0000", 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.968319708 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.669670+0000", 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015500299999999999 2024-06-14T23:00:38.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.669830+0000", 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015971 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672380+0000", 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025496999999999998 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674225+0000", 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018455019999999999 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674392+0000", 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000166751 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674447+0000", 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.49e-05 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674491+0000", 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3803000000000001e-05 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:24789 9.e 9:760cab86:::benchmark_data_smithi086_152062_object24788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:00:38.410 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:55:52.702965+0000", 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 285.62483152599998, 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97231838199999998, 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 24789 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.411 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.702965+0000", 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.702965+0000", 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.702965+0000", 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703013+0000", 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7966000000000001e-05 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703024+0000", 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1094000000000001e-05 2024-06-14T23:00:38.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703028+0000", 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5750000000000002e-06 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703068+0000", 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9561000000000003e-05 2024-06-14T23:00:38.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703075+0000", 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8410000000000002e-06 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.670935+0000", 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96786069299999999 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:38.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.671087+0000", 2024-06-14T23:00:38.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000151366 2024-06-14T23:00:38.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.671240+0000", 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015322999999999999 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.673153+0000", 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001913465 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674539+0000", 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013852230000000001 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.675086+0000", 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00054754700000000001 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.675170+0000", 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4192999999999998e-05 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.675284+0000", 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113508 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84914 9.e 9:7e1d350b:::benchmark_data_smithi086_152062_object84913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:30.093445+0000", 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 248.23435230000001, 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.105775729, 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84914 2024-06-14T23:00:38.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093445+0000", 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093445+0000", 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093444+0000", 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093508+0000", 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3757000000000006e-05 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093519+0000", 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0881e-05 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093524+0000", 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1619999999999999e-06 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.096870+0000", 2024-06-14T23:00:38.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033461860000000001 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.097013+0000", 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014290700000000001 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.097189+0000", 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000175835 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.100394+0000", 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032054779999999999 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.103706+0000", 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033121270000000002 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.199150+0000", 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095443745999999996 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.199190+0000", 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0014e-05 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.199220+0000", 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0354000000000001e-05 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.427 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84933 9.e 9:7994e0bc:::benchmark_data_smithi086_152062_object84932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:30.112138+0000", 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 248.215658574, 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.092303347999999993, 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84933 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112138+0000", 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112138+0000", 2024-06-14T23:00:38.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112137+0000", 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112208+0000", 2024-06-14T23:00:38.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1501999999999994e-05 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112224+0000", 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6008999999999999e-05 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112232+0000", 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6790000000000005e-06 2024-06-14T23:00:38.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112301+0000", 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9287000000000003e-05 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112439+0000", 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013751200000000001 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112628+0000", 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018896599999999999 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.115361+0000", 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002733564 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.115839+0000", 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047746600000000001 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.204341+0000", 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.088502239999999996 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.204397+0000", 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5859000000000002e-05 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.204442+0000", 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5028000000000003e-05 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:247149 9.c 9:38b204dd:::benchmark_data_smithi086_152062_object114667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:25.606397+0000", 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 72.721399704999996, 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045409085000000002, 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 247149 2024-06-14T23:00:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.606397+0000", 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.606397+0000", 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.606397+0000", 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.606401+0000", 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3760000000000001e-06 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.606409+0000", 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1799999999999996e-06 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.606413+0000", 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7440000000000001e-06 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.611546+0000", 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051331709999999997 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.611663+0000", 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117038 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.611737+0000", 2024-06-14T23:00:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3726000000000005e-05 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.618114+0000", 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063770140000000003 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.618488+0000", 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037401399999999999 2024-06-14T23:00:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.651572+0000", 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033084253000000001 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.651691+0000", 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011913099999999999 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:25.651806+0000", 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115074 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:247658 9.c 9:30c2aed1:::benchmark_data_smithi086_152062_object115176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:26.196997+0000", 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 72.130800072, 2024-06-14T23:00:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.038508997000000003, 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 247658 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.196997+0000", 2024-06-14T23:00:38.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.196997+0000", 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.196995+0000", 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:00:38.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.197002+0000", 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3450000000000002e-06 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.197015+0000", 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3531e-05 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.197020+0000", 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6689999999999997e-06 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.197073+0000", 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3254999999999997e-05 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.197254+0000", 2024-06-14T23:00:38.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180741 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.197399+0000", 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014527999999999999 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.201951+0000", 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045512679999999998 2024-06-14T23:00:38.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.203106+0000", 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011551199999999999 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.235439+0000", 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032333058999999997 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.235470+0000", 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0951999999999998e-05 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.235506+0000", 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6186000000000002e-05 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:248160 9.c 9:3dca8880:::benchmark_data_smithi086_152062_object115678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:26.756023+0000", 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 71.571774305999995, 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.040907477999999997, 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.451 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 248160 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.756023+0000", 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.756023+0000", 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.756022+0000", 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.756028+0000", 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.789e-06 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.756043+0000", 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5170999999999999e-05 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.756049+0000", 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8409999999999996e-06 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.765195+0000", 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091464470000000003 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.765324+0000", 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129445 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.765443+0000", 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118471 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.773701+0000", 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082583029999999998 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.778411+0000", 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047101790000000001 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.796850+0000", 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018438808000000001 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.796891+0000", 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0466000000000002e-05 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:26.796930+0000", 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9499000000000002e-05 2024-06-14T23:00:38.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:249098 9.c 9:33ee5914:::benchmark_data_smithi086_152062_object116616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:27.838253+0000", 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 70.489544131000002, 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.042374022999999997, 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.458 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 249098 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.838253+0000", 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.838253+0000", 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.838251+0000", 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.838258+0000", 2024-06-14T23:00:38.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7170000000000008e-06 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.838277+0000", 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8164000000000001e-05 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.838284+0000", 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1010000000000003e-06 2024-06-14T23:00:38.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.841606+0000", 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003322064 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.841795+0000", 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018946100000000001 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.841925+0000", 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129567 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.850899+0000", 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0089737989999999993 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.850999+0000", 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100616 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.880522+0000", 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029522535999999999 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.880582+0000", 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0355000000000001e-05 2024-06-14T23:00:38.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.880627+0000", 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4776e-05 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:249101 9.c 9:3043d3b6:::benchmark_data_smithi086_152062_object116619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:27.842164+0000", 2024-06-14T23:00:38.465 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 70.485632903999999, 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.038594590999999998, 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 249101 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.842164+0000", 2024-06-14T23:00:38.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.842164+0000", 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.842163+0000", 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.842168+0000", 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9010000000000002e-06 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.842180+0000", 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1762e-05 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.842185+0000", 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2039999999999996e-06 2024-06-14T23:00:38.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.846413+0000", 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042284540000000004 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.846488+0000", 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5119999999999999e-05 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.846586+0000", 2024-06-14T23:00:38.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7627000000000006e-05 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.855618+0000", 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090318810000000003 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.855752+0000", 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000134077 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.880687+0000", 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024934642 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.880727+0000", 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0500999999999997e-05 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:27.880759+0000", 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1572000000000002e-05 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:255150 9.e 9:7ca39f35:::benchmark_data_smithi086_152062_object122668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:34.822981+0000", 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 63.50481585, 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041914246000000002, 2024-06-14T23:00:38.472 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 255150 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.822981+0000", 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.822981+0000", 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.822981+0000", 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.822983+0000", 2024-06-14T23:00:38.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.785e-06 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.822990+0000", 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0759999999999999e-06 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.822993+0000", 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5829999999999998e-06 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.829120+0000", 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061264010000000001 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.829243+0000", 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123659 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.829364+0000", 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120819 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.844822+0000", 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015457820000000001 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.855146+0000", 2024-06-14T23:00:38.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010324261 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.864585+0000", 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094392710000000008 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.864752+0000", 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000166229 2024-06-14T23:00:38.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:34.864895+0000", 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014379599999999999 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:38.479 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:00:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:38.481 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:38.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:38.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:38.615 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:38.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:38.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:38.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:38.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:38.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:38.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:38.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:38.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:39.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:39.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:39.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:39.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:39.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:39.209 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 16 7429 7413 231.64 223.812 0.00541634 0.0043096 2024-06-14T23:00:39.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:39.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:39.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:39.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T23:00:39.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:39.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:39.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:39.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:39.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:39.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:39.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:39.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:39.738 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:00:39.785 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:00:39.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:39.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:39.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:39.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:39.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:39.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:40.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:39 smithi137 ceph-mon[90634]: pgmap v744: 99 pgs: 99 active+clean; 634 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:40.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:39 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2955776594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:00:40.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:39 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:00:40.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:39 smithi086 ceph-mon[97913]: pgmap v744: 99 pgs: 99 active+clean; 634 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:40.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:39 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2955776594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:00:40.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:39 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:00:40.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:39 smithi086 ceph-mon[102754]: pgmap v744: 99 pgs: 99 active+clean; 634 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-06-14T23:00:40.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:39 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2955776594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:00:40.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:39 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:00:40.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:40.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:40.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:40.209 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 16 10474 10458 217.858 190.312 0.00443036 0.00458397 2024-06-14T23:00:40.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:40.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:40.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:40.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:40.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:40.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:40.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:40.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:40.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:40.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:40.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:40.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:40.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:40.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:40.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:40.908 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:40.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:40.958 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:40.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:40.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:41.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:41.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:40 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:00:41.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:40 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:00:41.063 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:40 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:00:41.063 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:40 smithi086 ceph-mon[97913]: osdmap e191: 8 total, 8 up, 8 in 2024-06-14T23:00:41.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:40 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:00:41.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:40 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:00:41.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:40 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:00:41.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:40 smithi086 ceph-mon[102754]: osdmap e191: 8 total, 8 up, 8 in 2024-06-14T23:00:41.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:41.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:41.137 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:40 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:00:41.137 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:40 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:00:41.138 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:40 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:00:41.138 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:40 smithi137 ceph-mon[90634]: osdmap e191: 8 total, 8 up, 8 in 2024-06-14T23:00:41.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:41.209 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 16 13568 13552 211.733 193.375 0.00852503 0.00471582 2024-06-14T23:00:41.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:41.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:41.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:41.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:41.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:41.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:41.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:41.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:41.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:41.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:41.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:41.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:41.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:41.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:41.848 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:00:41.850 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:00:41.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:41.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:41.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:41.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:41 smithi137 ceph-mon[90634]: pgmap v746: 99 pgs: 99 active+clean; 386 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 48 MiB/s wr, 748 op/s 2024-06-14T23:00:42.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:42.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:42.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:42.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:41 smithi086 ceph-mon[102754]: pgmap v746: 99 pgs: 99 active+clean; 386 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 48 MiB/s wr, 748 op/s 2024-06-14T23:00:42.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:41 smithi086 ceph-mon[97913]: pgmap v746: 99 pgs: 99 active+clean; 386 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 48 MiB/s wr, 748 op/s 2024-06-14T23:00:42.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:42.209 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 16 16270 16254 203.158 168.875 0.00599677 0.00491579 2024-06-14T23:00:42.212 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:42.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:42.225 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:16286 10.6 10:6730eac7:::benchmark_data_smithi086_189745_object16285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:42.215298+0000", 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0085327340000000002, 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0085673220000000005, 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:42.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 16286 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.215298+0000", 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:42.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.215298+0000", 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.215297+0000", 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.215371+0000", 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4313999999999994e-05 2024-06-14T23:00:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.215383+0000", 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2081e-05 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.215389+0000", 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1840000000000004e-06 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.215439+0000", 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9731999999999998e-05 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.215563+0000", 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123688 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.215693+0000", 2024-06-14T23:00:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013045600000000001 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.217353+0000", 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016596600000000001 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.217588+0000", 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023516300000000001 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:16288 10.6 10:687edbcb:::benchmark_data_smithi086_189745_object16287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:42.219550+0000", 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0042809190000000002, 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004410977, 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:00:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 16288 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.219550+0000", 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.219550+0000", 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.219548+0000", 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.219586+0000", 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7481000000000002e-05 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.219612+0000", 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6682999999999998e-05 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.219620+0000", 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3590000000000002e-06 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.219659+0000", 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9551999999999997e-05 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.219781+0000", 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121519 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.219925+0000", 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014456000000000001 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:00:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.220873+0000", 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00094733000000000003 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:16296 10.6 10:6856c282:::benchmark_data_smithi086_189745_object16295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:42.221280+0000", 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.002550615, 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027543810000000002, 2024-06-14T23:00:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 16296 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.221280+0000", 2024-06-14T23:00:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.221280+0000", 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.221279+0000", 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.221397+0000", 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011772799999999999 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.221410+0000", 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3464e-05 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.221415+0000", 2024-06-14T23:00:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4360000000000001e-06 2024-06-14T23:00:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.221457+0000", 2024-06-14T23:00:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.125e-05 2024-06-14T23:00:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.221608+0000", 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000151168 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.222232+0000", 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00062461500000000002 2024-06-14T23:00:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:16300 10.6 10:69c2754f:::benchmark_data_smithi086_189745_object16299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:42.222087+0000", 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0017435790000000001, 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020295140000000001, 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:00:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 16300 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.222087+0000", 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.222087+0000", 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.222086+0000", 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.222168+0000", 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2237000000000006e-05 2024-06-14T23:00:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.222179+0000", 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0709e-05 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.222185+0000", 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.677e-06 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.223047+0000", 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00086278600000000004 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.223155+0000", 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107558 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.223271+0000", 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115785 2024-06-14T23:00:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.45821.0:16288 10.6 e191/186)", 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:42.223492+0000", 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00033824700000000002, 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000691235, 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:00:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.223492+0000", 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.223492+0000", 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:00:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.223492+0000", 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.223496+0000", 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3460000000000004e-06 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.223505+0000", 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7050000000000005e-06 2024-06-14T23:00:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:42.223509+0000", 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2289999999999997e-06 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 5 2024-06-14T23:00:42.251 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:00:42.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:42.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:42.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:42.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:42.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:42.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:42.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:42.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:42.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:42.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:42.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:42.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:42.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:42.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:42.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:42.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:00:42.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:42 smithi086 ceph-mon[97913]: from='client.35922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:00:42.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:42.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:42.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:42.951 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:42 smithi086 ceph-mon[102754]: from='client.35922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:00:42.951 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:42.951 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:42.951 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:42.993 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:42 smithi137 ceph-mon[90634]: from='client.35922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:00:42.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:42.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:42.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:42.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:43.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:43.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:43.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:43.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:43.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:43.173 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2024-06-14T23:00:40.671930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":404351340,"num_objects":6184,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6184,"num_whiteouts":0,"num_read":44,"num_read_kb":83,"num_write":6243,"num_write_kb":394489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7986,"ondisk_log_size":7986,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5459608,"kb_used_data":922324,"kb_used_omap":29,"kb_used_meta":4537186,"kb_avail":744501608,"statfs":{"total":767960285184,"available":762369646592,"internally_reserved":0,"allocated":944459776,"data_stored":936792302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":4646079141},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":403862769,"num_objects":6161,"num_object_clones":0,"num_object_copies":18483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6161,"num_whiteouts":0,"num_read":-353,"num_read_kb":-664,"num_write":5986,"num_write_kb":392483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-7,"num_bytes_recovered":-1878444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001940"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":121,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832717+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:00:34.832717+0000","last_peered":"2024-06-14T23:00:34.832717+0000","last_clean":"2024-06-14T23:00:34.832717+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:00:34.832717+0000","last_undegraded":"2024-06-14T23:00:34.832717+0000","last_fullsized":"2024-06-14T23:00:34.832717+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":153,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.568012+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T23:00:30.568012+0000","last_peered":"2024-06-14T23:00:30.568012+0000","last_clean":"2024-06-14T23:00:30.568012+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T23:00:30.568012+0000","last_undegraded":"2024-06-14T23:00:30.568012+0000","last_fullsized":"2024-06-14T23:00:30.568012+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":150,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567592+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:00:30.567592+0000","last_peered":"2024-06-14T23:00:30.567592+0000","last_clean":"2024-06-14T23:00:30.567592+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:00:30.567592+0000","last_undegraded":"2024-06-14T23:00:30.567592+0000","last_fullsized":"2024-06-14T23:00:30.567592+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":141,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.566901+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:00:30.566901+0000","last_peered":"2024-06-14T23:00:30.566901+0000","last_clean":"2024-06-14T23:00:30.566901+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:00:30.566901+0000","last_undegraded":"2024-06-14T23:00:30.566901+0000","last_fullsized":"2024-06-14T23:00:30.566901+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":142,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.566908+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T23:00:30.566908+0000","last_peered":"2024-06-14T23:00:30.566908+0000","last_clean":"2024-06-14T23:00:30.566908+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T23:00:30.566908+0000","last_undegraded":"2024-06-14T23:00:30.566908+0000","last_fullsized":"2024-06-14T23:00:30.566908+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":57,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018366+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T23:00:30.018366+0000","last_peered":"2024-06-14T23:00:30.018366+0000","last_clean":"2024-06-14T23:00:30.018366+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T23:00:30.018366+0000","last_undegraded":"2024-06-14T23:00:30.018366+0000","last_fullsized":"2024-06-14T23:00:30.018366+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":143,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:29.815247+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:00:29.815247+0000","last_peered":"2024-06-14T23:00:29.815247+0000","last_clean":"2024-06-14T23:00:29.815247+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:00:29.815247+0000","last_undegraded":"2024-06-14T23:00:29.815247+0000","last_fullsized":"2024-06-14T23:00:29.815247+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":159,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.541097+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T23:00:30.541097+0000","last_peered":"2024-06-14T23:00:30.541097+0000","last_clean":"2024-06-14T23:00:30.541097+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T23:00:30.541097+0000","last_undegraded":"2024-06-14T23:00:30.541097+0000","last_fullsized":"2024-06-14T23:00:30.541097+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"190'460","reported_seq":469,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:39.210137+0000","last_change":"2024-06-14T23:00:27.800011+0000","last_active":"2024-06-14T23:00:39.210137+0000","last_peered":"2024-06-14T23:00:39.210137+0000","last_clean":"2024-06-14T23:00:39.210137+0000","last_became_active":"2024-06-14T23:00:27.799716+0000","last_became_peered":"2024-06-14T23:00:27.799716+0000","last_unstale":"2024-06-14T23:00:39.210137+0000","last_undegraded":"2024-06-14T23:00:39.210137+0000","last_fullsized":"2024-06-14T23:00:39.210137+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:41:34.619251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":460,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":460,"num_write_kb":29440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":140,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567997+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:00:30.567997+0000","last_peered":"2024-06-14T23:00:30.567997+0000","last_clean":"2024-06-14T23:00:30.567997+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:00:30.567997+0000","last_undegraded":"2024-06-14T23:00:30.567997+0000","last_fullsized":"2024-06-14T23:00:30.567997+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":133,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567706+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T23:00:30.567706+0000","last_peered":"2024-06-14T23:00:30.567706+0000","last_clean":"2024-06-14T23:00:30.567706+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T23:00:30.567706+0000","last_undegraded":"2024-06-14T23:00:30.567706+0000","last_fullsized":"2024-06-14T23:00:30.567706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"190'349","reported_seq":361,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:38.753673+0000","last_change":"2024-06-14T23:00:27.798857+0000","last_active":"2024-06-14T23:00:38.753673+0000","last_peered":"2024-06-14T23:00:38.753673+0000","last_clean":"2024-06-14T23:00:38.753673+0000","last_became_active":"2024-06-14T23:00:27.798566+0000","last_became_peered":"2024-06-14T23:00:27.798566+0000","last_unstale":"2024-06-14T23:00:38.753673+0000","last_undegraded":"2024-06-14T23:00:38.753673+0000","last_fullsized":"2024-06-14T23:00:38.753673+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:38.237668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22872064,"num_objects":349,"num_object_clones":0,"num_object_copies":1047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":349,"num_write_kb":22336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":141,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567858+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:00:30.567858+0000","last_peered":"2024-06-14T23:00:30.567858+0000","last_clean":"2024-06-14T23:00:30.567858+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:00:30.567858+0000","last_undegraded":"2024-06-14T23:00:30.567858+0000","last_fullsized":"2024-06-14T23:00:30.567858+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":167,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018542+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:00:30.018542+0000","last_peered":"2024-06-14T23:00:30.018542+0000","last_clean":"2024-06-14T23:00:30.018542+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:00:30.018542+0000","last_undegraded":"2024-06-14T23:00:30.018542+0000","last_fullsized":"2024-06-14T23:00:30.018542+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:31.713352+0000","last_change":"2024-06-14T23:00:27.797882+0000","last_active":"2024-06-14T23:00:31.713352+0000","last_peered":"2024-06-14T23:00:31.713352+0000","last_clean":"2024-06-14T23:00:31.713352+0000","last_became_active":"2024-06-14T23:00:27.797600+0000","last_became_peered":"2024-06-14T23:00:27.797600+0000","last_unstale":"2024-06-14T23:00:31.713352+0000","last_undegraded":"2024-06-14T23:00:31.713352+0000","last_fullsized":"2024-06-14T23:00:31.713352+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:04:20.589709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":68,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567796+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T23:00:30.567796+0000","last_peered":"2024-06-14T23:00:30.567796+0000","last_clean":"2024-06-14T23:00:30.567796+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T23:00:30.567796+0000","last_undegraded":"2024-06-14T23:00:30.567796+0000","last_fullsized":"2024-06-14T23:00:30.567796+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":150,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.540767+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T23:00:30.540767+0000","last_peered":"2024-06-14T23:00:30.540767+0000","last_clean":"2024-06-14T23:00:30.540767+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T23:00:30.540767+0000","last_undegraded":"2024-06-14T23:00:30.540767+0000","last_fullsized":"2024-06-14T23:00:30.540767+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"191'631","reported_seq":645,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-06-14T23:00:40.202562+0000","last_change":"2024-06-14T23:00:27.794220+0000","last_active":"2024-06-14T23:00:40.202562+0000","last_peered":"2024-06-14T23:00:40.202562+0000","last_clean":"2024-06-14T23:00:40.202562+0000","last_became_active":"2024-06-14T23:00:27.793960+0000","last_became_peered":"2024-06-14T23:00:27.793960+0000","last_unstale":"2024-06-14T23:00:40.202562+0000","last_undegraded":"2024-06-14T23:00:40.202562+0000","last_fullsized":"2024-06-14T23:00:40.202562+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":631,"log_dups_size":0,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:04:26.933537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41353216,"num_objects":631,"num_object_clones":0,"num_object_copies":1893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":631,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":151,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.568095+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T23:00:30.568095+0000","last_peered":"2024-06-14T23:00:30.568095+0000","last_clean":"2024-06-14T23:00:30.568095+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T23:00:30.568095+0000","last_undegraded":"2024-06-14T23:00:30.568095+0000","last_fullsized":"2024-06-14T23:00:30.568095+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":143,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.852593+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T23:00:34.852593+0000","last_peered":"2024-06-14T23:00:34.852593+0000","last_clean":"2024-06-14T23:00:34.852593+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T23:00:34.852593+0000","last_undegraded":"2024-06-14T23:00:34.852593+0000","last_fullsized":"2024-06-14T23:00:34.852593+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"191'636","reported_seq":649,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-06-14T23:00:39.994459+0000","last_change":"2024-06-14T23:00:27.797355+0000","last_active":"2024-06-14T23:00:39.994459+0000","last_peered":"2024-06-14T23:00:39.994459+0000","last_clean":"2024-06-14T23:00:39.994459+0000","last_became_active":"2024-06-14T23:00:27.796666+0000","last_became_peered":"2024-06-14T23:00:27.796666+0000","last_unstale":"2024-06-14T23:00:39.994459+0000","last_undegraded":"2024-06-14T23:00:39.994459+0000","last_fullsized":"2024-06-14T23:00:39.994459+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:57:29.139721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":636,"num_write_kb":40704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":489,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.851157+0000","last_change":"2024-06-14T23:00:30.637455+0000","last_active":"2024-06-14T23:00:34.851157+0000","last_peered":"2024-06-14T23:00:34.851157+0000","last_clean":"2024-06-14T23:00:34.851157+0000","last_became_active":"2024-06-14T23:00:17.398732+0000","last_became_peered":"2024-06-14T23:00:17.398732+0000","last_unstale":"2024-06-14T23:00:34.851157+0000","last_undegraded":"2024-06-14T23:00:34.851157+0000","last_fullsized":"2024-06-14T23:00:34.851157+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.637373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.637373+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:10.033075+0000","scrub_duration":0.0016781470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":59,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567257+0000","last_change":"2024-06-14T22:58:09.752985+0000","last_active":"2024-06-14T23:00:30.567257+0000","last_peered":"2024-06-14T23:00:30.567257+0000","last_clean":"2024-06-14T23:00:30.567257+0000","last_became_active":"2024-06-14T22:58:09.752630+0000","last_became_peered":"2024-06-14T22:58:09.752630+0000","last_unstale":"2024-06-14T23:00:30.567257+0000","last_undegraded":"2024-06-14T23:00:30.567257+0000","last_fullsized":"2024-06-14T23:00:30.567257+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:22.320986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":150,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018724+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T23:00:30.018724+0000","last_peered":"2024-06-14T23:00:30.018724+0000","last_clean":"2024-06-14T23:00:30.018724+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T23:00:30.018724+0000","last_undegraded":"2024-06-14T23:00:30.018724+0000","last_fullsized":"2024-06-14T23:00:30.018724+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"191'653","reported_seq":667,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-06-14T23:00:40.198688+0000","last_change":"2024-06-14T23:00:27.797468+0000","last_active":"2024-06-14T23:00:40.198688+0000","last_peered":"2024-06-14T23:00:40.198688+0000","last_clean":"2024-06-14T23:00:40.198688+0000","last_became_active":"2024-06-14T23:00:27.797180+0000","last_became_peered":"2024-06-14T23:00:27.797180+0000","last_unstale":"2024-06-14T23:00:40.198688+0000","last_undegraded":"2024-06-14T23:00:40.198688+0000","last_fullsized":"2024-06-14T23:00:40.198688+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":653,"log_dups_size":0,"ondisk_log_size":653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:56:13.872188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42795008,"num_objects":653,"num_object_clones":0,"num_object_copies":1959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":653,"num_write_kb":41792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":487,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:33.815813+0000","last_change":"2024-06-14T23:00:33.815761+0000","last_active":"2024-06-14T23:00:33.815813+0000","last_peered":"2024-06-14T23:00:33.815813+0000","last_clean":"2024-06-14T23:00:33.815813+0000","last_became_active":"2024-06-14T23:00:17.400781+0000","last_became_peered":"2024-06-14T23:00:17.400781+0000","last_unstale":"2024-06-14T23:00:33.815813+0000","last_undegraded":"2024-06-14T23:00:33.815813+0000","last_fullsized":"2024-06-14T23:00:33.815813+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:21:41.196504+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":151,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.541100+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T23:00:30.541100+0000","last_peered":"2024-06-14T23:00:30.541100+0000","last_clean":"2024-06-14T23:00:30.541100+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T23:00:30.541100+0000","last_undegraded":"2024-06-14T23:00:30.541100+0000","last_fullsized":"2024-06-14T23:00:30.541100+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":160,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018999+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T23:00:30.018999+0000","last_peered":"2024-06-14T23:00:30.018999+0000","last_clean":"2024-06-14T23:00:30.018999+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T23:00:30.018999+0000","last_undegraded":"2024-06-14T23:00:30.018999+0000","last_fullsized":"2024-06-14T23:00:30.018999+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"190'347","reported_seq":359,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:38.753445+0000","last_change":"2024-06-14T23:00:27.802274+0000","last_active":"2024-06-14T23:00:38.753445+0000","last_peered":"2024-06-14T23:00:38.753445+0000","last_clean":"2024-06-14T23:00:38.753445+0000","last_became_active":"2024-06-14T23:00:27.801956+0000","last_became_peered":"2024-06-14T23:00:27.801956+0000","last_unstale":"2024-06-14T23:00:38.753445+0000","last_undegraded":"2024-06-14T23:00:38.753445+0000","last_fullsized":"2024-06-14T23:00:38.753445+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:23.791512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":347,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":151,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018576+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T23:00:30.018576+0000","last_peered":"2024-06-14T23:00:30.018576+0000","last_clean":"2024-06-14T23:00:30.018576+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T23:00:30.018576+0000","last_undegraded":"2024-06-14T23:00:30.018576+0000","last_fullsized":"2024-06-14T23:00:30.018576+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":150,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018972+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T23:00:30.018972+0000","last_peered":"2024-06-14T23:00:30.018972+0000","last_clean":"2024-06-14T23:00:30.018972+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T23:00:30.018972+0000","last_undegraded":"2024-06-14T23:00:30.018972+0000","last_fullsized":"2024-06-14T23:00:30.018972+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"190'366","reported_seq":378,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:38.715382+0000","last_change":"2024-06-14T23:00:27.801110+0000","last_active":"2024-06-14T23:00:38.715382+0000","last_peered":"2024-06-14T23:00:38.715382+0000","last_clean":"2024-06-14T23:00:38.715382+0000","last_became_active":"2024-06-14T23:00:27.800817+0000","last_became_peered":"2024-06-14T23:00:27.800817+0000","last_unstale":"2024-06-14T23:00:38.715382+0000","last_undegraded":"2024-06-14T23:00:38.715382+0000","last_fullsized":"2024-06-14T23:00:38.715382+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:48:04.422151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23986176,"num_objects":366,"num_object_clones":0,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":366,"num_write_kb":23424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":487,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:31.843471+0000","last_change":"2024-06-14T23:00:31.843436+0000","last_active":"2024-06-14T23:00:31.843471+0000","last_peered":"2024-06-14T23:00:31.843471+0000","last_clean":"2024-06-14T23:00:31.843471+0000","last_became_active":"2024-06-14T23:00:17.398181+0000","last_became_peered":"2024-06-14T23:00:17.398181+0000","last_unstale":"2024-06-14T23:00:31.843471+0000","last_undegraded":"2024-06-14T23:00:31.843471+0000","last_fullsized":"2024-06-14T23:00:31.843471+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:05:06.069424+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":145,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.533816+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:00:34.533816+0000","last_peered":"2024-06-14T23:00:34.533816+0000","last_clean":"2024-06-14T23:00:34.533816+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:00:34.533816+0000","last_undegraded":"2024-06-14T23:00:34.533816+0000","last_fullsized":"2024-06-14T23:00:34.533816+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":159,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-06-14T23:00:39.735669+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:00:39.735669+0000","last_peered":"2024-06-14T23:00:39.735669+0000","last_clean":"2024-06-14T23:00:39.735669+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:00:39.735669+0000","last_undegraded":"2024-06-14T23:00:39.735669+0000","last_fullsized":"2024-06-14T23:00:39.735669+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"190'376","reported_seq":388,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:38.747837+0000","last_change":"2024-06-14T23:00:27.797497+0000","last_active":"2024-06-14T23:00:38.747837+0000","last_peered":"2024-06-14T23:00:38.747837+0000","last_clean":"2024-06-14T23:00:38.747837+0000","last_became_active":"2024-06-14T23:00:27.797221+0000","last_became_peered":"2024-06-14T23:00:27.797221+0000","last_unstale":"2024-06-14T23:00:38.747837+0000","last_undegraded":"2024-06-14T23:00:38.747837+0000","last_fullsized":"2024-06-14T23:00:38.747837+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:24:58.530131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":376,"num_write_kb":24064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":479,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.610273+0000","last_change":"2024-06-14T23:00:30.610273+0000","last_active":"2024-06-14T23:00:30.610273+0000","last_peered":"2024-06-14T23:00:30.610273+0000","last_clean":"2024-06-14T23:00:30.610273+0000","last_became_active":"2024-06-14T23:00:09.766095+0000","last_became_peered":"2024-06-14T23:00:09.766095+0000","last_unstale":"2024-06-14T23:00:30.610273+0000","last_undegraded":"2024-06-14T23:00:30.610273+0000","last_fullsized":"2024-06-14T23:00:30.610273+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.610180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.610180+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:53.521891+0000","scrub_duration":0.00217396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":155,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567974+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T23:00:30.567974+0000","last_peered":"2024-06-14T23:00:30.567974+0000","last_clean":"2024-06-14T23:00:30.567974+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T23:00:30.567974+0000","last_undegraded":"2024-06-14T23:00:30.567974+0000","last_fullsized":"2024-06-14T23:00:30.567974+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":94,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.568439+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T23:00:30.568439+0000","last_peered":"2024-06-14T23:00:30.568439+0000","last_clean":"2024-06-14T23:00:30.568439+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T23:00:30.568439+0000","last_undegraded":"2024-06-14T23:00:30.568439+0000","last_fullsized":"2024-06-14T23:00:30.568439+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":133,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.541174+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T23:00:30.541174+0000","last_peered":"2024-06-14T23:00:30.541174+0000","last_clean":"2024-06-14T23:00:30.541174+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T23:00:30.541174+0000","last_undegraded":"2024-06-14T23:00:30.541174+0000","last_fullsized":"2024-06-14T23:00:30.541174+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":13,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:31.713287+0000","last_change":"2024-06-14T23:00:27.797246+0000","last_active":"2024-06-14T23:00:31.713287+0000","last_peered":"2024-06-14T23:00:31.713287+0000","last_clean":"2024-06-14T23:00:31.713287+0000","last_became_active":"2024-06-14T23:00:27.796988+0000","last_became_peered":"2024-06-14T23:00:27.796988+0000","last_unstale":"2024-06-14T23:00:31.713287+0000","last_undegraded":"2024-06-14T23:00:31.713287+0000","last_fullsized":"2024-06-14T23:00:31.713287+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:17.545715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":489,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832664+0000","last_change":"2024-06-14T23:00:30.719758+0000","last_active":"2024-06-14T23:00:34.832664+0000","last_peered":"2024-06-14T23:00:34.832664+0000","last_clean":"2024-06-14T23:00:34.832664+0000","last_became_active":"2024-06-14T23:00:17.403817+0000","last_became_peered":"2024-06-14T23:00:17.403817+0000","last_unstale":"2024-06-14T23:00:34.832664+0000","last_undegraded":"2024-06-14T23:00:34.832664+0000","last_fullsized":"2024-06-14T23:00:34.832664+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:03:47.728089+0000","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":149,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832216+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:00:34.832216+0000","last_peered":"2024-06-14T23:00:34.832216+0000","last_clean":"2024-06-14T23:00:34.832216+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:00:34.832216+0000","last_undegraded":"2024-06-14T23:00:34.832216+0000","last_fullsized":"2024-06-14T23:00:34.832216+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":142,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.566856+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T23:00:30.566856+0000","last_peered":"2024-06-14T23:00:30.566856+0000","last_clean":"2024-06-14T23:00:30.566856+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T23:00:30.566856+0000","last_undegraded":"2024-06-14T23:00:30.566856+0000","last_fullsized":"2024-06-14T23:00:30.566856+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":156,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567509+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T23:00:30.567509+0000","last_peered":"2024-06-14T23:00:30.567509+0000","last_clean":"2024-06-14T23:00:30.567509+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T23:00:30.567509+0000","last_undegraded":"2024-06-14T23:00:30.567509+0000","last_fullsized":"2024-06-14T23:00:30.567509+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"190'453","reported_seq":465,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:39.204899+0000","last_change":"2024-06-14T23:00:27.801838+0000","last_active":"2024-06-14T23:00:39.204899+0000","last_peered":"2024-06-14T23:00:39.204899+0000","last_clean":"2024-06-14T23:00:39.204899+0000","last_became_active":"2024-06-14T23:00:27.801477+0000","last_became_peered":"2024-06-14T23:00:27.801477+0000","last_unstale":"2024-06-14T23:00:39.204899+0000","last_undegraded":"2024-06-14T23:00:39.204899+0000","last_fullsized":"2024-06-14T23:00:39.204899+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:59:17.239200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29687808,"num_objects":453,"num_object_clones":0,"num_object_copies":1359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":453,"num_write_kb":28992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":496,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:31.643024+0000","last_change":"2024-06-14T23:00:31.643024+0000","last_active":"2024-06-14T23:00:31.643024+0000","last_peered":"2024-06-14T23:00:31.643024+0000","last_clean":"2024-06-14T23:00:31.643024+0000","last_became_active":"2024-06-14T23:00:17.400819+0000","last_became_peered":"2024-06-14T23:00:17.400819+0000","last_unstale":"2024-06-14T23:00:31.643024+0000","last_undegraded":"2024-06-14T23:00:31.643024+0000","last_fullsized":"2024-06-14T23:00:31.643024+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.642940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.642940+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:52:45.475940+0000","scrub_duration":0.001647253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":144,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.851428+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T23:00:34.851428+0000","last_peered":"2024-06-14T23:00:34.851428+0000","last_clean":"2024-06-14T23:00:34.851428+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T23:00:34.851428+0000","last_undegraded":"2024-06-14T23:00:34.851428+0000","last_fullsized":"2024-06-14T23:00:34.851428+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":159,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832396+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T23:00:34.832396+0000","last_peered":"2024-06-14T23:00:34.832396+0000","last_clean":"2024-06-14T23:00:34.832396+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T23:00:34.832396+0000","last_undegraded":"2024-06-14T23:00:34.832396+0000","last_fullsized":"2024-06-14T23:00:34.832396+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":144,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832555+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:00:34.832555+0000","last_peered":"2024-06-14T23:00:34.832555+0000","last_clean":"2024-06-14T23:00:34.832555+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:00:34.832555+0000","last_undegraded":"2024-06-14T23:00:34.832555+0000","last_fullsized":"2024-06-14T23:00:34.832555+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"190'378","reported_seq":390,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:38.751049+0000","last_change":"2024-06-14T23:00:27.799968+0000","last_active":"2024-06-14T23:00:38.751049+0000","last_peered":"2024-06-14T23:00:38.751049+0000","last_clean":"2024-06-14T23:00:38.751049+0000","last_became_active":"2024-06-14T23:00:27.799651+0000","last_became_peered":"2024-06-14T23:00:27.799651+0000","last_unstale":"2024-06-14T23:00:38.751049+0000","last_undegraded":"2024-06-14T23:00:38.751049+0000","last_fullsized":"2024-06-14T23:00:38.751049+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:23:57.581265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24772608,"num_objects":378,"num_object_clones":0,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":378,"num_write_kb":24192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":486,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:32.818174+0000","last_change":"2024-06-14T23:00:32.818134+0000","last_active":"2024-06-14T23:00:32.818174+0000","last_peered":"2024-06-14T23:00:32.818174+0000","last_clean":"2024-06-14T23:00:32.818174+0000","last_became_active":"2024-06-14T23:00:17.399852+0000","last_became_peered":"2024-06-14T23:00:17.399852+0000","last_unstale":"2024-06-14T23:00:32.818174+0000","last_undegraded":"2024-06-14T23:00:32.818174+0000","last_fullsized":"2024-06-14T23:00:32.818174+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.818076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.818076+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:12:39.666043+0000","scrub_duration":0.0013568390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":155,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.568501+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T23:00:30.568501+0000","last_peered":"2024-06-14T23:00:30.568501+0000","last_clean":"2024-06-14T23:00:30.568501+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T23:00:30.568501+0000","last_undegraded":"2024-06-14T23:00:30.568501+0000","last_fullsized":"2024-06-14T23:00:30.568501+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":151,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018713+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T23:00:30.018713+0000","last_peered":"2024-06-14T23:00:30.018713+0000","last_clean":"2024-06-14T23:00:30.018713+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T23:00:30.018713+0000","last_undegraded":"2024-06-14T23:00:30.018713+0000","last_fullsized":"2024-06-14T23:00:30.018713+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":174,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832497+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T23:00:34.832497+0000","last_peered":"2024-06-14T23:00:34.832497+0000","last_clean":"2024-06-14T23:00:34.832497+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T23:00:34.832497+0000","last_undegraded":"2024-06-14T23:00:34.832497+0000","last_fullsized":"2024-06-14T23:00:34.832497+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"190'408","reported_seq":419,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:38.986053+0000","last_change":"2024-06-14T23:00:27.803439+0000","last_active":"2024-06-14T23:00:38.986053+0000","last_peered":"2024-06-14T23:00:38.986053+0000","last_clean":"2024-06-14T23:00:38.986053+0000","last_became_active":"2024-06-14T23:00:27.802783+0000","last_became_peered":"2024-06-14T23:00:27.802783+0000","last_unstale":"2024-06-14T23:00:38.986053+0000","last_undegraded":"2024-06-14T23:00:38.986053+0000","last_fullsized":"2024-06-14T23:00:38.986053+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:35:39.179455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26738688,"num_objects":408,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":408,"num_write_kb":26112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":430,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.541205+0000","last_change":"2024-06-14T23:00:17.399219+0000","last_active":"2024-06-14T23:00:30.541205+0000","last_peered":"2024-06-14T23:00:30.541205+0000","last_clean":"2024-06-14T23:00:30.541205+0000","last_became_active":"2024-06-14T23:00:17.398852+0000","last_became_peered":"2024-06-14T23:00:17.398852+0000","last_unstale":"2024-06-14T23:00:30.541205+0000","last_undegraded":"2024-06-14T23:00:30.541205+0000","last_fullsized":"2024-06-14T23:00:30.541205+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T22:31:01.536459+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":83,"num_write":22,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":168,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567491+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T23:00:30.567491+0000","last_peered":"2024-06-14T23:00:30.567491+0000","last_clean":"2024-06-14T23:00:30.567491+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T23:00:30.567491+0000","last_undegraded":"2024-06-14T23:00:30.567491+0000","last_fullsized":"2024-06-14T23:00:30.567491+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":149,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832646+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:00:34.832646+0000","last_peered":"2024-06-14T23:00:34.832646+0000","last_clean":"2024-06-14T23:00:34.832646+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:00:34.832646+0000","last_undegraded":"2024-06-14T23:00:34.832646+0000","last_fullsized":"2024-06-14T23:00:34.832646+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":144,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832892+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:00:34.832892+0000","last_peered":"2024-06-14T23:00:34.832892+0000","last_clean":"2024-06-14T23:00:34.832892+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:00:34.832892+0000","last_undegraded":"2024-06-14T23:00:34.832892+0000","last_fullsized":"2024-06-14T23:00:34.832892+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"190'356","reported_seq":368,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:38.738678+0000","last_change":"2024-06-14T23:00:27.800097+0000","last_active":"2024-06-14T23:00:38.738678+0000","last_peered":"2024-06-14T23:00:38.738678+0000","last_clean":"2024-06-14T23:00:38.738678+0000","last_became_active":"2024-06-14T23:00:27.799790+0000","last_became_peered":"2024-06-14T23:00:27.799790+0000","last_unstale":"2024-06-14T23:00:38.738678+0000","last_undegraded":"2024-06-14T23:00:38.738678+0000","last_fullsized":"2024-06-14T23:00:38.738678+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:21:01.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23330816,"num_objects":356,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":356,"num_write_kb":22784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":440,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:31.555368+0000","last_change":"2024-06-14T23:00:31.555325+0000","last_active":"2024-06-14T23:00:31.555368+0000","last_peered":"2024-06-14T23:00:31.555368+0000","last_clean":"2024-06-14T23:00:31.555368+0000","last_became_active":"2024-06-14T23:00:17.400871+0000","last_became_peered":"2024-06-14T23:00:17.400871+0000","last_unstale":"2024-06-14T23:00:31.555368+0000","last_undegraded":"2024-06-14T23:00:31.555368+0000","last_fullsized":"2024-06-14T23:00:31.555368+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:58.191436+0000","scrub_duration":0.001665094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":159,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832574+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T23:00:34.832574+0000","last_peered":"2024-06-14T23:00:34.832574+0000","last_clean":"2024-06-14T23:00:34.832574+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T23:00:34.832574+0000","last_undegraded":"2024-06-14T23:00:34.832574+0000","last_fullsized":"2024-06-14T23:00:34.832574+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":155,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.568159+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:00:30.568159+0000","last_peered":"2024-06-14T23:00:30.568159+0000","last_clean":"2024-06-14T23:00:30.568159+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:00:30.568159+0000","last_undegraded":"2024-06-14T23:00:30.568159+0000","last_fullsized":"2024-06-14T23:00:30.568159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":128,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567317+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T23:00:30.567317+0000","last_peered":"2024-06-14T23:00:30.567317+0000","last_clean":"2024-06-14T23:00:30.567317+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T23:00:30.567317+0000","last_undegraded":"2024-06-14T23:00:30.567317+0000","last_fullsized":"2024-06-14T23:00:30.567317+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"190'355","reported_seq":367,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:38.711957+0000","last_change":"2024-06-14T23:00:27.797841+0000","last_active":"2024-06-14T23:00:38.711957+0000","last_peered":"2024-06-14T23:00:38.711957+0000","last_clean":"2024-06-14T23:00:38.711957+0000","last_became_active":"2024-06-14T23:00:27.797256+0000","last_became_peered":"2024-06-14T23:00:27.797256+0000","last_unstale":"2024-06-14T23:00:38.711957+0000","last_undegraded":"2024-06-14T23:00:38.711957+0000","last_fullsized":"2024-06-14T23:00:38.711957+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:52:24.873796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":355,"num_object_clones":0,"num_object_copies":1065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":355,"num_write_kb":22720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":487,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:32.174225+0000","last_change":"2024-06-14T23:00:32.174187+0000","last_active":"2024-06-14T23:00:32.174225+0000","last_peered":"2024-06-14T23:00:32.174225+0000","last_clean":"2024-06-14T23:00:32.174225+0000","last_became_active":"2024-06-14T23:00:17.395654+0000","last_became_peered":"2024-06-14T23:00:17.395654+0000","last_unstale":"2024-06-14T23:00:32.174225+0000","last_undegraded":"2024-06-14T23:00:32.174225+0000","last_fullsized":"2024-06-14T23:00:32.174225+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.174132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.174132+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:59.252795+0000","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":50,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567469+0000","last_change":"2024-06-14T22:58:09.752729+0000","last_active":"2024-06-14T23:00:30.567469+0000","last_peered":"2024-06-14T23:00:30.567469+0000","last_clean":"2024-06-14T23:00:30.567469+0000","last_became_active":"2024-06-14T22:58:09.752393+0000","last_became_peered":"2024-06-14T22:58:09.752393+0000","last_unstale":"2024-06-14T23:00:30.567469+0000","last_undegraded":"2024-06-14T23:00:30.567469+0000","last_fullsized":"2024-06-14T23:00:30.567469+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:21:58.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":143,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.833009+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T23:00:34.833009+0000","last_peered":"2024-06-14T23:00:34.833009+0000","last_clean":"2024-06-14T23:00:34.833009+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T23:00:34.833009+0000","last_undegraded":"2024-06-14T23:00:34.833009+0000","last_fullsized":"2024-06-14T23:00:34.833009+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":135,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567357+0000","last_change":"2024-06-14T22:58:32.754079+0000","last_active":"2024-06-14T23:00:30.567357+0000","last_peered":"2024-06-14T23:00:30.567357+0000","last_clean":"2024-06-14T23:00:30.567357+0000","last_became_active":"2024-06-14T22:58:32.753814+0000","last_became_peered":"2024-06-14T22:58:32.753814+0000","last_unstale":"2024-06-14T23:00:30.567357+0000","last_undegraded":"2024-06-14T23:00:30.567357+0000","last_fullsized":"2024-06-14T23:00:30.567357+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:37:07.414370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"190'392","reported_seq":404,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:38.750733+0000","last_change":"2024-06-14T23:00:27.798887+0000","last_active":"2024-06-14T23:00:38.750733+0000","last_peered":"2024-06-14T23:00:38.750733+0000","last_clean":"2024-06-14T23:00:38.750733+0000","last_became_active":"2024-06-14T23:00:27.798590+0000","last_became_peered":"2024-06-14T23:00:27.798590+0000","last_unstale":"2024-06-14T23:00:38.750733+0000","last_undegraded":"2024-06-14T23:00:38.750733+0000","last_fullsized":"2024-06-14T23:00:38.750733+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:02:33.396409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":392,"num_object_clones":0,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":392,"num_write_kb":25088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":488,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:32.716013+0000","last_change":"2024-06-14T23:00:32.716013+0000","last_active":"2024-06-14T23:00:32.716013+0000","last_peered":"2024-06-14T23:00:32.716013+0000","last_clean":"2024-06-14T23:00:32.716013+0000","last_became_active":"2024-06-14T23:00:17.398349+0000","last_became_peered":"2024-06-14T23:00:17.398349+0000","last_unstale":"2024-06-14T23:00:32.716013+0000","last_undegraded":"2024-06-14T23:00:32.716013+0000","last_fullsized":"2024-06-14T23:00:32.716013+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.715907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.715907+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:23:40.212866+0000","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":146,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567463+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T23:00:30.567463+0000","last_peered":"2024-06-14T23:00:30.567463+0000","last_clean":"2024-06-14T23:00:30.567463+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T23:00:30.567463+0000","last_undegraded":"2024-06-14T23:00:30.567463+0000","last_fullsized":"2024-06-14T23:00:30.567463+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":155,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.541340+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T23:00:30.541340+0000","last_peered":"2024-06-14T23:00:30.541340+0000","last_clean":"2024-06-14T23:00:30.541340+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T23:00:30.541340+0000","last_undegraded":"2024-06-14T23:00:30.541340+0000","last_fullsized":"2024-06-14T23:00:30.541340+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":133,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018384+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T23:00:30.018384+0000","last_peered":"2024-06-14T23:00:30.018384+0000","last_clean":"2024-06-14T23:00:30.018384+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T23:00:30.018384+0000","last_undegraded":"2024-06-14T23:00:30.018384+0000","last_fullsized":"2024-06-14T23:00:30.018384+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":150,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018638+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T23:00:30.018638+0000","last_peered":"2024-06-14T23:00:30.018638+0000","last_clean":"2024-06-14T23:00:30.018638+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T23:00:30.018638+0000","last_undegraded":"2024-06-14T23:00:30.018638+0000","last_fullsized":"2024-06-14T23:00:30.018638+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":156,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018780+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T23:00:30.018780+0000","last_peered":"2024-06-14T23:00:30.018780+0000","last_clean":"2024-06-14T23:00:30.018780+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T23:00:30.018780+0000","last_undegraded":"2024-06-14T23:00:30.018780+0000","last_fullsized":"2024-06-14T23:00:30.018780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":158,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832791+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T23:00:34.832791+0000","last_peered":"2024-06-14T23:00:34.832791+0000","last_clean":"2024-06-14T23:00:34.832791+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T23:00:34.832791+0000","last_undegraded":"2024-06-14T23:00:34.832791+0000","last_fullsized":"2024-06-14T23:00:34.832791+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":68,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:29.814592+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:00:29.814592+0000","last_peered":"2024-06-14T23:00:29.814592+0000","last_clean":"2024-06-14T23:00:29.814592+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:00:29.814592+0000","last_undegraded":"2024-06-14T23:00:29.814592+0000","last_fullsized":"2024-06-14T23:00:29.814592+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":151,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567820+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:00:30.567820+0000","last_peered":"2024-06-14T23:00:30.567820+0000","last_clean":"2024-06-14T23:00:30.567820+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:00:30.567820+0000","last_undegraded":"2024-06-14T23:00:30.567820+0000","last_fullsized":"2024-06-14T23:00:30.567820+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":94,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.540671+0000","last_change":"2024-06-14T22:58:09.753103+0000","last_active":"2024-06-14T23:00:30.540671+0000","last_peered":"2024-06-14T23:00:30.540671+0000","last_clean":"2024-06-14T23:00:30.540671+0000","last_became_active":"2024-06-14T22:58:09.752761+0000","last_became_peered":"2024-06-14T22:58:09.752761+0000","last_unstale":"2024-06-14T23:00:30.540671+0000","last_undegraded":"2024-06-14T23:00:30.540671+0000","last_fullsized":"2024-06-14T23:00:30.540671+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:20:20.619331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":141,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.566928+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T23:00:30.566928+0000","last_peered":"2024-06-14T23:00:30.566928+0000","last_clean":"2024-06-14T23:00:30.566928+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T23:00:30.566928+0000","last_undegraded":"2024-06-14T23:00:30.566928+0000","last_fullsized":"2024-06-14T23:00:30.566928+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":143,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:29.814420+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T23:00:29.814420+0000","last_peered":"2024-06-14T23:00:29.814420+0000","last_clean":"2024-06-14T23:00:29.814420+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T23:00:29.814420+0000","last_undegraded":"2024-06-14T23:00:29.814420+0000","last_fullsized":"2024-06-14T23:00:29.814420+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":150,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.541276+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T23:00:30.541276+0000","last_peered":"2024-06-14T23:00:30.541276+0000","last_clean":"2024-06-14T23:00:30.541276+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T23:00:30.541276+0000","last_undegraded":"2024-06-14T23:00:30.541276+0000","last_fullsized":"2024-06-14T23:00:30.541276+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":138,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.851276+0000","last_change":"2024-06-14T22:58:09.785074+0000","last_active":"2024-06-14T23:00:34.851276+0000","last_peered":"2024-06-14T23:00:34.851276+0000","last_clean":"2024-06-14T23:00:34.851276+0000","last_became_active":"2024-06-14T22:58:09.784309+0000","last_became_peered":"2024-06-14T22:58:09.784309+0000","last_unstale":"2024-06-14T23:00:34.851276+0000","last_undegraded":"2024-06-14T23:00:34.851276+0000","last_fullsized":"2024-06-14T23:00:34.851276+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:23.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":143,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.566889+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T23:00:30.566889+0000","last_peered":"2024-06-14T23:00:30.566889+0000","last_clean":"2024-06-14T23:00:30.566889+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T23:00:30.566889+0000","last_undegraded":"2024-06-14T23:00:30.566889+0000","last_fullsized":"2024-06-14T23:00:30.566889+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":158,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.832485+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:00:34.832485+0000","last_peered":"2024-06-14T23:00:34.832485+0000","last_clean":"2024-06-14T23:00:34.832485+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:00:34.832485+0000","last_undegraded":"2024-06-14T23:00:34.832485+0000","last_fullsized":"2024-06-14T23:00:34.832485+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":147,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-06-14T23:00:39.741846+0000","last_change":"2024-06-14T22:58:09.756040+0000","last_active":"2024-06-14T23:00:39.741846+0000","last_peered":"2024-06-14T23:00:39.741846+0000","last_clean":"2024-06-14T23:00:39.741846+0000","last_became_active":"2024-06-14T22:58:09.755659+0000","last_became_peered":"2024-06-14T22:58:09.755659+0000","last_unstale":"2024-06-14T23:00:39.741846+0000","last_undegraded":"2024-06-14T23:00:39.741846+0000","last_fullsized":"2024-06-14T23:00:39.741846+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:40.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":169,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018540+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T23:00:30.018540+0000","last_peered":"2024-06-14T23:00:30.018540+0000","last_clean":"2024-06-14T23:00:30.018540+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T23:00:30.018540+0000","last_undegraded":"2024-06-14T23:00:30.018540+0000","last_fullsized":"2024-06-14T23:00:30.018540+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":163,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:34.852624+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T23:00:34.852624+0000","last_peered":"2024-06-14T23:00:34.852624+0000","last_clean":"2024-06-14T23:00:34.852624+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T23:00:34.852624+0000","last_undegraded":"2024-06-14T23:00:34.852624+0000","last_fullsized":"2024-06-14T23:00:34.852624+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":130,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567271+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T23:00:30.567271+0000","last_peered":"2024-06-14T23:00:30.567271+0000","last_clean":"2024-06-14T23:00:30.567271+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T23:00:30.567271+0000","last_undegraded":"2024-06-14T23:00:30.567271+0000","last_fullsized":"2024-06-14T23:00:30.567271+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":167,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018634+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:00:30.018634+0000","last_peered":"2024-06-14T23:00:30.018634+0000","last_clean":"2024-06-14T23:00:30.018634+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:00:30.018634+0000","last_undegraded":"2024-06-14T23:00:30.018634+0000","last_fullsized":"2024-06-14T23:00:30.018634+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":153,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.568303+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:00:30.568303+0000","last_peered":"2024-06-14T23:00:30.568303+0000","last_clean":"2024-06-14T23:00:30.568303+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:00:30.568303+0000","last_undegraded":"2024-06-14T23:00:30.568303+0000","last_fullsized":"2024-06-14T23:00:30.568303+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":167,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018474+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:00:30.018474+0000","last_peered":"2024-06-14T23:00:30.018474+0000","last_clean":"2024-06-14T23:00:30.018474+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:00:30.018474+0000","last_undegraded":"2024-06-14T23:00:30.018474+0000","last_fullsized":"2024-06-14T23:00:30.018474+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":136,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567909+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T23:00:30.567909+0000","last_peered":"2024-06-14T23:00:30.567909+0000","last_clean":"2024-06-14T23:00:30.567909+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T23:00:30.567909+0000","last_undegraded":"2024-06-14T23:00:30.567909+0000","last_fullsized":"2024-06-14T23:00:30.567909+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":154,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-06-14T23:00:35.207901+0000","last_change":"2024-06-14T22:58:09.752281+0000","last_active":"2024-06-14T23:00:35.207901+0000","last_peered":"2024-06-14T23:00:35.207901+0000","last_clean":"2024-06-14T23:00:35.207901+0000","last_became_active":"2024-06-14T22:58:09.751952+0000","last_became_peered":"2024-06-14T22:58:09.751952+0000","last_unstale":"2024-06-14T23:00:35.207901+0000","last_undegraded":"2024-06-14T23:00:35.207901+0000","last_fullsized":"2024-06-14T23:00:35.207901+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:39:19.836734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":49,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.541165+0000","last_change":"2024-06-14T22:58:09.755622+0000","last_active":"2024-06-14T23:00:30.541165+0000","last_peered":"2024-06-14T23:00:30.541165+0000","last_clean":"2024-06-14T23:00:30.541165+0000","last_became_active":"2024-06-14T22:58:09.755383+0000","last_became_peered":"2024-06-14T22:58:09.755383+0000","last_unstale":"2024-06-14T23:00:30.541165+0000","last_undegraded":"2024-06-14T23:00:30.541165+0000","last_fullsized":"2024-06-14T23:00:30.541165+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:13.884346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":150,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.018259+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T23:00:30.018259+0000","last_peered":"2024-06-14T23:00:30.018259+0000","last_clean":"2024-06-14T23:00:30.018259+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T23:00:30.018259+0000","last_undegraded":"2024-06-14T23:00:30.018259+0000","last_fullsized":"2024-06-14T23:00:30.018259+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":146,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-06-14T23:00:30.567454+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T23:00:30.567454+0000","last_peered":"2024-06-14T23:00:30.567454+0000","last_clean":"2024-06-14T23:00:30.567454+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T23:00:30.567454+0000","last_undegraded":"2024-06-14T23:00:30.567454+0000","last_fullsized":"2024-06-14T23:00:30.567454+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":403701760,"num_objects":6160,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6160,"num_write_kb":394240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1167458304,"data_stored":1167458304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6160,"ondisk_log_size":6160,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":83,"num_write":22,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1771,"ondisk_log_size":1771,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234319,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1001812,"kb_used_data":94668,"kb_used_omap":1,"kb_used_meta":907134,"kb_avail":92743340,"statfs":{"total":95995035648,"available":94969180160,"internally_reserved":0,"allocated":96940032,"data_stored":95966843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":928905560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:00:14 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.1339999999999999,"15min":1.1220000000000001},"min":{"1min":0.35399999999999998,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":1.8420000000000001,"5min":6.9539999999999997,"15min":9.3919999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.748,"5min":1.0940000000000001,"15min":1.262},"min":{"1min":0.36299999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.8100000000000001,"5min":5.2130000000000001,"15min":10.416},"last":0.53200000000000003}]},{"osd":1,"last update":"Fri Jun 14 22:59:43 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":2.8380000000000001,"15min":1.833},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28100000000000003},"max":{"1min":5.7249999999999996,"5min":83.295000000000002,"15min":83.295000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":2.0449999999999999,"5min":2.8730000000000002,"15min":1.9159999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.223},"max":{"1min":6.3390000000000004,"5min":82.972999999999999,"15min":82.972999999999999},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:59:43 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.1080000000000001,"15min":0.75700000000000001},"min":{"1min":0.27100000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.9269999999999996,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.369,"5min":1.3029999999999999,"15min":0.81499999999999995},"min":{"1min":0.31900000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":4.9000000000000004,"5min":8.1150000000000002,"15min":8.1150000000000002},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":1.889,"15min":0.98699999999999999},"min":{"1min":0.35499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.4850000000000001,"5min":55.683999999999997,"15min":55.683999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.9279999999999999,"15min":1.042},"min":{"1min":0.36399999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.8319999999999999,"5min":55.533999999999999,"15min":55.533999999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.89500000000000002,"15min":0.68100000000000005},"min":{"1min":0.29699999999999999,"5min":0.245,"15min":0.22900000000000001},"max":{"1min":3.6320000000000001,"5min":6.7610000000000001,"15min":6.7610000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.0389999999999999,"15min":0.73899999999999999},"min":{"1min":0.27700000000000002,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":5.7320000000000002,"5min":16.879000000000001,"15min":16.879000000000001},"last":0.42899999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:00:01 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.262,"15min":0.84199999999999997},"min":{"1min":0.371,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":4.1559999999999997,"5min":6.9000000000000004,"15min":6.9000000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.1459999999999999,"15min":0.872},"min":{"1min":0.51900000000000002,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":6.2679999999999998,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":0.63100000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:00:01 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":1,"15min":0.79300000000000004},"min":{"1min":0.35999999999999999,"5min":0.31900000000000001,"15min":0.28100000000000003},"max":{"1min":2.8540000000000001,"5min":5.5,"15min":5.806},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.96099999999999997,"15min":0.76800000000000002},"min":{"1min":0.32000000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.992,"5min":12.951000000000001,"15min":12.951000000000001},"last":0.54700000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495955,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":745220,"kb_used_data":155388,"kb_used_omap":1,"kb_used_meta":589822,"kb_avail":92999932,"statfs":{"total":95995035648,"available":95231930368,"internally_reserved":0,"allocated":159117312,"data_stored":158160038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":603978072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":1.0549999999999999,"15min":1.26},"min":{"1min":0.25700000000000001,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":0.80400000000000005,"5min":9.9350000000000005,"15min":31.358000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.91600000000000004,"15min":1.298},"min":{"1min":0.318,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.9159999999999999,"5min":6.0730000000000004,"15min":29.399000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2330000000000001,"15min":1.2669999999999999},"min":{"1min":0.26400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.1280000000000001,"5min":6.6680000000000001,"15min":9.9559999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.7549999999999999,"5min":1.28,"15min":1.2769999999999999},"min":{"1min":0.30099999999999999,"5min":0.25,"15min":0.247},"max":{"1min":6.5510000000000002,"5min":6.7519999999999998,"15min":9.5510000000000002},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jun 14 22:59:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.91100000000000003,"15min":0.68600000000000005},"min":{"1min":0.312,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.544,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.495},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0089999999999999,"15min":0.74099999999999999},"min":{"1min":0.33400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.5030000000000001,"5min":6.6959999999999997,"15min":6.6959999999999997},"last":0.42599999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":1.6120000000000001,"15min":0.872},"min":{"1min":0.28599999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.0510000000000002,"5min":29.152000000000001,"15min":29.152000000000001},"last":0.439},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.617,"15min":0.92200000000000004},"min":{"1min":0.377,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.968,"5min":25.864000000000001,"15min":25.864000000000001},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:00:22 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":0.873,"15min":0.88500000000000001},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":7.7709999999999999,"5min":7.7709999999999999,"15min":47.698},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.75,"15min":0.79300000000000004},"min":{"1min":0.312,"5min":0.26800000000000002,"15min":0.23899999999999999},"max":{"1min":0.85699999999999998,"5min":9.9610000000000003,"15min":48.197000000000003},"last":0.315}]},{"osd":5,"last update":"Fri Jun 14 23:00:22 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.77700000000000002},"min":{"1min":0.39400000000000002,"5min":0.29699999999999999,"15min":0.26500000000000001},"max":{"1min":3.4359999999999999,"5min":5.8289999999999997,"15min":8.6500000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0329999999999999,"15min":0.85699999999999998},"min":{"1min":0.36799999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":6.1079999999999997,"5min":7.3769999999999998,"15min":8.6349999999999998},"last":0.33500000000000002}]},{"osd":7,"last update":"Fri Jun 14 22:59:56 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.03,"15min":0.83299999999999996},"min":{"1min":0.40000000000000002,"5min":0.311,"15min":0.28799999999999998},"max":{"1min":7.8579999999999997,"5min":7.8579999999999997,"15min":8.6950000000000003},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.569,"5min":1.0669999999999999,"15min":0.88800000000000001},"min":{"1min":0.45100000000000001,"5min":0.33500000000000002,"15min":0.28899999999999998},"max":{"1min":7.8120000000000003,"5min":7.8120000000000003,"15min":8.673},"last":0.379}]}]},{"osd":5,"up_from":38,"seq":163208757594,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1010880,"kb_used_data":115640,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":92734272,"statfs":{"total":95995035648,"available":94959894528,"internally_reserved":0,"allocated":118415360,"data_stored":117458540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:00:32 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.079,"15min":1.2070000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":3.5230000000000001,"5min":4.5819999999999999,"15min":17.766999999999999},"last":0.436},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.107,"15min":1.2150000000000001},"min":{"1min":0.29299999999999998,"5min":0.221,"15min":0.221},"max":{"1min":5.1950000000000003,"5min":5.1950000000000003,"15min":17.806000000000001},"last":0.40999999999999998}]},{"osd":1,"last update":"Fri Jun 14 22:59:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.105,"15min":1.26},"min":{"1min":0.27100000000000002,"5min":0.24199999999999999,"15min":0.23300000000000001},"max":{"1min":4.5250000000000004,"5min":5.6429999999999998,"15min":7.694},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.643,"5min":1.3169999999999999,"15min":1.2110000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":7.0700000000000003,"5min":7.0700000000000003,"15min":7.0700000000000003},"last":0.31900000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:59:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.05,"15min":0.71399999999999997},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.77,"5min":5.9379999999999997,"15min":5.9379999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.98099999999999998,"15min":0.70699999999999996},"min":{"1min":0.28100000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.6150000000000002,"5min":4.6050000000000004,"15min":4.6050000000000004},"last":0.40100000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:00:14 2024","interfaces":[{"interface":"back","average":{"1min":1.948,"5min":2.0550000000000002,"15min":1.0369999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":10.358000000000001,"5min":45.744,"15min":45.744},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.1179999999999999,"15min":1.1319999999999999},"min":{"1min":0.39300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":10.385,"5min":45.838000000000001,"15min":45.838000000000001},"last":0.46500000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.79600000000000004,"15min":0.626},"min":{"1min":0.307,"5min":0.26200000000000001,"15min":0.214},"max":{"1min":4.0899999999999999,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.81299999999999994,"15min":0.67400000000000004},"min":{"1min":0.34599999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.657,"5min":4.657,"15min":5.8109999999999999},"last":0.38400000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.88,"15min":0.78300000000000003},"min":{"1min":0.34200000000000003,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":4.6920000000000002,"5min":6.931,"15min":9.6020000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.81399999999999995,"15min":0.73899999999999999},"min":{"1min":0.41799999999999998,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":3.573,"5min":4.452,"15min":12.339},"last":0.41799999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:00:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.0580000000000001,"15min":0.79300000000000004},"min":{"1min":0.372,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.2380000000000004,"5min":5.3129999999999997,"15min":5.3129999999999997},"last":0.372},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.0620000000000001,"15min":0.78700000000000003},"min":{"1min":0.316,"5min":0.29999999999999999,"15min":0.25600000000000001},"max":{"1min":3.8580000000000001,"5min":5.6189999999999998,"15min":10.638999999999999},"last":0.30399999999999999}]}]},{"osd":4,"up_from":30,"seq":128849019230,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1027212,"kb_used_data":124228,"kb_used_omap":1,"kb_used_meta":902974,"kb_avail":92717940,"statfs":{"total":95995035648,"available":94943170560,"internally_reserved":0,"allocated":127209472,"data_stored":126244140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924645720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:00:23 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.88800000000000001,"15min":1.0289999999999999},"min":{"1min":0.216,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.1479999999999997,"5min":5.1479999999999997,"15min":7.5650000000000004},"last":0.17100000000000001},{"interface":"front","average":{"1min":1.5449999999999999,"5min":0.95999999999999996,"15min":1.1419999999999999},"min":{"1min":0.26900000000000002,"5min":0.23699999999999999,"15min":0.221},"max":{"1min":8.3390000000000004,"5min":8.3390000000000004,"15min":9.1470000000000002},"last":0.30299999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:59:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7010000000000001,"5min":1.1830000000000001,"15min":1.1659999999999999},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.4589999999999996,"5min":8.4960000000000004,"15min":8.4960000000000004},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.2390000000000001,"15min":1.2809999999999999},"min":{"1min":0.26600000000000001,"5min":0.26300000000000001,"15min":0.249},"max":{"1min":4.8529999999999998,"5min":6.5549999999999997,"15min":6.5549999999999997},"last":0.38700000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:59:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.80600000000000005,"15min":0.64600000000000002},"min":{"1min":0.27200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4809999999999999,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.091,"5min":0.89000000000000001,"15min":0.68000000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.48,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":0.32800000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.2150000000000001,"15min":0.76800000000000002},"min":{"1min":0.39900000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.5739999999999998,"5min":6.617,"15min":6.617},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.23,"15min":0.80600000000000005},"min":{"1min":0.33200000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.9130000000000003,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":0.27600000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:00:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":2.556,"15min":1.254},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.21199999999999999},"max":{"1min":5.7409999999999997,"5min":139.13200000000001,"15min":139.13200000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.4770000000000001,"5min":2.6669999999999998,"15min":1.3029999999999999},"min":{"1min":0.32100000000000001,"5min":0.26100000000000001,"15min":0.21199999999999999},"max":{"1min":9.1920000000000002,"5min":138.86000000000001,"15min":138.86000000000001},"last":0.36099999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:00:30 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.81799999999999995,"15min":0.65900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.254},"max":{"1min":4.2060000000000004,"5min":5.1210000000000004,"15min":5.1210000000000004},"last":0.376},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.70899999999999996,"15min":0.60499999999999998},"min":{"1min":0.29499999999999998,"5min":0.28899999999999998,"15min":0.254},"max":{"1min":0.67700000000000005,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.34200000000000003}]},{"osd":7,"last update":"Fri Jun 14 22:59:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":0.99299999999999999,"15min":0.71999999999999997},"min":{"1min":0.38700000000000001,"5min":0.30299999999999999,"15min":0.30099999999999999},"max":{"1min":3.9689999999999999,"5min":8.9489999999999998,"15min":8.9489999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.2070000000000001,"15min":0.83299999999999996},"min":{"1min":0.35599999999999998,"5min":0.308,"15min":0.29399999999999998},"max":{"1min":7.25,"5min":7.25,"15min":8.4730000000000008},"last":0.42199999999999999}]}]},{"osd":3,"up_from":138,"seq":592705486929,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312960,"kb_used_data":106808,"kb_used_omap":4,"kb_used_meta":206139,"kb_avail":93432192,"statfs":{"total":95995035648,"available":95674564608,"internally_reserved":0,"allocated":109371392,"data_stored":108414345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":211086814},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:00:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.2390000000000001,"15min":0.629},"min":{"1min":0.22,"5min":0.184,"15min":0.184},"max":{"1min":5.8209999999999997,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":0.97699999999999998,"15min":0.56399999999999995},"min":{"1min":0.254,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":5.8049999999999997,"5min":5.8049999999999997,"15min":5.8049999999999997},"last":2.3119999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:00:15 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.476,"15min":0.72799999999999998},"min":{"1min":0.25800000000000001,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":4.5220000000000002,"5min":24.097999999999999,"15min":24.097999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5189999999999999,"15min":0.76900000000000002},"min":{"1min":0.28999999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.7709999999999999,"5min":24.132999999999999,"15min":24.132999999999999},"last":1.6539999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:00:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0549999999999999,"15min":0.625},"min":{"1min":0.28599999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.4889999999999999,"5min":5.2910000000000004,"15min":5.2910000000000004},"last":1.548},{"interface":"front","average":{"1min":1,"5min":1.0169999999999999,"15min":0.57699999999999996},"min":{"1min":0.23200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.871,"5min":4.8070000000000004,"15min":4.8070000000000004},"last":0.28199999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:00:15 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.0660000000000001,"15min":0.77600000000000002},"min":{"1min":0.32900000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.9100000000000001,"5min":5.7960000000000003,"15min":5.7960000000000003},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.254,"5min":1.2310000000000001,"15min":0.82999999999999996},"min":{"1min":0.41699999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.6909999999999998,"5min":5.6660000000000004,"15min":5.6660000000000004},"last":0.69299999999999995}]},{"osd":5,"last update":"Fri Jun 14 23:00:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4370000000000001,"15min":0.90400000000000003},"min":{"1min":0.36099999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.7190000000000003,"5min":5.9359999999999999,"15min":5.9359999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.262,"5min":1.29,"15min":0.83799999999999997},"min":{"1min":0.44700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.4660000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":1.591}]},{"osd":6,"last update":"Fri Jun 14 23:00:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.141,"15min":0.80900000000000005},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.123,"15min":0.78200000000000003},"min":{"1min":0.375,"5min":0.314,"15min":0.314},"max":{"1min":3.9369999999999998,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":0.96999999999999997}]},{"osd":7,"last update":"Fri Jun 14 23:00:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.4379999999999999,"15min":0.91300000000000003},"min":{"1min":0.378,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.9609999999999999,"5min":7.7240000000000002,"15min":7.7240000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.502,"15min":0.92200000000000004},"min":{"1min":0.42499999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.7430000000000003,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.44600000000000001}]}]},{"osd":2,"up_from":133,"seq":571230650456,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":323648,"kb_used_data":214328,"kb_used_omap":6,"kb_used_meta":109305,"kb_avail":93421504,"statfs":{"total":95995035648,"available":95663620096,"internally_reserved":0,"allocated":219471872,"data_stored":218515039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":111928821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.75900000000000001,"15min":0.49299999999999999},"min":{"1min":0.20399999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":7.476,"5min":7.476,"15min":7.476},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":0.92100000000000004,"15min":0.54700000000000004},"min":{"1min":0.27300000000000002,"5min":0.182,"15min":0.182},"max":{"1min":9.0869999999999997,"5min":9.0869999999999997,"15min":9.0869999999999997},"last":0.48599999999999999}]},{"osd":1,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.3759999999999999,"15min":0.71199999999999997},"min":{"1min":0.22900000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.843,"5min":21.920000000000002,"15min":21.920000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.3400000000000001,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":13.054,"5min":21.573,"15min":21.573},"last":1.2769999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:00:18 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.4399999999999999,"15min":0.72199999999999998},"min":{"1min":0.249,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":7.9420000000000002,"5min":21.66,"15min":21.66},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.423,"15min":0.73599999999999999},"min":{"1min":0.28699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.9100000000000001,"5min":22.341000000000001,"15min":22.341000000000001},"last":0.57299999999999995}]},{"osd":4,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.0629999999999999,"15min":0.74299999999999999},"min":{"1min":0.45400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":7.8019999999999996,"5min":7.8019999999999996,"15min":7.8019999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.8069999999999999,"5min":1.0249999999999999,"15min":0.72699999999999998},"min":{"1min":0.30099999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.7839999999999998,"5min":7.7839999999999998,"15min":7.7839999999999998},"last":0.66100000000000003}]},{"osd":5,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.0289999999999999,"15min":0.71599999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.218,"5min":5.218,"15min":5.218},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.544,"5min":0.999,"15min":0.71999999999999997},"min":{"1min":0.42999999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.1790000000000003,"5min":6.1790000000000003,"15min":6.1790000000000003},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":0.877,"15min":0.66800000000000004},"min":{"1min":0.36399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.0099999999999998,"5min":5.8579999999999997,"15min":5.8579999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.95399999999999996,"15min":0.70199999999999996},"min":{"1min":0.36099999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.298,"5min":5.1829999999999998,"15min":5.1829999999999998},"last":1.3919999999999999}]},{"osd":7,"last update":"Fri Jun 14 22:59:39 2024","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":1.1679999999999999,"15min":0.78900000000000003},"min":{"1min":0.39900000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":3.9750000000000001,"5min":6.226,"15min":6.226},"last":0.627},{"interface":"front","average":{"1min":1.294,"5min":0.94599999999999995,"15min":0.70599999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.7080000000000002,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":1.343}]}]},{"osd":1,"up_from":116,"seq":498216206473,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542136,"kb_used_data":109040,"kb_used_omap":5,"kb_used_meta":433082,"kb_avail":93203016,"statfs":{"total":95995035648,"available":95439888384,"internally_reserved":0,"allocated":111656960,"data_stored":110707471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":443476248},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":1.3220000000000001,"15min":1.9339999999999999},"min":{"1min":0.29199999999999998,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":11.257999999999999,"5min":11.257999999999999,"15min":11.257999999999999},"last":1.357},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.069,"15min":1.4650000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":4.5090000000000003,"5min":5.0220000000000002,"15min":10.426},"last":0.97099999999999997}]},{"osd":2,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":2.1909999999999998,"5min":1.4430000000000001,"15min":0.70499999999999996},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.3759999999999994,"5min":9.3759999999999994,"15min":9.3759999999999994},"last":1.8600000000000001},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.94099999999999995,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.4100000000000001,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":2.2410000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:00:16 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.8600000000000001,"15min":0.83999999999999997},"min":{"1min":0.246,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.782,"5min":21.422000000000001,"15min":21.422000000000001},"last":0.37},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.919,"15min":0.88500000000000001},"min":{"1min":0.27400000000000002,"5min":0.214,"15min":0.214},"max":{"1min":3.6309999999999998,"5min":21.016999999999999,"15min":21.016999999999999},"last":0.505}]},{"osd":4,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3180000000000001,"15min":1.704},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.4990000000000001,"5min":7.673,"15min":8.8599999999999994},"last":0.45400000000000001},{"interface":"front","average":{"1min":2.1099999999999999,"5min":1.635,"15min":1.79},"min":{"1min":0.36599999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":9.4169999999999998,"5min":9.4169999999999998,"15min":11.161},"last":0.32800000000000001}]},{"osd":5,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":1.4890000000000001,"15min":1.696},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":10.103,"5min":10.103,"15min":10.103},"last":0.628},{"interface":"front","average":{"1min":1.151,"5min":1.3540000000000001,"15min":1.7430000000000001},"min":{"1min":0.32400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.5920000000000001,"5min":7.7439999999999998,"15min":8.125},"last":0.39400000000000002}]},{"osd":6,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.2490000000000001,"15min":1.677},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.6100000000000003,"5min":6.6100000000000003,"15min":9.7279999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.6970000000000001,"5min":1.371,"15min":1.742},"min":{"1min":0.38200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":8.4209999999999994,"5min":8.4209999999999994,"15min":8.4209999999999994},"last":0.59999999999999998}]},{"osd":7,"last update":"Fri Jun 14 22:59:44 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":1.772,"15min":1.8999999999999999},"min":{"1min":0.46999999999999997,"5min":0.35199999999999998,"15min":0.33700000000000002},"max":{"1min":9.4429999999999996,"5min":9.4429999999999996,"15min":9.4429999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.6459999999999999,"5min":1.383,"15min":1.5720000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.36499999999999999},"max":{"1min":4.3899999999999997,"5min":6.3529999999999998,"15min":9.5449999999999999},"last":0.66400000000000003}]}]},{"osd":0,"up_from":91,"seq":390842024127,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":495740,"kb_used_data":2224,"kb_used_omap":5,"kb_used_meta":493498,"kb_avail":93249412,"statfs":{"total":95995035648,"available":95487397888,"internally_reserved":0,"allocated":2277376,"data_stored":1325886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":505342042},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 22:59:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.496,"15min":1.603},"min":{"1min":0.186,"5min":0.186,"15min":0.18099999999999999},"max":{"1min":5.3419999999999996,"5min":16.305,"15min":16.305},"last":0.307},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.5980000000000001,"15min":1.992},"min":{"1min":0.25,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.6820000000000004,"5min":16.184999999999999,"15min":16.184999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Fri Jun 14 22:59:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.157,"15min":0.624},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.326000000000001,"5min":16.757000000000001,"15min":16.757000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.3720000000000001,"15min":0.72199999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":11.519,"5min":16.055,"15min":16.055},"last":0.28100000000000003}]},{"osd":3,"last update":"Fri Jun 14 23:00:18 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.4590000000000001,"15min":0.72899999999999998},"min":{"1min":0.26500000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.238,"5min":16.873999999999999,"15min":16.873999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.4370000000000001,"15min":0.71799999999999997},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":5.1150000000000002,"5min":16.085999999999999,"15min":16.085999999999999},"last":0.33800000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:00:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.1779999999999999,"15min":1.387},"min":{"1min":0.38800000000000001,"5min":0.307,"15min":0.27600000000000002},"max":{"1min":5.1539999999999999,"5min":16.018000000000001,"15min":16.018000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":1.1930000000000001,"15min":1.325},"min":{"1min":0.34100000000000003,"5min":0.29699999999999999,"15min":0.28599999999999998},"max":{"1min":1.145,"5min":15.976000000000001,"15min":15.976000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Jun 14 23:00:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.5860000000000001,"15min":1.486},"min":{"1min":0.307,"5min":0.30399999999999999,"15min":0.29299999999999998},"max":{"1min":2.6880000000000002,"5min":16.108000000000001,"15min":16.108000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.548,"15min":1.5309999999999999},"min":{"1min":0.32600000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":11.436,"5min":16.006,"15min":16.006},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Jun 14 23:00:23 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.2829999999999999,"15min":1.407},"min":{"1min":0.33600000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.113,"5min":16.152000000000001,"15min":16.152000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.333,"15min":1.4550000000000001},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.29399999999999998},"max":{"1min":11.48,"5min":16.227,"15min":16.227},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Jun 14 23:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.476,"15min":1.496},"min":{"1min":0.38400000000000001,"5min":0.35899999999999999,"15min":0.33100000000000002},"max":{"1min":11.364000000000001,"5min":16.199999999999999,"15min":16.199999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.153,"5min":1.462,"15min":1.5509999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.314},"max":{"1min":5.1779999999999999,"5min":16.259,"15min":16.259},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124125184,"data_stored":124125184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254803968,"data_stored":254803968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":174653440,"data_stored":174653440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":186384384,"data_stored":186384384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":150994944,"data_stored":150994944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":167837696,"data_stored":167837696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":108658688,"data_stored":108658688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:00:43.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-06-14T23:00:43.209 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 19003 18987 197.765 170.812 0.00409986 0.0050499 2024-06-14T23:00:43.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:43.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:43.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:43.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:43.434 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:43.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:43.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:43.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:43.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:43.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:43.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:43.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:43.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:43.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:43.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:43.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:43.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:43.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:44.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:44.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:43 smithi086 ceph-mon[97913]: 1.7 scrub starts 2024-06-14T23:00:44.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:43 smithi086 ceph-mon[97913]: 1.7 scrub ok 2024-06-14T23:00:44.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:43 smithi086 ceph-mon[97913]: pgmap v747: 99 pgs: 99 active+clean; 502 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 63 MiB/s wr, 997 op/s 2024-06-14T23:00:44.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:43 smithi086 ceph-mon[102754]: 1.7 scrub starts 2024-06-14T23:00:44.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:43 smithi086 ceph-mon[102754]: 1.7 scrub ok 2024-06-14T23:00:44.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:43 smithi086 ceph-mon[102754]: pgmap v747: 99 pgs: 99 active+clean; 502 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 63 MiB/s wr, 997 op/s 2024-06-14T23:00:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:44.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:44.209 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 16 21353 21337 190.493 146.875 0.00514862 0.00524218 2024-06-14T23:00:44.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:44.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:43 smithi137 ceph-mon[90634]: 1.7 scrub starts 2024-06-14T23:00:44.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:43 smithi137 ceph-mon[90634]: 1.7 scrub ok 2024-06-14T23:00:44.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:43 smithi137 ceph-mon[90634]: pgmap v747: 99 pgs: 99 active+clean; 502 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 63 MiB/s wr, 997 op/s 2024-06-14T23:00:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:44.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:44.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:44.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:44.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:44.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:44.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:44.526 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:44.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:44.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:44.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:44.684 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:44.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:44.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:44.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:44.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:44.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:44.944 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:00:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:00:44] "GET /metrics HTTP/1.1" 200 36592 "" "Prometheus/2.43.0" 2024-06-14T23:00:44.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:44.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:44.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-14T23:00:45.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:45.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:45.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:45.210 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 23869 23853 186.318 157.25 0.00621183 0.00536174 2024-06-14T23:00:45.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:45.254 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:45.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:45.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:45.395 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:45.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:45.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:45.504 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:00:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:45.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:45.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:45.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:45.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:45.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:45.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:45.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:45.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:45.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:45.883 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:00:45.883 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:00:45.900 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:00:45.901 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:00:45.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:45.932 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:00:45.932 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:00:45.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[97913]: pgmap v748: 99 pgs: 99 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 114 MiB/s wr, 1.81k op/s 2024-06-14T23:00:45.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:00:45.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.933 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[102754]: pgmap v748: 99 pgs: 99 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 114 MiB/s wr, 1.81k op/s 2024-06-14T23:00:45.933 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:00:45.933 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.933 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.933 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.933 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:45.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:45.965 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:00:45.965 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:00:45.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:45 smithi137 ceph-mon[90634]: pgmap v748: 99 pgs: 99 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 114 MiB/s wr, 1.81k op/s 2024-06-14T23:00:45.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:00:45.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:45.997 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:00:45.997 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:00:46.027 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:00:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:00:46.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:46.061 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:00:46.062 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:00:46.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:46.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:46.089 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:00:46.089 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:00:46.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:46.210 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 26101 26085 181.114 139.5 0.00681302 0.00551565 2024-06-14T23:00:46.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:46.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:46.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:46.362 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:46.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:46.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:46.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:46.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:46.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:46.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:46.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:46.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:46.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:46.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2657490656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:46.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2657490656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:46.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:46.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:46.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:46.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2657490656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:47.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:47.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:47.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:47.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:47.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:47.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:47.210 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 16 28493 28477 177.952 149.5 0.00558581 0.00561472 2024-06-14T23:00:47.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:47.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:47.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:47.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:47.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:47.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:47.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:47.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:47.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:47.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:47.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:47.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:47.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:47.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:47.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:47.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:47.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:47.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:47 smithi137 ceph-mon[90634]: pgmap v749: 99 pgs: 99 active+clean; 1.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-06-14T23:00:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:48.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:48.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:48.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:47 smithi086 ceph-mon[102754]: pgmap v749: 99 pgs: 99 active+clean; 1.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-06-14T23:00:48.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:47 smithi086 ceph-mon[97913]: pgmap v749: 99 pgs: 99 active+clean; 1.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-06-14T23:00:48.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:48.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:48.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:48.210 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 30672 30656 174.154 136.188 0.0024932 0.00573638 2024-06-14T23:00:48.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:48.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:48.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:48.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:48.430 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:48.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:48.454 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:00:48.454 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:00:48.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:48.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:48.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:48.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:48.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:48.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:48.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:48.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:48.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:48.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:48.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:48.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:48.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:48.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:49.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:49.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:49.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:49.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:49.210 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 16 32835 32819 170.906 135.188 0.0101702 0.0058454 2024-06-14T23:00:49.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:49.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:49.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:49.365 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:49.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:49.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:49.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:49.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:49.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:49.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:49.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:49.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:49.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:49.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:49.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:49.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:49.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:49.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:50.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:50.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:50.057 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:49 smithi086 ceph-mon[102754]: pgmap v750: 99 pgs: 99 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-06-14T23:00:50.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:49 smithi086 ceph-mon[97913]: pgmap v750: 99 pgs: 99 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-06-14T23:00:50.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:50.167 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:50.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:50.211 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 34699 34683 166.715 116.5 0.0059946 0.0059937 2024-06-14T23:00:50.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:49 smithi137 ceph-mon[90634]: pgmap v750: 99 pgs: 99 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-06-14T23:00:50.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:50.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:50.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:50.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:50.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:50.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:50.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:50.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:50.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:50.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:50.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:50.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:50.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:50.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:00:50.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:50.827 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:50.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:50.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:50.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:50.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:50.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:51.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:51.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:51.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:51.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:51.211 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 36498 36482 162.838 112.438 0.00961413 0.00613687 2024-06-14T23:00:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:51.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:51.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:51.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:51.346 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:51.362 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:51.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:51.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:51.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:51.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:51.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:51.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:51.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:51.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:51.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:51.801 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:00:51.801 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 1 pgp_num to 11 2024-06-14T23:00:51.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:51.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:51.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:51.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-14T23:00:52.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:52.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:52.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[97913]: pgmap v751: 99 pgs: 99 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 158 MiB/s wr, 2.53k op/s 2024-06-14T23:00:52.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4263354423' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:00:52.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:00:52.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/815160882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:52.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[102754]: pgmap v751: 99 pgs: 99 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 158 MiB/s wr, 2.53k op/s 2024-06-14T23:00:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4263354423' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:00:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:00:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/815160882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:52.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:52.193 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:52.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:52.211 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 38395 38379 159.886 118.562 0.00341257 0.00624879 2024-06-14T23:00:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:51 smithi137 ceph-mon[90634]: pgmap v751: 99 pgs: 99 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 158 MiB/s wr, 2.53k op/s 2024-06-14T23:00:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4263354423' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:00:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:00:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/815160882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:00:52.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:52.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:52.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:52.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:52.476 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:52.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:52.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:52.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:52.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:52.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:52.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:52.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:52.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:52.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:52.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:52.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:52.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:52.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:53.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:53.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:53.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:52 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:53.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:52 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-06-14T23:00:53.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:52 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:00:53.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:52 smithi086 ceph-mon[97913]: osdmap e192: 8 total, 8 up, 8 in 2024-06-14T23:00:53.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:52 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:53.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:52 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-06-14T23:00:53.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:52 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:00:53.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:52 smithi086 ceph-mon[102754]: osdmap e192: 8 total, 8 up, 8 in 2024-06-14T23:00:53.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:53.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:53.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:53.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 40096 40080 156.536 106.312 0.00807793 0.00638333 2024-06-14T23:00:53.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:52 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:00:53.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:52 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-06-14T23:00:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:52 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:00:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:52 smithi137 ceph-mon[90634]: osdmap e192: 8 total, 8 up, 8 in 2024-06-14T23:00:53.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:53.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:53.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:53.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:53.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:53.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:53.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:53.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:53.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:53.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:53.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:53.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:53.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:53.858 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:00:53.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:53.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:53 smithi086 ceph-mon[97913]: pgmap v753: 99 pgs: 99 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 157 MiB/s wr, 2.52k op/s 2024-06-14T23:00:53.980 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:53 smithi086 ceph-mon[102754]: pgmap v753: 99 pgs: 99 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 157 MiB/s wr, 2.52k op/s 2024-06-14T23:00:53.986 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:53.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:54.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:54.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:54.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:54.143 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:53 smithi137 ceph-mon[90634]: pgmap v753: 99 pgs: 99 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 157 MiB/s wr, 2.52k op/s 2024-06-14T23:00:54.211 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 16 42031 42015 154.441 120.938 0.00790098 0.00646923 2024-06-14T23:00:54.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:54.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:54.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:54.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:54.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:54.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:54.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:54.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:54.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:54.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:54.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:54.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:54.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:00:54.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:54.816 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:54.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:54.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:54.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:54.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:55.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:55.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:55.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:55.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:00:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:00:54] "GET /metrics HTTP/1.1" 200 36592 "" "Prometheus/2.43.0" 2024-06-14T23:00:55.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:55.211 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 43880 43864 152.281 115.562 0.00600948 0.00656063 2024-06-14T23:00:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:55.234 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:55.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:55.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:55.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:55.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:55.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:55.498 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:55.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:55.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:55.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:55.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:55.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:55.764 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:55.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:55.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:55.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:55.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:55.961 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:55 smithi137 ceph-mon[90634]: pgmap v754: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-06-14T23:00:55.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:56.022 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:00:56.022 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:00:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:56.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:56.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:55 smithi086 ceph-mon[97913]: pgmap v754: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-06-14T23:00:56.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:55 smithi086 ceph-mon[102754]: pgmap v754: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-06-14T23:00:56.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:56.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:56.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:56.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:56.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 45775 45759 150.5 118.438 0.00563814 0.00663975 2024-06-14T23:00:56.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:56.297 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:56.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:56.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:56.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:56.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:56.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:56.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:56.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:56.698 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:56.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:56.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:56.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:56.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:56.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T23:00:56.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:56.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:57.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:57.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:57.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:57.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:00:57.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:00:57.212403+0000 min lat: 0.00119713 max lat: 0.0961691 avg lat: 0.00675453 2024-06-14T23:00:57.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:00:57.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 47329 47313 147.831 97.125 0.0606004 0.00675453 2024-06-14T23:00:57.252 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:57.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:00:57.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:57.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:57.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:57.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:57.534 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:57.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:57.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:57.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:57.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:57.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:57.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:57.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:57.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:57.847 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:00:57.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:57.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:57.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:58.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:57 smithi086 ceph-mon[97913]: pgmap v755: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-06-14T23:00:58.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:58.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:58.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:57 smithi086 ceph-mon[102754]: pgmap v755: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-06-14T23:00:58.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:58.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:58.092 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:57 smithi137 ceph-mon[90634]: pgmap v755: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-06-14T23:00:58.092 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:00:58.092 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:00:58.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:58.132 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:58.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:58.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 16 48981 48965 145.707 103.25 0.00556474 0.00685705 2024-06-14T23:00:58.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:00:58.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:58.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:00:58.276 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:00:58.338 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:00:58.338 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:00:58.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:58.375 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:00:58.375 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:00:58.400 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:00:58.401 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:00:58.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:58.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:58.428 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:00:58.428 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:00:58.459 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:00:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:00:58.491 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:00:58.491 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:00:58.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:58.524 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:00:58.525 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:00:58.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:58.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:58.560 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:00:58.560 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:00:58.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:00:58.691 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:58.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:00:58.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:58.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:58.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:58.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:58.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:58.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:58.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3244409138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:59.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:00:59.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3244409138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:59.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3244409138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:00:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:59.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:00:59.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:59.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 50570 50554 143.596 99.3125 0.00525267 0.00695851 2024-06-14T23:00:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:59.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:59.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:00:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:59.376 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:00:59.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:00:59.428 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:00:59.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 0 1 2024-06-14T23:00:59.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:59.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:59.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:59.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:59.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:00:59.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:59.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:00:59.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:00:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:59.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:59.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:00:59.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:00:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:00:59.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:00:59.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:00:59.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:00:59 smithi137 ceph-mon[90634]: pgmap v756: 99 pgs: 99 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-06-14T23:01:00.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:00.073 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:00.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:00.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:00:59 smithi086 ceph-mon[102754]: pgmap v756: 99 pgs: 99 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-06-14T23:01:00.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:00:59 smithi086 ceph-mon[97913]: pgmap v756: 99 pgs: 99 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-06-14T23:01:00.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:00.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 52120 52104 141.564 96.875 0.00942256 0.00705962 2024-06-14T23:01:00.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:00.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:00.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:00.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:00.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:00.443 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:00.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:00.507 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:00.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:00.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:00.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:00.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:00.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:00.785 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:00.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:00.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:00.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:00.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:01.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:01.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:01.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:01.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:01.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:01.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:01.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 53785 53769 140.002 104.062 0.00705441 0.00713749 2024-06-14T23:01:01.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:01.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:01.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:01.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:01.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:01.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:01.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:01.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:01.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:01.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:01.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:01.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:01.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:01.852 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:01:01.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:01.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:01.880 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:01:01.880 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:01:01.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.880 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:54186 10.b 10:d74f9cbc:::benchmark_data_smithi086_189745_object54185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:01.467580+0000", 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0031925410000000001, 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032262599999999999, 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 54186 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:01.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467580+0000", 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467580+0000", 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467578+0000", 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467812+0000", 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023414599999999999 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467827+0000", 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4603000000000001e-05 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467834+0000", 2024-06-14T23:01:01.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8199999999999999e-06 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468356+0000", 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052188099999999997 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468475+0000", 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119503 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:01:01.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468607+0000", 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131886 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:54187 10.f 10:f3141bb7:::benchmark_data_smithi086_189745_object54186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:01.467843+0000", 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0029289199999999998, 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030484230000000002, 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:01.885 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 54187 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467843+0000", 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467843+0000", 2024-06-14T23:01:01.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467843+0000", 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467880+0000", 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7166999999999998e-05 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:01.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467888+0000", 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6410000000000008e-06 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467893+0000", 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9509999999999997e-06 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467945+0000", 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1616999999999997e-05 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468064+0000", 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118639 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468210+0000", 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014644899999999999 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:54188 10.7 10:ef2b3539:::benchmark_data_smithi086_189745_object54187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-14T23:01:01.889 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:01.467906+0000", 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0028662179999999998, 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003067953, 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 54188 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:01.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467906+0000", 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467906+0000", 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467905+0000", 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467942+0000", 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6928999999999998e-05 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467951+0000", 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7299999999999994e-06 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.467956+0000", 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.809e-06 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468004+0000", 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8489000000000003e-05 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468132+0000", 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127589 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468266+0000", 2024-06-14T23:01:01.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013354099999999999 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:54190 10.7 10.61697917 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:01.468051+0000", 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.002721261, 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029970779999999998, 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:01.894 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 54190 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468051+0000", 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468051+0000", 2024-06-14T23:01:01.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468050+0000", 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468456+0000", 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040579899999999998 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:01.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468464+0000", 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1580000000000005e-06 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468469+0000", 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.181e-06 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.45821.0:54182 10.7 e192/186)", 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:01.468482+0000", 2024-06-14T23:01:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0022903559999999999, 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026123309999999999, 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468482+0000", 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468482+0000", 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468481+0000", 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:01.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468484+0000", 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0410000000000002e-06 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468490+0000", 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.823e-06 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468494+0000", 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.473e-06 2024-06-14T23:01:01.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:54185 10.e e192/186 10:7b2baf5c:::benchmark_data_smithi086_189745_object54184:head v 192'3392, mlcod=192'3389)", 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:01.469346+0000", 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0014262999999999999, 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001813678, 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:01:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.469346+0000", 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.469346+0000", 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.469345+0000", 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.469372+0000", 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.656e-05 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.469384+0000", 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1926999999999999e-05 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.469390+0000", 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9950000000000002e-06 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:01.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.469449+0000", 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8950000000000003e-05 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.469467+0000", 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8527000000000001e-05 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.905 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:54172 10.b 10:dfbde4a8:::benchmark_data_smithi086_189745_object54171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-14T23:01:01.906 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:01.456019+0000", 2024-06-14T23:01:01.906 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.014753194000000001, 2024-06-14T23:01:01.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015225664, 2024-06-14T23:01:01.906 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:01.906 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:01:01.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:01.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:01.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:01.906 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 54172 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.456019+0000", 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.456019+0000", 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.456018+0000", 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.456049+0000", 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1133000000000002e-05 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:01.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.456062+0000", 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2952999999999999e-05 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.456068+0000", 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5849999999999996e-06 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.457096+0000", 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001027708 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.457203+0000", 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010673799999999999 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.457323+0000", 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120217 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.459994+0000", 2024-06-14T23:01:01.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026713150000000001 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.461705+0000", 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001710526 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:54179 10.b 10:d9954f9a:::benchmark_data_smithi086_189745_object54178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:01.461802+0000", 2024-06-14T23:01:01.911 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0089706660000000004, 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095438840000000007, 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 54179 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.461802+0000", 2024-06-14T23:01:01.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.461802+0000", 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.461801+0000", 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.461827+0000", 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5497999999999999e-05 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.461833+0000", 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7290000000000003e-06 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.461837+0000", 2024-06-14T23:01:01.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1060000000000002e-06 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.461906+0000", 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8062e-05 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.462038+0000", 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000132588 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.462176+0000", 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137601 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468108+0000", 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059321840000000001 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.468336+0000", 2024-06-14T23:01:01.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000227665 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:54182 10.7 10:e474cecc:::benchmark_data_smithi086_189745_object54181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:01.462694+0000", 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0080778629999999994, 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0087351939999999999, 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:01:01.917 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 54182 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.462694+0000", 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.462694+0000", 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.462693+0000", 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.462778+0000", 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4975999999999994e-05 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.462810+0000", 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1491000000000001e-05 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:01.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.462835+0000", 2024-06-14T23:01:01.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5253e-05 2024-06-14T23:01:01.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:01.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.462893+0000", 2024-06-14T23:01:01.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8326999999999999e-05 2024-06-14T23:01:01.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.463021+0000", 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012752099999999999 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.463188+0000", 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016747299999999999 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.465437+0000", 2024-06-14T23:01:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022480880000000001 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:01.466286+0000", 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00084992300000000004 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 9 2024-06-14T23:01:01.923 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:01:01.936 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:01 smithi086 ceph-mon[102754]: pgmap v757: 99 pgs: 99 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-06-14T23:01:01.936 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/491075899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:01:01.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:01 smithi086 ceph-mon[97913]: pgmap v757: 99 pgs: 99 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-06-14T23:01:01.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/491075899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:01:01.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:01.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:01 smithi137 ceph-mon[90634]: pgmap v757: 99 pgs: 99 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-06-14T23:01:01.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/491075899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:01:02.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:02.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:02.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:02.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:02.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:02.213 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 55367 55351 138.356 98.875 0.0150233 0.00722289 2024-06-14T23:01:02.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:02.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:02.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:02.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:02.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:02.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:02.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:01:02.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:02.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:02.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:02.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:02.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:02.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:02.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:02.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:02.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:02.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:02.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:02.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:02.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:02.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:03.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:03.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:03.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:03.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:03.212 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 56870 56854 136.648 93.9375 0.0325782 0.00731351 2024-06-14T23:01:03.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:03.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:03.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:02 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:01:03.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/491075899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:01:03.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:02 smithi137 ceph-mon[90634]: osdmap e193: 8 total, 8 up, 8 in 2024-06-14T23:01:03.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:02 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:01:03.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/491075899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:01:03.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:02 smithi086 ceph-mon[97913]: osdmap e193: 8 total, 8 up, 8 in 2024-06-14T23:01:03.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:02 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:01:03.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/491075899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:01:03.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:02 smithi086 ceph-mon[102754]: osdmap e193: 8 total, 8 up, 8 in 2024-06-14T23:01:03.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:03.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:03.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:03.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:03.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:03.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:03.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:03.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:03.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:03.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:03.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:03.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:03.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:03.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:03.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:04.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:04.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:04.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:04.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:04.198 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:03 smithi086 ceph-mon[102754]: pgmap v759: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-06-14T23:01:04.199 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:03 smithi086 ceph-mon[97913]: pgmap v759: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-06-14T23:01:04.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:04.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:04.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 58551 58535 135.474 105.062 0.00824909 0.00737507 2024-06-14T23:01:04.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:03 smithi137 ceph-mon[90634]: pgmap v759: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-06-14T23:01:04.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:04.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:04.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:04.451 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:04.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:04.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:04.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:04.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-14T23:01:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:04.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:04.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:04.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:04.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:04.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:04.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:04.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:05.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:05.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:05.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:05.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:01:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:01:04] "GET /metrics HTTP/1.1" 200 36581 "" "Prometheus/2.43.0" 2024-06-14T23:01:05.127 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:01:05.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:05.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:05.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:05.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 60001 59985 133.872 90.625 0.0106541 0.00746441 2024-06-14T23:01:05.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:05.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:05.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:05.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:05.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:05.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:05.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:05.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:05.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:05.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:05.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:05.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:05.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:05.815 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:05.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:05.885 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:05 smithi086 ceph-mon[102754]: pgmap v760: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-14T23:01:05.886 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:05.886 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:05.886 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1127640062' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:01:05.886 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.0 to 1 (10000) 2024-06-14T23:01:05.886 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:05 smithi086 ceph-mon[97913]: pgmap v760: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-14T23:01:05.886 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:05.887 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:05.887 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1127640062' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:01:05.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:05.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:05.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:05 smithi137 ceph-mon[90634]: pgmap v760: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-14T23:01:05.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:05.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:05.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1127640062' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:01:06.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:06.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:06.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:06.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:06.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:06.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:06.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:06.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:06.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 61590 61574 132.68 99.3125 0.0119621 0.00753242 2024-06-14T23:01:06.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:01:06.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:06.337 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:06.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:06.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:06.473 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:06.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:06.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:06.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:06.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:06.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:06.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:06.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:06.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:06.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 1 1 2024-06-14T23:01:07.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:07.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:07.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:07.153 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1127640062' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-14T23:01:07.153 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:06 smithi137 ceph-mon[90634]: osdmap e194: 8 total, 8 up, 8 in 2024-06-14T23:01:07.153 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:06 smithi137 ceph-mon[90634]: pgmap v762: 99 pgs: 99 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-06-14T23:01:07.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:07.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:07.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:07.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1127640062' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-14T23:01:07.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:06 smithi086 ceph-mon[97913]: osdmap e194: 8 total, 8 up, 8 in 2024-06-14T23:01:07.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:06 smithi086 ceph-mon[97913]: pgmap v762: 99 pgs: 99 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-06-14T23:01:07.188 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1127640062' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-14T23:01:07.188 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:06 smithi086 ceph-mon[102754]: osdmap e194: 8 total, 8 up, 8 in 2024-06-14T23:01:07.189 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:06 smithi086 ceph-mon[102754]: pgmap v762: 99 pgs: 99 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-06-14T23:01:07.198 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:01:07.198 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:01:07.198 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:01:07.198 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:01:07.198 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:01:07.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 63099 63083 131.401 94.3125 0.0113134 0.00760558 2024-06-14T23:01:07.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:07.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:07.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:07.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:07.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:07.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:07.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:07.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:07.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:07.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:07.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:07.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:07.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:07.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:07.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:07.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-14T23:01:07.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:07.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:07.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:08.013 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:08.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:08.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:08.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:08.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 64582 64566 130.152 92.6875 0.0214874 0.00767821 2024-06-14T23:01:08.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:08.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:08.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:08.311 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:01:08.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:08.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:08.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:08.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:08.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:08.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:08.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:08.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:08.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:08.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:08.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:08.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:08.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:08.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:08.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:09.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:09.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:09.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:09.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:09.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 66046 66030 128.944 91.5 0.0109813 0.00775014 2024-06-14T23:01:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:09.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:09.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:09.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:09.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:09.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:09.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:09.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:09.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:09.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:09.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:09.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:09.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:09.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:09.809 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:09 smithi137 ceph-mon[90634]: pgmap v763: 99 pgs: 99 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:01:09.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:09.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:09.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:09.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:09 smithi086 ceph-mon[102754]: pgmap v763: 99 pgs: 99 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:01:09.992 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:09 smithi086 ceph-mon[97913]: pgmap v763: 99 pgs: 99 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:01:10.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:10.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:10.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:10.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:10.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:10.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:10.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 16 67584 67568 127.95 96.125 0.00571223 0.00780968 2024-06-14T23:01:10.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:10.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:10.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:10.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:10.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:10.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:10.512 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:01:10.512 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:01:10.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:01:10.512 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:01:10.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.512 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189735 9.1 9:8c974211:::benchmark_data_smithi086_152062_object57253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:01:10.512 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:19.288420+0000", 2024-06-14T23:01:10.512 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 171.21922639300001, 2024-06-14T23:01:10.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.060468734000000003, 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189735 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288420+0000", 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288420+0000", 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288419+0000", 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288425+0000", 2024-06-14T23:01:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5589999999999996e-06 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288441+0000", 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6191000000000001e-05 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.288447+0000", 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9580000000000004e-06 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.306503+0000", 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018056493999999999 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.306775+0000", 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000271439 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.307076+0000", 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030119800000000003 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.327473+0000", 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020397408999999998 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.332910+0000", 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054364610000000001 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.348788+0000", 2024-06-14T23:01:10.517 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015878006 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.348846+0000", 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8294999999999997e-05 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.348889+0000", 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2718e-05 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.518 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189739 9.1 9:8f3b18eb:::benchmark_data_smithi086_152062_object57257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:19.293852+0000", 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 171.213794347, 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059466487999999998, 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189739 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.519 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293852+0000", 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293852+0000", 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293851+0000", 2024-06-14T23:01:10.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293859+0000", 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8529999999999992e-06 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293877+0000", 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8148e-05 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.293883+0000", 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7320000000000001e-06 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.311872+0000", 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017989092000000002 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.312176+0000", 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000303696 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.312487+0000", 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000310365 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.327500+0000", 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015013253000000001 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.332925+0000", 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054252670000000001 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.353087+0000", 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020162269999999999 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.353204+0000", 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011688099999999999 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:19.353319+0000", 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114548 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.524 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:230781 9.1 9:8c197635:::benchmark_data_smithi086_152062_object98299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:06.622879+0000", 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 123.884767936, 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.063133849000000006, 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 230781 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.622879+0000", 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.622879+0000", 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.622878+0000", 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.622882+0000", 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.63e-06 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.622896+0000", 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4448e-05 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.622901+0000", 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6120000000000001e-06 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.634162+0000", 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011260857000000001 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.634268+0000", 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010642200000000001 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.634420+0000", 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015242899999999999 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.639489+0000", 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050688499999999997 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.658025+0000", 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018535303999999999 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.685922+0000", 2024-06-14T23:01:10.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027897675 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.685970+0000", 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.816e-05 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:06.686013+0000", 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2070000000000002e-05 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.530 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:20261 10.0 10:043bc759:::benchmark_data_smithi086_189745_object20260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:43.696782+0000", 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 26.810864247000001, 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059781049000000003, 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20261 2024-06-14T23:01:10.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.696782+0000", 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.696782+0000", 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.696781+0000", 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.696822+0000", 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0160000000000002e-05 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.696842+0000", 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0353e-05 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.696848+0000", 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7919999999999998e-06 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.697355+0000", 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050731300000000001 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.697461+0000", 2024-06-14T23:01:10.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010618799999999999 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.697737+0000", 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000276166 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698749+0000", 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010114919999999999 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.756118+0000", 2024-06-14T23:01:10.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.057369321000000001 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.756479+0000", 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036090500000000001 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.756524+0000", 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4524999999999998e-05 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.756563+0000", 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9749000000000002e-05 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:20263 10.0 10:000b15c3:::benchmark_data_smithi086_189745_object20262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:43.696873+0000", 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 26.810773442999999, 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059772782000000003, 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.537 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20263 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.696873+0000", 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.696873+0000", 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.696873+0000", 2024-06-14T23:01:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698811+0000", 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019382830000000001 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698820+0000", 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4429999999999992e-06 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698826+0000", 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2970000000000003e-06 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698962+0000", 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013659100000000001 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.699066+0000", 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010371599999999999 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.699274+0000", 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000207784 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700218+0000", 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00094478400000000005 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.756152+0000", 2024-06-14T23:01:10.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.055933604999999997 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.756603+0000", 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00045099799999999998 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.756627+0000", 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4034000000000001e-05 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.756646+0000", 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8853999999999999e-05 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:20265 10.7 10:efeb30ea:::benchmark_data_smithi086_189745_object20264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:43.698839+0000", 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 26.808807764000001, 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.060167368999999998, 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.544 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20265 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698839+0000", 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698839+0000", 2024-06-14T23:01:10.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698838+0000", 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698872+0000", 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4452999999999997e-05 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698879+0000", 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8580000000000002e-06 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.698883+0000", 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9400000000000004e-06 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.699158+0000", 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000274913 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.699279+0000", 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121073 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.699409+0000", 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012975399999999999 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.758069+0000", 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.058659783 2024-06-14T23:01:10.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.758889+0000", 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00081973500000000004 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.758926+0000", 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7150000000000002e-05 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.758971+0000", 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5457999999999997e-05 2024-06-14T23:01:10.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.759006+0000", 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5125e-05 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:20269 10.7 10:e56718ef:::benchmark_data_smithi086_189745_object20268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:43.700236+0000", 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 26.807410599000001, 2024-06-14T23:01:10.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.078286028999999993, 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20269 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700236+0000", 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700236+0000", 2024-06-14T23:01:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700235+0000", 2024-06-14T23:01:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:10.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700390+0000", 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155193 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700402+0000", 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1793999999999999e-05 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700408+0000", 2024-06-14T23:01:10.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6899999999999997e-06 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700451+0000", 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2982999999999998e-05 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700572+0000", 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012118800000000001 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.757188+0000", 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.056615784000000002 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.759035+0000", 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001847642 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.759058+0000", 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2367000000000001e-05 2024-06-14T23:01:10.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.778436+0000", 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019378543000000002 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.778488+0000", 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1490000000000003e-05 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.778522+0000", 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4486999999999997e-05 2024-06-14T23:01:10.556 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:20272 10.f 10:f0d57aeb:::benchmark_data_smithi086_189745_object20271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:43.700628+0000", 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 26.807018594999999, 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.079060627999999994, 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.557 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20272 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700628+0000", 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700628+0000", 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700627+0000", 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.700660+0000", 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2518999999999998e-05 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.755890+0000", 2024-06-14T23:01:10.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.055230393000000003 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.755904+0000", 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3866000000000001e-05 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.758092+0000", 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021877569999999998 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.758194+0000", 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102056 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.758309+0000", 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115445 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.777133+0000", 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018823745999999999 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.777193+0000", 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9728999999999997e-05 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.779611+0000", 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024181150000000002 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.779654+0000", 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2886000000000002e-05 2024-06-14T23:01:10.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.779689+0000", 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4771999999999998e-05 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:20847 10.f 10:f1a4f58b:::benchmark_data_smithi086_189745_object20846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:43.964147+0000", 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 26.54350006, 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.064336246, 2024-06-14T23:01:10.563 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20847 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.964147+0000", 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.564 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.964147+0000", 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.964146+0000", 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.964177+0000", 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1656999999999999e-05 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.964188+0000", 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0180000000000001e-05 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.964192+0000", 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6550000000000003e-06 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.964535+0000", 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034278100000000001 2024-06-14T23:01:10.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:43.964667+0000", 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013221899999999999 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.015988+0000", 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.051320426000000002 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.018310+0000", 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023225139999999999 2024-06-14T23:01:10.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.018331+0000", 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0956999999999999e-05 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.028021+0000", 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096895839999999994 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.028449+0000", 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042854200000000002 2024-06-14T23:01:10.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.028483+0000", 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3469000000000001e-05 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:22263 10.a 10:5dcaed5e:::benchmark_data_smithi086_189745_object22262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:44.547357+0000", 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 25.960289698, 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.062149778000000003, 2024-06-14T23:01:10.569 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22263 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.547357+0000", 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.570 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.547357+0000", 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.547356+0000", 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.547390+0000", 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3997e-05 2024-06-14T23:01:10.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.547398+0000", 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2339999999999997e-06 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.547402+0000", 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6070000000000001e-06 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.547461+0000", 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8656999999999998e-05 2024-06-14T23:01:10.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.547566+0000", 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010501399999999999 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.547684+0000", 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118188 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.548715+0000", 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010305189999999999 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.550585+0000", 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001870013 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.609431+0000", 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.058846843000000003 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.609471+0000", 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0024999999999997e-05 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.609507+0000", 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5213999999999999e-05 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:22270 10.a 10:541dca84:::benchmark_data_smithi086_189745_object22269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:44.553895+0000", 2024-06-14T23:01:10.575 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 25.953751628999999, 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.061469335, 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22270 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.553895+0000", 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.553895+0000", 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.553895+0000", 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.553937+0000", 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2861000000000002e-05 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.553944+0000", 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8689999999999998e-06 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.553948+0000", 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1099999999999996e-06 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.609784+0000", 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.055835695999999997 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.609913+0000", 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012934700000000001 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.610867+0000", 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00095340900000000001 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.615181+0000", 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043140510000000002 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.615244+0000", 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3613000000000002e-05 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.615278+0000", 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3018999999999997e-05 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.615320+0000", 2024-06-14T23:01:10.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2215999999999997e-05 2024-06-14T23:01:10.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.615364+0000", 2024-06-14T23:01:10.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4610999999999997e-05 2024-06-14T23:01:10.581 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.581 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:55503 10.f 10:f2095c8e:::benchmark_data_smithi086_189745_object55502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:02.289161+0000", 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.2184853400000009, 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.071015054999999994, 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 55503 2024-06-14T23:01:10.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.289161+0000", 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.289161+0000", 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.289161+0000", 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.289191+0000", 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0543999999999998e-05 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.289198+0000", 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4400000000000002e-06 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.289201+0000", 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4860000000000002e-06 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296047+0000", 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068452920000000002 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296158+0000", 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011163999999999999 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:01:10.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296351+0000", 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000192249 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.353038+0000", 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.056687560999999997 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.353116+0000", 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8280000000000003e-05 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360098+0000", 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0069818270000000003 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360140+0000", 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1789999999999998e-05 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360176+0000", 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6371000000000001e-05 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:55506 10.7 10:e8b32ac9:::benchmark_data_smithi086_189745_object55505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:02.291024+0000", 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.2166228629999996, 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.068001303999999999, 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 55506 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.588 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291024+0000", 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291024+0000", 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291022+0000", 2024-06-14T23:01:10.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291077+0000", 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4165000000000001e-05 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291089+0000", 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2475e-05 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291095+0000", 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8390000000000003e-06 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292315+0000", 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001220219 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292421+0000", 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010555700000000001 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292538+0000", 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011680600000000001 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.353007+0000", 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.060469138999999998 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.353064+0000", 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6929000000000003e-05 2024-06-14T23:01:10.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.358941+0000", 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058777930000000001 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.358990+0000", 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8239999999999999e-05 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.359025+0000", 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5468e-05 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.593 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:55507 10.7 10:eac47abd:::benchmark_data_smithi086_189745_object55506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:02.291256+0000", 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.216391003, 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.067842354999999993, 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.594 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 55507 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291256+0000", 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291256+0000", 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291255+0000", 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291374+0000", 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118663 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291385+0000", 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1214e-05 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.291389+0000", 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.827e-06 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.294407+0000", 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030185339999999998 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.294503+0000", 2024-06-14T23:01:10.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6040999999999997e-05 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.294616+0000", 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112575 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.353089+0000", 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.058473063999999998 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.353142+0000", 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2787999999999997e-05 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.359033+0000", 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005891248 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.359070+0000", 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7171e-05 2024-06-14T23:01:10.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.359098+0000", 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7889e-05 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:55513 10.a 10:539eb3e6:::benchmark_data_smithi086_189745_object55512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:02.292338+0000", 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.2153089000000001, 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.061837120000000002, 2024-06-14T23:01:10.600 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 55513 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292338+0000", 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292338+0000", 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292337+0000", 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292368+0000", 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0446e-05 2024-06-14T23:01:10.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292377+0000", 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4630000000000007e-06 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292382+0000", 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7659999999999998e-06 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292426+0000", 2024-06-14T23:01:10.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4749000000000001e-05 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292529+0000", 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010259699999999999 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.292648+0000", 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119212 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296792+0000", 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004143702 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296820+0000", 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8215e-05 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.352782+0000", 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.055961840999999998 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.352834+0000", 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1693999999999997e-05 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.354175+0000", 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013411040000000001 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.606 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.45821.0:55502 10.f e193/186 ondisk, result = 0)", 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:02.296196+0000", 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.2114504410000002, 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.062904294999999999, 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296196+0000", 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296196+0000", 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296196+0000", 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296209+0000", 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3623e-05 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296217+0000", 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8110000000000003e-06 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296221+0000", 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1350000000000002e-06 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.298278+0000", 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020563589999999998 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.298283+0000", 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3009999999999997e-06 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.359100+0000", 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.060817587999999999 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.610 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:55517 10.7 10:e23006dd:::benchmark_data_smithi086_189745_object55516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:02.296295+0000", 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.2113513959999995, 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.063969401999999995, 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 55517 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296295+0000", 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296295+0000", 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296295+0000", 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296324+0000", 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9147000000000001e-05 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296331+0000", 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1119999999999998e-06 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296335+0000", 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.369e-06 2024-06-14T23:01:10.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.298245+0000", 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019096320000000001 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.350755+0000", 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.052510514000000001 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:01:10.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.350928+0000", 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017232799999999999 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.355318+0000", 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043906600000000002 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.357871+0000", 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025524240000000002 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360188+0000", 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023175079999999998 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360231+0000", 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2821000000000002e-05 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360265+0000", 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3433000000000003e-05 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.616 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:55520 10.7 10:e61e51c4:::benchmark_data_smithi086_189745_object55519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:02.296811+0000", 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.2108358579999994, 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.063571406999999996, 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 55520 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296811+0000", 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296811+0000", 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296810+0000", 2024-06-14T23:01:10.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296886+0000", 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6297999999999994e-05 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296896+0000", 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8649999999999996e-06 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.296901+0000", 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8899999999999998e-06 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.353164+0000", 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.056262965999999998 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.353281+0000", 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011715 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.353410+0000", 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000128823 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.357846+0000", 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044359719999999998 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360118+0000", 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002271739 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360272+0000", 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015442400000000001 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360350+0000", 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8059000000000006e-05 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.360382+0000", 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1977000000000002e-05 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:55523 10.a 10:539362aa:::benchmark_data_smithi086_189745_object55522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:02.298587+0000", 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.2090601220000003, 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.061335413999999998, 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.624 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 55523 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.298587+0000", 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.298587+0000", 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.298586+0000", 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.298645+0000", 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9469000000000003e-05 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.350705+0000", 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.052059893000000003 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.350718+0000", 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3081999999999999e-05 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.354371+0000", 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036528440000000001 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.354486+0000", 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114695 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.354608+0000", 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122514 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.358903+0000", 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042947860000000001 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.358912+0000", 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8349999999999993e-06 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.359831+0000", 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00091909399999999997 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.359880+0000", 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9561000000000002e-05 2024-06-14T23:01:10.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.359922+0000", 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1489999999999997e-05 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:55773 10.7 10:e6db8ff0:::benchmark_data_smithi086_189745_object55772:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:02.484628+0000", 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.0230183979999996, 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.060752701999999999, 2024-06-14T23:01:10.630 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 55773 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.484628+0000", 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.484628+0000", 2024-06-14T23:01:10.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.484628+0000", 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.484703+0000", 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5119999999999999e-05 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.484712+0000", 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.448e-06 2024-06-14T23:01:10.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.484716+0000", 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0899999999999998e-06 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.489354+0000", 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046377479999999997 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.489481+0000", 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126888 2024-06-14T23:01:10.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.489605+0000", 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012442900000000001 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.492689+0000", 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030837880000000001 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.495741+0000", 2024-06-14T23:01:10.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030517360000000002 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.545254+0000", 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.049513458000000003 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.545346+0000", 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1799999999999995e-05 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:02.545381+0000", 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4721999999999997e-05 2024-06-14T23:01:10.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:10.636 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:10.636 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:01:10.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:10.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:10.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:10.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:10.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:10.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:10.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:10.947 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:10.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:11.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:11.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:11.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:11.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:11.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 68984 68968 126.76 87.5 0.0149433 0.0078836 2024-06-14T23:01:11.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:11.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:11.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:11.357 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:11.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:11.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:11.492 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:11.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:11.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:11.629 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:11.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:11.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-14T23:01:11.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:11.764 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:11.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:11.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:11.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:11.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:11.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:11.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:11 smithi137 ceph-mon[90634]: pgmap v764: 99 pgs: 99 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-06-14T23:01:11.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:11.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:12.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:12.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:12.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:11 smithi086 ceph-mon[97913]: pgmap v764: 99 pgs: 99 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-06-14T23:01:12.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:12.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:12.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:11 smithi086 ceph-mon[102754]: pgmap v764: 99 pgs: 99 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-06-14T23:01:12.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:12.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:12.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:12.148 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:01:12.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:12.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:12.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:12.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 70342 70326 125.563 84.875 0.00460772 0.00795969 2024-06-14T23:01:12.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:12.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:12.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:12.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:12.478 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:12.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:12.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:12.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:12.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:12.687 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:01:12.688 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:01:12.713 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:01:12.713 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:01:12.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:12.738 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:01:12.738 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:01:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:12.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:12.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:12.774 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:01:12.774 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:01:12.817 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:01:12.817 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:01:12.858 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:01:12.858 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:01:12.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:12.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:12.902 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:01:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:01:12.931 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:01:12.931 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:01:12.958 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:12.958 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:01:12.958 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:01:12.958 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:12.958 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4233621289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:12.958 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:12.958 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:01:12.958 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:01:12.958 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:12.959 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4233621289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:12.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:13.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:13.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:13.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:13.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:13.182 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:13.182 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:01:13.182 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:01:13.182 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:13.182 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4233621289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:13.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 71575 71559 124.216 77.0625 0.0269821 0.00804431 2024-06-14T23:01:13.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:13.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:13.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:13.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:13.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:13.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:13.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:13.591 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:13.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:13.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:13.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:13.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:13.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:13.856 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.1 to 1 (10000) 2024-06-14T23:01:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:13.879 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:13.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:13.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:14.014 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:14.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:14.053 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:13 smithi137 ceph-mon[90634]: pgmap v765: 99 pgs: 99 active+clean; 4.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-14T23:01:14.053 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1364548794' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:01:14.053 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:13 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:01:14.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:13 smithi086 ceph-mon[97913]: pgmap v765: 99 pgs: 99 active+clean; 4.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-14T23:01:14.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1364548794' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:01:14.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:13 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:01:14.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:13 smithi086 ceph-mon[102754]: pgmap v765: 99 pgs: 99 active+clean; 4.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-14T23:01:14.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1364548794' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:01:14.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:13 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:01:14.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:14.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:14.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:14.214 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:01:14.214 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:01:14.215 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 72907 72891 123.108 83.25 0.00793733 0.00811738 2024-06-14T23:01:14.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:14.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:14.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:14.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:14.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:14.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:14.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:14.546 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:14.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:14.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:14.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:14.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:14.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:14.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:14.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:14.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:14.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:14.955 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:14.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:15.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:15.078 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:15.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:15.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:15.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 74141 74125 121.898 77.125 0.0123468 0.00819862 2024-06-14T23:01:15.217 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:15.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:15.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:14 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-14T23:01:15.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:14 smithi137 ceph-mon[90634]: osdmap e195: 8 total, 8 up, 8 in 2024-06-14T23:01:15.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:15.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:14 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-14T23:01:15.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:14 smithi086 ceph-mon[97913]: osdmap e195: 8 total, 8 up, 8 in 2024-06-14T23:01:15.335 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:01:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:01:14] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.43.0" 2024-06-14T23:01:15.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:14 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-14T23:01:15.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:14 smithi086 ceph-mon[102754]: osdmap e195: 8 total, 8 up, 8 in 2024-06-14T23:01:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:15.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:15.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:15.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:15.490 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:15.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:15.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:15.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:15.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:15.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:15.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:15.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:15.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:15.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:16.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:16.041 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:16.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:16.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:16.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:16.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:16.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 16 75582 75566 121.082 90.0625 0.0139703 0.00825412 2024-06-14T23:01:16.242 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:15 smithi086 ceph-mon[97913]: pgmap v767: 99 pgs: 99 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-14T23:01:16.243 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:15 smithi086 ceph-mon[102754]: pgmap v767: 99 pgs: 99 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-14T23:01:16.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:15 smithi137 ceph-mon[90634]: pgmap v767: 99 pgs: 99 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-06-14T23:01:16.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:16.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:16.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:16.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:16.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:16.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:16.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:16.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:16.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:16.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:16.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 2 1 2024-06-14T23:01:16.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:16.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:16.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:16.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:16.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:16.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:16.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:16 smithi086 ceph-mon[97913]: pgmap v768: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-06-14T23:01:16.945 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:16 smithi086 ceph-mon[102754]: pgmap v768: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-06-14T23:01:16.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:16.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:17.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:17.100 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:17.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:17.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:17.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:01:17.215087+0000 min lat: 0.00119713 max lat: 0.110442 avg lat: 0.00833493 2024-06-14T23:01:17.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:01:17.215 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 76766 76750 119.905 74 0.0129491 0.00833493 2024-06-14T23:01:17.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:17.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:17.243 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:16 smithi137 ceph-mon[90634]: pgmap v768: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-06-14T23:01:17.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:17.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:17.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:17.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:17.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:17.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:17.636 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:17.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:17.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:17.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:17.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:17.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:17.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-14T23:01:17.865 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:17.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:17.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:17.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:18.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:18.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:18.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:18.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:18.170 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:01:18.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:18.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:18.214 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 77745 77729 118.472 61.1875 0.00845036 0.00843601 2024-06-14T23:01:18.284 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:01:18.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:18.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1904462011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:01:18.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:18 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:01:18.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1904462011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:01:18.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:18 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:01:18.347 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:18.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:18.437 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1904462011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:01:18.437 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:18 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:01:18.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:18.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:18.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:18.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:18.622 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:18.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:18.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:18.745 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:18.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:18.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:18.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:18.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:18.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:19.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:19.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:19.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:19.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:19.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:19.215 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 16 79083 79067 117.643 83.625 0.0112415 0.008496 2024-06-14T23:01:19.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:19.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:19.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:19.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:19.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:19.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:19.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:19 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:01:19.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:19 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:01:19.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:19 smithi137 ceph-mon[90634]: osdmap e196: 8 total, 8 up, 8 in 2024-06-14T23:01:19.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:19 smithi137 ceph-mon[90634]: pgmap v770: 99 pgs: 99 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s 2024-06-14T23:01:19.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:01:19.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:19.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:19.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:19.561 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:19 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:01:19.561 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:19 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:01:19.561 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:19 smithi086 ceph-mon[102754]: osdmap e196: 8 total, 8 up, 8 in 2024-06-14T23:01:19.562 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:19 smithi086 ceph-mon[102754]: pgmap v770: 99 pgs: 99 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s 2024-06-14T23:01:19.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:19 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:01:19.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:19 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:01:19.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:19 smithi086 ceph-mon[97913]: osdmap e196: 8 total, 8 up, 8 in 2024-06-14T23:01:19.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:19 smithi086 ceph-mon[97913]: pgmap v770: 99 pgs: 99 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s 2024-06-14T23:01:19.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:19.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:19.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:19.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:19.822 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:19.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:19.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:19.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:20.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:20.122 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:20.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:20.216 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 80418 80402 116.843 83.4375 0.0193183 0.00855343 2024-06-14T23:01:20.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:20.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:20.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:20.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:20.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:20.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:20.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:20.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:20.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:20.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:20.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:20.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:20.682 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:01:20.682 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:01:20.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:20.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:20.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:20.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:20.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:20.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:21.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:21.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:21.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:21.216 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 15 81685 81670 115.989 79.25 0.0180018 0.00861693 2024-06-14T23:01:21.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:21.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:21.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:21.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-14T23:01:21.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:21.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:21.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:21.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:21.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:21.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:21.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:21.754 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.2 to 1 (10000) 2024-06-14T23:01:21.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:21.806 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:01:21.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:21.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:21.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:21 smithi086 ceph-mon[97913]: pgmap v771: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T23:01:21.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/146772778' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:01:21.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:21 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:01:21.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:21 smithi086 ceph-mon[102754]: pgmap v771: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T23:01:21.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/146772778' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:01:21.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:21 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:01:21.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:21.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:22.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:21 smithi137 ceph-mon[90634]: pgmap v771: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-06-14T23:01:22.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/146772778' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:01:22.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:21 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:01:22.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:22.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:22.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:22.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:22.217 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 82937 82921 115.148 78.1875 0.0106385 0.00867968 2024-06-14T23:01:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:22.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:22.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:22.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:22.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:22.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:22.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:01:22.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:22.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:22.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:22.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:22.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:22.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:22.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:22.759 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:22 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:01:22.759 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:22 smithi086 ceph-mon[97913]: osdmap e197: 8 total, 8 up, 8 in 2024-06-14T23:01:22.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:22.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:22.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:22.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 3 1 2024-06-14T23:01:22.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:22.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:22.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:22 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:01:22.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:22 smithi137 ceph-mon[90634]: osdmap e197: 8 total, 8 up, 8 in 2024-06-14T23:01:23.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:23.066 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:22 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:01:23.066 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:22 smithi086 ceph-mon[102754]: osdmap e197: 8 total, 8 up, 8 in 2024-06-14T23:01:23.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:23.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:23.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:23.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:23.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:23.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:23.217 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 16 84145 84129 114.286 75.5 0.0101125 0.00874506 2024-06-14T23:01:23.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:23.353 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:23.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:23.404 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:23.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:23.500 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:23.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:23.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:23.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:23.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:23.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:23.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:23.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:23.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:23.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:23.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:23.990 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:23 smithi086 ceph-mon[97913]: pgmap v773: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-14T23:01:23.990 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:23 smithi086 ceph-mon[102754]: pgmap v773: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-14T23:01:24.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:24.035 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:23 smithi137 ceph-mon[90634]: pgmap v773: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-14T23:01:24.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:24.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:24.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:24.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:24.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:24.217 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 85352 85336 113.46 75.4375 0.00746966 0.00880911 2024-06-14T23:01:24.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:24.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:24.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:24.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:24.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:24.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:24.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:24.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:24.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:24.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:24.624 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:24.698 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:24.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:24.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:24.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:24.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:24.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:24.913 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:01:24.913 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:01:24.942 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:01:24.942 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:01:24.975 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:01:24.975 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:01:24.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:25.010 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:01:25.010 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:01:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:25.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:25.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:25.052 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:01:25.052 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:01:25.052 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:01:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:01:24] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.43.0" 2024-06-14T23:01:25.083 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:01:25.083 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:01:25.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:25.119 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:01:25.119 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:01:25.150 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:01:25.150 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:01:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:25.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:25.217 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 86575 86559 112.688 76.4375 0.0190304 0.0088691 2024-06-14T23:01:25.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:25.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:25.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:25.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:25.420 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:25.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:25.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:25.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:25.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:25.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:25.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:25.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:25.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:25.812 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:01:25.812 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:01:25.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:01:25.812 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:01:25.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.812 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:156912 9.e e159/145 9:764b212a:::benchmark_data_smithi086_152062_object24430:head v 159'9773, mlcod=159'9771)", 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:57:40.983294+0000", 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 224.82620439600001, 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.04572201, 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:40.983294+0000", 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:40.983294+0000", 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:40.983293+0000", 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:40.983305+0000", 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2092999999999999e-05 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:40.983319+0000", 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4141000000000001e-05 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:40.983324+0000", 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2390000000000002e-06 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:40.983411+0000", 2024-06-14T23:01:25.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6508000000000006e-05 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:40.983429+0000", 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8099000000000001e-05 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:41.028867+0000", 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045437907999999999 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:41.029016+0000", 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148818 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:171243 9.9 e161/140 9:9c1de09a:::benchmark_data_smithi086_152062_object38761:head v 161'10578, mlcod=161'10574)", 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:57:56.826965+0000", 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.982532663, 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.048573951999999997, 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.817 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.826965+0000", 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.826965+0000", 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.826965+0000", 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.826971+0000", 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0950000000000001e-06 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.826982+0000", 2024-06-14T23:01:25.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0703999999999999e-05 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.826987+0000", 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7429999999999999e-06 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.839966+0000", 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012978658000000001 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.839981+0000", 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5457000000000001e-05 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.875499+0000", 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.035517264 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:56.875539+0000", 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0646999999999999e-05 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.821 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:173125 9.9 e161/140 9:94e094fe:::benchmark_data_smithi086_152062_object40643:head v 161'10705, mlcod=161'10702)", 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:57:59.083632+0000", 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 206.72586602600001, 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.051271118999999997, 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.083632+0000", 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.083632+0000", 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.083631+0000", 2024-06-14T23:01:25.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.083638+0000", 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0360000000000001e-06 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.083659+0000", 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0738e-05 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.083664+0000", 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0490000000000003e-06 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.083876+0000", 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021181699999999999 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.088222+0000", 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043458840000000004 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.134751+0000", 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046528937999999999 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.134903+0000", 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015232599999999999 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.827 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:173454 9.9 e161/140 9:96c86ccc:::benchmark_data_smithi086_152062_object40972:head v 161'10735, mlcod=161'10730)", 2024-06-14T23:01:25.827 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:57:59.490034+0000", 2024-06-14T23:01:25.827 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 206.31946405100001, 2024-06-14T23:01:25.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050372973000000001, 2024-06-14T23:01:25.827 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.827 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.827 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.490034+0000", 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.490034+0000", 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.490033+0000", 2024-06-14T23:01:25.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.490040+0000", 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2810000000000003e-06 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.490053+0000", 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2473000000000001e-05 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.490059+0000", 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0460000000000002e-06 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.506307+0000", 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016248123999999999 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.506362+0000", 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4960000000000002e-05 2024-06-14T23:01:25.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.540357+0000", 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033994675000000002 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.540407+0000", 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0287999999999997e-05 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:173460 9.1 e161/140 9:83e8fb35:::benchmark_data_smithi086_152062_object40978:head v 161'10763, mlcod=161'10760)", 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:57:59.496180+0000", 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 206.31331827, 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.044254052000000002, 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.496180+0000", 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.496180+0000", 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.496179+0000", 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.496189+0000", 2024-06-14T23:01:25.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0644e-05 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.496207+0000", 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8155000000000002e-05 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.496215+0000", 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.011e-06 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.516524+0000", 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020308771999999999 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.516544+0000", 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9649000000000001e-05 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.540411+0000", 2024-06-14T23:01:25.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023867427 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:57:59.540434+0000", 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2606999999999999e-05 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:174174 9.e e161/145 9:7cbe29f7:::benchmark_data_smithi086_152062_object41692:head v 161'10887, mlcod=161'10884)", 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:00.383048+0000", 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 205.426450034, 2024-06-14T23:01:25.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047146953999999998, 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.383048+0000", 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.383048+0000", 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.383047+0000", 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.383052+0000", 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5530000000000003e-06 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.383063+0000", 2024-06-14T23:01:25.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.048e-05 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.383067+0000", 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0759999999999996e-06 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.386496+0000", 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034297680000000001 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.386522+0000", 2024-06-14T23:01:25.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5805000000000001e-05 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.429909+0000", 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.043387296999999998 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.430195+0000", 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000285448 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:174509 9.1 e161/140 9:83768f14:::benchmark_data_smithi086_152062_object42027:head v 161'10844, mlcod=161'10840)", 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:00.749788+0000", 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 205.059709594, 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045778934, 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.749788+0000", 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.749788+0000", 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.749788+0000", 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.749797+0000", 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0059999999999998e-06 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.749807+0000", 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0686000000000001e-05 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.749813+0000", 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1390000000000003e-06 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.760787+0000", 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010974526 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.760806+0000", 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8921000000000001e-05 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.795530+0000", 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034724123000000003 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:00.795567+0000", 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7264000000000001e-05 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:179899 9.5 e162/140 9:a92301e2:::benchmark_data_smithi086_152062_object47417:head v 162'11581, mlcod=162'11576)", 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:06.828152+0000", 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 198.98134599900001, 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046118851000000002, 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.845 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.828152+0000", 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.828152+0000", 2024-06-14T23:01:25.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.828151+0000", 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.828159+0000", 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1380000000000007e-06 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.828171+0000", 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2252000000000001e-05 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.828177+0000", 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6749999999999999e-06 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.842166+0000", 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013988799999999999 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.842214+0000", 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7788999999999999e-05 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.874127+0000", 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031913888000000001 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.874271+0000", 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014336899999999999 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.849 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:179904 9.5 e162/140 9:a7937b89:::benchmark_data_smithi086_152062_object47422:head v 162'11583, mlcod=162'11576)", 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:06.838201+0000", 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 198.97129665200001, 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046956533000000002, 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.838201+0000", 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.838201+0000", 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.838200+0000", 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.838207+0000", 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9330000000000004e-06 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.838219+0000", 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1192999999999999e-05 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.838224+0000", 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6110000000000001e-06 2024-06-14T23:01:25.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.856850+0000", 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018625791999999999 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.856869+0000", 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9241000000000001e-05 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.885026+0000", 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028157056999999999 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:06.885158+0000", 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131586 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:188813 9.7 e165/140 9:ea5168f4:::benchmark_data_smithi086_152062_object56331:head v 165'11765, mlcod=165'11763)", 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.163287+0000", 2024-06-14T23:01:25.854 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 187.646210586, 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.04645875, 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.163287+0000", 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.163287+0000", 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.163287+0000", 2024-06-14T23:01:25.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:25.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.163294+0000", 2024-06-14T23:01:25.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1729999999999998e-06 2024-06-14T23:01:25.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.163308+0000", 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4236000000000001e-05 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.163313+0000", 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3619999999999997e-06 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.163747+0000", 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00043359699999999998 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.163774+0000", 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6529e-05 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.209669+0000", 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045895762999999999 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.209746+0000", 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6804999999999994e-05 2024-06-14T23:01:25.859 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:189338 9.d e165/145 9:bd408c6d:::benchmark_data_smithi086_152062_object56856:head v 165'11887, mlcod=165'11884)", 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.765890+0000", 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 187.04360843500001, 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.102441778, 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.860 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765890+0000", 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765890+0000", 2024-06-14T23:01:25.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765889+0000", 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765895+0000", 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7049999999999996e-06 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765903+0000", 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1899999999999995e-06 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765907+0000", 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8670000000000004e-06 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765942+0000", 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5522999999999999e-05 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765955+0000", 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2515e-05 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.868243+0000", 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.102288822 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.868331+0000", 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7859e-05 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:192539 9.5 e165/140 9:aa0144f6:::benchmark_data_smithi086_152062_object60057:head v 165'12436, mlcod=165'12434)", 2024-06-14T23:01:25.865 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:22.504414+0000", 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.305084099, 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.044139537, 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.504414+0000", 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.504414+0000", 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.504413+0000", 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.504421+0000", 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3479999999999998e-06 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.504430+0000", 2024-06-14T23:01:25.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9080000000000007e-06 2024-06-14T23:01:25.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.504436+0000", 2024-06-14T23:01:25.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4820000000000002e-06 2024-06-14T23:01:25.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.504474+0000", 2024-06-14T23:01:25.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8387999999999999e-05 2024-06-14T23:01:25.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.504507+0000", 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3033e-05 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.548451+0000", 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.043943833000000002 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:22.548553+0000", 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102246 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.871 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:207309 9.9 e168/140 9:91eb3e3e:::benchmark_data_smithi086_152062_object74827:head v 168'12857, mlcod=168'12853)", 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:40.093579+0000", 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 165.715919001, 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.04934798, 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.093579+0000", 2024-06-14T23:01:25.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.093579+0000", 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.093578+0000", 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.093586+0000", 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3220000000000004e-06 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.093596+0000", 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0696999999999999e-05 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.093602+0000", 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2120000000000002e-06 2024-06-14T23:01:25.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.108729+0000", 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015127866 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.108749+0000", 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9296999999999998e-05 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.142781+0000", 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034032449999999999 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.142927+0000", 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000145859 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.876 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:207673 9.7 e168/140 9:e969cca6:::benchmark_data_smithi086_152062_object75191:head v 168'12918, mlcod=168'12914)", 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:40.519965+0000", 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 165.289533256, 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047746117999999997, 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.519965+0000", 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.519965+0000", 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.519964+0000", 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.519971+0000", 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0319999999999998e-06 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.519984+0000", 2024-06-14T23:01:25.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3648e-05 2024-06-14T23:01:25.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.519990+0000", 2024-06-14T23:01:25.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4199999999999998e-06 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.524143+0000", 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041534759999999997 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.524165+0000", 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1959000000000001e-05 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.567663+0000", 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.043498165999999998 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:40.567711+0000", 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7487000000000001e-05 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:223214 9.3 9:c2765e8b:::benchmark_data_smithi086_152062_object90732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:58.031263+0000", 2024-06-14T23:01:25.881 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 147.77823526099999, 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050272850000000001, 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 223214 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.031263+0000", 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.031263+0000", 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.031262+0000", 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.031266+0000", 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.506e-06 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.031276+0000", 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0703000000000001e-05 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.031281+0000", 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1849999999999997e-06 2024-06-14T23:01:25.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.036901+0000", 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056202470000000001 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.037228+0000", 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032728800000000001 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3", 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.037339+0000", 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110956 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.052708+0000", 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015368671 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.059260+0000", 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065518709999999999 2024-06-14T23:01:25.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.081446+0000", 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022186648999999999 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.081495+0000", 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9277000000000003e-05 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.081536+0000", 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0096999999999999e-05 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:223219 9.3 9:c8646127:::benchmark_data_smithi086_152062_object90737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:58.035875+0000", 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 147.773623061, 2024-06-14T23:01:25.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045831269000000001, 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 223219 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.035875+0000", 2024-06-14T23:01:25.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.035875+0000", 2024-06-14T23:01:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:25.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:25.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:25.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.035873+0000", 2024-06-14T23:01:25.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:25.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.035880+0000", 2024-06-14T23:01:25.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0180000000000004e-06 2024-06-14T23:01:25.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.035898+0000", 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7499999999999998e-05 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.035905+0000", 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8390000000000001e-06 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.040671+0000", 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047658750000000001 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.040782+0000", 2024-06-14T23:01:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110938 2024-06-14T23:01:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3", 2024-06-14T23:01:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.040864+0000", 2024-06-14T23:01:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2259e-05 2024-06-14T23:01:25.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.052712+0000", 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011848088999999999 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.068811+0000", 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016099156999999999 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.081576+0000", 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012764967 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.081666+0000", 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9818e-05 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:58.081706+0000", 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0377000000000003e-05 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:229720 9.1 e172/163 9:86a824b7:::benchmark_data_smithi086_152062_object97238:head v 172'14297, mlcod=172'14293)", 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:05.502503+0000", 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 140.30699528100001, 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.048541991, 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.502503+0000", 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.502503+0000", 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.502502+0000", 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:25.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.502507+0000", 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5290000000000002e-06 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.502515+0000", 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1259999999999998e-06 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.502519+0000", 2024-06-14T23:01:25.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2209999999999999e-06 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.510038+0000", 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075190559999999997 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.510125+0000", 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6391999999999995e-05 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.550901+0000", 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.040776537000000002 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:05.551045+0000", 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014360100000000001 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:249821 9.3 9:c4aa2f05:::benchmark_data_smithi086_152062_object117339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:01:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:28.730412+0000", 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 117.07908578200001, 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047338207, 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 249821 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.730412+0000", 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.730412+0000", 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.730411+0000", 2024-06-14T23:01:25.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.730417+0000", 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9359999999999996e-06 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.730430+0000", 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.342e-05 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.730436+0000", 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7339999999999996e-06 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.730484+0000", 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8591000000000003e-05 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.730595+0000", 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011069399999999999 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3", 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.730705+0000", 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109466 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.733675+0000", 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029703989999999999 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.734011+0000", 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000335853 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.777658+0000", 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.043647569999999997 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.777708+0000", 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0009000000000002e-05 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:28.777750+0000", 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2061999999999998e-05 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:252803 9.1 e174/163 9:86961f78:::benchmark_data_smithi086_152062_object120321:head v 174'15654, mlcod=174'15652)", 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:59:32.191148+0000", 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 113.618350405, 2024-06-14T23:01:25.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045300034000000003, 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.191148+0000", 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.191148+0000", 2024-06-14T23:01:25.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.191147+0000", 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.191153+0000", 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9920000000000004e-06 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.191163+0000", 2024-06-14T23:01:25.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8630000000000003e-06 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.191168+0000", 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7849999999999999e-06 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.192491+0000", 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013234550000000001 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.192509+0000", 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8148e-05 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.236361+0000", 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.043852131000000003 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:59:32.236448+0000", 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6286999999999996e-05 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.914 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:22282 10.c 10:3d60334c:::benchmark_data_smithi086_189745_object22281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:00:44.559390+0000", 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 41.250108480000002, 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.049624823999999998, 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22282 2024-06-14T23:01:25.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.559390+0000", 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.559390+0000", 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.559388+0000", 2024-06-14T23:01:25.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.559451+0000", 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2561e-05 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.559459+0000", 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7029999999999995e-06 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.559463+0000", 2024-06-14T23:01:25.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7850000000000002e-06 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.559504+0000", 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0299999999999997e-05 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.559591+0000", 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7453000000000004e-05 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.559669+0000", 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8233999999999997e-05 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.560410+0000", 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00074086699999999998 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.561040+0000", 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00062983699999999995 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.608926+0000", 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047886384999999997 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.608978+0000", 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1978000000000003e-05 2024-06-14T23:01:25.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:00:44.609014+0000", 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6025999999999997e-05 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:01:25.921 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:01:25.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:25.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:25 smithi137 ceph-mon[90634]: pgmap v774: 99 pgs: 99 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-06-14T23:01:25.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4030002431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:26.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:26.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:26.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:25 smithi086 ceph-mon[97913]: pgmap v774: 99 pgs: 99 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-06-14T23:01:26.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4030002431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:26.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:25 smithi086 ceph-mon[102754]: pgmap v774: 99 pgs: 99 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-06-14T23:01:26.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4030002431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:26.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:26.177 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:26.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:26.217 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 87840 87824 112.002 79.0625 0.0198817 0.00892362 2024-06-14T23:01:26.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:26.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:26.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:26.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:26.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:26.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:26.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:26.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:26.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:26.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:26.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:26.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-14T23:01:26.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:26.879 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:26.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:26.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:27.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:27.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:27.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:27.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:27.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:27.220 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 89052 89036 111.27 75.75 0.0237359 0.00898197 2024-06-14T23:01:27.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:27.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:27.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:27.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:27.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:27.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:27.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:27.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:27.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:27.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:27.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:27.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:27.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:27.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:27.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:27.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:27.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:28.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:27 smithi086 ceph-mon[102754]: pgmap v775: 99 pgs: 99 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T23:01:28.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:28.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:27 smithi086 ceph-mon[97913]: pgmap v775: 99 pgs: 99 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T23:01:28.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:28.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:28.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:28.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:28.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:28.220 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 90181 90165 110.472 70.5625 0.00871198 0.00904699 2024-06-14T23:01:28.235 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:27 smithi137 ceph-mon[90634]: pgmap v775: 99 pgs: 99 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T23:01:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:28.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:28.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:28.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:01:28.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:28.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:28.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:28.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:28.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:28.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:28.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:28.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:28.652 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:28.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:28.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:28.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:28.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:28.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:28.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:29.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:29.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:29.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:29.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:29.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:29.217 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:29.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:29.220 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 16 91183 91167 109.552 62.625 0.0164031 0.00912334 2024-06-14T23:01:29.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:29.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:29.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:29.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:29.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:29.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:29.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:29.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:29.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:29.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:29.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:29.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.3 to 1 (10000) 2024-06-14T23:01:29.827 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:29.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:29.901 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:29.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:30.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:30.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:30.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:30.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:29 smithi086 ceph-mon[97913]: pgmap v776: 99 pgs: 99 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T23:01:30.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:29 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/99821607' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:01:30.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:29 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:01:30.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:29 smithi086 ceph-mon[102754]: pgmap v776: 99 pgs: 99 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T23:01:30.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:29 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/99821607' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:01:30.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:29 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:01:30.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:30.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:30.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:30.220 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 16 92373 92357 108.888 74.375 0.0136112 0.00917898 2024-06-14T23:01:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:29 smithi137 ceph-mon[90634]: pgmap v776: 99 pgs: 99 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T23:01:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:29 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/99821607' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:01:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:29 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:01:30.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:30.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:30.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:30.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:30.456 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:30.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:30.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:30.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:30.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:30.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:30.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:30.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:30.814 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:01:30.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:30.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:30.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:30.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:31.002 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:31.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:31.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 4 1 2024-06-14T23:01:31.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:30 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:01:31.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:30 smithi086 ceph-mon[102754]: osdmap e198: 8 total, 8 up, 8 in 2024-06-14T23:01:31.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3951276381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:01:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:30 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:01:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:30 smithi086 ceph-mon[97913]: osdmap e198: 8 total, 8 up, 8 in 2024-06-14T23:01:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3951276381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:01:31.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:31.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:31.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:31.220 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 93605 93589 108.297 77 0.00640218 0.00922927 2024-06-14T23:01:31.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:30 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:01:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:30 smithi137 ceph-mon[90634]: osdmap e198: 8 total, 8 up, 8 in 2024-06-14T23:01:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3951276381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:01:31.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:31.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:31.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:31.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:31.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:31.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:31.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:31.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:31.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:31.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:31.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:31.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:31.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:31.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:31.958 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:31.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:32.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:32.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[102754]: pgmap v778: 99 pgs: 99 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-14T23:01:32.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:01:32.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:01:32.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3951276381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:01:32.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[102754]: osdmap e199: 8 total, 8 up, 8 in 2024-06-14T23:01:32.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[97913]: pgmap v778: 99 pgs: 99 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-14T23:01:32.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:01:32.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:01:32.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3951276381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:01:32.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:31 smithi086 ceph-mon[97913]: osdmap e199: 8 total, 8 up, 8 in 2024-06-14T23:01:32.093 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:32.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:32.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:32.220 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 94816 94800 107.704 75.6875 0.0129764 0.00927993 2024-06-14T23:01:32.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:32.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:32.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:31 smithi137 ceph-mon[90634]: pgmap v778: 99 pgs: 99 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-14T23:01:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:31 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:01:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:31 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:01:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3951276381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:01:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:31 smithi137 ceph-mon[90634]: osdmap e199: 8 total, 8 up, 8 in 2024-06-14T23:01:32.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:32.360 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:32.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:32.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:32.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:32.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:32.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:32.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:32.723 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:01:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:01:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.45821.0:95351 10.1 e199/186 10:8200ac79:::benchmark_data_smithi086_189745_object95350:head v 199'6041, mlcod=199'6039)", 2024-06-14T23:01:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:32.661010+0000", 2024-06-14T23:01:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0077450369999999998, 2024-06-14T23:01:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0077825669999999998, 2024-06-14T23:01:32.723 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.661010+0000", 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.661010+0000", 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.661009+0000", 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.661042+0000", 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3290000000000001e-05 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.661054+0000", 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1571e-05 2024-06-14T23:01:32.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.661060+0000", 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3520000000000003e-06 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.661143+0000", 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2516999999999996e-05 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.661161+0000", 2024-06-14T23:01:32.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7978e-05 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.45821.0:95353 10.5 e199/186 10:a1c6ed5c:::benchmark_data_smithi086_189745_object95352:head v 199'6068, mlcod=199'6066)", 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:32.664156+0000", 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0045991180000000001, 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046964859999999997, 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:32.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664156+0000", 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664156+0000", 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664155+0000", 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664178+0000", 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3098000000000001e-05 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664190+0000", 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2318e-05 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664197+0000", 2024-06-14T23:01:32.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7399999999999998e-06 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664378+0000", 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018116000000000001 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664394+0000", 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6345e-05 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.730 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:95355 10.4 10:27169c45:::benchmark_data_smithi086_189745_object95354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:32.664654+0000", 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0041009419999999998, 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042574680000000004, 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 95355 2024-06-14T23:01:32.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664654+0000", 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664654+0000", 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664652+0000", 2024-06-14T23:01:32.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664675+0000", 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3020999999999999e-05 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664689+0000", 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3540000000000001e-05 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:32.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664696+0000", 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0999999999999998e-06 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664822+0000", 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000125621 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:32.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.664994+0000", 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017211200000000001 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.665132+0000", 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000137763 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.45821.0:95356 10.1 e199/186 10:8c30e478:::benchmark_data_smithi086_189745_object95355:head v 199'6042, mlcod=199'6039)", 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:32.665280+0000", 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0034746820000000002, 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037043610000000002, 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.665280+0000", 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.665280+0000", 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.665280+0000", 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.665337+0000", 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6950999999999997e-05 2024-06-14T23:01:32.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.665350+0000", 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3694e-05 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.665357+0000", 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3260000000000001e-06 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:32.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.667430+0000", 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020732620000000002 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.667449+0000", 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8760999999999999e-05 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:95358 10.4 10:29bc1cda:::benchmark_data_smithi086_189745_object95357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-14T23:01:32.739 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:32.666239+0000", 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.002515757, 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028018840000000001, 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 95358 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666239+0000", 2024-06-14T23:01:32.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666239+0000", 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666238+0000", 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666256+0000", 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8196999999999999e-05 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666267+0000", 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0365e-05 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666275+0000", 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7659999999999998e-06 2024-06-14T23:01:32.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.668694+0000", 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024193610000000001 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.668825+0000", 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000131202 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:01:32.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.668958+0000", 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013257500000000001 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.45821.0:95357 10.6 e199/186 10:6549df4f:::benchmark_data_smithi086_189745_object95356:head v 199'5901, mlcod=199'5899)", 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:32.666664+0000", 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0020910579999999998, 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024396259999999999, 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:32.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666664+0000", 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666664+0000", 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666656+0000", 2024-06-14T23:01:32.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999919 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666703+0000", 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7812000000000002e-05 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666719+0000", 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5191000000000001e-05 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:32.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666725+0000", 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9789999999999998e-06 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666775+0000", 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.003e-05 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:32.666793+0000", 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8437999999999999e-05 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:32.747 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-06-14T23:01:32.748 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 6 2024-06-14T23:01:32.748 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:01:32.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:32.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:32.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:32.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:32.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:32.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:33.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:33.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:33.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:33.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:33.194 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:33.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:33.221 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 96031 96015 107.137 75.9375 0.014209 0.00932869 2024-06-14T23:01:33.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:33.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:33.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:33.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:33.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:33.467 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:33.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:33.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:33.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:33.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:33.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:33.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:33.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:33.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:33.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:33.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:33.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:34.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:34.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:34.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:33 smithi086 ceph-mon[97913]: pgmap v780: 99 pgs: 99 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-06-14T23:01:34.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:33 smithi086 ceph-mon[102754]: pgmap v780: 99 pgs: 99 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-06-14T23:01:34.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:34.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:34.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:34.221 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 97237 97221 106.58 75.375 0.00763801 0.00937804 2024-06-14T23:01:34.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:33 smithi137 ceph-mon[90634]: pgmap v780: 99 pgs: 99 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 60 MiB/s wr, 966 op/s 2024-06-14T23:01:34.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:34.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:34.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:34.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:34.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:34.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:34.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:34.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:34.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:34.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-14T23:01:34.657 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:34.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:34.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:34.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:34.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:34.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:34.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:34.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:34.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:35.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:35.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:35.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:35.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:35.221 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 16 98402 98386 105.998 72.8125 0.00814537 0.00942955 2024-06-14T23:01:35.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:35.281 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:01:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:01:34] "GET /metrics HTTP/1.1" 200 36589 "" "Prometheus/2.43.0" 2024-06-14T23:01:35.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:35.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:35.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:35.425 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:35.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:35.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:01:35.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:35.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:35.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:35.650 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:01:35.651 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:01:35.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:35.684 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:01:35.684 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:01:35.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:35.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:35.716 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:01:35.716 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:01:35.751 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:01:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:01:35.783 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:01:35.784 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:01:35.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:35.820 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:01:35.820 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:01:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:35.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:35.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:35.854 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:01:35.854 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:01:35.884 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:01:35.885 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:01:35.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:35.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:35.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:36.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:36.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:35 smithi086 ceph-mon[97913]: pgmap v781: 99 pgs: 99 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T23:01:36.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2185087739' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:36.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:35 smithi086 ceph-mon[102754]: pgmap v781: 99 pgs: 99 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T23:01:36.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2185087739' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:36.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:36.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:36.222 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 99452 99436 105.313 65.625 0.0202314 0.00949105 2024-06-14T23:01:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:35 smithi137 ceph-mon[90634]: pgmap v781: 99 pgs: 99 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T23:01:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2185087739' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:36.253 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:36.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:36.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:36.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:36.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:36.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:36.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:36.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:36.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:36.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:36.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:36.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:36.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:36.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:36.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:36.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:36.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:36.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:37.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:37.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:37.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:37.221 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:01:37.221432+0000 min lat: 0.00119713 max lat: 0.179812 avg lat: 0.00954283 2024-06-14T23:01:37.221 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:01:37.221 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 100587 100571 104.74 70.9375 0.00855502 0.00954283 2024-06-14T23:01:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:37.226 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:37.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:37.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:37.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:37.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:37.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:37.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:38.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:38.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:38.289 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:37 smithi137 ceph-mon[90634]: pgmap v782: 99 pgs: 99 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-14T23:01:38.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:37 smithi086 ceph-mon[97913]: pgmap v782: 99 pgs: 99 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-14T23:01:38.289 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:37 smithi086 ceph-mon[102754]: pgmap v782: 99 pgs: 99 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-14T23:01:38.290 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 101770 101754 104.235 73.9375 0.00882323 0.00958803 2024-06-14T23:01:38.290 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:38.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:38.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:38.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:38.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:38.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:38.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:38.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:38.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:38.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:38.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:38.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:38.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:38.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:38.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:38.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:39.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:39.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:39.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:39.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:39.221 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 102898 102882 103.691 70.5 0.00968022 0.0096391 2024-06-14T23:01:39.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:39.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:39.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:39.381 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:39.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:39.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:39.511 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:39.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:39.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:39.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:39.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:39.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:39.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:39.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:39.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:39.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:39.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:39.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:40.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:40.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:40.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:40.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:39 smithi086 ceph-mon[102754]: pgmap v783: 99 pgs: 99 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 55 MiB/s wr, 877 op/s 2024-06-14T23:01:40.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:39 smithi086 ceph-mon[97913]: pgmap v783: 99 pgs: 99 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 55 MiB/s wr, 877 op/s 2024-06-14T23:01:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:40.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:40.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:40.221 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 104006 103990 103.144 69.25 0.00914983 0.00969032 2024-06-14T23:01:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:40.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:40.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:39 smithi137 ceph-mon[90634]: pgmap v783: 99 pgs: 99 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 55 MiB/s wr, 877 op/s 2024-06-14T23:01:40.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:01:40.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:40.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:40.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:40.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:40.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:40.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:40.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:40.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:40.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:40.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:40.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:40.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:40.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:40.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:40.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:41.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:41.101 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:41.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:41.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:41.222 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 105013 104997 102.516 62.9375 0.0248906 0.00974961 2024-06-14T23:01:41.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:41.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:41.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:41.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:41.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:41.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:41.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:41.510 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:41.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:41.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:41.649 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:41.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:41.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:41.779 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:41.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:41.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:41.918 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:41.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:42.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:42.053 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:42.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:42.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:42.185 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:42.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:42.201 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:41 smithi086 ceph-mon[102754]: pgmap v784: 99 pgs: 99 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-14T23:01:42.201 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:41 smithi086 ceph-mon[97913]: pgmap v784: 99 pgs: 99 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-14T23:01:42.202 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:41 smithi137 ceph-mon[90634]: pgmap v784: 99 pgs: 99 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-14T23:01:42.222 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 105994 105978 101.882 61.3125 0.0100134 0.00981007 2024-06-14T23:01:42.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:42.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:42.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:42.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:42.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:42.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:42.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:42.603 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:42.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:42.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:42.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:42.742 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:42.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:42.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:42.882 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:42.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:42.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:43.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:43.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:43.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:43.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:43.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:43.222 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 107016 107000 101.306 63.875 0.0151118 0.00986625 2024-06-14T23:01:43.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:43.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:43.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:43.382 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.4 to 1 (10000) 2024-06-14T23:01:43.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:43.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:43.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:43.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:43.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:43.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:43.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:43.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:43.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:43 smithi086 ceph-mon[97913]: pgmap v785: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-14T23:01:43.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2333030517' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:01:43.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:43.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:43.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:43 smithi086 ceph-mon[102754]: pgmap v785: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-14T23:01:43.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2333030517' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:01:43.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:43.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:43.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:43.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:43.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:43.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:43 smithi137 ceph-mon[90634]: pgmap v785: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-14T23:01:43.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2333030517' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:01:43.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:43.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:43.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:43.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:43.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:43.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:44.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:44.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:44.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:44.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:44.222 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 108123 108107 100.826 69.1875 0.0144131 0.00991294 2024-06-14T23:01:44.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:44.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:44.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:44.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:44.372 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:44.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:44.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:44.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:44.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:44.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:44.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:44.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:44.686 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2333030517' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:01:44.686 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:44 smithi086 ceph-mon[102754]: osdmap e200: 8 total, 8 up, 8 in 2024-06-14T23:01:44.686 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2333030517' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:01:44.686 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:44 smithi086 ceph-mon[97913]: osdmap e200: 8 total, 8 up, 8 in 2024-06-14T23:01:44.733 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:01:44.733 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:01:44.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2333030517' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:01:44.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:44 smithi137 ceph-mon[90634]: osdmap e200: 8 total, 8 up, 8 in 2024-06-14T23:01:44.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:44.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:44.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:44.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:44.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:44.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:45.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:45.038 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:01:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:01:44] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.43.0" 2024-06-14T23:01:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:45.062 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:45.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:45.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:45.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:45.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:45.222 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 109177 109161 100.312 65.875 0.00669753 0.00996388 2024-06-14T23:01:45.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:45.347 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:45.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:45.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:45.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 5 1 2024-06-14T23:01:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:45.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:45.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:45.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:45.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:45.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:45.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:45.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:45 smithi137 ceph-mon[90634]: pgmap v787: 99 pgs: 99 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-06-14T23:01:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:45.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:45.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:45 smithi086 ceph-mon[97913]: pgmap v787: 99 pgs: 99 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-06-14T23:01:45.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:45 smithi086 ceph-mon[102754]: pgmap v787: 99 pgs: 99 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-06-14T23:01:45.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:45.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:45.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:45.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:46.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:46.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:46.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:46.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:46.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:46.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:46.222 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 110219 110203 99.8024 65.125 0.0251533 0.0100154 2024-06-14T23:01:46.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:46.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:46.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:46.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:46.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:46.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:46.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:46.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:46.575 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:46.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:46.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:46.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:46.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-14T23:01:46.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:46.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:46.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:46.967 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:47.013 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:47.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:47.093 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:47.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:47.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:47.223 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 111195 111179 99.2467 61 0.00724054 0.0100705 2024-06-14T23:01:47.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:47.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:47.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:47.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:47.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:47.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:47.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:47.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:47.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:47.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:47.665 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:47.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:47.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:47.780 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:01:47.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:47.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:47.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:47.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:47.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:48.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:48.073 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:48.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:48.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:47 smithi086 ceph-mon[97913]: pgmap v788: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-14T23:01:48.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4292345395' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:01:48.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:47 smithi086 ceph-mon[102754]: pgmap v788: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-14T23:01:48.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4292345395' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:01:48.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:48.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:48.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:48.223 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 112254 112238 98.781 66.1875 0.0307019 0.0101183 2024-06-14T23:01:48.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:47 smithi137 ceph-mon[90634]: pgmap v788: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-14T23:01:48.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4292345395' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:01:48.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:48.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:48.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:48.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:48.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:48.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:48.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:48.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:48.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:48.754 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:48.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:48.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:48.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:48.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:48.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:49.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:49.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:49.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:48 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:01:49.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4292345395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:01:49.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:48 smithi086 ceph-mon[97913]: osdmap e201: 8 total, 8 up, 8 in 2024-06-14T23:01:49.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:48 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:01:49.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4292345395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:01:49.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:48 smithi086 ceph-mon[102754]: osdmap e201: 8 total, 8 up, 8 in 2024-06-14T23:01:49.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:49.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:49.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:49.224 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 113353 113337 98.363 68.6875 0.0111232 0.0101617 2024-06-14T23:01:49.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:48 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:01:49.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4292345395' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:01:49.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:48 smithi137 ceph-mon[90634]: osdmap e201: 8 total, 8 up, 8 in 2024-06-14T23:01:49.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:49.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:49.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:49.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:49.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:49.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:49.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:49.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:49.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:49.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:49.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:49.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:49.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:49.868 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:49.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:49.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:50.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:50.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:50.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:49 smithi086 ceph-mon[97913]: pgmap v790: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 733 op/s 2024-06-14T23:01:50.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:49 smithi086 ceph-mon[102754]: pgmap v790: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 733 op/s 2024-06-14T23:01:50.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:50.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:50.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:50.223 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 114425 114409 97.9334 67 0.0134839 0.0102066 2024-06-14T23:01:50.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:49 smithi137 ceph-mon[90634]: pgmap v790: 99 pgs: 99 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 733 op/s 2024-06-14T23:01:50.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:50.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:50.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:50.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:50.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:50.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:50.471 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:50.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:50.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:50.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:50.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:50.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:50.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:50.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:50.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:50.858 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:50.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:50.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:50.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:50.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:51.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:51.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:51.224 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 16 115464 115448 97.4874 64.9375 0.0131115 0.0102528 2024-06-14T23:01:51.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:51.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:51.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:51.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:51.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:51.413 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:51.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:51.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:51.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:51.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:51.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:51.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:51.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:51.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:01:51.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:51.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:51.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:51.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:51.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:51.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:52.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:51 smithi086 ceph-mon[97913]: pgmap v791: 99 pgs: 99 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T23:01:52.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4034828917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:52.032 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:51 smithi086 ceph-mon[102754]: pgmap v791: 99 pgs: 99 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T23:01:52.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4034828917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:52.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:52.078 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:52.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:52.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:52.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:52.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:52.224 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 116441 116425 97.0012 61.0625 0.0124928 0.0103042 2024-06-14T23:01:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:51 smithi137 ceph-mon[90634]: pgmap v791: 99 pgs: 99 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T23:01:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4034828917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:01:52.254 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:01:52.254 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:01:52.254 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:01:52.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:01:52.256 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:01:52.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:52.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:52.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:52.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:52.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:52.506 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:52.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:52.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:52.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:52.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:52.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:52.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:52.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:52.791 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.5 to 1 (10000) 2024-06-14T23:01:52.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/47290761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:01:52.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:52 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:01:52.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/47290761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:01:52.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:52 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:01:52.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:52.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:52.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:53.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:53.054 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:53.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:53.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:53.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:53.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:53.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:53.224 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 117363 117347 96.4831 57.625 0.00705184 0.0103597 2024-06-14T23:01:53.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/47290761' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:01:53.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:52 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:01:53.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:53.321 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:53.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:53.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:53.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:53.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:53.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:53.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:53.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:53.739 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:53.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:53.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:53.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:53.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:53.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:53.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:54.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:53 smithi086 ceph-mon[97913]: pgmap v792: 99 pgs: 99 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:01:54.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:53 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:01:54.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:53 smithi086 ceph-mon[97913]: osdmap e202: 8 total, 8 up, 8 in 2024-06-14T23:01:54.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:53 smithi086 ceph-mon[102754]: pgmap v792: 99 pgs: 99 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:01:54.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:53 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:01:54.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:53 smithi086 ceph-mon[102754]: osdmap e202: 8 total, 8 up, 8 in 2024-06-14T23:01:54.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:54.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:54.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:54.224 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 16 118398 118382 96.0702 64.6875 0.00896563 0.0104039 2024-06-14T23:01:54.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:54.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:53 smithi137 ceph-mon[90634]: pgmap v792: 99 pgs: 99 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:01:54.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:53 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:01:54.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:53 smithi137 ceph-mon[90634]: osdmap e202: 8 total, 8 up, 8 in 2024-06-14T23:01:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:54.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:54.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:54.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 6 1 2024-06-14T23:01:54.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:54.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:54.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:54.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:54.624 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:54.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:54.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:54.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:54.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:54.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:54.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:54.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:54.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:55.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:55.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83971 9.9 9:99f5752d:::benchmark_data_smithi086_152062_object83970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.088245+0000", 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.44539216499999, 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.169417085, 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83971 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088245+0000", 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088245+0000", 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088241+0000", 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999967 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088265+0000", 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4185999999999998e-05 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088276+0000", 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0818999999999999e-05 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088282+0000", 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2559999999999999e-06 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088318+0000", 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5987e-05 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088428+0000", 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109157 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.088618+0000", 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000190936 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.095042+0000", 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064230329999999999 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105426+0000", 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010384167999999999 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257565+0000", 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15213905999999999 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257619+0000", 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4005999999999998e-05 2024-06-14T23:01:55.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257662+0000", 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2976000000000003e-05 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83974 9.9 9:99e21f48:::benchmark_data_smithi086_152062_object83973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.089191+0000", 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.44444536200001, 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.16863312799999999, 2024-06-14T23:01:55.089 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83974 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089191+0000", 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089191+0000", 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089191+0000", 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089220+0000", 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9252999999999999e-05 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089230+0000", 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8130000000000007e-06 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.089236+0000", 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.268e-06 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.091590+0000", 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023543259999999999 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.091768+0000", 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017736499999999999 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.091883+0000", 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011573099999999999 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.095048+0000", 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031641579999999998 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105462+0000", 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010414526 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257737+0000", 2024-06-14T23:01:55.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.152274889 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257791+0000", 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4196000000000001e-05 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257825+0000", 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3441000000000001e-05 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.095 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83983 9.b 9:d50ba4e2:::benchmark_data_smithi086_152062_object83982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.097967+0000", 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.43566968200003, 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.159703397, 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83983 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.096 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.097967+0000", 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.097967+0000", 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.097966+0000", 2024-06-14T23:01:55.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.097987+0000", 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1386000000000002e-05 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098000+0000", 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2683999999999999e-05 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098006+0000", 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4840000000000001e-06 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.099132+0000", 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011255939999999999 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.099231+0000", 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9570999999999999e-05 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:01:55.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.099318+0000", 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6347999999999994e-05 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105456+0000", 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061385839999999999 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105470+0000", 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4134999999999999e-05 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257539+0000", 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.152069074 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257635+0000", 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5579000000000001e-05 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257671+0000", 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5451000000000003e-05 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.101 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83985 9.b 9:d28a972b:::benchmark_data_smithi086_152062_object83984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.098330+0000", 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.43530711800003, 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15945879099999999, 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.102 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83985 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098330+0000", 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098330+0000", 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098329+0000", 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098346+0000", 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6656000000000001e-05 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098351+0000", 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6629999999999998e-06 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.098355+0000", 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5190000000000001e-06 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102075+0000", 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037196130000000001 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102215+0000", 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014068300000000001 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102357+0000", 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014141199999999999 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105599+0000", 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0032428119999999999 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108779+0000", 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031796620000000002 2024-06-14T23:01:55.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257712+0000", 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14893310400000001 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257759+0000", 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6999999999999997e-05 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257788+0000", 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9207999999999999e-05 2024-06-14T23:01:55.107 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83989 9.9 9:9deded04:::benchmark_data_smithi086_152062_object83988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.102222+0000", 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.43141435799998, 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15581730699999999, 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83989 2024-06-14T23:01:55.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102222+0000", 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102222+0000", 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102221+0000", 2024-06-14T23:01:55.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102245+0000", 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4485e-05 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102259+0000", 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4005999999999999e-05 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102268+0000", 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2279999999999998e-06 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102421+0000", 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015389099999999999 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102550+0000", 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000128279 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.102676+0000", 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000125964 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.105594+0000", 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002918258 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257705+0000", 2024-06-14T23:01:55.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15211116299999999 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257913+0000", 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000208216 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257990+0000", 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6217999999999994e-05 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258040+0000", 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0238000000000002e-05 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83994 9.9 9:92cd671a:::benchmark_data_smithi086_152062_object83993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108378+0000", 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.42525842600003, 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14977987000000001, 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.114 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83994 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108378+0000", 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108378+0000", 2024-06-14T23:01:55.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108377+0000", 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108399+0000", 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2325999999999999e-05 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108411+0000", 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1615e-05 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108417+0000", 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0340000000000002e-06 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108456+0000", 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8645e-05 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108609+0000", 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015293699999999999 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108775+0000", 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016661099999999999 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257869+0000", 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14909354699999999 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257878+0000", 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7390000000000005e-06 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258082+0000", 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000204184 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258125+0000", 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2861999999999997e-05 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258158+0000", 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3636000000000001e-05 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83995 9.9 9:91dc56da:::benchmark_data_smithi086_152062_object83994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108431+0000", 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.42520625200001, 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15774317700000001, 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.120 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83995 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108431+0000", 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108431+0000", 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108430+0000", 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108449+0000", 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8578000000000001e-05 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108455+0000", 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3119999999999999e-06 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108460+0000", 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6999999999999999e-06 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254637+0000", 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.146177636 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254830+0000", 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000192404 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254981+0000", 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015106399999999999 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.262344+0000", 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073629960000000001 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.262359+0000", 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4986e-05 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.266048+0000", 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036887959999999998 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.266134+0000", 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6101999999999997e-05 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.266174+0000", 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0074000000000003e-05 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83971 9.9 e150/140 ondisk, result = 0)", 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108484+0000", 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.42515263600001, 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.149187404, 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108484+0000", 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108484+0000", 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108484+0000", 2024-06-14T23:01:55.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108489+0000", 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.575e-06 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108502+0000", 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2952999999999999e-05 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108506+0000", 2024-06-14T23:01:55.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0640000000000004e-06 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257550+0000", 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14904347300000001 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257558+0000", 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9370000000000004e-06 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257672+0000", 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114052 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83974 9.9 e150/140 ondisk, result = 0)", 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108513+0000", 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.425123864, 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.149319274, 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108513+0000", 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108513+0000", 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108513+0000", 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108514+0000", 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7129999999999999e-06 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108518+0000", 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1350000000000001e-06 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108520+0000", 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2809999999999998e-06 2024-06-14T23:01:55.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257726+0000", 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14920634999999999 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257732+0000", 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7590000000000003e-06 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257832+0000", 2024-06-14T23:01:55.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100278 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83989 9.9 e150/140 ondisk, result = 0)", 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108621+0000", 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.42501529399999, 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14942707399999999, 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108621+0000", 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108621+0000", 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108621+0000", 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108625+0000", 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3640000000000001e-06 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108634+0000", 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5699999999999993e-06 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108639+0000", 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2469999999999999e-06 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257903+0000", 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14926384300000001 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257909+0000", 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.913e-06 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258049+0000", 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000139808 2024-06-14T23:01:55.137 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83997 9.b 9:dc8147a4:::benchmark_data_smithi086_152062_object83996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108697+0000", 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.42493970800001, 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15510373, 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83997 2024-06-14T23:01:55.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108697+0000", 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108697+0000", 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108696+0000", 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108716+0000", 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9347e-05 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108726+0000", 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0271999999999999e-05 2024-06-14T23:01:55.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108734+0000", 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7020000000000001e-06 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108882+0000", 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014828399999999999 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109035+0000", 2024-06-14T23:01:55.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015296 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251864+0000", 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14282935599999999 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257801+0000", 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059364630000000003 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257872+0000", 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0970999999999995e-05 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.263662+0000", 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057900950000000003 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.263762+0000", 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100052 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.263801+0000", 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.909e-05 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83989 9.9 e150/140 ondisk, result = 0)", 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108796+0000", 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.42484099199999, 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14891169800000001, 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108796+0000", 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108796+0000", 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108795+0000", 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108828+0000", 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2316e-05 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108837+0000", 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.217e-06 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108843+0000", 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8839999999999999e-06 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257693+0000", 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14885061399999999 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257700+0000", 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5259999999999999e-06 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257707+0000", 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7069999999999992e-06 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.147 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83998 9.b 9:dbbacc67:::benchmark_data_smithi086_152062_object83997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.108876+0000", 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.42476065900001, 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15535863599999999, 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83998 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.148 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108876+0000", 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108876+0000", 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108875+0000", 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108895+0000", 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9137e-05 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108904+0000", 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7950000000000002e-06 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.108910+0000", 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6359999999999999e-06 2024-06-14T23:01:55.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254780+0000", 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.145869586 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.254884+0000", 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104069 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.255046+0000", 2024-06-14T23:01:55.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016279700000000001 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257805+0000", 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027586429999999999 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258788+0000", 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0009826940000000001 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.263865+0000", 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0050769819999999998 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.264197+0000", 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000332434 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.264235+0000", 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7564999999999997e-05 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:83999 9.9 9:9a69a847:::benchmark_data_smithi086_152062_object83998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.109179+0000", 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.42445784799997, 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.15805195, 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 83999 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.154 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109179+0000", 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109179+0000", 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109178+0000", 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.109200+0000", 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1909e-05 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251700+0000", 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14249978799999999 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251713+0000", 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3613e-05 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.258720+0000", 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070072509999999999 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.259074+0000", 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000353815 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.259462+0000", 2024-06-14T23:01:55.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00038739500000000001 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.262363+0000", 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029017909999999999 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.266703+0000", 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043395769999999998 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.267058+0000", 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035485300000000003 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.267188+0000", 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013003099999999999 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.267231+0000", 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2994999999999997e-05 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.159 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.35580.0:83983 9.b e150/140 ondisk, result = 0)", 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:56:29.110471+0000", 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 325.42316586499999, 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.147209854, 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.110471+0000", 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.110471+0000", 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.110470+0000", 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.110475+0000", 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9019999999999998e-06 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251660+0000", 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14118467000000001 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.251668+0000", 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5520000000000005e-06 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257523+0000", 2024-06-14T23:01:55.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058546839999999998 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257533+0000", 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0125e-05 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:56:29.257681+0000", 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147687 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:189328 9.b 9:ded1c998:::benchmark_data_smithi086_152062_object56846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:58:18.751601+0000", 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 215.78203600800001, 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14561811199999999, 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 189328 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751601+0000", 2024-06-14T23:01:55.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751601+0000", 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751599+0000", 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751606+0000", 2024-06-14T23:01:55.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.297e-06 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751620+0000", 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4623e-05 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751628+0000", 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0949999999999996e-06 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751707+0000", 2024-06-14T23:01:55.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8498000000000006e-05 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751886+0000", 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017868199999999999 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.751997+0000", 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011175699999999999 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.757120+0000", 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005122817 2024-06-14T23:01:55.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.757780+0000", 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00065953400000000003 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897069+0000", 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13928911799999999 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897155+0000", 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6499e-05 2024-06-14T23:01:55.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897219+0000", 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3600000000000001e-05 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:189340 9.b 9:d0a0718a:::benchmark_data_smithi086_152062_object56858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:58:18.765759+0000", 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 215.767877457, 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13161884700000001, 2024-06-14T23:01:55.169 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 189340 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765759+0000", 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765759+0000", 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765758+0000", 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765764+0000", 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6969999999999998e-06 2024-06-14T23:01:55.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765775+0000", 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1484000000000001e-05 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765782+0000", 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.905e-06 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765830+0000", 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7967999999999999e-05 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.765967+0000", 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013704799999999999 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.766092+0000", 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012515099999999999 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.774208+0000", 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0081152389999999998 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.774256+0000", 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8254000000000003e-05 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897253+0000", 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.122996893 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897342+0000", 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9734000000000005e-05 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:58:18.897378+0000", 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5734e-05 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.175 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:257289 9.b 9:d2511226:::benchmark_data_smithi086_152062_object124807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:37.247644+0000", 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 137.28599256699999, 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.065775379999999994, 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 257289 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247644+0000", 2024-06-14T23:01:55.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247644+0000", 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247643+0000", 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247649+0000", 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7230000000000001e-06 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247661+0000", 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2775999999999999e-05 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247668+0000", 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3520000000000003e-06 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.248312+0000", 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00064445300000000002 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.248425+0000", 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011311099999999999 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.248539+0000", 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114169 2024-06-14T23:01:55.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.293262+0000", 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044722164000000002 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.293294+0000", 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2137000000000001e-05 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.313132+0000", 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019837995000000001 2024-06-14T23:01:55.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.313299+0000", 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016760599999999999 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.313420+0000", 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012020500000000001 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:257292 9.b 9:d93a6c57:::benchmark_data_smithi086_152062_object124810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:37.247673+0000", 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 137.28596366400001, 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.066030509000000001, 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 257292 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.182 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247673+0000", 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247673+0000", 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247673+0000", 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247675+0000", 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5739999999999998e-06 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247680+0000", 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1649999999999997e-06 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247685+0000", 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2590000000000002e-06 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.286355+0000", 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038670537999999997 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.286548+0000", 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019231700000000001 2024-06-14T23:01:55.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.286668+0000", 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120241 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.293323+0000", 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066555130000000001 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.312938+0000", 2024-06-14T23:01:55.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019614392000000001 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.313441+0000", 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00050290600000000001 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.313590+0000", 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014919299999999999 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.313704+0000", 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000113863 2024-06-14T23:01:55.187 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:117256 10.4 10:28d7c4fe:::benchmark_data_smithi086_189745_object117255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:53.050978+0000", 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.4826586070000001, 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.088860252000000001, 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:01:55.188 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117256 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.050978+0000", 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.050978+0000", 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.050977+0000", 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.050995+0000", 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8261e-05 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.051006+0000", 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1347e-05 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.051012+0000", 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4299999999999997e-06 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.051050+0000", 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7984000000000001e-05 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.051155+0000", 2024-06-14T23:01:55.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000105505 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.051250+0000", 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5192000000000006e-05 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.054347+0000", 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003096915 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.054869+0000", 2024-06-14T23:01:55.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00052167200000000004 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.139772+0000", 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.084903159000000006 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.139807+0000", 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4415999999999997e-05 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.139838+0000", 2024-06-14T23:01:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1794000000000001e-05 2024-06-14T23:01:55.194 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.194 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.194 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.194 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:01:55.194 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:01:55.194 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:01:55.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:55.224 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 119276 119260 95.542 54.875 0.0106815 0.0104611 2024-06-14T23:01:55.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:55.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:55.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:55.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:01:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:01:54] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.43.0" 2024-06-14T23:01:55.369 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:55.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:55.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:55.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:55.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:55.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:55.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:55.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:55.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:55.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:55.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:55.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:55.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:56.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:56.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:56.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:56.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:56.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:55 smithi086 ceph-mon[102754]: pgmap v794: 99 pgs: 99 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-14T23:01:56.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:55 smithi086 ceph-mon[97913]: pgmap v794: 99 pgs: 99 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-14T23:01:56.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:56.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:56.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:56.224 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 120205 120189 95.0676 58.0625 0.0167126 0.0105142 2024-06-14T23:01:56.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:55 smithi137 ceph-mon[90634]: pgmap v794: 99 pgs: 99 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-14T23:01:56.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-14T23:01:56.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:56.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:56.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:56.350 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:56.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:56.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:56.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:56.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:56.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:56.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:56.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:56.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:56.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:56.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:56.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:56.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:01:56.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:57.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:57.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:57.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:57.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:57.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:57.225 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:01:57.225117+0000 min lat: 0.00119713 max lat: 0.179812 avg lat: 0.0105649 2024-06-14T23:01:57.225 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:01:57.225 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 121132 121116 94.6032 57.9375 0.0196474 0.0105649 2024-06-14T23:01:57.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:01:57.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:57.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:01:57.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:57.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:57.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:57.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:57.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:57.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:57.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:57.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:57.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:57.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:57.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:57.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:57.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:57.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:57.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:58.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:57 smithi086 ceph-mon[102754]: pgmap v795: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-14T23:01:58.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:58.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:58.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:57 smithi086 ceph-mon[97913]: pgmap v795: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-14T23:01:58.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:58.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:58.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:58.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:58.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:58.225 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 122142 122126 94.2146 63.125 0.00543011 0.0106092 2024-06-14T23:01:58.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:57 smithi137 ceph-mon[90634]: pgmap v795: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-14T23:01:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:01:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:01:58.280 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:58.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:58.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:01:58.425 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:58.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:01:58.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:58.563 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:58.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:58.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:58.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:58.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:58.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:58.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:58.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:58.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:58.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:01:58.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:58.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:01:59.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:59.121 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:01:59.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:59.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:59.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:59.226 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 123130 123114 93.817 61.75 0.0143787 0.0106542 2024-06-14T23:01:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:59.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:59.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:59.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:01:59.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:59.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:01:59.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:01:59.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:59.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:01:59.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:01:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:01:59.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:01:59.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:01:59.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:59.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:01:59.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:01:59.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:01:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:01:59.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:01:59.994 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:01:59 smithi086 ceph-mon[97913]: pgmap v796: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-14T23:01:59.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:01:59 smithi086 ceph-mon[102754]: pgmap v796: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-14T23:02:00.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:00.090 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:00.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:00.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:00.226 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 16 124120 124104 93.4321 61.875 0.0546016 0.0106969 2024-06-14T23:02:00.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:00.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:00.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:01:59 smithi137 ceph-mon[90634]: pgmap v796: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-14T23:02:00.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:00.361 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:00.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:00.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:00.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:00.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:00.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:00.632 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:00.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:00.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:00.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:00.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:00.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:00.855 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.6 to 1 (10000) 2024-06-14T23:02:00.880 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4130115987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:02:00.880 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:00 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:02:00.880 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4130115987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:02:00.880 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:00 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:02:00.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:00.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:00.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:01.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:01.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:01.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:01.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:01.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:01.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:01.227 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 125060 125044 93.0192 58.75 0.0109816 0.0107457 2024-06-14T23:02:01.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4130115987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:02:01.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:00 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:02:01.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:01.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:01.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:01.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:01.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:01.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:01.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:01.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:01.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:01.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:01.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:01.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:01.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:01.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:01.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:02:01.906 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:01.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:02.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:02.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:02.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:02.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[97913]: pgmap v797: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 983 op/s 2024-06-14T23:02:02.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:02:02.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[97913]: osdmap e203: 8 total, 8 up, 8 in 2024-06-14T23:02:02.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/669143686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:02.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:02.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[102754]: pgmap v797: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 983 op/s 2024-06-14T23:02:02.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:02:02.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[102754]: osdmap e203: 8 total, 8 up, 8 in 2024-06-14T23:02:02.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/669143686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:02.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:01 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:02.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:02.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:02.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:02.227 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 126057 126041 92.657 62.3125 0.0145414 0.0107874 2024-06-14T23:02:02.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:01 smithi137 ceph-mon[90634]: pgmap v797: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 983 op/s 2024-06-14T23:02:02.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:01 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:02:02.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:01 smithi137 ceph-mon[90634]: osdmap e203: 8 total, 8 up, 8 in 2024-06-14T23:02:02.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/669143686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:02.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:01 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:02.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:02.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:02.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:02.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:02.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:02.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:02.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:02.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:02.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:02.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:02.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:02.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:02.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:02.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:02.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:02.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:02.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:03.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:03.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:03.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:02 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:02:03.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:02 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:02:03.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:02 smithi086 ceph-mon[97913]: osdmap e204: 8 total, 8 up, 8 in 2024-06-14T23:02:03.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:02 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:02:03.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:02 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:02:03.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:02 smithi086 ceph-mon[102754]: osdmap e204: 8 total, 8 up, 8 in 2024-06-14T23:02:03.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:03.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:03.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:03.227 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 127086 127070 92.3274 64.3125 0.0231004 0.0108263 2024-06-14T23:02:03.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:02 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:02:03.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:02 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:02:03.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:02 smithi137 ceph-mon[90634]: osdmap e204: 8 total, 8 up, 8 in 2024-06-14T23:02:03.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:03.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:03.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:03.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:03.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 7 1 2024-06-14T23:02:03.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:03.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:03.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:03.573 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:03.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:03.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:03.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:03.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:03.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:03.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:03.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:03.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:03.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:03.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:03.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:03.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:04.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:03 smithi086 ceph-mon[97913]: pgmap v800: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-06-14T23:02:04.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:03 smithi086 ceph-mon[102754]: pgmap v800: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-06-14T23:02:04.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:04.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:04.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:04.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:04.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 128068 128052 91.9716 61.375 0.0122755 0.0108682 2024-06-14T23:02:04.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:03 smithi137 ceph-mon[90634]: pgmap v800: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-06-14T23:02:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:04.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:04.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:04.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:04.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:04.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:04.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:04.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:04.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:04.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:04.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:04.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:04.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:04.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:04.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:04.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:04.943 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:02:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:02:04] "GET /metrics HTTP/1.1" 200 36587 "" "Prometheus/2.43.0" 2024-06-14T23:02:04.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:04.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:05.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:05.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:05.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:05.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:05.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:05.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:05.228 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 129068 129052 91.6367 62.5 0.0101465 0.0109078 2024-06-14T23:02:05.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:05.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:05.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:05.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:05.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:05.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:05.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:05.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:05.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:05.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:05.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:02:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:05.778 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:02:05.778 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:02:05.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.778 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:129569 10.2 10:41c9bb56:::benchmark_data_smithi086_189745_object129568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-06-14T23:02:05.778 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:02:05.748135+0000", 2024-06-14T23:02:05.778 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.021333570999999999, 2024-06-14T23:02:05.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021374723000000002, 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 129569 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.748135+0000", 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:05.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.748135+0000", 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.748133+0000", 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.748157+0000", 2024-06-14T23:02:05.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3456e-05 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.748174+0000", 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7039000000000001e-05 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.748181+0000", 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2910000000000002e-06 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.748964+0000", 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00078304600000000002 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.749108+0000", 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000144246 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.749248+0000", 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013978999999999999 2024-06-14T23:02:05.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.755987+0000", 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067390200000000001 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.756073+0000", 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6347000000000005e-05 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.45821.0:129576 10.0 e204/186 10:0540b07c:::benchmark_data_smithi086_189745_object129575:head v 204'8099, mlcod=204'8097)", 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:02:05.761122+0000", 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0083461590000000006, 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084742589999999996, 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.761122+0000", 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.784 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.761122+0000", 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.761121+0000", 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.761158+0000", 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7045000000000002e-05 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.761171+0000", 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3055e-05 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.761178+0000", 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1409999999999999e-06 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.761237+0000", 2024-06-14T23:02:05.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9073000000000002e-05 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.761257+0000", 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9905999999999999e-05 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.45821.0:129579 10.0 e204/186 10:0a17670a:::benchmark_data_smithi086_189745_object129578:head v 204'8100, mlcod=204'8097)", 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:02:05.764598+0000", 2024-06-14T23:02:05.787 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0048707580000000002, 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0050542240000000004, 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.764598+0000", 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.764598+0000", 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:05.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.764597+0000", 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.764653+0000", 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6428000000000003e-05 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.764666+0000", 2024-06-14T23:02:05.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3312999999999999e-05 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.764674+0000", 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6090000000000003e-06 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.765256+0000", 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00058251400000000001 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:05.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:05.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:05.765275+0000", 2024-06-14T23:02:05.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8672999999999999e-05 2024-06-14T23:02:05.791 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:05.791 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:05.791 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:05.791 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:05.791 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-06-14T23:02:05.791 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 3 2024-06-14T23:02:05.791 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:02:05.792 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:05.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:05.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:05.948 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:05.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:06.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:06.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:06.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:06.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:06.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:05 smithi086 ceph-mon[97913]: pgmap v801: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-14T23:02:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:05 smithi086 ceph-mon[102754]: pgmap v801: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-14T23:02:06.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:06.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:06.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:06.228 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 129998 129982 91.2594 58.125 0.0103566 0.010953 2024-06-14T23:02:06.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:05 smithi137 ceph-mon[90634]: pgmap v801: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-14T23:02:06.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:06.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:06.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:06.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:06.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:06.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:06.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:06.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:06.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:06.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:06.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:06.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:06.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:06.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:06.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:06.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:07.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:07.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:07.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:07.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:07.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:07.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0088 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 130965 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 90.9391 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 37.6168 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 239.5 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 54.875 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1455 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 601.869 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 3832 2024-06-14T23:02:07.229 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 878 2024-06-14T23:02:07.230 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0109922 2024-06-14T23:02:07.230 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.00756836 2024-06-14T23:02:07.230 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 0.179812 2024-06-14T23:02:07.230 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00119713 2024-06-14T23:02:07.234 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T23:02:07.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:07.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:07.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:07.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:07.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:07.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:07.481 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:02:07.481 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:02:07.523 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:02:07.524 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:02:07.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:07.570 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:02:07.571 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:02:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:07.579 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:07.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:07.606 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:02:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:02:07.636 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:02:07.636 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:02:07.664 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:02:07.664 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:02:07.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:07.698 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:02:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:02:07.735 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:02:07.735 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:02:07.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:07.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:07.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:07.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:07.882 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:07.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:07.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:08.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:08.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:08.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:07 smithi086 ceph-mon[97913]: pgmap v802: 99 pgs: 99 active+clean; 7.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T23:02:08.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3247141834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:08.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:07 smithi086 ceph-mon[102754]: pgmap v802: 99 pgs: 99 active+clean; 7.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T23:02:08.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3247141834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:08.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:08.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:08.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:07 smithi137 ceph-mon[90634]: pgmap v802: 99 pgs: 99 active+clean; 7.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T23:02:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3247141834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:08.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:08.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:08.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:08.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:08.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:08.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:08.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:08.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:08.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:08.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:08.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:08.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:08.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:08.864 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:08.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:08.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:08.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:08.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:09.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:09.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:09.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:09.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:09.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:09.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:09.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:09.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:09.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:09.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:09.545 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:09.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:09.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:09.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:09.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:09.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:09.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:09.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:09.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:09.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:09.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:10.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:10.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:10.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:10.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:10.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:10.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:10.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:09 smithi137 ceph-mon[90634]: pgmap v803: 99 pgs: 99 active+clean; 7.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 697 op/s 2024-06-14T23:02:10.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:10.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:09 smithi086 ceph-mon[97913]: pgmap v803: 99 pgs: 99 active+clean; 7.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 697 op/s 2024-06-14T23:02:10.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:09 smithi086 ceph-mon[102754]: pgmap v803: 99 pgs: 99 active+clean; 7.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 697 op/s 2024-06-14T23:02:10.353 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:10.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:10.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:10.490 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:10.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:10.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:10.625 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:10.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:10.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:10.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:10.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:10.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:10.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:10.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:10.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:11.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:11.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:11.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:11.171 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:11.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:11.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:11.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:11.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:11.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:11.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:11.434 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:11.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:11.495 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:11.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:11.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:11.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:11.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:11.721 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:11.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:11.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:11.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:11.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:11.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:11.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:12.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:12.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:12.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:12.159 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:11 smithi086 ceph-mon[97913]: pgmap v804: 99 pgs: 99 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 103 B/s rd, 51 MiB/s wr, 991 op/s 2024-06-14T23:02:12.159 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:11 smithi086 ceph-mon[102754]: pgmap v804: 99 pgs: 99 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 103 B/s rd, 51 MiB/s wr, 991 op/s 2024-06-14T23:02:12.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:12.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:11 smithi137 ceph-mon[90634]: pgmap v804: 99 pgs: 99 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 103 B/s rd, 51 MiB/s wr, 991 op/s 2024-06-14T23:02:12.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:12.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:12.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:12.384 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:12.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:12.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:12.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:12.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:12.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:12.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:12.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:12.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:12.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:12.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:12.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:12.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:12.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:12.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:13.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:13.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:13.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:13.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:13.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:02:13.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:13.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:02:13.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:13.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:13.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:13.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:13.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:02:13.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:13.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:13.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:13.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:13.490 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:13.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:13.587 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:13.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:13.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:13.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:13.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:13.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:13.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:13.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:13.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:13.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:13.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:13.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:14.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:14.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:14.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:14.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:13 smithi086 ceph-mon[97913]: pgmap v805: 99 pgs: 99 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 94 B/s rd, 47 MiB/s wr, 960 op/s 2024-06-14T23:02:14.168 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:13 smithi086 ceph-mon[102754]: pgmap v805: 99 pgs: 99 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 94 B/s rd, 47 MiB/s wr, 960 op/s 2024-06-14T23:02:14.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:14.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:14.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:02:14.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:13 smithi137 ceph-mon[90634]: pgmap v805: 99 pgs: 99 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 94 B/s rd, 47 MiB/s wr, 960 op/s 2024-06-14T23:02:14.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:14.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:14.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:14.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:14.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:14.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:14.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:14.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:14.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:14.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:14.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:14.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:14.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:14.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:14.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:15.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:15.028 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:02:15.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:15.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:15.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:15.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:15.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:15.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3788402549' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:02:15.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:14 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:02:15.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:15.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3788402549' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:02:15.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:14 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:02:15.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3788402549' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:02:15.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:14 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:02:15.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:02:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:02:14] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.43.0" 2024-06-14T23:02:15.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:15.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:15.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:15.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:15.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:15.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:15.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:15.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:15.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:15.750 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:15.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:15.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:15.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:15.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:15.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:15.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:15 smithi137 ceph-mon[90634]: pgmap v806: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 85 B/s rd, 39 MiB/s wr, 1.10k op/s 2024-06-14T23:02:15.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:15 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:02:15.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:15 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:02:15.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:15 smithi137 ceph-mon[90634]: osdmap e205: 8 total, 8 up, 8 in 2024-06-14T23:02:15.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/411427308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:02:15.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:15 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:02:16.023 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:16.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:16.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:16.089 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.7 to 1 (10000) 2024-06-14T23:02:16.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:16.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:16.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:16.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:16.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:16.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:16.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[97913]: pgmap v806: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 85 B/s rd, 39 MiB/s wr, 1.10k op/s 2024-06-14T23:02:16.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:02:16.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:02:16.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[97913]: osdmap e205: 8 total, 8 up, 8 in 2024-06-14T23:02:16.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/411427308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:02:16.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:02:16.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[102754]: pgmap v806: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 85 B/s rd, 39 MiB/s wr, 1.10k op/s 2024-06-14T23:02:16.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:02:16.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:02:16.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[102754]: osdmap e205: 8 total, 8 up, 8 in 2024-06-14T23:02:16.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/411427308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:02:16.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:15 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:02:16.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:16.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:16.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:16.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:16.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:16.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:16.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:16.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:16.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:16.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:16.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:16.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:16.874 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:16.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:16.904 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-14T23:02:16.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:02:16.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:17.023 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:17.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:17.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:17.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:17.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:17.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:16 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:02:17.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:16 smithi137 ceph-mon[90634]: osdmap e206: 8 total, 8 up, 8 in 2024-06-14T23:02:17.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:16 smithi137 ceph-mon[90634]: pgmap v809: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 127 B/s rd, 20 MiB/s wr, 1.15k op/s 2024-06-14T23:02:17.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:17.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:17.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:17.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:16 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:02:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:16 smithi086 ceph-mon[97913]: osdmap e206: 8 total, 8 up, 8 in 2024-06-14T23:02:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:16 smithi086 ceph-mon[97913]: pgmap v809: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 127 B/s rd, 20 MiB/s wr, 1.15k op/s 2024-06-14T23:02:17.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:16 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:02:17.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:16 smithi086 ceph-mon[102754]: osdmap e206: 8 total, 8 up, 8 in 2024-06-14T23:02:17.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:16 smithi086 ceph-mon[102754]: pgmap v809: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 127 B/s rd, 20 MiB/s wr, 1.15k op/s 2024-06-14T23:02:17.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:17.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:17.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:17.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:17.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:17.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:17.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:17.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:17.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:17.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:17.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:17.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:18.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:18.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:18.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:18.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:18.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:18.190 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:18.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:18.282 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:18.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:18.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:18.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:18.434 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:18.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:18.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:18.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:18.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:18.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:18.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:18.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:18.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:18.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:18.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:18.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:18.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:18.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:19.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:19.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:19.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:19.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:19.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:19.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:19.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:19.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:19.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:19.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:19.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:02:19.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:19.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:19.768 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:02:19.768 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:02:19.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:19.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:19.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:19.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:19.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:19.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:19.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:19 smithi137 ceph-mon[90634]: pgmap v810: 99 pgs: 99 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 288 KiB/s wr, 696 op/s 2024-06-14T23:02:20.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:20.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:19 smithi086 ceph-mon[102754]: pgmap v810: 99 pgs: 99 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 288 KiB/s wr, 696 op/s 2024-06-14T23:02:20.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:19 smithi086 ceph-mon[97913]: pgmap v810: 99 pgs: 99 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 288 KiB/s wr, 696 op/s 2024-06-14T23:02:20.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:20.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:20.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:20.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:20.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:20.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:20.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:20.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:20.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:20.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:20.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:20.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:20.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-14T23:02:20.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:20.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:20.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:20.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:20.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:20.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:20.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:20.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:21.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:21.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:21.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:21.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:21.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:21.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:21.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:21.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:21.321 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:21.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:21.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:21.476 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:21.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:21.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:21.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:21.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:21.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:21.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:21.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:21.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:21.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:21.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:21.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:22.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:22.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:21 smithi086 ceph-mon[102754]: pgmap v811: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-14T23:02:22.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:22.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:22.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:21 smithi086 ceph-mon[97913]: pgmap v811: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-14T23:02:22.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:22.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:22.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:22.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:22.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:22.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:21 smithi137 ceph-mon[90634]: pgmap v811: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-06-14T23:02:22.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:22.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:22.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:22.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:22.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:22.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:22.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:22.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:22.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:22.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:22.547 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:22.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:22.623 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:22.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:22.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:22.703 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:22.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:22.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:22.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:22.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:22.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:22.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:22.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:23.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:23.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:23.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:23.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:23.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:23.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:23.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:23.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:23.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:23.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:23.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:23.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:23.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:23.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:23.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:23.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:23.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:23.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:23.952 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:23.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:24.026 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:24.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:24.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:24.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:24.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:23 smithi086 ceph-mon[97913]: pgmap v812: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-06-14T23:02:24.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:23 smithi086 ceph-mon[102754]: pgmap v812: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-06-14T23:02:24.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:24.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:24.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:24.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:23 smithi137 ceph-mon[90634]: pgmap v812: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-06-14T23:02:24.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:24.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:24.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:24.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:24.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:24.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:24.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:24.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:24.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:24.789 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:02:24.790 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:02:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:24.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:24.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:24.812 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:02:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:02:24.847 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:02:24.848 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:02:24.885 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:02:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:02:24.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:24.921 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:02:24.921 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:02:24.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:24.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:24.961 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:02:24.961 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:02:24.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:24 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3901106990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:24.962 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:24 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3901106990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:24.962 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:02:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:02:24] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.43.0" 2024-06-14T23:02:24.997 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:02:24.997 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:02:25.030 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:02:25.030 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:02:25.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:25.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:25.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:25.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:25.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:25.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:25.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:24 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3901106990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:25.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:25.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:25.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:25.451 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:25.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:25.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:25.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:25.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:25.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:25.750 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:25.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:25.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:25.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:25.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:25.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:26.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:26.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:26.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:25 smithi086 ceph-mon[97913]: pgmap v813: 99 pgs: 99 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 963 op/s 2024-06-14T23:02:26.067 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:25 smithi086 ceph-mon[102754]: pgmap v813: 99 pgs: 99 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 963 op/s 2024-06-14T23:02:26.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:26.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:26.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:26.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:25 smithi137 ceph-mon[90634]: pgmap v813: 99 pgs: 99 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 963 op/s 2024-06-14T23:02:26.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:26.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:26.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:26.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:26.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:26.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:26.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:26.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:26.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:26.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:26.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:26.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:26.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:26.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:26.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:26.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:26.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:26.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:26.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:26.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:27.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:27.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:27.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:27.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:27.261 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:27.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:27.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:27.390 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:27.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:27.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:27.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:27.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:27.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:27.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:27.697 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:02:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:02:27.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:27.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:27.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:27.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:27.975 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:27.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:28.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:28.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:27 smithi086 ceph-mon[97913]: pgmap v814: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 876 op/s 2024-06-14T23:02:28.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:28.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:27 smithi086 ceph-mon[102754]: pgmap v814: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 876 op/s 2024-06-14T23:02:28.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:28.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:28.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:28.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:28.241 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:28.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:27 smithi137 ceph-mon[90634]: pgmap v814: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 876 op/s 2024-06-14T23:02:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:28.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:28.376 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:28.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:28.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:28.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:28.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:28.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:28.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:28.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:28.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:28.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:28.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:28.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:28.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:28.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:28.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:28.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:29.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:29.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:28 smithi086 ceph-mon[97913]: from='client.55507 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:02:29.045 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:28 smithi086 ceph-mon[102754]: from='client.55507 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:02:29.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:29.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:29.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:29.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:29.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:29.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:29.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:28 smithi137 ceph-mon[90634]: from='client.55507 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:02:29.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:29.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:29.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:29.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:29.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:29.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:29.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:29.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:29.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:29.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:29.725 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":814,"stamp":"2024-06-14T23:02:26.718870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7529568222,"num_objects":114908,"num_object_clones":0,"num_object_copies":344724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114908,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147109,"num_write_kb":8381790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130974,"ondisk_log_size":130974,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":27392480,"kb_used_data":22207072,"kb_used_omap":29,"kb_used_meta":5185058,"kb_avail":722568736,"statfs":{"total":767960285184,"available":739910385664,"internally_reserved":0,"allocated":22740041728,"data_stored":22731859398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":5309500069},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-620494848,"num_objects":-9468,"num_object_clones":0,"num_object_copies":-28404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.798529"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":136,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.933157+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:02:14.933157+0000","last_peered":"2024-06-14T23:02:14.933157+0000","last_clean":"2024-06-14T23:02:14.933157+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:02:14.933157+0000","last_undegraded":"2024-06-14T23:02:14.933157+0000","last_fullsized":"2024-06-14T23:02:14.933157+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":169,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.915953+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T23:02:14.915953+0000","last_peered":"2024-06-14T23:02:14.915953+0000","last_clean":"2024-06-14T23:02:14.915953+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T23:02:14.915953+0000","last_undegraded":"2024-06-14T23:02:14.915953+0000","last_fullsized":"2024-06-14T23:02:14.915953+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":166,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920129+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:02:14.920129+0000","last_peered":"2024-06-14T23:02:14.920129+0000","last_clean":"2024-06-14T23:02:14.920129+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:02:14.920129+0000","last_undegraded":"2024-06-14T23:02:14.920129+0000","last_fullsized":"2024-06-14T23:02:14.920129+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":158,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.930488+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:02:14.930488+0000","last_peered":"2024-06-14T23:02:14.930488+0000","last_clean":"2024-06-14T23:02:14.930488+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:02:14.930488+0000","last_undegraded":"2024-06-14T23:02:14.930488+0000","last_fullsized":"2024-06-14T23:02:14.930488+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":159,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.931433+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T23:02:14.931433+0000","last_peered":"2024-06-14T23:02:14.931433+0000","last_clean":"2024-06-14T23:02:14.931433+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T23:02:14.931433+0000","last_undegraded":"2024-06-14T23:02:14.931433+0000","last_fullsized":"2024-06-14T23:02:14.931433+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929403+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T23:02:14.929403+0000","last_peered":"2024-06-14T23:02:14.929403+0000","last_clean":"2024-06-14T23:02:14.929403+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T23:02:14.929403+0000","last_undegraded":"2024-06-14T23:02:14.929403+0000","last_fullsized":"2024-06-14T23:02:14.929403+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":160,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920941+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:02:14.920941+0000","last_peered":"2024-06-14T23:02:14.920941+0000","last_clean":"2024-06-14T23:02:14.920941+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:02:14.920941+0000","last_undegraded":"2024-06-14T23:02:14.920941+0000","last_fullsized":"2024-06-14T23:02:14.920941+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":175,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928856+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T23:02:14.928856+0000","last_peered":"2024-06-14T23:02:14.928856+0000","last_clean":"2024-06-14T23:02:14.928856+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T23:02:14.928856+0000","last_undegraded":"2024-06-14T23:02:14.928856+0000","last_fullsized":"2024-06-14T23:02:14.928856+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"206'9170","reported_seq":9198,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:24.247210+0000","last_change":"2024-06-14T23:00:27.800011+0000","last_active":"2024-06-14T23:02:24.247210+0000","last_peered":"2024-06-14T23:02:24.247210+0000","last_clean":"2024-06-14T23:02:24.247210+0000","last_became_active":"2024-06-14T23:00:27.799716+0000","last_became_peered":"2024-06-14T23:00:27.799716+0000","last_unstale":"2024-06-14T23:02:24.247210+0000","last_undegraded":"2024-06-14T23:02:24.247210+0000","last_fullsized":"2024-06-14T23:02:24.247210+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9170,"log_dups_size":0,"ondisk_log_size":9170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:41:34.619251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466878464,"num_objects":7124,"num_object_clones":0,"num_object_copies":21372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9170,"num_write_kb":521408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":156,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.916997+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:02:14.916997+0000","last_peered":"2024-06-14T23:02:14.916997+0000","last_clean":"2024-06-14T23:02:14.916997+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:02:14.916997+0000","last_undegraded":"2024-06-14T23:02:14.916997+0000","last_fullsized":"2024-06-14T23:02:14.916997+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":149,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.919827+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T23:02:14.919827+0000","last_peered":"2024-06-14T23:02:14.919827+0000","last_clean":"2024-06-14T23:02:14.919827+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T23:02:14.919827+0000","last_undegraded":"2024-06-14T23:02:14.919827+0000","last_fullsized":"2024-06-14T23:02:14.919827+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"206'9198","reported_seq":9226,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:23.849803+0000","last_change":"2024-06-14T23:00:27.798857+0000","last_active":"2024-06-14T23:02:23.849803+0000","last_peered":"2024-06-14T23:02:23.849803+0000","last_clean":"2024-06-14T23:02:23.849803+0000","last_became_active":"2024-06-14T23:00:27.798566+0000","last_became_peered":"2024-06-14T23:00:27.798566+0000","last_unstale":"2024-06-14T23:02:23.849803+0000","last_undegraded":"2024-06-14T23:02:23.849803+0000","last_fullsized":"2024-06-14T23:02:23.849803+0000","mapping_epoch":186,"log_start":"191'1500","ondisk_log_start":"191'1500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:38.237668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471203840,"num_objects":7190,"num_object_clones":0,"num_object_copies":21570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9198,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":157,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.916887+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:02:14.916887+0000","last_peered":"2024-06-14T23:02:14.916887+0000","last_clean":"2024-06-14T23:02:14.916887+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:02:14.916887+0000","last_undegraded":"2024-06-14T23:02:14.916887+0000","last_fullsized":"2024-06-14T23:02:14.916887+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":183,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928777+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:02:14.928777+0000","last_peered":"2024-06-14T23:02:14.928777+0000","last_clean":"2024-06-14T23:02:14.928777+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:02:14.928777+0000","last_undegraded":"2024-06-14T23:02:14.928777+0000","last_fullsized":"2024-06-14T23:02:14.928777+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"206'8990","reported_seq":9019,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:21.922211+0000","last_change":"2024-06-14T23:00:27.797882+0000","last_active":"2024-06-14T23:02:21.922211+0000","last_peered":"2024-06-14T23:02:21.922211+0000","last_clean":"2024-06-14T23:02:21.922211+0000","last_became_active":"2024-06-14T23:00:27.797600+0000","last_became_peered":"2024-06-14T23:00:27.797600+0000","last_unstale":"2024-06-14T23:02:21.922211+0000","last_undegraded":"2024-06-14T23:02:21.922211+0000","last_fullsized":"2024-06-14T23:02:21.922211+0000","mapping_epoch":186,"log_start":"191'1200","ondisk_log_start":"191'1200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7790,"log_dups_size":0,"ondisk_log_size":7790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:04:20.589709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473169920,"num_objects":7220,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":518720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":84,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920108+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T23:02:14.920108+0000","last_peered":"2024-06-14T23:02:14.920108+0000","last_clean":"2024-06-14T23:02:14.920108+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T23:02:14.920108+0000","last_undegraded":"2024-06-14T23:02:14.920108+0000","last_fullsized":"2024-06-14T23:02:14.920108+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":166,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929062+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T23:02:14.929062+0000","last_peered":"2024-06-14T23:02:14.929062+0000","last_clean":"2024-06-14T23:02:14.929062+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T23:02:14.929062+0000","last_undegraded":"2024-06-14T23:02:14.929062+0000","last_fullsized":"2024-06-14T23:02:14.929062+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"206'9453","reported_seq":9482,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:25.319260+0000","last_change":"2024-06-14T23:00:27.794220+0000","last_active":"2024-06-14T23:02:25.319260+0000","last_peered":"2024-06-14T23:02:25.319260+0000","last_clean":"2024-06-14T23:02:25.319260+0000","last_became_active":"2024-06-14T23:00:27.793960+0000","last_became_peered":"2024-06-14T23:00:27.793960+0000","last_unstale":"2024-06-14T23:02:25.319260+0000","last_undegraded":"2024-06-14T23:02:25.319260+0000","last_fullsized":"2024-06-14T23:02:25.319260+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:04:26.933537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480706560,"num_objects":7335,"num_object_clones":0,"num_object_copies":22005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9453,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":167,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.919999+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T23:02:14.919999+0000","last_peered":"2024-06-14T23:02:14.919999+0000","last_clean":"2024-06-14T23:02:14.919999+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T23:02:14.919999+0000","last_undegraded":"2024-06-14T23:02:14.919999+0000","last_fullsized":"2024-06-14T23:02:14.919999+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":159,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.930695+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T23:02:14.930695+0000","last_peered":"2024-06-14T23:02:14.930695+0000","last_clean":"2024-06-14T23:02:14.930695+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T23:02:14.930695+0000","last_undegraded":"2024-06-14T23:02:14.930695+0000","last_fullsized":"2024-06-14T23:02:14.930695+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"206'9325","reported_seq":9354,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:25.030168+0000","last_change":"2024-06-14T23:00:27.797355+0000","last_active":"2024-06-14T23:02:25.030168+0000","last_peered":"2024-06-14T23:02:25.030168+0000","last_clean":"2024-06-14T23:02:25.030168+0000","last_became_active":"2024-06-14T23:00:27.796666+0000","last_became_peered":"2024-06-14T23:00:27.796666+0000","last_unstale":"2024-06-14T23:02:25.030168+0000","last_undegraded":"2024-06-14T23:02:25.030168+0000","last_fullsized":"2024-06-14T23:02:25.030168+0000","mapping_epoch":186,"log_start":"191'900","ondisk_log_start":"191'900","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:57:29.139721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472055808,"num_objects":7203,"num_object_clones":0,"num_object_copies":21609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":528896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":504,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.940278+0000","last_change":"2024-06-14T23:00:30.637455+0000","last_active":"2024-06-14T23:02:14.940278+0000","last_peered":"2024-06-14T23:02:14.940278+0000","last_clean":"2024-06-14T23:02:14.940278+0000","last_became_active":"2024-06-14T23:00:17.398732+0000","last_became_peered":"2024-06-14T23:00:17.398732+0000","last_unstale":"2024-06-14T23:02:14.940278+0000","last_undegraded":"2024-06-14T23:02:14.940278+0000","last_fullsized":"2024-06-14T23:02:14.940278+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.637373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.637373+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:10.033075+0000","scrub_duration":0.0016781470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":75,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.930439+0000","last_change":"2024-06-14T22:58:09.752985+0000","last_active":"2024-06-14T23:02:14.930439+0000","last_peered":"2024-06-14T23:02:14.930439+0000","last_clean":"2024-06-14T23:02:14.930439+0000","last_became_active":"2024-06-14T22:58:09.752630+0000","last_became_peered":"2024-06-14T22:58:09.752630+0000","last_unstale":"2024-06-14T23:02:14.930439+0000","last_undegraded":"2024-06-14T23:02:14.930439+0000","last_fullsized":"2024-06-14T23:02:14.930439+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:22.320986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":166,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.931364+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T23:02:14.931364+0000","last_peered":"2024-06-14T23:02:14.931364+0000","last_clean":"2024-06-14T23:02:14.931364+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T23:02:14.931364+0000","last_undegraded":"2024-06-14T23:02:14.931364+0000","last_fullsized":"2024-06-14T23:02:14.931364+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"206'9362","reported_seq":9391,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:25.284667+0000","last_change":"2024-06-14T23:00:27.797468+0000","last_active":"2024-06-14T23:02:25.284667+0000","last_peered":"2024-06-14T23:02:25.284667+0000","last_clean":"2024-06-14T23:02:25.284667+0000","last_became_active":"2024-06-14T23:00:27.797180+0000","last_became_peered":"2024-06-14T23:00:27.797180+0000","last_unstale":"2024-06-14T23:02:25.284667+0000","last_undegraded":"2024-06-14T23:02:25.284667+0000","last_fullsized":"2024-06-14T23:02:25.284667+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9362,"log_dups_size":0,"ondisk_log_size":9362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:56:13.872188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466485248,"num_objects":7118,"num_object_clones":0,"num_object_copies":21354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9362,"num_write_kb":527360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":503,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.919745+0000","last_change":"2024-06-14T23:00:33.815761+0000","last_active":"2024-06-14T23:02:14.919745+0000","last_peered":"2024-06-14T23:02:14.919745+0000","last_clean":"2024-06-14T23:02:14.919745+0000","last_became_active":"2024-06-14T23:00:17.400781+0000","last_became_peered":"2024-06-14T23:00:17.400781+0000","last_unstale":"2024-06-14T23:02:14.919745+0000","last_undegraded":"2024-06-14T23:02:14.919745+0000","last_fullsized":"2024-06-14T23:02:14.919745+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:21:41.196504+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":167,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929185+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T23:02:14.929185+0000","last_peered":"2024-06-14T23:02:14.929185+0000","last_clean":"2024-06-14T23:02:14.929185+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T23:02:14.929185+0000","last_undegraded":"2024-06-14T23:02:14.929185+0000","last_fullsized":"2024-06-14T23:02:14.929185+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":176,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929223+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T23:02:14.929223+0000","last_peered":"2024-06-14T23:02:14.929223+0000","last_clean":"2024-06-14T23:02:14.929223+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T23:02:14.929223+0000","last_undegraded":"2024-06-14T23:02:14.929223+0000","last_fullsized":"2024-06-14T23:02:14.929223+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"206'9122","reported_seq":9150,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:23.839171+0000","last_change":"2024-06-14T23:00:27.802274+0000","last_active":"2024-06-14T23:02:23.839171+0000","last_peered":"2024-06-14T23:02:23.839171+0000","last_clean":"2024-06-14T23:02:23.839171+0000","last_became_active":"2024-06-14T23:00:27.801956+0000","last_became_peered":"2024-06-14T23:00:27.801956+0000","last_unstale":"2024-06-14T23:02:23.839171+0000","last_undegraded":"2024-06-14T23:02:23.839171+0000","last_fullsized":"2024-06-14T23:02:23.839171+0000","mapping_epoch":186,"log_start":"191'1400","ondisk_log_start":"191'1400","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:23.791512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473563136,"num_objects":7226,"num_object_clones":0,"num_object_copies":21678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9122,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":167,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928847+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T23:02:14.928847+0000","last_peered":"2024-06-14T23:02:14.928847+0000","last_clean":"2024-06-14T23:02:14.928847+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T23:02:14.928847+0000","last_undegraded":"2024-06-14T23:02:14.928847+0000","last_fullsized":"2024-06-14T23:02:14.928847+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":166,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929496+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T23:02:14.929496+0000","last_peered":"2024-06-14T23:02:14.929496+0000","last_clean":"2024-06-14T23:02:14.929496+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T23:02:14.929496+0000","last_undegraded":"2024-06-14T23:02:14.929496+0000","last_fullsized":"2024-06-14T23:02:14.929496+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"206'9217","reported_seq":9245,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:23.836532+0000","last_change":"2024-06-14T23:00:27.801110+0000","last_active":"2024-06-14T23:02:23.836532+0000","last_peered":"2024-06-14T23:02:23.836532+0000","last_clean":"2024-06-14T23:02:23.836532+0000","last_became_active":"2024-06-14T23:00:27.800817+0000","last_became_peered":"2024-06-14T23:00:27.800817+0000","last_unstale":"2024-06-14T23:02:23.836532+0000","last_undegraded":"2024-06-14T23:02:23.836532+0000","last_fullsized":"2024-06-14T23:02:23.836532+0000","mapping_epoch":186,"log_start":"191'2300","ondisk_log_start":"191'2300","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":6917,"log_dups_size":0,"ondisk_log_size":6917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:48:04.422151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467337216,"num_objects":7131,"num_object_clones":0,"num_object_copies":21393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9217,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":503,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920079+0000","last_change":"2024-06-14T23:00:31.843436+0000","last_active":"2024-06-14T23:02:14.920079+0000","last_peered":"2024-06-14T23:02:14.920079+0000","last_clean":"2024-06-14T23:02:14.920079+0000","last_became_active":"2024-06-14T23:00:17.398181+0000","last_became_peered":"2024-06-14T23:00:17.398181+0000","last_unstale":"2024-06-14T23:02:14.920079+0000","last_undegraded":"2024-06-14T23:02:14.920079+0000","last_fullsized":"2024-06-14T23:02:14.920079+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:05:06.069424+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":161,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.921494+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:02:14.921494+0000","last_peered":"2024-06-14T23:02:14.921494+0000","last_clean":"2024-06-14T23:02:14.921494+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:02:14.921494+0000","last_undegraded":"2024-06-14T23:02:14.921494+0000","last_fullsized":"2024-06-14T23:02:14.921494+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":174,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.933386+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:02:14.933386+0000","last_peered":"2024-06-14T23:02:14.933386+0000","last_clean":"2024-06-14T23:02:14.933386+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:02:14.933386+0000","last_undegraded":"2024-06-14T23:02:14.933386+0000","last_fullsized":"2024-06-14T23:02:14.933386+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"206'9345","reported_seq":9373,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:23.880175+0000","last_change":"2024-06-14T23:00:27.797497+0000","last_active":"2024-06-14T23:02:23.880175+0000","last_peered":"2024-06-14T23:02:23.880175+0000","last_clean":"2024-06-14T23:02:23.880175+0000","last_became_active":"2024-06-14T23:00:27.797221+0000","last_became_peered":"2024-06-14T23:00:27.797221+0000","last_unstale":"2024-06-14T23:02:23.880175+0000","last_undegraded":"2024-06-14T23:02:23.880175+0000","last_fullsized":"2024-06-14T23:02:23.880175+0000","mapping_epoch":186,"log_start":"191'1600","ondisk_log_start":"191'1600","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:24:58.530131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472842240,"num_objects":7215,"num_object_clones":0,"num_object_copies":21645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":496,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929966+0000","last_change":"2024-06-14T23:00:30.610273+0000","last_active":"2024-06-14T23:02:14.929966+0000","last_peered":"2024-06-14T23:02:14.929966+0000","last_clean":"2024-06-14T23:02:14.929966+0000","last_became_active":"2024-06-14T23:00:09.766095+0000","last_became_peered":"2024-06-14T23:00:09.766095+0000","last_unstale":"2024-06-14T23:02:14.929966+0000","last_undegraded":"2024-06-14T23:02:14.929966+0000","last_fullsized":"2024-06-14T23:02:14.929966+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.610180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.610180+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:53.521891+0000","scrub_duration":0.00217396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":171,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920041+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T23:02:14.920041+0000","last_peered":"2024-06-14T23:02:14.920041+0000","last_clean":"2024-06-14T23:02:14.920041+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T23:02:14.920041+0000","last_undegraded":"2024-06-14T23:02:14.920041+0000","last_fullsized":"2024-06-14T23:02:14.920041+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":110,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920165+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T23:02:14.920165+0000","last_peered":"2024-06-14T23:02:14.920165+0000","last_clean":"2024-06-14T23:02:14.920165+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T23:02:14.920165+0000","last_undegraded":"2024-06-14T23:02:14.920165+0000","last_fullsized":"2024-06-14T23:02:14.920165+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":149,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929711+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T23:02:14.929711+0000","last_peered":"2024-06-14T23:02:14.929711+0000","last_clean":"2024-06-14T23:02:14.929711+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T23:02:14.929711+0000","last_undegraded":"2024-06-14T23:02:14.929711+0000","last_fullsized":"2024-06-14T23:02:14.929711+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"206'8980","reported_seq":9009,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:21.938575+0000","last_change":"2024-06-14T23:00:27.797246+0000","last_active":"2024-06-14T23:02:21.938575+0000","last_peered":"2024-06-14T23:02:21.938575+0000","last_clean":"2024-06-14T23:02:21.938575+0000","last_became_active":"2024-06-14T23:00:27.796988+0000","last_became_peered":"2024-06-14T23:00:27.796988+0000","last_unstale":"2024-06-14T23:02:21.938575+0000","last_undegraded":"2024-06-14T23:02:21.938575+0000","last_fullsized":"2024-06-14T23:02:21.938575+0000","mapping_epoch":186,"log_start":"191'1200","ondisk_log_start":"191'1200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7780,"log_dups_size":0,"ondisk_log_size":7780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:17.545715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474349568,"num_objects":7238,"num_object_clones":0,"num_object_copies":21714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":505,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.921162+0000","last_change":"2024-06-14T23:00:30.719758+0000","last_active":"2024-06-14T23:02:15.921162+0000","last_peered":"2024-06-14T23:02:15.921162+0000","last_clean":"2024-06-14T23:02:15.921162+0000","last_became_active":"2024-06-14T23:00:17.403817+0000","last_became_peered":"2024-06-14T23:00:17.403817+0000","last_unstale":"2024-06-14T23:02:15.921162+0000","last_undegraded":"2024-06-14T23:02:15.921162+0000","last_fullsized":"2024-06-14T23:02:15.921162+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:03:47.728089+0000","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.922931+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:02:14.922931+0000","last_peered":"2024-06-14T23:02:14.922931+0000","last_clean":"2024-06-14T23:02:14.922931+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:02:14.922931+0000","last_undegraded":"2024-06-14T23:02:14.922931+0000","last_fullsized":"2024-06-14T23:02:14.922931+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":159,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.930889+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T23:02:14.930889+0000","last_peered":"2024-06-14T23:02:14.930889+0000","last_clean":"2024-06-14T23:02:14.930889+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T23:02:14.930889+0000","last_undegraded":"2024-06-14T23:02:14.930889+0000","last_fullsized":"2024-06-14T23:02:14.930889+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":173,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.931557+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T23:02:14.931557+0000","last_peered":"2024-06-14T23:02:14.931557+0000","last_clean":"2024-06-14T23:02:14.931557+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T23:02:14.931557+0000","last_undegraded":"2024-06-14T23:02:14.931557+0000","last_fullsized":"2024-06-14T23:02:14.931557+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"206'9055","reported_seq":9083,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:24.265768+0000","last_change":"2024-06-14T23:00:27.801838+0000","last_active":"2024-06-14T23:02:24.265768+0000","last_peered":"2024-06-14T23:02:24.265768+0000","last_clean":"2024-06-14T23:02:24.265768+0000","last_became_active":"2024-06-14T23:00:27.801477+0000","last_became_peered":"2024-06-14T23:00:27.801477+0000","last_unstale":"2024-06-14T23:02:24.265768+0000","last_undegraded":"2024-06-14T23:02:24.265768+0000","last_fullsized":"2024-06-14T23:02:24.265768+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9055,"log_dups_size":0,"ondisk_log_size":9055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:59:17.239200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464322560,"num_objects":7085,"num_object_clones":0,"num_object_copies":21255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9055,"num_write_kb":516480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":513,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927444+0000","last_change":"2024-06-14T23:00:31.643024+0000","last_active":"2024-06-14T23:02:15.927444+0000","last_peered":"2024-06-14T23:02:15.927444+0000","last_clean":"2024-06-14T23:02:15.927444+0000","last_became_active":"2024-06-14T23:00:17.400819+0000","last_became_peered":"2024-06-14T23:00:17.400819+0000","last_unstale":"2024-06-14T23:02:15.927444+0000","last_undegraded":"2024-06-14T23:02:15.927444+0000","last_fullsized":"2024-06-14T23:02:15.927444+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.642940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.642940+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:52:45.475940+0000","scrub_duration":0.001647253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":160,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.940877+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T23:02:14.940877+0000","last_peered":"2024-06-14T23:02:14.940877+0000","last_clean":"2024-06-14T23:02:14.940877+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T23:02:14.940877+0000","last_undegraded":"2024-06-14T23:02:14.940877+0000","last_fullsized":"2024-06-14T23:02:14.940877+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":175,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.923496+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T23:02:14.923496+0000","last_peered":"2024-06-14T23:02:14.923496+0000","last_clean":"2024-06-14T23:02:14.923496+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T23:02:14.923496+0000","last_undegraded":"2024-06-14T23:02:14.923496+0000","last_fullsized":"2024-06-14T23:02:14.923496+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":160,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.921234+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:02:15.921234+0000","last_peered":"2024-06-14T23:02:15.921234+0000","last_clean":"2024-06-14T23:02:15.921234+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:02:15.921234+0000","last_undegraded":"2024-06-14T23:02:15.921234+0000","last_fullsized":"2024-06-14T23:02:15.921234+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"206'9204","reported_seq":9232,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:23.875177+0000","last_change":"2024-06-14T23:00:27.799968+0000","last_active":"2024-06-14T23:02:23.875177+0000","last_peered":"2024-06-14T23:02:23.875177+0000","last_clean":"2024-06-14T23:02:23.875177+0000","last_became_active":"2024-06-14T23:00:27.799651+0000","last_became_peered":"2024-06-14T23:00:27.799651+0000","last_unstale":"2024-06-14T23:02:23.875177+0000","last_undegraded":"2024-06-14T23:02:23.875177+0000","last_fullsized":"2024-06-14T23:02:23.875177+0000","mapping_epoch":186,"log_start":"191'1500","ondisk_log_start":"191'1500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7704,"log_dups_size":0,"ondisk_log_size":7704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:23:57.581265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470810624,"num_objects":7184,"num_object_clones":0,"num_object_copies":21552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9204,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":502,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.919888+0000","last_change":"2024-06-14T23:00:32.818134+0000","last_active":"2024-06-14T23:02:14.919888+0000","last_peered":"2024-06-14T23:02:14.919888+0000","last_clean":"2024-06-14T23:02:14.919888+0000","last_became_active":"2024-06-14T23:00:17.399852+0000","last_became_peered":"2024-06-14T23:00:17.399852+0000","last_unstale":"2024-06-14T23:02:14.919888+0000","last_undegraded":"2024-06-14T23:02:14.919888+0000","last_fullsized":"2024-06-14T23:02:14.919888+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.818076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.818076+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:12:39.666043+0000","scrub_duration":0.0013568390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":171,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920269+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T23:02:14.920269+0000","last_peered":"2024-06-14T23:02:14.920269+0000","last_clean":"2024-06-14T23:02:14.920269+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T23:02:14.920269+0000","last_undegraded":"2024-06-14T23:02:14.920269+0000","last_fullsized":"2024-06-14T23:02:14.920269+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":167,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929176+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T23:02:14.929176+0000","last_peered":"2024-06-14T23:02:14.929176+0000","last_clean":"2024-06-14T23:02:14.929176+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T23:02:14.929176+0000","last_undegraded":"2024-06-14T23:02:14.929176+0000","last_fullsized":"2024-06-14T23:02:14.929176+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":190,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928608+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T23:02:14.928608+0000","last_peered":"2024-06-14T23:02:14.928608+0000","last_clean":"2024-06-14T23:02:14.928608+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T23:02:14.928608+0000","last_undegraded":"2024-06-14T23:02:14.928608+0000","last_fullsized":"2024-06-14T23:02:14.928608+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"206'9166","reported_seq":9194,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:23.908082+0000","last_change":"2024-06-14T23:00:27.803439+0000","last_active":"2024-06-14T23:02:23.908082+0000","last_peered":"2024-06-14T23:02:23.908082+0000","last_clean":"2024-06-14T23:02:23.908082+0000","last_became_active":"2024-06-14T23:00:27.802783+0000","last_became_peered":"2024-06-14T23:00:27.802783+0000","last_unstale":"2024-06-14T23:02:23.908082+0000","last_undegraded":"2024-06-14T23:02:23.908082+0000","last_fullsized":"2024-06-14T23:02:23.908082+0000","mapping_epoch":186,"log_start":"191'1200","ondisk_log_start":"191'1200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7966,"log_dups_size":0,"ondisk_log_size":7966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:35:39.179455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472514560,"num_objects":7210,"num_object_clones":0,"num_object_copies":21630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9166,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":456,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929822+0000","last_change":"2024-06-14T23:00:42.661659+0000","last_active":"2024-06-14T23:02:14.929822+0000","last_peered":"2024-06-14T23:02:14.929822+0000","last_clean":"2024-06-14T23:02:14.929822+0000","last_became_active":"2024-06-14T23:00:17.398852+0000","last_became_peered":"2024-06-14T23:00:17.398852+0000","last_unstale":"2024-06-14T23:02:14.929822+0000","last_undegraded":"2024-06-14T23:02:14.929822+0000","last_fullsized":"2024-06-14T23:02:14.929822+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:47:14.444805+0000","scrub_duration":0.0053445919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":184,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920215+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T23:02:14.920215+0000","last_peered":"2024-06-14T23:02:14.920215+0000","last_clean":"2024-06-14T23:02:14.920215+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T23:02:14.920215+0000","last_undegraded":"2024-06-14T23:02:14.920215+0000","last_fullsized":"2024-06-14T23:02:14.920215+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":165,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.933329+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:02:14.933329+0000","last_peered":"2024-06-14T23:02:14.933329+0000","last_clean":"2024-06-14T23:02:14.933329+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:02:14.933329+0000","last_undegraded":"2024-06-14T23:02:14.933329+0000","last_fullsized":"2024-06-14T23:02:14.933329+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":160,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929150+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:02:14.929150+0000","last_peered":"2024-06-14T23:02:14.929150+0000","last_clean":"2024-06-14T23:02:14.929150+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:02:14.929150+0000","last_undegraded":"2024-06-14T23:02:14.929150+0000","last_fullsized":"2024-06-14T23:02:14.929150+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"206'9227","reported_seq":9255,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:23.848228+0000","last_change":"2024-06-14T23:00:27.800097+0000","last_active":"2024-06-14T23:02:23.848228+0000","last_peered":"2024-06-14T23:02:23.848228+0000","last_clean":"2024-06-14T23:02:23.848228+0000","last_became_active":"2024-06-14T23:00:27.799790+0000","last_became_peered":"2024-06-14T23:00:27.799790+0000","last_unstale":"2024-06-14T23:02:23.848228+0000","last_undegraded":"2024-06-14T23:02:23.848228+0000","last_fullsized":"2024-06-14T23:02:23.848228+0000","mapping_epoch":186,"log_start":"191'1500","ondisk_log_start":"191'1500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7727,"log_dups_size":0,"ondisk_log_size":7727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:21:01.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470155288,"num_objects":7175,"num_object_clones":0,"num_object_copies":21525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9227,"num_write_kb":524801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":456,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929686+0000","last_change":"2024-06-14T23:00:31.555325+0000","last_active":"2024-06-14T23:02:14.929686+0000","last_peered":"2024-06-14T23:02:14.929686+0000","last_clean":"2024-06-14T23:02:14.929686+0000","last_became_active":"2024-06-14T23:00:17.400871+0000","last_became_peered":"2024-06-14T23:00:17.400871+0000","last_unstale":"2024-06-14T23:02:14.929686+0000","last_undegraded":"2024-06-14T23:02:14.929686+0000","last_fullsized":"2024-06-14T23:02:14.929686+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:58.191436+0000","scrub_duration":0.001665094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":175,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928531+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T23:02:14.928531+0000","last_peered":"2024-06-14T23:02:14.928531+0000","last_clean":"2024-06-14T23:02:14.928531+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T23:02:14.928531+0000","last_undegraded":"2024-06-14T23:02:14.928531+0000","last_fullsized":"2024-06-14T23:02:14.928531+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":171,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.917108+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:02:14.917108+0000","last_peered":"2024-06-14T23:02:14.917108+0000","last_clean":"2024-06-14T23:02:14.917108+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:02:14.917108+0000","last_undegraded":"2024-06-14T23:02:14.917108+0000","last_fullsized":"2024-06-14T23:02:14.917108+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":145,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.930627+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T23:02:14.930627+0000","last_peered":"2024-06-14T23:02:14.930627+0000","last_clean":"2024-06-14T23:02:14.930627+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T23:02:14.930627+0000","last_undegraded":"2024-06-14T23:02:14.930627+0000","last_fullsized":"2024-06-14T23:02:14.930627+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"206'9250","reported_seq":9278,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:23.859356+0000","last_change":"2024-06-14T23:00:27.797841+0000","last_active":"2024-06-14T23:02:23.859356+0000","last_peered":"2024-06-14T23:02:23.859356+0000","last_clean":"2024-06-14T23:02:23.859356+0000","last_became_active":"2024-06-14T23:00:27.797256+0000","last_became_peered":"2024-06-14T23:00:27.797256+0000","last_unstale":"2024-06-14T23:02:23.859356+0000","last_undegraded":"2024-06-14T23:02:23.859356+0000","last_fullsized":"2024-06-14T23:02:23.859356+0000","mapping_epoch":186,"log_start":"192'2400","ondisk_log_start":"192'2400","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:52:24.873796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475791360,"num_objects":7260,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9250,"num_write_kb":528320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":503,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928966+0000","last_change":"2024-06-14T23:00:32.174187+0000","last_active":"2024-06-14T23:02:14.928966+0000","last_peered":"2024-06-14T23:02:14.928966+0000","last_clean":"2024-06-14T23:02:14.928966+0000","last_became_active":"2024-06-14T23:00:17.395654+0000","last_became_peered":"2024-06-14T23:00:17.395654+0000","last_unstale":"2024-06-14T23:02:14.928966+0000","last_undegraded":"2024-06-14T23:02:14.928966+0000","last_fullsized":"2024-06-14T23:02:14.928966+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.174132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.174132+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:59.252795+0000","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":66,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.930010+0000","last_change":"2024-06-14T22:58:09.752729+0000","last_active":"2024-06-14T23:02:14.930010+0000","last_peered":"2024-06-14T23:02:14.930010+0000","last_clean":"2024-06-14T23:02:14.930010+0000","last_became_active":"2024-06-14T22:58:09.752393+0000","last_became_peered":"2024-06-14T22:58:09.752393+0000","last_unstale":"2024-06-14T23:02:14.930010+0000","last_undegraded":"2024-06-14T23:02:14.930010+0000","last_fullsized":"2024-06-14T23:02:14.930010+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:21:58.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":159,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.921106+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T23:02:15.921106+0000","last_peered":"2024-06-14T23:02:15.921106+0000","last_clean":"2024-06-14T23:02:15.921106+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T23:02:15.921106+0000","last_undegraded":"2024-06-14T23:02:15.921106+0000","last_fullsized":"2024-06-14T23:02:15.921106+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":151,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.915071+0000","last_change":"2024-06-14T22:58:32.754079+0000","last_active":"2024-06-14T23:02:14.915071+0000","last_peered":"2024-06-14T23:02:14.915071+0000","last_clean":"2024-06-14T23:02:14.915071+0000","last_became_active":"2024-06-14T22:58:32.753814+0000","last_became_peered":"2024-06-14T22:58:32.753814+0000","last_unstale":"2024-06-14T23:02:14.915071+0000","last_undegraded":"2024-06-14T23:02:14.915071+0000","last_fullsized":"2024-06-14T23:02:14.915071+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:37:07.414370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"206'8984","reported_seq":9012,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:23.849559+0000","last_change":"2024-06-14T23:00:27.798887+0000","last_active":"2024-06-14T23:02:23.849559+0000","last_peered":"2024-06-14T23:02:23.849559+0000","last_clean":"2024-06-14T23:02:23.849559+0000","last_became_active":"2024-06-14T23:00:27.798590+0000","last_became_peered":"2024-06-14T23:00:27.798590+0000","last_unstale":"2024-06-14T23:02:23.849559+0000","last_undegraded":"2024-06-14T23:02:23.849559+0000","last_fullsized":"2024-06-14T23:02:23.849559+0000","mapping_epoch":186,"log_start":"191'1200","ondisk_log_start":"191'1200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:02:33.396409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456785920,"num_objects":6970,"num_object_clones":0,"num_object_copies":20910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8984,"num_write_kb":510528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":505,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.930664+0000","last_change":"2024-06-14T23:00:32.716013+0000","last_active":"2024-06-14T23:02:15.930664+0000","last_peered":"2024-06-14T23:02:15.930664+0000","last_clean":"2024-06-14T23:02:15.930664+0000","last_became_active":"2024-06-14T23:00:17.398349+0000","last_became_peered":"2024-06-14T23:00:17.398349+0000","last_unstale":"2024-06-14T23:02:15.930664+0000","last_undegraded":"2024-06-14T23:02:15.930664+0000","last_fullsized":"2024-06-14T23:02:15.930664+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.715907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.715907+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:23:40.212866+0000","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":163,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.931356+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T23:02:14.931356+0000","last_peered":"2024-06-14T23:02:14.931356+0000","last_clean":"2024-06-14T23:02:14.931356+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T23:02:14.931356+0000","last_undegraded":"2024-06-14T23:02:14.931356+0000","last_fullsized":"2024-06-14T23:02:14.931356+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929896+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T23:02:14.929896+0000","last_peered":"2024-06-14T23:02:14.929896+0000","last_clean":"2024-06-14T23:02:14.929896+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T23:02:14.929896+0000","last_undegraded":"2024-06-14T23:02:14.929896+0000","last_fullsized":"2024-06-14T23:02:14.929896+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":149,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928648+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T23:02:14.928648+0000","last_peered":"2024-06-14T23:02:14.928648+0000","last_clean":"2024-06-14T23:02:14.928648+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T23:02:14.928648+0000","last_undegraded":"2024-06-14T23:02:14.928648+0000","last_fullsized":"2024-06-14T23:02:14.928648+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":166,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929107+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T23:02:14.929107+0000","last_peered":"2024-06-14T23:02:14.929107+0000","last_clean":"2024-06-14T23:02:14.929107+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T23:02:14.929107+0000","last_undegraded":"2024-06-14T23:02:14.929107+0000","last_fullsized":"2024-06-14T23:02:14.929107+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":172,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.931379+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T23:02:14.931379+0000","last_peered":"2024-06-14T23:02:14.931379+0000","last_clean":"2024-06-14T23:02:14.931379+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T23:02:14.931379+0000","last_undegraded":"2024-06-14T23:02:14.931379+0000","last_fullsized":"2024-06-14T23:02:14.931379+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":174,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.922783+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T23:02:14.922783+0000","last_peered":"2024-06-14T23:02:14.922783+0000","last_clean":"2024-06-14T23:02:14.922783+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T23:02:14.922783+0000","last_undegraded":"2024-06-14T23:02:14.922783+0000","last_fullsized":"2024-06-14T23:02:14.922783+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":85,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.921421+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:02:14.921421+0000","last_peered":"2024-06-14T23:02:14.921421+0000","last_clean":"2024-06-14T23:02:14.921421+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:02:14.921421+0000","last_undegraded":"2024-06-14T23:02:14.921421+0000","last_fullsized":"2024-06-14T23:02:14.921421+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":167,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920304+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:02:14.920304+0000","last_peered":"2024-06-14T23:02:14.920304+0000","last_clean":"2024-06-14T23:02:14.920304+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:02:14.920304+0000","last_undegraded":"2024-06-14T23:02:14.920304+0000","last_fullsized":"2024-06-14T23:02:14.920304+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":110,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928546+0000","last_change":"2024-06-14T22:58:09.753103+0000","last_active":"2024-06-14T23:02:14.928546+0000","last_peered":"2024-06-14T23:02:14.928546+0000","last_clean":"2024-06-14T23:02:14.928546+0000","last_became_active":"2024-06-14T22:58:09.752761+0000","last_became_peered":"2024-06-14T22:58:09.752761+0000","last_unstale":"2024-06-14T23:02:14.928546+0000","last_undegraded":"2024-06-14T23:02:14.928546+0000","last_fullsized":"2024-06-14T23:02:14.928546+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:20:20.619331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":158,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929931+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T23:02:14.929931+0000","last_peered":"2024-06-14T23:02:14.929931+0000","last_clean":"2024-06-14T23:02:14.929931+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T23:02:14.929931+0000","last_undegraded":"2024-06-14T23:02:14.929931+0000","last_fullsized":"2024-06-14T23:02:14.929931+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":160,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920153+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T23:02:14.920153+0000","last_peered":"2024-06-14T23:02:14.920153+0000","last_clean":"2024-06-14T23:02:14.920153+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T23:02:14.920153+0000","last_undegraded":"2024-06-14T23:02:14.920153+0000","last_fullsized":"2024-06-14T23:02:14.920153+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":166,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929822+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T23:02:14.929822+0000","last_peered":"2024-06-14T23:02:14.929822+0000","last_clean":"2024-06-14T23:02:14.929822+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T23:02:14.929822+0000","last_undegraded":"2024-06-14T23:02:14.929822+0000","last_fullsized":"2024-06-14T23:02:14.929822+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":154,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929842+0000","last_change":"2024-06-14T22:58:09.785074+0000","last_active":"2024-06-14T23:02:14.929842+0000","last_peered":"2024-06-14T23:02:14.929842+0000","last_clean":"2024-06-14T23:02:14.929842+0000","last_became_active":"2024-06-14T22:58:09.784309+0000","last_became_peered":"2024-06-14T22:58:09.784309+0000","last_unstale":"2024-06-14T23:02:14.929842+0000","last_undegraded":"2024-06-14T23:02:14.929842+0000","last_fullsized":"2024-06-14T23:02:14.929842+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:23.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":160,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.931496+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T23:02:14.931496+0000","last_peered":"2024-06-14T23:02:14.931496+0000","last_clean":"2024-06-14T23:02:14.931496+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T23:02:14.931496+0000","last_undegraded":"2024-06-14T23:02:14.931496+0000","last_fullsized":"2024-06-14T23:02:14.931496+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":174,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.922466+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:02:14.922466+0000","last_peered":"2024-06-14T23:02:14.922466+0000","last_clean":"2024-06-14T23:02:14.922466+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:02:14.922466+0000","last_undegraded":"2024-06-14T23:02:14.922466+0000","last_fullsized":"2024-06-14T23:02:14.922466+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":162,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.942705+0000","last_change":"2024-06-14T22:58:09.756040+0000","last_active":"2024-06-14T23:02:14.942705+0000","last_peered":"2024-06-14T23:02:14.942705+0000","last_clean":"2024-06-14T23:02:14.942705+0000","last_became_active":"2024-06-14T22:58:09.755659+0000","last_became_peered":"2024-06-14T22:58:09.755659+0000","last_unstale":"2024-06-14T23:02:14.942705+0000","last_undegraded":"2024-06-14T23:02:14.942705+0000","last_fullsized":"2024-06-14T23:02:14.942705+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:40.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":185,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929166+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T23:02:14.929166+0000","last_peered":"2024-06-14T23:02:14.929166+0000","last_clean":"2024-06-14T23:02:14.929166+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T23:02:14.929166+0000","last_undegraded":"2024-06-14T23:02:14.929166+0000","last_fullsized":"2024-06-14T23:02:14.929166+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":179,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.940030+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T23:02:14.940030+0000","last_peered":"2024-06-14T23:02:14.940030+0000","last_clean":"2024-06-14T23:02:14.940030+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T23:02:14.940030+0000","last_undegraded":"2024-06-14T23:02:14.940030+0000","last_fullsized":"2024-06-14T23:02:14.940030+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":147,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.926383+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T23:02:15.926383+0000","last_peered":"2024-06-14T23:02:15.926383+0000","last_clean":"2024-06-14T23:02:15.926383+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T23:02:15.926383+0000","last_undegraded":"2024-06-14T23:02:15.926383+0000","last_fullsized":"2024-06-14T23:02:15.926383+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":183,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929041+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:02:14.929041+0000","last_peered":"2024-06-14T23:02:14.929041+0000","last_clean":"2024-06-14T23:02:14.929041+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:02:14.929041+0000","last_undegraded":"2024-06-14T23:02:14.929041+0000","last_fullsized":"2024-06-14T23:02:14.929041+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":169,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.919989+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:02:14.919989+0000","last_peered":"2024-06-14T23:02:14.919989+0000","last_clean":"2024-06-14T23:02:14.919989+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:02:14.919989+0000","last_undegraded":"2024-06-14T23:02:14.919989+0000","last_fullsized":"2024-06-14T23:02:14.919989+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":183,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929038+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:02:14.929038+0000","last_peered":"2024-06-14T23:02:14.929038+0000","last_clean":"2024-06-14T23:02:14.929038+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:02:14.929038+0000","last_undegraded":"2024-06-14T23:02:14.929038+0000","last_fullsized":"2024-06-14T23:02:14.929038+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":152,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.920190+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T23:02:14.920190+0000","last_peered":"2024-06-14T23:02:14.920190+0000","last_clean":"2024-06-14T23:02:14.920190+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T23:02:14.920190+0000","last_undegraded":"2024-06-14T23:02:14.920190+0000","last_fullsized":"2024-06-14T23:02:14.920190+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":169,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.940518+0000","last_change":"2024-06-14T22:58:09.752281+0000","last_active":"2024-06-14T23:02:14.940518+0000","last_peered":"2024-06-14T23:02:14.940518+0000","last_clean":"2024-06-14T23:02:14.940518+0000","last_became_active":"2024-06-14T22:58:09.751952+0000","last_became_peered":"2024-06-14T22:58:09.751952+0000","last_unstale":"2024-06-14T23:02:14.940518+0000","last_undegraded":"2024-06-14T23:02:14.940518+0000","last_fullsized":"2024-06-14T23:02:14.940518+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:39:19.836734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":65,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929675+0000","last_change":"2024-06-14T22:58:09.755622+0000","last_active":"2024-06-14T23:02:14.929675+0000","last_peered":"2024-06-14T23:02:14.929675+0000","last_clean":"2024-06-14T23:02:14.929675+0000","last_became_active":"2024-06-14T22:58:09.755383+0000","last_became_peered":"2024-06-14T22:58:09.755383+0000","last_unstale":"2024-06-14T23:02:14.929675+0000","last_undegraded":"2024-06-14T23:02:14.929675+0000","last_fullsized":"2024-06-14T23:02:14.929675+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:13.884346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":166,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929065+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T23:02:14.929065+0000","last_peered":"2024-06-14T23:02:14.929065+0000","last_clean":"2024-06-14T23:02:14.929065+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T23:02:14.929065+0000","last_undegraded":"2024-06-14T23:02:14.929065+0000","last_fullsized":"2024-06-14T23:02:14.929065+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":163,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929834+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T23:02:14.929834+0000","last_peered":"2024-06-14T23:02:14.929834+0000","last_clean":"2024-06-14T23:02:14.929834+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T23:02:14.929834+0000","last_undegraded":"2024-06-14T23:02:14.929834+0000","last_fullsized":"2024-06-14T23:02:14.929834+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7528972312,"num_objects":114884,"num_object_clones":0,"num_object_copies":344652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147048,"num_write_kb":8381761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22605869056,"data_stored":22605856840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129148,"ondisk_log_size":129148,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1771,"ondisk_log_size":1771,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234340,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2743892,"kb_used_data":1832652,"kb_used_omap":1,"kb_used_meta":911230,"kb_avail":91001260,"statfs":{"total":95995035648,"available":93185290240,"internally_reserved":0,"allocated":1876635648,"data_stored":1875599629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":933099864},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.105,"15min":1.0840000000000001},"min":{"1min":0.224,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":2.351,"5min":6.9539999999999997,"15min":9.3919999999999995},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.661,"5min":1.242,"15min":1.24},"min":{"1min":0.29099999999999998,"5min":0.26800000000000002,"15min":0.20300000000000001},"max":{"1min":4.806,"5min":5.2130000000000001,"15min":10.416},"last":3.8260000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.3680000000000001,"15min":1.7490000000000001},"min":{"1min":0.31900000000000001,"5min":0.29199999999999998,"15min":0.28100000000000003},"max":{"1min":3.3660000000000001,"5min":8.5340000000000007,"15min":83.295000000000002},"last":3.8580000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.4510000000000001,"15min":1.821},"min":{"1min":0.29199999999999998,"5min":0.245,"15min":0.223},"max":{"1min":2.5459999999999998,"5min":6.8620000000000001,"15min":82.972999999999999},"last":1.3169999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1299999999999999,"15min":0.79500000000000004},"min":{"1min":0.26700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.9820000000000002,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":1.296},{"interface":"front","average":{"1min":1.03,"5min":1.2749999999999999,"15min":0.84999999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.0199999999999996,"5min":8.1150000000000002,"15min":8.1150000000000002},"last":1.3380000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.4199999999999999,"15min":1.0669999999999999},"min":{"1min":0.34899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":6.2220000000000004,"5min":12.992000000000001,"15min":55.683999999999997},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.3180000000000001,"15min":1.0920000000000001},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.528,"5min":9.5559999999999992,"15min":55.533999999999999},"last":3.899}]},{"osd":4,"last update":"Fri Jun 14 23:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.002,"15min":0.73899999999999999},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.245},"max":{"1min":3.1480000000000001,"5min":6.7610000000000001,"15min":6.7610000000000001},"last":4.0970000000000004},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.0840000000000001,"15min":0.79400000000000004},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":6.1600000000000001,"5min":16.879000000000001,"15min":16.879000000000001},"last":0.44600000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.331,"15min":0.91500000000000004},"min":{"1min":0.317,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.4320000000000004,"5min":6.9000000000000004,"15min":6.9000000000000004},"last":1.363},{"interface":"front","average":{"1min":1.216,"5min":1.137,"15min":0.90700000000000003},"min":{"1min":0.33200000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.9380000000000002,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":3.879}]},{"osd":6,"last update":"Fri Jun 14 23:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.0920000000000001,"15min":0.84199999999999997},"min":{"1min":0.35799999999999998,"5min":0.307,"15min":0.28100000000000003},"max":{"1min":2.9689999999999999,"5min":5.5,"15min":5.806},"last":4.319},{"interface":"front","average":{"1min":1.518,"5min":1.099,"15min":0.84899999999999998},"min":{"1min":0.38700000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.1980000000000004,"5min":12.951000000000001,"15min":12.951000000000001},"last":3.4209999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495976,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4129928,"kb_used_data":3232704,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":89615224,"statfs":{"total":95995035648,"available":91765989376,"internally_reserved":0,"allocated":3310288896,"data_stored":3309264720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:14 2024","interfaces":[{"interface":"back","average":{"1min":2.431,"5min":1.4059999999999999,"15min":1.351},"min":{"1min":0.36399999999999999,"5min":0.23699999999999999,"15min":0.218},"max":{"1min":10.348000000000001,"5min":10.348000000000001,"15min":31.358000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.859,"5min":1.119,"15min":1.2929999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.21299999999999999},"max":{"1min":10.348000000000001,"5min":10.348000000000001,"15min":29.399000000000001},"last":0.34300000000000003}]},{"osd":1,"last update":"Fri Jun 14 23:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.3500000000000001,"15min":1.21},"min":{"1min":0.32200000000000001,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":4.6260000000000003,"5min":6.6680000000000001,"15min":9.9559999999999995},"last":3.0489999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.351,"15min":1.2330000000000001},"min":{"1min":0.27400000000000002,"5min":0.25,"15min":0.247},"max":{"1min":3.0830000000000002,"5min":6.7519999999999998,"15min":9.5510000000000002},"last":0.748}]},{"osd":2,"last update":"Fri Jun 14 23:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.1160000000000001,"15min":0.76500000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":9.9600000000000009,"5min":9.9600000000000009,"15min":9.9600000000000009},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.2050000000000001,"15min":0.82099999999999995},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":9.9870000000000001,"5min":9.9870000000000001,"15min":9.9870000000000001},"last":4.9749999999999996}]},{"osd":3,"last update":"Fri Jun 14 23:02:14 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.3149999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.28599999999999998,"15min":0.27300000000000002},"max":{"1min":3.1560000000000001,"5min":9.8970000000000002,"15min":29.152000000000001},"last":3.1669999999999998},{"interface":"front","average":{"1min":1.081,"5min":1.3069999999999999,"15min":0.98999999999999999},"min":{"1min":0.379,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.621,"5min":6.3040000000000003,"15min":25.864000000000001},"last":0.38600000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:01:26 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":0.96599999999999997,"15min":0.92600000000000005},"min":{"1min":0.29299999999999998,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":3.1179999999999999,"5min":7.7709999999999999,"15min":47.698},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.1659999999999999,"5min":0.86099999999999999,"15min":0.84599999999999997},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":4.6790000000000003,"5min":9.9610000000000003,"15min":48.197000000000003},"last":1.2909999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:01:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.0349999999999999,"15min":0.83199999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.26500000000000001},"max":{"1min":7.3540000000000001,"5min":7.3540000000000001,"15min":8.6500000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.087,"15min":0.88600000000000001},"min":{"1min":0.33500000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.145,"5min":7.3769999999999998,"15min":8.6349999999999998},"last":2.714}]},{"osd":7,"last update":"Fri Jun 14 23:02:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.224,"15min":0.89800000000000002},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.29299999999999998},"max":{"1min":4.4720000000000004,"5min":7.8579999999999997,"15min":8.6950000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.079,"5min":1.147,"15min":0.92900000000000005},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.28899999999999998},"max":{"1min":3.6869999999999998,"5min":7.8120000000000003,"15min":8.673},"last":0.71899999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757615,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3200640,"kb_used_data":2305400,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":90544512,"statfs":{"total":95995035648,"available":92717580288,"internally_reserved":0,"allocated":2360729600,"data_stored":2359709950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:01:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.2050000000000001,"15min":1.2370000000000001},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.219},"max":{"1min":5.7089999999999996,"5min":5.7089999999999996,"15min":17.766999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.1890000000000001,"15min":1.224},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.964,"5min":5.1950000000000003,"15min":17.806000000000001},"last":4.2190000000000003}]},{"osd":1,"last update":"Fri Jun 14 23:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.0209999999999999,"15min":1.157},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23300000000000001},"max":{"1min":1.6719999999999999,"5min":5.6429999999999998,"15min":7.694},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.3200000000000001,"15min":1.1699999999999999},"min":{"1min":0.312,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":3.1930000000000001,"5min":7.0700000000000003,"15min":7.0700000000000003},"last":1.2889999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.0980000000000001,"15min":0.78300000000000003},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.8470000000000004,"5min":6.8470000000000004,"15min":6.8470000000000004},"last":1.236},{"interface":"front","average":{"1min":1.754,"5min":1.169,"15min":0.79000000000000004},"min":{"1min":0.379,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.9000000000000004,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":1.1970000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:02:19 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.534,"15min":1.1319999999999999},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.4580000000000002,"5min":10.358000000000001,"15min":45.744},"last":4.1849999999999996},{"interface":"front","average":{"1min":1.845,"5min":1.5900000000000001,"15min":1.222},"min":{"1min":0.378,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.4199999999999999,"5min":10.385,"15min":45.838000000000001},"last":1.264}]},{"osd":4,"last update":"Fri Jun 14 23:02:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.872,"15min":0.66200000000000003},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.214},"max":{"1min":2.5299999999999998,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.312,"5min":0.28299999999999997,"15min":0.22700000000000001},"max":{"1min":4.4359999999999999,"5min":4.657,"15min":5.8109999999999999},"last":4.2469999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.038,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":4.0090000000000003,"5min":6.931,"15min":9.6020000000000003},"last":4.1529999999999996},{"interface":"front","average":{"1min":1.4119999999999999,"5min":0.96199999999999997,"15min":0.78800000000000003},"min":{"1min":0.373,"5min":0.30099999999999999,"15min":0.25},"max":{"1min":5.056,"5min":5.056,"15min":12.339},"last":5.625}]},{"osd":7,"last update":"Fri Jun 14 23:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.1040000000000001,"15min":0.83299999999999996},"min":{"1min":0.30499999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.4369999999999998,"5min":5.3129999999999997,"15min":5.3129999999999997},"last":1.167},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.1220000000000001,"15min":0.83899999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":5.1820000000000004,"5min":5.6189999999999998,"15min":10.638999999999999},"last":0.34399999999999997}]}]},{"osd":4,"up_from":30,"seq":128849019251,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4575952,"kb_used_data":3689352,"kb_used_omap":1,"kb_used_meta":886590,"kb_avail":89169200,"statfs":{"total":95995035648,"available":91309260800,"internally_reserved":0,"allocated":3777896448,"data_stored":3776864214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907868504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:01:27 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.97799999999999998,"15min":1.036},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":3.1819999999999999,"5min":5.1479999999999997,"15min":7.5650000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.0289999999999999,"15min":1.1060000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.221},"max":{"1min":3.3330000000000002,"5min":8.3390000000000004,"15min":9.1470000000000002},"last":0.379}]},{"osd":1,"last update":"Fri Jun 14 23:01:44 2024","interfaces":[{"interface":"back","average":{"1min":4.0469999999999997,"5min":1.901,"15min":1.345},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":56.917000000000002,"5min":56.917000000000002,"15min":56.917000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":4.1479999999999997,"5min":1.98,"15min":1.4410000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":56.789999999999999,"5min":56.789999999999999,"15min":56.789999999999999},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:01:44 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.89500000000000002,"15min":0.67400000000000004},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.5750000000000002,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.95899999999999996,"15min":0.70799999999999996},"min":{"1min":0.30299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.8480000000000001,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":0.41699999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:01:27 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.3,"15min":0.80200000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.8340000000000001,"5min":6.617,"15min":6.617},"last":0.62},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.24,"15min":0.82499999999999996},"min":{"1min":0.27600000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.2120000000000002,"5min":5.9580000000000002,"15min":5.9580000000000002},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Jun 14 23:02:07 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":2.6579999999999999,"15min":1.3240000000000001},"min":{"1min":0.32100000000000001,"5min":0.253,"15min":0.22500000000000001},"max":{"1min":5.0629999999999997,"5min":139.13200000000001,"15min":139.13200000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.0069999999999999,"5min":2.7759999999999998,"15min":1.353},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.23699999999999999},"max":{"1min":2.3069999999999999,"5min":138.86000000000001,"15min":138.86000000000001},"last":0.45000000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:01:31 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":0.94299999999999995,"15min":0.71399999999999997},"min":{"1min":0.33700000000000002,"5min":0.28899999999999998,"15min":0.254},"max":{"1min":4.335,"5min":5.1210000000000004,"15min":5.1210000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.456,"5min":0.88700000000000001,"15min":0.67300000000000004},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.254},"max":{"1min":8,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.35899999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:01:57 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.056,"15min":0.755},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.0009999999999999,"5min":8.9489999999999998,"15min":8.9489999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.2410000000000001,"15min":0.871},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.308},"max":{"1min":2.9409999999999998,"5min":7.25,"15min":8.4730000000000008},"last":0.64600000000000002}]}]},{"osd":3,"up_from":138,"seq":592705486950,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2611264,"kb_used_data":2292664,"kb_used_omap":4,"kb_used_meta":318523,"kb_avail":91133888,"statfs":{"total":95995035648,"available":93321101312,"internally_reserved":0,"allocated":2347687936,"data_stored":2346668059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":326168030},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.4570000000000001,"15min":0.78100000000000003},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.184},"max":{"1min":5.8499999999999996,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.0009999999999999,"5min":1.2569999999999999,"15min":0.70899999999999996},"min":{"1min":0.26100000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2330000000000005,"5min":8.2330000000000005,"15min":8.2330000000000005},"last":1.0780000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.4890000000000001,"15min":0.82999999999999996},"min":{"1min":0.249,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":4.4189999999999996,"5min":24.097999999999999,"15min":24.097999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.5680000000000001,"15min":0.86199999999999999},"min":{"1min":0.35299999999999998,"5min":0.26600000000000001,"15min":0.24099999999999999},"max":{"1min":3.5859999999999999,"5min":24.132999999999999,"15min":24.132999999999999},"last":1.1000000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.1950000000000001,"15min":0.72399999999999998},"min":{"1min":0.29499999999999998,"5min":0.245,"15min":0.23599999999999999},"max":{"1min":3.831,"5min":5.2910000000000004,"15min":5.2910000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.9159999999999999,"5min":1.2,"15min":0.71599999999999997},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":10.369,"5min":10.369,"15min":10.369},"last":0.248}]},{"osd":4,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.2170000000000001,"15min":0.877},"min":{"1min":0.40600000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.976,"5min":5.1189999999999998,"15min":5.7960000000000003},"last":1.163},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3680000000000001,"15min":0.92800000000000005},"min":{"1min":0.379,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.1580000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"last":2.6040000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.405,"15min":0.96699999999999997},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.29699999999999999},"max":{"1min":5.117,"5min":5.9359999999999999,"15min":5.9359999999999999},"last":2.4889999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.373,"15min":0.92100000000000004},"min":{"1min":0.32400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.6360000000000001,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":1.1499999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.27,"15min":0.89600000000000002},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.762,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":1.048},{"interface":"front","average":{"1min":1.401,"5min":1.252,"15min":0.88100000000000001},"min":{"1min":0.32200000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.3760000000000003,"5min":5.0439999999999996,"15min":5.8259999999999996},"last":0.97599999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.3919999999999999,"15min":0.997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35299999999999998},"max":{"1min":4.351,"5min":7.7240000000000002,"15min":7.7240000000000002},"last":0.442},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.423,"15min":0.98199999999999998},"min":{"1min":0.41299999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.2160000000000002,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":3.032}]}]},{"osd":2,"up_from":133,"seq":571230650477,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2915776,"kb_used_data":2742456,"kb_used_omap":6,"kb_used_meta":173305,"kb_avail":90829376,"statfs":{"total":95995035648,"available":93009281024,"internally_reserved":0,"allocated":2808274944,"data_stored":2807255281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":177464821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.78800000000000003,"15min":0.54100000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":2.883,"5min":7.476,"15min":7.476},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.1399999999999999,"15min":0.63600000000000001},"min":{"1min":0.24099999999999999,"5min":0.182,"15min":0.182},"max":{"1min":11.369,"5min":11.369,"15min":11.369},"last":1.429}]},{"osd":1,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.099,"15min":0.747},"min":{"1min":0.221,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.613,"5min":5.8200000000000003,"15min":21.920000000000002},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.0920000000000001,"15min":0.73399999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":5.7439999999999998,"5min":13.054,"15min":21.573},"last":1.1319999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.4119999999999999,"15min":0.83099999999999996},"min":{"1min":0.23999999999999999,"5min":0.219,"15min":0.219},"max":{"1min":11.24,"5min":11.24,"15min":21.66},"last":0.41299999999999998},{"interface":"front","average":{"1min":2.0099999999999998,"5min":1.3149999999999999,"15min":0.872},"min":{"1min":0.27100000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":11.271000000000001,"5min":11.271000000000001,"15min":22.341000000000001},"last":0.442}]},{"osd":4,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6139999999999999,"5min":1.4430000000000001,"15min":0.88400000000000001},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.318},"max":{"1min":14.438000000000001,"5min":14.438000000000001,"15min":14.438000000000001},"last":0.501},{"interface":"front","average":{"1min":2.1629999999999998,"5min":1.333,"15min":0.83899999999999997},"min":{"1min":0.33100000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":14.423,"5min":14.423,"15min":14.423},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":1.4239999999999999,"15min":0.86399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":31.539999999999999,"5min":31.539999999999999,"15min":31.539999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":2.778,"5min":1.405,"15min":0.875},"min":{"1min":0.437,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":31.481999999999999,"5min":31.481999999999999,"15min":31.481999999999999},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":1.1819999999999999,"15min":0.78300000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":11.454000000000001,"5min":11.454000000000001,"15min":11.454000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.1599999999999999,"15min":0.77700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.30399999999999999},"max":{"1min":12.227,"5min":12.227,"15min":12.227},"last":2.4049999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3560000000000001,"15min":0.86499999999999999},"min":{"1min":0.36499999999999999,"5min":0.36199999999999999,"15min":0.35899999999999999},"max":{"1min":10.614000000000001,"5min":10.614000000000001,"15min":10.614000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.02,"5min":1.038,"15min":0.752},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":3.1230000000000002,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":2.4409999999999998}]}]},{"osd":1,"up_from":116,"seq":498216206494,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1826488,"kb_used_data":1387632,"kb_used_omap":5,"kb_used_meta":438842,"kb_avail":91918664,"statfs":{"total":95995035648,"available":94124711936,"internally_reserved":0,"allocated":1420935168,"data_stored":1419922849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":449374488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.385,"15min":1.712},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.6269999999999998,"5min":11.257999999999999,"15min":11.257999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.635,"5min":1.234,"15min":1.3520000000000001},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":10.426},"last":0.72999999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.4610000000000001,"15min":0.77700000000000002},"min":{"1min":0.22500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.6790000000000003,"5min":9.3759999999999994,"15min":9.3759999999999994},"last":5.2110000000000003},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.054,"15min":0.60899999999999999},"min":{"1min":0.255,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":3.7799999999999998,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":1.2210000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.3380000000000001,"15min":0.89600000000000002},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.8929999999999998,"5min":8.2089999999999996,"15min":21.422000000000001},"last":5.7670000000000003},{"interface":"front","average":{"1min":0.88,"5min":1.462,"15min":0.94999999999999996},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.214},"max":{"1min":4.2450000000000001,"5min":9.4990000000000006,"15min":21.016999999999999},"last":5.1879999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.238,"15min":1.518},"min":{"1min":0.28899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.806,"5min":6.6360000000000001,"15min":8.8599999999999994},"last":6.5350000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.5840000000000001,"15min":1.6419999999999999},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.9589999999999996,"5min":9.4169999999999998,"15min":11.161},"last":5.1630000000000003}]},{"osd":5,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.482,"15min":1.5580000000000001},"min":{"1min":0.38100000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.855,"5min":10.103,"15min":10.103},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.2210000000000001,"15min":1.536},"min":{"1min":0.308,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.5,"5min":6.5730000000000004,"15min":8.125},"last":5.8250000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":4.5919999999999996,"5min":1.9910000000000001,"15min":1.7649999999999999},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":63.795999999999999,"5min":63.795999999999999,"15min":63.795999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":4.2080000000000002,"5min":1.927,"15min":1.7569999999999999},"min":{"1min":0.35699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":63.811,"5min":63.811,"15min":63.811},"last":5.8019999999999996}]},{"osd":7,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.641,"15min":1.6919999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.923,"5min":9.4429999999999996,"15min":9.4429999999999996},"last":5.117},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.3520000000000001,"15min":1.45},"min":{"1min":0.433,"5min":0.371,"15min":0.36499999999999999},"max":{"1min":3.8340000000000001,"5min":6.3529999999999998,"15min":9.5449999999999999},"last":0.505}]}]},{"osd":0,"up_from":91,"seq":390842024148,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5388540,"kb_used_data":4724212,"kb_used_omap":5,"kb_used_meta":664122,"kb_avail":88356612,"statfs":{"total":95995035648,"available":90477170688,"internally_reserved":0,"allocated":4837593088,"data_stored":4836574696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":680061018},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.341,"15min":1.482},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.6000000000000001,"5min":15.547000000000001,"15min":16.305},"last":4.383},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.454,"15min":1.7609999999999999},"min":{"1min":0.20499999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.9249999999999998,"5min":6.9889999999999999,"15min":16.184999999999999},"last":2.835}]},{"osd":2,"last update":"Fri Jun 14 23:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.24,"15min":0.73399999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.053000000000001,"5min":11.326000000000001,"15min":16.757000000000001},"last":1.722},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.4059999999999999,"15min":0.81299999999999994},"min":{"1min":0.24299999999999999,"5min":0.222,"15min":0.222},"max":{"1min":9.2769999999999992,"5min":11.519,"15min":16.055},"last":1.538}]},{"osd":3,"last update":"Fri Jun 14 23:01:23 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.373,"15min":0.75900000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":2.343,"5min":7.04,"15min":16.873999999999999},"last":2.887},{"interface":"front","average":{"1min":0.78800000000000003,"5min":1.3200000000000001,"15min":0.747},"min":{"1min":0.23699999999999999,"5min":0.224,"15min":0.224},"max":{"1min":2.5750000000000002,"5min":12.308999999999999,"15min":16.085999999999999},"last":4.4260000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:01:27 2024","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.25,"15min":1.3799999999999999},"min":{"1min":0.41699999999999998,"5min":0.307,"15min":0.27600000000000002},"max":{"1min":16.527000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":12.436},{"interface":"front","average":{"1min":1.7669999999999999,"5min":1.28,"15min":1.3660000000000001},"min":{"1min":0.42299999999999999,"5min":0.33200000000000002,"15min":0.28599999999999998},"max":{"1min":14.475,"5min":14.475,"15min":15.976000000000001},"last":2.9359999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:01:27 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.5349999999999999,"15min":1.4910000000000001},"min":{"1min":0.34399999999999997,"5min":0.30399999999999999,"15min":0.29299999999999998},"max":{"1min":2.9460000000000002,"5min":7.1269999999999998,"15min":16.108000000000001},"last":2.9540000000000002},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.446,"15min":1.5149999999999999},"min":{"1min":0.436,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.4329999999999998,"5min":11.436,"15min":16.006},"last":1.329}]},{"osd":6,"last update":"Fri Jun 14 23:01:27 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":1.427,"15min":1.466},"min":{"1min":0.42499999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":14.247,"5min":14.247,"15min":16.152000000000001},"last":2.9119999999999999},{"interface":"front","average":{"1min":2.0830000000000002,"5min":1.4810000000000001,"15min":1.4930000000000001},"min":{"1min":0.40100000000000002,"5min":0.312,"15min":0.29399999999999998},"max":{"1min":14.523999999999999,"5min":14.523999999999999,"15min":16.227},"last":1.3029999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.3680000000000001,"15min":1.448},"min":{"1min":0.379,"5min":0.35899999999999999,"15min":0.33100000000000002},"max":{"1min":3.3679999999999999,"5min":11.364000000000001,"15min":16.199999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.415,"15min":1.4990000000000001},"min":{"1min":0.47999999999999998,"5min":0.36899999999999999,"15min":0.314},"max":{"1min":3.73,"5min":12.361000000000001,"15min":16.259},"last":4.5190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4796452864,"data_stored":4796448792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1408827392,"data_stored":1408827392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2789408768,"data_stored":2789408768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2339635200,"data_stored":2339635200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3770880000,"data_stored":3770875928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2340552704,"data_stored":2340552704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3290894336,"data_stored":3290890264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1869217792,"data_stored":1869217792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:02:29.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:02:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:29.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:29.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:29.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:29.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:29.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:29.974 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:29 smithi086 ceph-mon[102754]: pgmap v815: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 840 op/s 2024-06-14T23:02:29.975 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:29 smithi086 ceph-mon[97913]: pgmap v815: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 840 op/s 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24715 9.d 9:b7390049:::benchmark_data_smithi086_152062_object24714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.661292+0000", 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 397.31012042700002, 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.018053425, 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24715 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661292+0000", 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661292+0000", 2024-06-14T23:02:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661290+0000", 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661315+0000", 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5228999999999998e-05 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661325+0000", 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8689999999999999e-06 2024-06-14T23:02:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661332+0000", 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8990000000000004e-06 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661364+0000", 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:02:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661375+0000", 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0645999999999999e-05 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.665805+0000", 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.004429807 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.666631+0000", 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082573 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:02:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.666788+0000", 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000157484 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673421+0000", 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066326279999999998 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673465+0000", 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4218999999999998e-05 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679261+0000", 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057959120000000003 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679308+0000", 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6998e-05 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679345+0000", 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7234999999999998e-05 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24723 9.d 9:b2f904ff:::benchmark_data_smithi086_152062_object24722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.664488+0000", 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 397.30692364999999, 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0150036360000001, 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24723 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664488+0000", 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664488+0000", 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664487+0000", 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664523+0000", 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5485000000000003e-05 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664538+0000", 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5412000000000001e-05 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664545+0000", 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.781e-06 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664598+0000", 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2976000000000002e-05 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664607+0000", 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6290000000000005e-06 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668290+0000", 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0036830910000001 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668426+0000", 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013543800000000001 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668573+0000", 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147498 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673521+0000", 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049472129999999998 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.676833+0000", 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0033120250000000001 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679379+0000", 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025462699999999998 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679445+0000", 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6476999999999994e-05 2024-06-14T23:02:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679492+0000", 2024-06-14T23:02:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6640000000000001e-05 2024-06-14T23:02:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24729 9.d 9:b6335b33:::benchmark_data_smithi086_152062_object24728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.667823+0000", 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 397.30358867699999, 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0118812850000001, 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24729 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667823+0000", 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667823+0000", 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667822+0000", 2024-06-14T23:02:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667847+0000", 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4666000000000001e-05 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667864+0000", 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7611000000000001e-05 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667878+0000", 2024-06-14T23:02:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3182e-05 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668333+0000", 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000454931 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668342+0000", 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1630000000000002e-06 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.669852+0000", 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.00151035 2024-06-14T23:02:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.670010+0000", 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000158257 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.670183+0000", 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017219199999999999 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.677648+0000", 2024-06-14T23:02:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007465546 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679516+0000", 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001868429 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679566+0000", 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9395999999999999e-05 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679670+0000", 2024-06-14T23:02:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010412900000000001 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679705+0000", 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4591999999999997e-05 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24730 9.d 9:be481d2c:::benchmark_data_smithi086_152062_object24729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.668425+0000", 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 397.30298680099997, 2024-06-14T23:02:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0114436330000001, 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24730 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668447+0000", 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2699000000000001e-05 2024-06-14T23:02:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668454+0000", 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6880000000000002e-06 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668459+0000", 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4940000000000003e-06 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668500+0000", 2024-06-14T23:02:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0577000000000001e-05 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668508+0000", 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5229999999999998e-06 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.671828+0000", 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.003320011 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.671965+0000", 2024-06-14T23:02:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000137081 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.672100+0000", 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013549800000000001 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.677685+0000", 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055849710000000002 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679521+0000", 2024-06-14T23:02:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001835761 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679784+0000", 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000263465 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679825+0000", 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0980999999999999e-05 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679869+0000", 2024-06-14T23:02:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3461000000000003e-05 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24738 9.d 9:b3d2a961:::benchmark_data_smithi086_152062_object24737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.671698+0000", 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 397.29971368399998, 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.009193183, 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24738 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671698+0000", 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671698+0000", 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671697+0000", 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671720+0000", 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3345000000000001e-05 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671734+0000", 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3669000000000001e-05 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671742+0000", 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7470000000000005e-06 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671822+0000", 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.051e-05 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671834+0000", 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1847e-05 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673548+0000", 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.001713718 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673671+0000", 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012297600000000001 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673796+0000", 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012489900000000001 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679254+0000", 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054580119999999999 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679525+0000", 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00027125800000000002 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680809+0000", 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012839290000000001 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680856+0000", 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7094e-05 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680891+0000", 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5438999999999997e-05 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249750 9.2 9:4ea8c052:::benchmark_data_smithi086_152062_object117268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:02:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.641870+0000", 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 181.32954223300001, 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.071249919999999994, 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249750 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641870+0000", 2024-06-14T23:02:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641870+0000", 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641869+0000", 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641874+0000", 2024-06-14T23:02:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.605e-06 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641889+0000", 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5305999999999999e-05 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641896+0000", 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7360000000000004e-06 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.680196+0000", 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038300010000000002 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.680308+0000", 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112149 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.680416+0000", 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108577 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693896+0000", 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013479804999999999 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693903+0000", 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4049999999999997e-06 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.713033+0000", 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019130945999999999 2024-06-14T23:02:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.713081+0000", 2024-06-14T23:02:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7457999999999999e-05 2024-06-14T23:02:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.713120+0000", 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8748999999999998e-05 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249768 9.2 9:4907852a:::benchmark_data_smithi086_152062_object117286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.667370+0000", 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 181.30404186199999, 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.057787167, 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249768 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667370+0000", 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667370+0000", 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667369+0000", 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667374+0000", 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5079999999999996e-06 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667384+0000", 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0035e-05 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.667391+0000", 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3339999999999998e-06 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.686242+0000", 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018851422 2024-06-14T23:02:30.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.686343+0000", 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100976 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.686444+0000", 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100659 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693937+0000", 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074934650000000004 2024-06-14T23:02:30.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.699813+0000", 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058754219999999999 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.724891+0000", 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025078644000000001 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.725125+0000", 2024-06-14T23:02:30.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00023403900000000001 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.725157+0000", 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1955999999999997e-05 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249775 9.2 9:4b4108a1:::benchmark_data_smithi086_152062_object117293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.674316+0000", 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 181.29709609400001, 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.079108740999999996, 2024-06-14T23:02:30.021 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249775 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674316+0000", 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674316+0000", 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674315+0000", 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674321+0000", 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2870000000000001e-06 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674334+0000", 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3183999999999999e-05 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674341+0000", 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7009999999999999e-06 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.690107+0000", 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015766227000000001 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.690222+0000", 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115213 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.690337+0000", 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114992 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.697790+0000", 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074525579999999998 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.705669+0000", 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0078787340000000001 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753341+0000", 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047672842999999999 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753388+0000", 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7018999999999998e-05 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753425+0000", 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6276000000000003e-05 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249790 9.2 9:429f8418:::benchmark_data_smithi086_152062_object117308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.692884+0000", 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 181.278528263, 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060712204999999998, 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249790 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692884+0000", 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692884+0000", 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692882+0000", 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692888+0000", 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7150000000000003e-06 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692898+0000", 2024-06-14T23:02:30.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0008e-05 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.692905+0000", 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6200000000000001e-06 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693956+0000", 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010514299999999999 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.694076+0000", 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012010099999999999 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.694192+0000", 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115711 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.699378+0000", 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0051855069999999998 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.705966+0000", 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0065888329999999997 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753529+0000", 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047562184 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753564+0000", 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5240000000000001e-05 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753596+0000", 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2150999999999998e-05 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:254239 9.2 9:4c339b56:::benchmark_data_smithi086_152062_object121757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:33.777110+0000", 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 176.194301854, 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.054250944000000002, 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.033 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 254239 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777110+0000", 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777110+0000", 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777108+0000", 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777119+0000", 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0147e-05 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777135+0000", 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6742999999999999e-05 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.777145+0000", 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.499e-06 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.783108+0000", 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059630259999999997 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.783213+0000", 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104886 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.783322+0000", 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108861 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.798666+0000", 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015344750000000001 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.805636+0000", 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069697200000000004 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.831285+0000", 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025648908000000002 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.831325+0000", 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0095999999999997e-05 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.831361+0000", 2024-06-14T23:02:30.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6087999999999997e-05 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.35580.0:254354 9.f e174/140 9:f2e6ba9e:::benchmark_data_smithi086_152062_object121872:head v 174'15910, mlcod=174'15907)", 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:33.940059+0000", 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 176.03135334199999, 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.055034489999999998, 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.940059+0000", 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.940059+0000", 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.940058+0000", 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.940064+0000", 2024-06-14T23:02:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5910000000000003e-06 2024-06-14T23:02:30.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.940070+0000", 2024-06-14T23:02:30.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.015e-06 2024-06-14T23:02:30.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:30.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.940074+0000", 2024-06-14T23:02:30.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3749999999999996e-06 2024-06-14T23:02:30.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.943150+0000", 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003076086 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.943168+0000", 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7351999999999999e-05 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.995052+0000", 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051884595999999998 2024-06-14T23:02:30.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:33.995093+0000", 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0886000000000001e-05 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:257294 9.4 9:28fa20c2:::benchmark_data_smithi086_152062_object124812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:37.247652+0000", 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 172.723760183, 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.065028322999999999, 2024-06-14T23:02:30.044 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 257294 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247652+0000", 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247652+0000", 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247651+0000", 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247654+0000", 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.915e-06 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247661+0000", 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4899999999999997e-06 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.247666+0000", 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4519999999999996e-06 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.272333+0000", 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024666819999999999 2024-06-14T23:02:30.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.272703+0000", 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00037041800000000002 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.273193+0000", 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00048966000000000005 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.281391+0000", 2024-06-14T23:02:30.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0081982349999999999 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.283700+0000", 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023091129999999998 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.312394+0000", 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028693315 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.312635+0000", 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00024135899999999999 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:37.312680+0000", 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.507e-05 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:22285 10.2 10:48f891f3:::benchmark_data_smithi086_189745_object22284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:00:44.559598+0000", 2024-06-14T23:02:30.050 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.41181438, 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051528321000000002, 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 22285 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.559598+0000", 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.559598+0000", 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.559595+0000", 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.559626+0000", 2024-06-14T23:02:30.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0069999999999998e-05 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.559642+0000", 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6175e-05 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.559649+0000", 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8349999999999998e-06 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.559684+0000", 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5472999999999997e-05 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.559817+0000", 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013344799999999999 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.559940+0000", 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012236300000000001 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.561169+0000", 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001229061 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.561393+0000", 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000224479 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.611047+0000", 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.049653769 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.611087+0000", 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9538e-05 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:00:44.611126+0000", 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9233999999999997e-05 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.056 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:55530 10.2 10:4a4d3962:::benchmark_data_smithi086_189745_object55529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:02.303970+0000", 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 87.667441652999997, 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.056785932999999997, 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 55530 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.303970+0000", 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.303970+0000", 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.303969+0000", 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.303991+0000", 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1654e-05 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.304003+0000", 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2388000000000001e-05 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.304011+0000", 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.729e-06 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.304050+0000", 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8965999999999999e-05 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.304159+0000", 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109121 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.304256+0000", 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6601000000000006e-05 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.306031+0000", 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001775205 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.306182+0000", 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015092799999999999 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.360657+0000", 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.054475618000000003 2024-06-14T23:02:30.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.360716+0000", 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8736999999999997e-05 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:02.360756+0000", 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0170000000000003e-05 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:90605 10.2 10:43913749:::benchmark_data_smithi086_189745_object90604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:28.678591+0000", 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 61.292820704999997, 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.057962947000000001, 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 90605 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.678591+0000", 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.678591+0000", 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.678590+0000", 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.678612+0000", 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1466000000000001e-05 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.678626+0000", 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3879e-05 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.678633+0000", 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0500000000000003e-06 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.685512+0000", 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0068796430000000004 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.685861+0000", 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00034824299999999998 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.689330+0000", 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034697170000000002 2024-06-14T23:02:30.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.694872+0000", 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055418919999999997 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.698772+0000", 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038995129999999999 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.736273+0000", 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037501161999999998 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.736418+0000", 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014535000000000001 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:28.736554+0000", 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013596300000000001 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:117237 10.2 10:4e671e84:::benchmark_data_smithi086_189745_object117236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:53.026979+0000", 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 36.944432513000002, 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.096224995999999993, 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117237 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.026979+0000", 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.026979+0000", 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.026978+0000", 2024-06-14T23:02:30.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027001+0000", 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2802999999999998e-05 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027016+0000", 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5668999999999999e-05 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027023+0000", 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9890000000000003e-06 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027062+0000", 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8475e-05 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027171+0000", 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109152 2024-06-14T23:02:30.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027268+0000", 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6864e-05 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.035630+0000", 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083618860000000007 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.042402+0000", 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067718520000000001 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.123118+0000", 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.080716689999999994 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.123175+0000", 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6718999999999997e-05 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.123204+0000", 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9386999999999998e-05 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:117241 10.2 10:40bacbc2:::benchmark_data_smithi086_189745_object117240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:53.030165+0000", 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 36.941246552000003, 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.097320450000000003, 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117241 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030165+0000", 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030165+0000", 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030164+0000", 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030214+0000", 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0188999999999997e-05 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030233+0000", 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8879999999999999e-05 2024-06-14T23:02:30.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030240+0000", 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2690000000000002e-06 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.031168+0000", 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000928151 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.031297+0000", 2024-06-14T23:02:30.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012866900000000001 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.031433+0000", 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013556899999999999 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.035638+0000", 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042048800000000002 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.042528+0000", 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0068906940000000002 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.127425+0000", 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.084896588999999995 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.127459+0000", 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3997e-05 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.127486+0000", 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7046000000000002e-05 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:117247 10.2 10:4a6dbd53:::benchmark_data_smithi086_189745_object117246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:53.038191+0000", 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 36.933221308, 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.096470944000000003, 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.081 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117247 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038191+0000", 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038191+0000", 2024-06-14T23:02:30.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038189+0000", 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038215+0000", 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5704e-05 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038233+0000", 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8318999999999999e-05 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038241+0000", 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.276e-06 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038326+0000", 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4302999999999996e-05 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038564+0000", 2024-06-14T23:02:30.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000238741 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038720+0000", 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000155718 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.042551+0000", 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038304789999999999 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.043231+0000", 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068008399999999996 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.134573+0000", 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.091342603999999994 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.134622+0000", 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8588000000000003e-05 2024-06-14T23:02:30.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.134662+0000", 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9749000000000002e-05 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:150612 10.2 10:473b038c:::benchmark_data_smithi086_189745_object19644:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:02:27.604540+0000", 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 2.366871669, 2024-06-14T23:02:30.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.053662131000000002, 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 150612 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604540+0000", 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604540+0000", 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604538+0000", 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604544+0000", 2024-06-14T23:02:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0859999999999998e-06 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604554+0000", 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9229999999999997e-06 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604560+0000", 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8950000000000003e-06 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.607691+0000", 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031311590000000001 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.607962+0000", 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00027103799999999998 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.608298+0000", 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00033543100000000002 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.621228+0000", 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012929942999999999 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.643578+0000", 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022349974000000002 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.657952+0000", 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014374203 2024-06-14T23:02:30.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.658095+0000", 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000143177 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.658202+0000", 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010755300000000001 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:150613 10.2 10:4613e5d8:::benchmark_data_smithi086_189745_object19645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:02:27.604570+0000", 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 2.3668418899999999, 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.053867962999999998, 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 150613 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:02:30.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604570+0000", 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604570+0000", 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604570+0000", 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:02:30.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604573+0000", 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9950000000000001e-06 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604578+0000", 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7520000000000002e-06 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:02:30.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.604583+0000", 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5410000000000002e-06 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.621315+0000", 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016732196000000001 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.622111+0000", 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00079561499999999995 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.622666+0000", 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00055500099999999995 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.643742+0000", 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021075824999999999 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.643900+0000", 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015812700000000001 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.658227+0000", 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.01432755 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.658335+0000", 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010807299999999999 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:02:27.658438+0000", 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102728 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.100 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.101 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:02:30.101 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:02:30.101 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:02:30.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:30.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:30.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:30.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:02:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:29 smithi137 ceph-mon[90634]: pgmap v815: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 840 op/s 2024-06-14T23:02:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:30.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:30.357 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:30.390 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:30.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:30.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:30.539 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:30.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:30.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:30.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:30.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:30.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:30.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:30.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:30.831 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:02:30.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:30.940 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:30.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:31.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:31.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:31.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:31.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1545592965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:31.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:30 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1545592965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:31.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:30 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:31.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:31.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T23:02:31.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:31.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1545592965' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:31.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:30 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:31.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:31.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:31.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:31.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:31.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:31.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:31.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:31.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:31.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:31.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:31.756 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:31.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:31.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:31.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:31.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:32.027 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:32.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:32.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[102754]: pgmap v816: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T23:02:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:02:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:02:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:02:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[102754]: osdmap e207: 8 total, 8 up, 8 in 2024-06-14T23:02:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[97913]: pgmap v816: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T23:02:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:02:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:02:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:02:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:31 smithi086 ceph-mon[97913]: osdmap e207: 8 total, 8 up, 8 in 2024-06-14T23:02:32.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:32.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:32.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:31 smithi137 ceph-mon[90634]: pgmap v816: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T23:02:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:31 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:02:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:31 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:02:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:31 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:02:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:31 smithi137 ceph-mon[90634]: osdmap e207: 8 total, 8 up, 8 in 2024-06-14T23:02:32.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:32.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:32.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:32.358 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:32.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:32.439 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:32.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:32.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:32.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:32.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:32.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:32.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:32.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:32.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:32.868 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:32.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:32.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:33.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:33.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:33.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:33.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:33.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:33.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:33.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:33.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:33.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:33.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:33.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:33.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:33.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:33.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:33.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:33.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:33.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:33.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:33.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:33.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:33.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:33.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:33.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:34.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:34.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:34.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:33 smithi086 ceph-mon[102754]: pgmap v818: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 958 op/s 2024-06-14T23:02:34.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:33 smithi086 ceph-mon[97913]: pgmap v818: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 958 op/s 2024-06-14T23:02:34.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:34.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:34.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:33 smithi137 ceph-mon[90634]: pgmap v818: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 958 op/s 2024-06-14T23:02:34.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:34.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:34.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:34.368 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:34.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:02:34.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:34.427 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:34.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:34.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:34.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:34.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:34.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:34.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:34.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:34.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:34.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:34.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:34.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:34.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:35.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:35.000 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:02:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:02:34] "GET /metrics HTTP/1.1" 200 36592 "" "Prometheus/2.43.0" 2024-06-14T23:02:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:35.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:35.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:35.137 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:35.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:35.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:35.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:35.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:35.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:02:35.389 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:02:35.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:35.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:35.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:35.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:35.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:35.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:35.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:35.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:35.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:35.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:35.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:35.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:35 smithi086 ceph-mon[102754]: pgmap v819: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 938 op/s 2024-06-14T23:02:35.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:35 smithi086 ceph-mon[97913]: pgmap v819: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 938 op/s 2024-06-14T23:02:35.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:35.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:35.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:35.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:36.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:36.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:36.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:36.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:36.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:35 smithi137 ceph-mon[90634]: pgmap v819: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 938 op/s 2024-06-14T23:02:36.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:36.343 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:36.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:36.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:36.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:36.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:36.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:36.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:36.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:36.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:36.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:36.748 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:36.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:36.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:36.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:36.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:36.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:37.023 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:37.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:37.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:36 smithi086 ceph-mon[97913]: from='client.45962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:02:37.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:36 smithi086 ceph-mon[102754]: from='client.45962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:02:37.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:37.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:37.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:37.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:36 smithi137 ceph-mon[90634]: from='client.45962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:02:37.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:37.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:37.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:37.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:37.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:37.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:37.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:37.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:37.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:37.701 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:37.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:37.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:37.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:37.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:37.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:37.979 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:37.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:38.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:38.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:37 smithi086 ceph-mon[97913]: pgmap v820: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:02:38.109 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:37 smithi086 ceph-mon[102754]: pgmap v820: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:02:38.111 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":819,"stamp":"2024-06-14T23:02:34.722117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6974609374,"num_objects":106440,"num_object_clones":0,"num_object_copies":319320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155577,"num_write_kb":8381790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132742,"ondisk_log_size":132742,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":25751284,"kb_used_data":20577960,"kb_used_omap":29,"kb_used_meta":5172770,"kb_avail":724209932,"statfs":{"total":767960285184,"available":741590970368,"internally_reserved":0,"allocated":21071831040,"data_stored":21063624986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":5296917157},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,6,5,7,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-615186432,"num_objects":-9387,"num_object_clones":0,"num_object_copies":-28161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9387,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003824"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":136,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.933157+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:02:14.933157+0000","last_peered":"2024-06-14T23:02:14.933157+0000","last_clean":"2024-06-14T23:02:14.933157+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:02:14.933157+0000","last_undegraded":"2024-06-14T23:02:14.933157+0000","last_fullsized":"2024-06-14T23:02:14.933157+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":170,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928666+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T23:02:15.928666+0000","last_peered":"2024-06-14T23:02:15.928666+0000","last_clean":"2024-06-14T23:02:15.928666+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T23:02:15.928666+0000","last_undegraded":"2024-06-14T23:02:15.928666+0000","last_fullsized":"2024-06-14T23:02:15.928666+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":167,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.930165+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:02:15.930165+0000","last_peered":"2024-06-14T23:02:15.930165+0000","last_clean":"2024-06-14T23:02:15.930165+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:02:15.930165+0000","last_undegraded":"2024-06-14T23:02:15.930165+0000","last_fullsized":"2024-06-14T23:02:15.930165+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":159,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927516+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:02:15.927516+0000","last_peered":"2024-06-14T23:02:15.927516+0000","last_clean":"2024-06-14T23:02:15.927516+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:02:15.927516+0000","last_undegraded":"2024-06-14T23:02:15.927516+0000","last_fullsized":"2024-06-14T23:02:15.927516+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":160,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927262+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T23:02:15.927262+0000","last_peered":"2024-06-14T23:02:15.927262+0000","last_clean":"2024-06-14T23:02:15.927262+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T23:02:15.927262+0000","last_undegraded":"2024-06-14T23:02:15.927262+0000","last_fullsized":"2024-06-14T23:02:15.927262+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":74,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928626+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T23:02:15.928626+0000","last_peered":"2024-06-14T23:02:15.928626+0000","last_clean":"2024-06-14T23:02:15.928626+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T23:02:15.928626+0000","last_undegraded":"2024-06-14T23:02:15.928626+0000","last_fullsized":"2024-06-14T23:02:15.928626+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":161,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.933405+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:02:15.933405+0000","last_peered":"2024-06-14T23:02:15.933405+0000","last_clean":"2024-06-14T23:02:15.933405+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:02:15.933405+0000","last_undegraded":"2024-06-14T23:02:15.933405+0000","last_fullsized":"2024-06-14T23:02:15.933405+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":176,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.936570+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T23:02:15.936570+0000","last_peered":"2024-06-14T23:02:15.936570+0000","last_clean":"2024-06-14T23:02:15.936570+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T23:02:15.936570+0000","last_undegraded":"2024-06-14T23:02:15.936570+0000","last_fullsized":"2024-06-14T23:02:15.936570+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"207'9723","reported_seq":9751,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:34.300665+0000","last_change":"2024-06-14T23:00:27.800011+0000","last_active":"2024-06-14T23:02:34.300665+0000","last_peered":"2024-06-14T23:02:34.300665+0000","last_clean":"2024-06-14T23:02:34.300665+0000","last_became_active":"2024-06-14T23:00:27.799716+0000","last_became_peered":"2024-06-14T23:00:27.799716+0000","last_unstale":"2024-06-14T23:02:34.300665+0000","last_undegraded":"2024-06-14T23:02:34.300665+0000","last_fullsized":"2024-06-14T23:02:34.300665+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9723,"log_dups_size":0,"ondisk_log_size":9723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:41:34.619251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430637056,"num_objects":6571,"num_object_clones":0,"num_object_copies":19713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9723,"num_write_kb":521408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":157,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928960+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:02:15.928960+0000","last_peered":"2024-06-14T23:02:15.928960+0000","last_clean":"2024-06-14T23:02:15.928960+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:02:15.928960+0000","last_undegraded":"2024-06-14T23:02:15.928960+0000","last_fullsized":"2024-06-14T23:02:15.928960+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":150,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.929671+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T23:02:15.929671+0000","last_peered":"2024-06-14T23:02:15.929671+0000","last_clean":"2024-06-14T23:02:15.929671+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T23:02:15.929671+0000","last_undegraded":"2024-06-14T23:02:15.929671+0000","last_fullsized":"2024-06-14T23:02:15.929671+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"207'9786","reported_seq":9812,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:33.871192+0000","last_change":"2024-06-14T23:00:27.798857+0000","last_active":"2024-06-14T23:02:33.871192+0000","last_peered":"2024-06-14T23:02:33.871192+0000","last_clean":"2024-06-14T23:02:33.871192+0000","last_became_active":"2024-06-14T23:00:27.798566+0000","last_became_peered":"2024-06-14T23:00:27.798566+0000","last_unstale":"2024-06-14T23:02:33.871192+0000","last_undegraded":"2024-06-14T23:02:33.871192+0000","last_fullsized":"2024-06-14T23:02:33.871192+0000","mapping_epoch":186,"log_start":"191'2000","ondisk_log_start":"191'2000","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7786,"log_dups_size":0,"ondisk_log_size":7786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:38.237668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432668672,"num_objects":6602,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":158,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928716+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:02:15.928716+0000","last_peered":"2024-06-14T23:02:15.928716+0000","last_clean":"2024-06-14T23:02:15.928716+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:02:15.928716+0000","last_undegraded":"2024-06-14T23:02:15.928716+0000","last_fullsized":"2024-06-14T23:02:15.928716+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":184,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928773+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:02:15.928773+0000","last_peered":"2024-06-14T23:02:15.928773+0000","last_clean":"2024-06-14T23:02:15.928773+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:02:15.928773+0000","last_undegraded":"2024-06-14T23:02:15.928773+0000","last_fullsized":"2024-06-14T23:02:15.928773+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"207'9589","reported_seq":9619,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:31.960904+0000","last_change":"2024-06-14T23:00:27.797882+0000","last_active":"2024-06-14T23:02:31.960904+0000","last_peered":"2024-06-14T23:02:31.960904+0000","last_clean":"2024-06-14T23:02:31.960904+0000","last_became_active":"2024-06-14T23:00:27.797600+0000","last_became_peered":"2024-06-14T23:00:27.797600+0000","last_unstale":"2024-06-14T23:02:31.960904+0000","last_undegraded":"2024-06-14T23:02:31.960904+0000","last_fullsized":"2024-06-14T23:02:31.960904+0000","mapping_epoch":186,"log_start":"191'1800","ondisk_log_start":"191'1800","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:04:20.589709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433913856,"num_objects":6621,"num_object_clones":0,"num_object_copies":19863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9589,"num_write_kb":518720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":85,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.930095+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T23:02:15.930095+0000","last_peered":"2024-06-14T23:02:15.930095+0000","last_clean":"2024-06-14T23:02:15.930095+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T23:02:15.930095+0000","last_undegraded":"2024-06-14T23:02:15.930095+0000","last_fullsized":"2024-06-14T23:02:15.930095+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":167,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.931321+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T23:02:15.931321+0000","last_peered":"2024-06-14T23:02:15.931321+0000","last_clean":"2024-06-14T23:02:15.931321+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T23:02:15.931321+0000","last_undegraded":"2024-06-14T23:02:15.931321+0000","last_fullsized":"2024-06-14T23:02:15.931321+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"206'9757","reported_seq":9785,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.332509+0000","last_change":"2024-06-14T23:00:27.794220+0000","last_active":"2024-06-14T23:02:30.332509+0000","last_peered":"2024-06-14T23:02:30.332509+0000","last_clean":"2024-06-14T23:02:30.332509+0000","last_became_active":"2024-06-14T23:00:27.793960+0000","last_became_peered":"2024-06-14T23:00:27.793960+0000","last_unstale":"2024-06-14T23:02:30.332509+0000","last_undegraded":"2024-06-14T23:02:30.332509+0000","last_fullsized":"2024-06-14T23:02:30.332509+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9757,"log_dups_size":0,"ondisk_log_size":9757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:04:26.933537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460783616,"num_objects":7031,"num_object_clones":0,"num_object_copies":21093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9757,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":168,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.929950+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T23:02:15.929950+0000","last_peered":"2024-06-14T23:02:15.929950+0000","last_clean":"2024-06-14T23:02:15.929950+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T23:02:15.929950+0000","last_undegraded":"2024-06-14T23:02:15.929950+0000","last_fullsized":"2024-06-14T23:02:15.929950+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":160,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.926537+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T23:02:15.926537+0000","last_peered":"2024-06-14T23:02:15.926537+0000","last_clean":"2024-06-14T23:02:15.926537+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T23:02:15.926537+0000","last_undegraded":"2024-06-14T23:02:15.926537+0000","last_fullsized":"2024-06-14T23:02:15.926537+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"206'9645","reported_seq":9672,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.012745+0000","last_change":"2024-06-14T23:00:27.797355+0000","last_active":"2024-06-14T23:02:30.012745+0000","last_peered":"2024-06-14T23:02:30.012745+0000","last_clean":"2024-06-14T23:02:30.012745+0000","last_became_active":"2024-06-14T23:00:27.796666+0000","last_became_peered":"2024-06-14T23:00:27.796666+0000","last_unstale":"2024-06-14T23:02:30.012745+0000","last_undegraded":"2024-06-14T23:02:30.012745+0000","last_fullsized":"2024-06-14T23:02:30.012745+0000","mapping_epoch":186,"log_start":"191'1300","ondisk_log_start":"191'1300","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":8345,"log_dups_size":0,"ondisk_log_size":8345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:57:29.139721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451084288,"num_objects":6883,"num_object_clones":0,"num_object_copies":20649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9645,"num_write_kb":528896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":505,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928992+0000","last_change":"2024-06-14T23:00:30.637455+0000","last_active":"2024-06-14T23:02:15.928992+0000","last_peered":"2024-06-14T23:02:15.928992+0000","last_clean":"2024-06-14T23:02:15.928992+0000","last_became_active":"2024-06-14T23:00:17.398732+0000","last_became_peered":"2024-06-14T23:00:17.398732+0000","last_unstale":"2024-06-14T23:02:15.928992+0000","last_undegraded":"2024-06-14T23:02:15.928992+0000","last_fullsized":"2024-06-14T23:02:15.928992+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.637373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.637373+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:10.033075+0000","scrub_duration":0.0016781470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":76,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.926134+0000","last_change":"2024-06-14T22:58:09.752985+0000","last_active":"2024-06-14T23:02:15.926134+0000","last_peered":"2024-06-14T23:02:15.926134+0000","last_clean":"2024-06-14T23:02:15.926134+0000","last_became_active":"2024-06-14T22:58:09.752630+0000","last_became_peered":"2024-06-14T22:58:09.752630+0000","last_unstale":"2024-06-14T23:02:15.926134+0000","last_undegraded":"2024-06-14T23:02:15.926134+0000","last_fullsized":"2024-06-14T23:02:15.926134+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:22.320986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":167,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.932156+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T23:02:15.932156+0000","last_peered":"2024-06-14T23:02:15.932156+0000","last_clean":"2024-06-14T23:02:15.932156+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T23:02:15.932156+0000","last_undegraded":"2024-06-14T23:02:15.932156+0000","last_fullsized":"2024-06-14T23:02:15.932156+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"206'9629","reported_seq":9658,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.298788+0000","last_change":"2024-06-14T23:00:27.797468+0000","last_active":"2024-06-14T23:02:30.298788+0000","last_peered":"2024-06-14T23:02:30.298788+0000","last_clean":"2024-06-14T23:02:30.298788+0000","last_became_active":"2024-06-14T23:00:27.797180+0000","last_became_peered":"2024-06-14T23:00:27.797180+0000","last_unstale":"2024-06-14T23:02:30.298788+0000","last_undegraded":"2024-06-14T23:02:30.298788+0000","last_fullsized":"2024-06-14T23:02:30.298788+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:56:13.872188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448987136,"num_objects":6851,"num_object_clones":0,"num_object_copies":20553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":527360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":504,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.929529+0000","last_change":"2024-06-14T23:00:33.815761+0000","last_active":"2024-06-14T23:02:15.929529+0000","last_peered":"2024-06-14T23:02:15.929529+0000","last_clean":"2024-06-14T23:02:15.929529+0000","last_became_active":"2024-06-14T23:00:17.400781+0000","last_became_peered":"2024-06-14T23:00:17.400781+0000","last_unstale":"2024-06-14T23:02:15.929529+0000","last_undegraded":"2024-06-14T23:02:15.929529+0000","last_fullsized":"2024-06-14T23:02:15.929529+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:21:41.196504+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":168,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.931433+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T23:02:15.931433+0000","last_peered":"2024-06-14T23:02:15.931433+0000","last_clean":"2024-06-14T23:02:15.931433+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T23:02:15.931433+0000","last_undegraded":"2024-06-14T23:02:15.931433+0000","last_fullsized":"2024-06-14T23:02:15.931433+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":177,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928035+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T23:02:15.928035+0000","last_peered":"2024-06-14T23:02:15.928035+0000","last_clean":"2024-06-14T23:02:15.928035+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T23:02:15.928035+0000","last_undegraded":"2024-06-14T23:02:15.928035+0000","last_fullsized":"2024-06-14T23:02:15.928035+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"207'9734","reported_seq":9763,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:33.856045+0000","last_change":"2024-06-14T23:00:27.802274+0000","last_active":"2024-06-14T23:02:33.856045+0000","last_peered":"2024-06-14T23:02:33.856045+0000","last_clean":"2024-06-14T23:02:33.856045+0000","last_became_active":"2024-06-14T23:00:27.801956+0000","last_became_peered":"2024-06-14T23:00:27.801956+0000","last_unstale":"2024-06-14T23:02:33.856045+0000","last_undegraded":"2024-06-14T23:02:33.856045+0000","last_fullsized":"2024-06-14T23:02:33.856045+0000","mapping_epoch":186,"log_start":"191'2000","ondisk_log_start":"191'2000","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:23.791512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433455104,"num_objects":6614,"num_object_clones":0,"num_object_copies":19842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9734,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":168,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928848+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T23:02:15.928848+0000","last_peered":"2024-06-14T23:02:15.928848+0000","last_clean":"2024-06-14T23:02:15.928848+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T23:02:15.928848+0000","last_undegraded":"2024-06-14T23:02:15.928848+0000","last_fullsized":"2024-06-14T23:02:15.928848+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":167,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927939+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T23:02:15.927939+0000","last_peered":"2024-06-14T23:02:15.927939+0000","last_clean":"2024-06-14T23:02:15.927939+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T23:02:15.927939+0000","last_undegraded":"2024-06-14T23:02:15.927939+0000","last_fullsized":"2024-06-14T23:02:15.927939+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"207'9780","reported_seq":9809,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:33.798611+0000","last_change":"2024-06-14T23:00:27.801110+0000","last_active":"2024-06-14T23:02:33.798611+0000","last_peered":"2024-06-14T23:02:33.798611+0000","last_clean":"2024-06-14T23:02:33.798611+0000","last_became_active":"2024-06-14T23:00:27.800817+0000","last_became_peered":"2024-06-14T23:00:27.800817+0000","last_unstale":"2024-06-14T23:02:33.798611+0000","last_undegraded":"2024-06-14T23:02:33.798611+0000","last_fullsized":"2024-06-14T23:02:33.798611+0000","mapping_epoch":186,"log_start":"192'2900","ondisk_log_start":"192'2900","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:48:04.422151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430440448,"num_objects":6568,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":504,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.929628+0000","last_change":"2024-06-14T23:00:31.843436+0000","last_active":"2024-06-14T23:02:15.929628+0000","last_peered":"2024-06-14T23:02:15.929628+0000","last_clean":"2024-06-14T23:02:15.929628+0000","last_became_active":"2024-06-14T23:00:17.398181+0000","last_became_peered":"2024-06-14T23:00:17.398181+0000","last_unstale":"2024-06-14T23:02:15.929628+0000","last_undegraded":"2024-06-14T23:02:15.929628+0000","last_fullsized":"2024-06-14T23:02:15.929628+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:05:06.069424+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":162,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.930347+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:02:15.930347+0000","last_peered":"2024-06-14T23:02:15.930347+0000","last_clean":"2024-06-14T23:02:15.930347+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:02:15.930347+0000","last_undegraded":"2024-06-14T23:02:15.930347+0000","last_fullsized":"2024-06-14T23:02:15.930347+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":174,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.933386+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:02:14.933386+0000","last_peered":"2024-06-14T23:02:14.933386+0000","last_clean":"2024-06-14T23:02:14.933386+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:02:14.933386+0000","last_undegraded":"2024-06-14T23:02:14.933386+0000","last_fullsized":"2024-06-14T23:02:14.933386+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"207'9942","reported_seq":9971,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:33.871443+0000","last_change":"2024-06-14T23:00:27.797497+0000","last_active":"2024-06-14T23:02:33.871443+0000","last_peered":"2024-06-14T23:02:33.871443+0000","last_clean":"2024-06-14T23:02:33.871443+0000","last_became_active":"2024-06-14T23:00:27.797221+0000","last_became_peered":"2024-06-14T23:00:27.797221+0000","last_unstale":"2024-06-14T23:02:33.871443+0000","last_undegraded":"2024-06-14T23:02:33.871443+0000","last_fullsized":"2024-06-14T23:02:33.871443+0000","mapping_epoch":186,"log_start":"191'2200","ondisk_log_start":"191'2200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7742,"log_dups_size":0,"ondisk_log_size":7742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:24:58.530131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433717248,"num_objects":6618,"num_object_clones":0,"num_object_copies":19854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9942,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":497,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.926802+0000","last_change":"2024-06-14T23:00:30.610273+0000","last_active":"2024-06-14T23:02:15.926802+0000","last_peered":"2024-06-14T23:02:15.926802+0000","last_clean":"2024-06-14T23:02:15.926802+0000","last_became_active":"2024-06-14T23:00:09.766095+0000","last_became_peered":"2024-06-14T23:00:09.766095+0000","last_unstale":"2024-06-14T23:02:15.926802+0000","last_undegraded":"2024-06-14T23:02:15.926802+0000","last_fullsized":"2024-06-14T23:02:15.926802+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.610180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.610180+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:53.521891+0000","scrub_duration":0.00217396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":172,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.930023+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T23:02:15.930023+0000","last_peered":"2024-06-14T23:02:15.930023+0000","last_clean":"2024-06-14T23:02:15.930023+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T23:02:15.930023+0000","last_undegraded":"2024-06-14T23:02:15.930023+0000","last_fullsized":"2024-06-14T23:02:15.930023+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":111,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.929746+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T23:02:15.929746+0000","last_peered":"2024-06-14T23:02:15.929746+0000","last_clean":"2024-06-14T23:02:15.929746+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T23:02:15.929746+0000","last_undegraded":"2024-06-14T23:02:15.929746+0000","last_fullsized":"2024-06-14T23:02:15.929746+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":150,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.931139+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T23:02:15.931139+0000","last_peered":"2024-06-14T23:02:15.931139+0000","last_clean":"2024-06-14T23:02:15.931139+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T23:02:15.931139+0000","last_undegraded":"2024-06-14T23:02:15.931139+0000","last_fullsized":"2024-06-14T23:02:15.931139+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"207'9590","reported_seq":9620,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:31.950667+0000","last_change":"2024-06-14T23:00:27.797246+0000","last_active":"2024-06-14T23:02:31.950667+0000","last_peered":"2024-06-14T23:02:31.950667+0000","last_clean":"2024-06-14T23:02:31.950667+0000","last_became_active":"2024-06-14T23:00:27.796988+0000","last_became_peered":"2024-06-14T23:00:27.796988+0000","last_unstale":"2024-06-14T23:02:31.950667+0000","last_undegraded":"2024-06-14T23:02:31.950667+0000","last_fullsized":"2024-06-14T23:02:31.950667+0000","mapping_epoch":186,"log_start":"191'1800","ondisk_log_start":"191'1800","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7790,"log_dups_size":0,"ondisk_log_size":7790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:17.545715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434372608,"num_objects":6628,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9590,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":505,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.921162+0000","last_change":"2024-06-14T23:00:30.719758+0000","last_active":"2024-06-14T23:02:15.921162+0000","last_peered":"2024-06-14T23:02:15.921162+0000","last_clean":"2024-06-14T23:02:15.921162+0000","last_became_active":"2024-06-14T23:00:17.403817+0000","last_became_peered":"2024-06-14T23:00:17.403817+0000","last_unstale":"2024-06-14T23:02:15.921162+0000","last_undegraded":"2024-06-14T23:02:15.921162+0000","last_fullsized":"2024-06-14T23:02:15.921162+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:03:47.728089+0000","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.922931+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:02:14.922931+0000","last_peered":"2024-06-14T23:02:14.922931+0000","last_clean":"2024-06-14T23:02:14.922931+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:02:14.922931+0000","last_undegraded":"2024-06-14T23:02:14.922931+0000","last_fullsized":"2024-06-14T23:02:14.922931+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":160,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.924798+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T23:02:15.924798+0000","last_peered":"2024-06-14T23:02:15.924798+0000","last_clean":"2024-06-14T23:02:15.924798+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T23:02:15.924798+0000","last_undegraded":"2024-06-14T23:02:15.924798+0000","last_fullsized":"2024-06-14T23:02:15.924798+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":174,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927391+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T23:02:15.927391+0000","last_peered":"2024-06-14T23:02:15.927391+0000","last_clean":"2024-06-14T23:02:15.927391+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T23:02:15.927391+0000","last_undegraded":"2024-06-14T23:02:15.927391+0000","last_fullsized":"2024-06-14T23:02:15.927391+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"207'9632","reported_seq":9661,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:34.275646+0000","last_change":"2024-06-14T23:00:27.801838+0000","last_active":"2024-06-14T23:02:34.275646+0000","last_peered":"2024-06-14T23:02:34.275646+0000","last_clean":"2024-06-14T23:02:34.275646+0000","last_became_active":"2024-06-14T23:00:27.801477+0000","last_became_peered":"2024-06-14T23:00:27.801477+0000","last_unstale":"2024-06-14T23:02:34.275646+0000","last_undegraded":"2024-06-14T23:02:34.275646+0000","last_fullsized":"2024-06-14T23:02:34.275646+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9632,"log_dups_size":0,"ondisk_log_size":9632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:59:17.239200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426508288,"num_objects":6508,"num_object_clones":0,"num_object_copies":19524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9632,"num_write_kb":516480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":514,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:16.946721+0000","last_change":"2024-06-14T23:00:31.643024+0000","last_active":"2024-06-14T23:02:16.946721+0000","last_peered":"2024-06-14T23:02:16.946721+0000","last_clean":"2024-06-14T23:02:16.946721+0000","last_became_active":"2024-06-14T23:00:17.400819+0000","last_became_peered":"2024-06-14T23:00:17.400819+0000","last_unstale":"2024-06-14T23:02:16.946721+0000","last_undegraded":"2024-06-14T23:02:16.946721+0000","last_fullsized":"2024-06-14T23:02:16.946721+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.642940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.642940+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:52:45.475940+0000","scrub_duration":0.001647253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":160,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.940877+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T23:02:14.940877+0000","last_peered":"2024-06-14T23:02:14.940877+0000","last_clean":"2024-06-14T23:02:14.940877+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T23:02:14.940877+0000","last_undegraded":"2024-06-14T23:02:14.940877+0000","last_fullsized":"2024-06-14T23:02:14.940877+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":175,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.923496+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T23:02:14.923496+0000","last_peered":"2024-06-14T23:02:14.923496+0000","last_clean":"2024-06-14T23:02:14.923496+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T23:02:14.923496+0000","last_undegraded":"2024-06-14T23:02:14.923496+0000","last_fullsized":"2024-06-14T23:02:14.923496+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":160,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.921234+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:02:15.921234+0000","last_peered":"2024-06-14T23:02:15.921234+0000","last_clean":"2024-06-14T23:02:15.921234+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:02:15.921234+0000","last_undegraded":"2024-06-14T23:02:15.921234+0000","last_fullsized":"2024-06-14T23:02:15.921234+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"207'9764","reported_seq":9792,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:33.862629+0000","last_change":"2024-06-14T23:00:27.799968+0000","last_active":"2024-06-14T23:02:33.862629+0000","last_peered":"2024-06-14T23:02:33.862629+0000","last_clean":"2024-06-14T23:02:33.862629+0000","last_became_active":"2024-06-14T23:00:27.799651+0000","last_became_peered":"2024-06-14T23:00:27.799651+0000","last_unstale":"2024-06-14T23:02:33.862629+0000","last_undegraded":"2024-06-14T23:02:33.862629+0000","last_fullsized":"2024-06-14T23:02:33.862629+0000","mapping_epoch":186,"log_start":"191'2000","ondisk_log_start":"191'2000","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:23:57.581265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434110464,"num_objects":6624,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9764,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":503,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.929488+0000","last_change":"2024-06-14T23:00:32.818134+0000","last_active":"2024-06-14T23:02:15.929488+0000","last_peered":"2024-06-14T23:02:15.929488+0000","last_clean":"2024-06-14T23:02:15.929488+0000","last_became_active":"2024-06-14T23:00:17.399852+0000","last_became_peered":"2024-06-14T23:00:17.399852+0000","last_unstale":"2024-06-14T23:02:15.929488+0000","last_undegraded":"2024-06-14T23:02:15.929488+0000","last_fullsized":"2024-06-14T23:02:15.929488+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.818076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.818076+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:12:39.666043+0000","scrub_duration":0.0013568390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":172,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.937254+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T23:02:15.937254+0000","last_peered":"2024-06-14T23:02:15.937254+0000","last_clean":"2024-06-14T23:02:15.937254+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T23:02:15.937254+0000","last_undegraded":"2024-06-14T23:02:15.937254+0000","last_fullsized":"2024-06-14T23:02:15.937254+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":168,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928241+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T23:02:15.928241+0000","last_peered":"2024-06-14T23:02:15.928241+0000","last_clean":"2024-06-14T23:02:15.928241+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T23:02:15.928241+0000","last_undegraded":"2024-06-14T23:02:15.928241+0000","last_fullsized":"2024-06-14T23:02:15.928241+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":190,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928608+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T23:02:14.928608+0000","last_peered":"2024-06-14T23:02:14.928608+0000","last_clean":"2024-06-14T23:02:14.928608+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T23:02:14.928608+0000","last_undegraded":"2024-06-14T23:02:14.928608+0000","last_fullsized":"2024-06-14T23:02:14.928608+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"207'9780","reported_seq":9809,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:34.040292+0000","last_change":"2024-06-14T23:00:27.803439+0000","last_active":"2024-06-14T23:02:34.040292+0000","last_peered":"2024-06-14T23:02:34.040292+0000","last_clean":"2024-06-14T23:02:34.040292+0000","last_became_active":"2024-06-14T23:00:27.802783+0000","last_became_peered":"2024-06-14T23:00:27.802783+0000","last_unstale":"2024-06-14T23:02:34.040292+0000","last_undegraded":"2024-06-14T23:02:34.040292+0000","last_fullsized":"2024-06-14T23:02:34.040292+0000","mapping_epoch":186,"log_start":"191'1800","ondisk_log_start":"191'1800","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7980,"log_dups_size":0,"ondisk_log_size":7980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:35:39.179455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432275456,"num_objects":6596,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":457,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.937855+0000","last_change":"2024-06-14T23:00:42.661659+0000","last_active":"2024-06-14T23:02:15.937855+0000","last_peered":"2024-06-14T23:02:15.937855+0000","last_clean":"2024-06-14T23:02:15.937855+0000","last_became_active":"2024-06-14T23:00:17.398852+0000","last_became_peered":"2024-06-14T23:00:17.398852+0000","last_unstale":"2024-06-14T23:02:15.937855+0000","last_undegraded":"2024-06-14T23:02:15.937855+0000","last_fullsized":"2024-06-14T23:02:15.937855+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:47:14.444805+0000","scrub_duration":0.0053445919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":185,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.930315+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T23:02:15.930315+0000","last_peered":"2024-06-14T23:02:15.930315+0000","last_clean":"2024-06-14T23:02:15.930315+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T23:02:15.930315+0000","last_undegraded":"2024-06-14T23:02:15.930315+0000","last_fullsized":"2024-06-14T23:02:15.930315+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":165,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.933329+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:02:14.933329+0000","last_peered":"2024-06-14T23:02:14.933329+0000","last_clean":"2024-06-14T23:02:14.933329+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:02:14.933329+0000","last_undegraded":"2024-06-14T23:02:14.933329+0000","last_fullsized":"2024-06-14T23:02:14.933329+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":160,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.929150+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:02:14.929150+0000","last_peered":"2024-06-14T23:02:14.929150+0000","last_clean":"2024-06-14T23:02:14.929150+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:02:14.929150+0000","last_undegraded":"2024-06-14T23:02:14.929150+0000","last_fullsized":"2024-06-14T23:02:14.929150+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"207'9833","reported_seq":9862,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:33.862050+0000","last_change":"2024-06-14T23:00:27.800097+0000","last_active":"2024-06-14T23:02:33.862050+0000","last_peered":"2024-06-14T23:02:33.862050+0000","last_clean":"2024-06-14T23:02:33.862050+0000","last_became_active":"2024-06-14T23:00:27.799790+0000","last_became_peered":"2024-06-14T23:00:27.799790+0000","last_unstale":"2024-06-14T23:02:33.862050+0000","last_undegraded":"2024-06-14T23:02:33.862050+0000","last_fullsized":"2024-06-14T23:02:33.862050+0000","mapping_epoch":186,"log_start":"191'2100","ondisk_log_start":"191'2100","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7733,"log_dups_size":0,"ondisk_log_size":7733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:21:01.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430440472,"num_objects":6569,"num_object_clones":0,"num_object_copies":19707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9833,"num_write_kb":524801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":457,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.931244+0000","last_change":"2024-06-14T23:00:31.555325+0000","last_active":"2024-06-14T23:02:15.931244+0000","last_peered":"2024-06-14T23:02:15.931244+0000","last_clean":"2024-06-14T23:02:15.931244+0000","last_became_active":"2024-06-14T23:00:17.400871+0000","last_became_peered":"2024-06-14T23:00:17.400871+0000","last_unstale":"2024-06-14T23:02:15.931244+0000","last_undegraded":"2024-06-14T23:02:15.931244+0000","last_fullsized":"2024-06-14T23:02:15.931244+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:58.191436+0000","scrub_duration":0.001665094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":175,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.928531+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T23:02:14.928531+0000","last_peered":"2024-06-14T23:02:14.928531+0000","last_clean":"2024-06-14T23:02:14.928531+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T23:02:14.928531+0000","last_undegraded":"2024-06-14T23:02:14.928531+0000","last_fullsized":"2024-06-14T23:02:14.928531+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.929194+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:02:15.929194+0000","last_peered":"2024-06-14T23:02:15.929194+0000","last_clean":"2024-06-14T23:02:15.929194+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:02:15.929194+0000","last_undegraded":"2024-06-14T23:02:15.929194+0000","last_fullsized":"2024-06-14T23:02:15.929194+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":146,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.926462+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T23:02:15.926462+0000","last_peered":"2024-06-14T23:02:15.926462+0000","last_clean":"2024-06-14T23:02:15.926462+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T23:02:15.926462+0000","last_undegraded":"2024-06-14T23:02:15.926462+0000","last_fullsized":"2024-06-14T23:02:15.926462+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"207'9795","reported_seq":9824,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:33.869701+0000","last_change":"2024-06-14T23:00:27.797841+0000","last_active":"2024-06-14T23:02:33.869701+0000","last_peered":"2024-06-14T23:02:33.869701+0000","last_clean":"2024-06-14T23:02:33.869701+0000","last_became_active":"2024-06-14T23:00:27.797256+0000","last_became_peered":"2024-06-14T23:00:27.797256+0000","last_unstale":"2024-06-14T23:02:33.869701+0000","last_undegraded":"2024-06-14T23:02:33.869701+0000","last_fullsized":"2024-06-14T23:02:33.869701+0000","mapping_epoch":186,"log_start":"192'2900","ondisk_log_start":"192'2900","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:52:24.873796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440074240,"num_objects":6715,"num_object_clones":0,"num_object_copies":20145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9795,"num_write_kb":528320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":504,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927838+0000","last_change":"2024-06-14T23:00:32.174187+0000","last_active":"2024-06-14T23:02:15.927838+0000","last_peered":"2024-06-14T23:02:15.927838+0000","last_clean":"2024-06-14T23:02:15.927838+0000","last_became_active":"2024-06-14T23:00:17.395654+0000","last_became_peered":"2024-06-14T23:00:17.395654+0000","last_unstale":"2024-06-14T23:02:15.927838+0000","last_undegraded":"2024-06-14T23:02:15.927838+0000","last_fullsized":"2024-06-14T23:02:15.927838+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.174132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.174132+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:59.252795+0000","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":68,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.841140+0000","last_change":"2024-06-14T22:58:09.752729+0000","last_active":"2024-06-14T23:02:30.841140+0000","last_peered":"2024-06-14T23:02:30.841140+0000","last_clean":"2024-06-14T23:02:30.841140+0000","last_became_active":"2024-06-14T22:58:09.752393+0000","last_became_peered":"2024-06-14T22:58:09.752393+0000","last_unstale":"2024-06-14T23:02:30.841140+0000","last_undegraded":"2024-06-14T23:02:30.841140+0000","last_fullsized":"2024-06-14T23:02:30.841140+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:21:58.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":159,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.921106+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T23:02:15.921106+0000","last_peered":"2024-06-14T23:02:15.921106+0000","last_clean":"2024-06-14T23:02:15.921106+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T23:02:15.921106+0000","last_undegraded":"2024-06-14T23:02:15.921106+0000","last_fullsized":"2024-06-14T23:02:15.921106+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":152,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927419+0000","last_change":"2024-06-14T22:58:32.754079+0000","last_active":"2024-06-14T23:02:15.927419+0000","last_peered":"2024-06-14T23:02:15.927419+0000","last_clean":"2024-06-14T23:02:15.927419+0000","last_became_active":"2024-06-14T22:58:32.753814+0000","last_became_peered":"2024-06-14T22:58:32.753814+0000","last_unstale":"2024-06-14T23:02:15.927419+0000","last_undegraded":"2024-06-14T23:02:15.927419+0000","last_fullsized":"2024-06-14T23:02:15.927419+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:37:07.414370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"207'9537","reported_seq":9566,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:33.857056+0000","last_change":"2024-06-14T23:00:27.798887+0000","last_active":"2024-06-14T23:02:33.857056+0000","last_peered":"2024-06-14T23:02:33.857056+0000","last_clean":"2024-06-14T23:02:33.857056+0000","last_became_active":"2024-06-14T23:00:27.798590+0000","last_became_peered":"2024-06-14T23:00:27.798590+0000","last_unstale":"2024-06-14T23:02:33.857056+0000","last_undegraded":"2024-06-14T23:02:33.857056+0000","last_fullsized":"2024-06-14T23:02:33.857056+0000","mapping_epoch":186,"log_start":"191'1800","ondisk_log_start":"191'1800","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7737,"log_dups_size":0,"ondisk_log_size":7737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:02:33.396409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420544512,"num_objects":6417,"num_object_clones":0,"num_object_copies":19251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9537,"num_write_kb":510528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":506,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:24.596187+0000","last_change":"2024-06-14T23:00:32.716013+0000","last_active":"2024-06-14T23:02:24.596187+0000","last_peered":"2024-06-14T23:02:24.596187+0000","last_clean":"2024-06-14T23:02:24.596187+0000","last_became_active":"2024-06-14T23:00:17.398349+0000","last_became_peered":"2024-06-14T23:00:17.398349+0000","last_unstale":"2024-06-14T23:02:24.596187+0000","last_undegraded":"2024-06-14T23:02:24.596187+0000","last_fullsized":"2024-06-14T23:02:24.596187+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.715907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.715907+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:23:40.212866+0000","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":164,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927192+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T23:02:15.927192+0000","last_peered":"2024-06-14T23:02:15.927192+0000","last_clean":"2024-06-14T23:02:15.927192+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T23:02:15.927192+0000","last_undegraded":"2024-06-14T23:02:15.927192+0000","last_fullsized":"2024-06-14T23:02:15.927192+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":172,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.938082+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T23:02:15.938082+0000","last_peered":"2024-06-14T23:02:15.938082+0000","last_clean":"2024-06-14T23:02:15.938082+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T23:02:15.938082+0000","last_undegraded":"2024-06-14T23:02:15.938082+0000","last_fullsized":"2024-06-14T23:02:15.938082+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":150,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928929+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T23:02:15.928929+0000","last_peered":"2024-06-14T23:02:15.928929+0000","last_clean":"2024-06-14T23:02:15.928929+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T23:02:15.928929+0000","last_undegraded":"2024-06-14T23:02:15.928929+0000","last_fullsized":"2024-06-14T23:02:15.928929+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":167,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928173+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T23:02:15.928173+0000","last_peered":"2024-06-14T23:02:15.928173+0000","last_clean":"2024-06-14T23:02:15.928173+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T23:02:15.928173+0000","last_undegraded":"2024-06-14T23:02:15.928173+0000","last_fullsized":"2024-06-14T23:02:15.928173+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":173,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.932241+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T23:02:15.932241+0000","last_peered":"2024-06-14T23:02:15.932241+0000","last_clean":"2024-06-14T23:02:15.932241+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T23:02:15.932241+0000","last_undegraded":"2024-06-14T23:02:15.932241+0000","last_fullsized":"2024-06-14T23:02:15.932241+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":174,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.922783+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T23:02:14.922783+0000","last_peered":"2024-06-14T23:02:14.922783+0000","last_clean":"2024-06-14T23:02:14.922783+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T23:02:14.922783+0000","last_undegraded":"2024-06-14T23:02:14.922783+0000","last_fullsized":"2024-06-14T23:02:14.922783+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":86,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.931245+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:02:15.931245+0000","last_peered":"2024-06-14T23:02:15.931245+0000","last_clean":"2024-06-14T23:02:15.931245+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:02:15.931245+0000","last_undegraded":"2024-06-14T23:02:15.931245+0000","last_fullsized":"2024-06-14T23:02:15.931245+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":168,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.938572+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:02:15.938572+0000","last_peered":"2024-06-14T23:02:15.938572+0000","last_clean":"2024-06-14T23:02:15.938572+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:02:15.938572+0000","last_undegraded":"2024-06-14T23:02:15.938572+0000","last_fullsized":"2024-06-14T23:02:15.938572+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":111,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.930381+0000","last_change":"2024-06-14T22:58:09.753103+0000","last_active":"2024-06-14T23:02:15.930381+0000","last_peered":"2024-06-14T23:02:15.930381+0000","last_clean":"2024-06-14T23:02:15.930381+0000","last_became_active":"2024-06-14T22:58:09.752761+0000","last_became_peered":"2024-06-14T22:58:09.752761+0000","last_unstale":"2024-06-14T23:02:15.930381+0000","last_undegraded":"2024-06-14T23:02:15.930381+0000","last_fullsized":"2024-06-14T23:02:15.930381+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:20:20.619331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":159,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.925435+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T23:02:15.925435+0000","last_peered":"2024-06-14T23:02:15.925435+0000","last_clean":"2024-06-14T23:02:15.925435+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T23:02:15.925435+0000","last_undegraded":"2024-06-14T23:02:15.925435+0000","last_fullsized":"2024-06-14T23:02:15.925435+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":161,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.930263+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T23:02:15.930263+0000","last_peered":"2024-06-14T23:02:15.930263+0000","last_clean":"2024-06-14T23:02:15.930263+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T23:02:15.930263+0000","last_undegraded":"2024-06-14T23:02:15.930263+0000","last_fullsized":"2024-06-14T23:02:15.930263+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":167,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.937931+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T23:02:15.937931+0000","last_peered":"2024-06-14T23:02:15.937931+0000","last_clean":"2024-06-14T23:02:15.937931+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T23:02:15.937931+0000","last_undegraded":"2024-06-14T23:02:15.937931+0000","last_fullsized":"2024-06-14T23:02:15.937931+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":155,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.926733+0000","last_change":"2024-06-14T22:58:09.785074+0000","last_active":"2024-06-14T23:02:15.926733+0000","last_peered":"2024-06-14T23:02:15.926733+0000","last_clean":"2024-06-14T23:02:15.926733+0000","last_became_active":"2024-06-14T22:58:09.784309+0000","last_became_peered":"2024-06-14T22:58:09.784309+0000","last_unstale":"2024-06-14T23:02:15.926733+0000","last_undegraded":"2024-06-14T23:02:15.926733+0000","last_fullsized":"2024-06-14T23:02:15.926733+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:23.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":161,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927327+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T23:02:15.927327+0000","last_peered":"2024-06-14T23:02:15.927327+0000","last_clean":"2024-06-14T23:02:15.927327+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T23:02:15.927327+0000","last_undegraded":"2024-06-14T23:02:15.927327+0000","last_fullsized":"2024-06-14T23:02:15.927327+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":174,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.922466+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:02:14.922466+0000","last_peered":"2024-06-14T23:02:14.922466+0000","last_clean":"2024-06-14T23:02:14.922466+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:02:14.922466+0000","last_undegraded":"2024-06-14T23:02:14.922466+0000","last_fullsized":"2024-06-14T23:02:14.922466+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":162,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.942705+0000","last_change":"2024-06-14T22:58:09.756040+0000","last_active":"2024-06-14T23:02:14.942705+0000","last_peered":"2024-06-14T23:02:14.942705+0000","last_clean":"2024-06-14T23:02:14.942705+0000","last_became_active":"2024-06-14T22:58:09.755659+0000","last_became_peered":"2024-06-14T22:58:09.755659+0000","last_unstale":"2024-06-14T23:02:14.942705+0000","last_undegraded":"2024-06-14T23:02:14.942705+0000","last_fullsized":"2024-06-14T23:02:14.942705+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:40.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":186,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927853+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T23:02:15.927853+0000","last_peered":"2024-06-14T23:02:15.927853+0000","last_clean":"2024-06-14T23:02:15.927853+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T23:02:15.927853+0000","last_undegraded":"2024-06-14T23:02:15.927853+0000","last_fullsized":"2024-06-14T23:02:15.927853+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":179,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.940030+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T23:02:14.940030+0000","last_peered":"2024-06-14T23:02:14.940030+0000","last_clean":"2024-06-14T23:02:14.940030+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T23:02:14.940030+0000","last_undegraded":"2024-06-14T23:02:14.940030+0000","last_fullsized":"2024-06-14T23:02:14.940030+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":148,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:21.953873+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T23:02:21.953873+0000","last_peered":"2024-06-14T23:02:21.953873+0000","last_clean":"2024-06-14T23:02:21.953873+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T23:02:21.953873+0000","last_undegraded":"2024-06-14T23:02:21.953873+0000","last_fullsized":"2024-06-14T23:02:21.953873+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":184,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928102+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:02:15.928102+0000","last_peered":"2024-06-14T23:02:15.928102+0000","last_clean":"2024-06-14T23:02:15.928102+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:02:15.928102+0000","last_undegraded":"2024-06-14T23:02:15.928102+0000","last_fullsized":"2024-06-14T23:02:15.928102+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":170,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.929869+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:02:15.929869+0000","last_peered":"2024-06-14T23:02:15.929869+0000","last_clean":"2024-06-14T23:02:15.929869+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:02:15.929869+0000","last_undegraded":"2024-06-14T23:02:15.929869+0000","last_fullsized":"2024-06-14T23:02:15.929869+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":184,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.927778+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:02:15.927778+0000","last_peered":"2024-06-14T23:02:15.927778+0000","last_clean":"2024-06-14T23:02:15.927778+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:02:15.927778+0000","last_undegraded":"2024-06-14T23:02:15.927778+0000","last_fullsized":"2024-06-14T23:02:15.927778+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":153,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.930239+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T23:02:15.930239+0000","last_peered":"2024-06-14T23:02:15.930239+0000","last_clean":"2024-06-14T23:02:15.930239+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T23:02:15.930239+0000","last_undegraded":"2024-06-14T23:02:15.930239+0000","last_fullsized":"2024-06-14T23:02:15.930239+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":169,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-06-14T23:02:14.940518+0000","last_change":"2024-06-14T22:58:09.752281+0000","last_active":"2024-06-14T23:02:14.940518+0000","last_peered":"2024-06-14T23:02:14.940518+0000","last_clean":"2024-06-14T23:02:14.940518+0000","last_became_active":"2024-06-14T22:58:09.751952+0000","last_became_peered":"2024-06-14T22:58:09.751952+0000","last_unstale":"2024-06-14T23:02:14.940518+0000","last_undegraded":"2024-06-14T23:02:14.940518+0000","last_fullsized":"2024-06-14T23:02:14.940518+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:39:19.836734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":66,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.931323+0000","last_change":"2024-06-14T22:58:09.755622+0000","last_active":"2024-06-14T23:02:15.931323+0000","last_peered":"2024-06-14T23:02:15.931323+0000","last_clean":"2024-06-14T23:02:15.931323+0000","last_became_active":"2024-06-14T22:58:09.755383+0000","last_became_peered":"2024-06-14T22:58:09.755383+0000","last_unstale":"2024-06-14T23:02:15.931323+0000","last_undegraded":"2024-06-14T23:02:15.931323+0000","last_fullsized":"2024-06-14T23:02:15.931323+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:13.884346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":167,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.928713+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T23:02:15.928713+0000","last_peered":"2024-06-14T23:02:15.928713+0000","last_clean":"2024-06-14T23:02:15.928713+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T23:02:15.928713+0000","last_undegraded":"2024-06-14T23:02:15.928713+0000","last_fullsized":"2024-06-14T23:02:15.928713+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":164,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-06-14T23:02:15.925358+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T23:02:15.925358+0000","last_peered":"2024-06-14T23:02:15.925358+0000","last_clean":"2024-06-14T23:02:15.925358+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T23:02:15.925358+0000","last_undegraded":"2024-06-14T23:02:15.925358+0000","last_fullsized":"2024-06-14T23:02:15.925358+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6974013464,"num_objects":106416,"num_object_clones":0,"num_object_copies":319248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106416,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155516,"num_write_kb":8381761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20935028736,"data_stored":20935016520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130916,"ondisk_log_size":130916,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1771,"ondisk_log_size":1771,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234342,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2585696,"kb_used_data":1682648,"kb_used_omap":1,"kb_used_meta":903038,"kb_avail":91159456,"statfs":{"total":95995035648,"available":93347282944,"internally_reserved":0,"allocated":1723031552,"data_stored":1721991579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924711256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.105,"15min":1.0840000000000001},"min":{"1min":0.224,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":2.351,"5min":6.9539999999999997,"15min":9.3919999999999995},"last":2.738},{"interface":"front","average":{"1min":1.661,"5min":1.242,"15min":1.24},"min":{"1min":0.29099999999999998,"5min":0.26800000000000002,"15min":0.20300000000000001},"max":{"1min":4.806,"5min":5.2130000000000001,"15min":10.416},"last":6.4489999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.3680000000000001,"15min":1.7490000000000001},"min":{"1min":0.31900000000000001,"5min":0.29199999999999998,"15min":0.28100000000000003},"max":{"1min":3.3660000000000001,"5min":8.5340000000000007,"15min":83.295000000000002},"last":3.2850000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.4510000000000001,"15min":1.821},"min":{"1min":0.29199999999999998,"5min":0.245,"15min":0.223},"max":{"1min":2.5459999999999998,"5min":6.8620000000000001,"15min":82.972999999999999},"last":0.28499999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1299999999999999,"15min":0.79500000000000004},"min":{"1min":0.26700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.9820000000000002,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":7.468},{"interface":"front","average":{"1min":1.03,"5min":1.2749999999999999,"15min":0.84999999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.0199999999999996,"5min":8.1150000000000002,"15min":8.1150000000000002},"last":0.25900000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.4199999999999999,"15min":1.0669999999999999},"min":{"1min":0.34899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":6.2220000000000004,"5min":12.992000000000001,"15min":55.683999999999997},"last":5.4589999999999996},{"interface":"front","average":{"1min":1.401,"5min":1.3180000000000001,"15min":1.0920000000000001},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.528,"5min":9.5559999999999992,"15min":55.533999999999999},"last":0.79800000000000004}]},{"osd":4,"last update":"Fri Jun 14 23:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.002,"15min":0.73899999999999999},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.245},"max":{"1min":3.1480000000000001,"5min":6.7610000000000001,"15min":6.7610000000000001},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.0840000000000001,"15min":0.79400000000000004},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":6.1600000000000001,"5min":16.879000000000001,"15min":16.879000000000001},"last":1.444}]},{"osd":5,"last update":"Fri Jun 14 23:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.331,"15min":0.91500000000000004},"min":{"1min":0.317,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.4320000000000004,"5min":6.9000000000000004,"15min":6.9000000000000004},"last":0.316},{"interface":"front","average":{"1min":1.216,"5min":1.137,"15min":0.90700000000000003},"min":{"1min":0.33200000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.9380000000000002,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":0.82099999999999995}]},{"osd":6,"last update":"Fri Jun 14 23:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.0920000000000001,"15min":0.84199999999999997},"min":{"1min":0.35799999999999998,"5min":0.307,"15min":0.28100000000000003},"max":{"1min":2.9689999999999999,"5min":5.5,"15min":5.806},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.518,"5min":1.099,"15min":0.84899999999999998},"min":{"1min":0.38700000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.1980000000000004,"5min":12.951000000000001,"15min":12.951000000000001},"last":0.59899999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495978,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3868120,"kb_used_data":2974988,"kb_used_omap":1,"kb_used_meta":893118,"kb_avail":89877032,"statfs":{"total":95995035648,"available":92034080768,"internally_reserved":0,"allocated":3046387712,"data_stored":3045359582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914553176},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:14 2024","interfaces":[{"interface":"back","average":{"1min":2.431,"5min":1.4059999999999999,"15min":1.351},"min":{"1min":0.36399999999999999,"5min":0.23699999999999999,"15min":0.218},"max":{"1min":10.348000000000001,"5min":10.348000000000001,"15min":31.358000000000001},"last":3.8650000000000002},{"interface":"front","average":{"1min":1.859,"5min":1.119,"15min":1.2929999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.21299999999999999},"max":{"1min":10.348000000000001,"5min":10.348000000000001,"15min":29.399000000000001},"last":1.4079999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.3500000000000001,"15min":1.21},"min":{"1min":0.32200000000000001,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":4.6260000000000003,"5min":6.6680000000000001,"15min":9.9559999999999995},"last":1.8779999999999999},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.351,"15min":1.2330000000000001},"min":{"1min":0.27400000000000002,"5min":0.25,"15min":0.247},"max":{"1min":3.0830000000000002,"5min":6.7519999999999998,"15min":9.5510000000000002},"last":0.27600000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.1160000000000001,"15min":0.76500000000000001},"min":{"1min":0.30299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":9.9600000000000009,"5min":9.9600000000000009,"15min":9.9600000000000009},"last":0.31},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.2050000000000001,"15min":0.82099999999999995},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":9.9870000000000001,"5min":9.9870000000000001,"15min":9.9870000000000001},"last":0.36799999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:02:14 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.3149999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.28599999999999998,"15min":0.27300000000000002},"max":{"1min":3.1560000000000001,"5min":9.8970000000000002,"15min":29.152000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.081,"5min":1.3069999999999999,"15min":0.98999999999999999},"min":{"1min":0.379,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.621,"5min":6.3040000000000003,"15min":25.864000000000001},"last":3.1139999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:02:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.98699999999999999,"15min":0.92400000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.264,"5min":7.7709999999999999,"15min":47.698},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.93600000000000005,"15min":0.877},"min":{"1min":0.28899999999999998,"5min":0.27600000000000002,"15min":0.24299999999999999},"max":{"1min":2.3999999999999999,"5min":9.9610000000000003,"15min":48.197000000000003},"last":0.433}]},{"osd":5,"last update":"Fri Jun 14 23:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1459999999999999,"15min":0.88400000000000001},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.26500000000000001},"max":{"1min":5.0010000000000003,"5min":7.3540000000000001,"15min":8.6500000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.103,"15min":0.90300000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":2.714,"5min":7.3769999999999998,"15min":8.6349999999999998},"last":0.41999999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:02:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.224,"15min":0.89800000000000002},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.29299999999999998},"max":{"1min":4.4720000000000004,"5min":7.8579999999999997,"15min":8.6950000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.147,"15min":0.92900000000000005},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.28899999999999998},"max":{"1min":3.6869999999999998,"5min":7.8120000000000003,"15min":8.673},"last":0.39500000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757617,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3011088,"kb_used_data":2124036,"kb_used_omap":1,"kb_used_meta":887038,"kb_avail":90734064,"statfs":{"total":95995035648,"available":92911681536,"internally_reserved":0,"allocated":2175012864,"data_stored":2173989260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908327256},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:01:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.2050000000000001,"15min":1.2370000000000001},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.219},"max":{"1min":5.7089999999999996,"5min":5.7089999999999996,"15min":17.766999999999999},"last":0.434},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.1890000000000001,"15min":1.224},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.964,"5min":5.1950000000000003,"15min":17.806000000000001},"last":1.288}]},{"osd":1,"last update":"Fri Jun 14 23:01:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.0209999999999999,"15min":1.157},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23300000000000001},"max":{"1min":1.6719999999999999,"5min":5.6429999999999998,"15min":7.694},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.3200000000000001,"15min":1.1699999999999999},"min":{"1min":0.312,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":3.1930000000000001,"5min":7.0700000000000003,"15min":7.0700000000000003},"last":0.61299999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:01:48 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.0980000000000001,"15min":0.78300000000000003},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.8470000000000004,"5min":6.8470000000000004,"15min":6.8470000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.754,"5min":1.169,"15min":0.79000000000000004},"min":{"1min":0.379,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.9000000000000004,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":2.3050000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:02:19 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.534,"15min":1.1319999999999999},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.4580000000000002,"5min":10.358000000000001,"15min":45.744},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.845,"5min":1.5900000000000001,"15min":1.222},"min":{"1min":0.378,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.4199999999999999,"5min":10.385,"15min":45.838000000000001},"last":0.65800000000000003}]},{"osd":4,"last update":"Fri Jun 14 23:02:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.872,"15min":0.66200000000000003},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.214},"max":{"1min":2.5299999999999998,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.312,"5min":0.28299999999999997,"15min":0.22700000000000001},"max":{"1min":4.4359999999999999,"5min":4.657,"15min":5.8109999999999999},"last":1.3120000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:01:41 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.038,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.253,"15min":0.23000000000000001},"max":{"1min":4.0090000000000003,"5min":6.931,"15min":9.6020000000000003},"last":4.2039999999999997},{"interface":"front","average":{"1min":1.4119999999999999,"5min":0.96199999999999997,"15min":0.78800000000000003},"min":{"1min":0.373,"5min":0.30099999999999999,"15min":0.25},"max":{"1min":5.056,"5min":5.056,"15min":12.339},"last":0.80500000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.1040000000000001,"15min":0.83299999999999996},"min":{"1min":0.30499999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.4369999999999998,"5min":5.3129999999999997,"15min":5.3129999999999997},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.1220000000000001,"15min":0.83899999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":5.1820000000000004,"5min":5.6189999999999998,"15min":10.638999999999999},"last":0.41599999999999998}]}]},{"osd":4,"up_from":30,"seq":128849019253,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4274268,"kb_used_data":3387604,"kb_used_omap":1,"kb_used_meta":886590,"kb_avail":89470884,"statfs":{"total":95995035648,"available":91618185216,"internally_reserved":0,"allocated":3468906496,"data_stored":3467870308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907868504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4220000000000002,"5min":1.3140000000000001,"15min":1.125},"min":{"1min":0.25800000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":10.567,"5min":10.567,"15min":10.567},"last":5.8600000000000003},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.2110000000000001,"15min":1.139},"min":{"1min":0.23999999999999999,"5min":0.22700000000000001,"15min":0.221},"max":{"1min":5.9189999999999996,"5min":8.3390000000000004,"15min":9.1470000000000002},"last":6.4669999999999996}]},{"osd":1,"last update":"Fri Jun 14 23:01:44 2024","interfaces":[{"interface":"back","average":{"1min":4.0469999999999997,"5min":1.901,"15min":1.345},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":56.917000000000002,"5min":56.917000000000002,"15min":56.917000000000002},"last":0.317},{"interface":"front","average":{"1min":4.1479999999999997,"5min":1.98,"15min":1.4410000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":56.789999999999999,"5min":56.789999999999999,"15min":56.789999999999999},"last":0.39500000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:01:44 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.89500000000000002,"15min":0.67400000000000004},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.5750000000000002,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.95899999999999996,"15min":0.70799999999999996},"min":{"1min":0.30299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.8480000000000001,"5min":4.0490000000000004,"15min":4.0490000000000004},"last":1.764}]},{"osd":3,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.268,"15min":0.86399999999999999},"min":{"1min":0.28599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.327,"5min":6.617,"15min":6.617},"last":0.64300000000000002},{"interface":"front","average":{"1min":2.1269999999999998,"5min":1.3819999999999999,"15min":0.92700000000000005},"min":{"1min":0.33100000000000002,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":10.510999999999999,"5min":10.510999999999999,"15min":10.510999999999999},"last":4.1680000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:02:07 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":2.6579999999999999,"15min":1.3240000000000001},"min":{"1min":0.32100000000000001,"5min":0.253,"15min":0.22500000000000001},"max":{"1min":5.0629999999999997,"5min":139.13200000000001,"15min":139.13200000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":2.7759999999999998,"15min":1.353},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.23699999999999999},"max":{"1min":2.3069999999999999,"5min":138.86000000000001,"15min":138.86000000000001},"last":0.42499999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0629999999999999,"15min":0.75700000000000001},"min":{"1min":0.32600000000000001,"5min":0.28899999999999998,"15min":0.254},"max":{"1min":4.6269999999999998,"5min":5.1210000000000004,"15min":5.1210000000000004},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.1120000000000001,"15min":0.748},"min":{"1min":0.35899999999999999,"5min":0.27900000000000003,"15min":0.254},"max":{"1min":5.8680000000000003,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.46100000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:01:57 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.056,"15min":0.755},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.0009999999999999,"5min":8.9489999999999998,"15min":8.9489999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.2410000000000001,"15min":0.871},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.308},"max":{"1min":2.9409999999999998,"5min":7.25,"15min":8.4730000000000008},"last":0.54100000000000004}]}]},{"osd":3,"up_from":138,"seq":592705486952,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2418380,"kb_used_data":2108036,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":91326772,"statfs":{"total":95995035648,"available":93518614528,"internally_reserved":0,"allocated":2158628864,"data_stored":2157605033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":317779422},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.4570000000000001,"15min":0.78100000000000003},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.184},"max":{"1min":5.8499999999999996,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":2.0009999999999999,"5min":1.2569999999999999,"15min":0.70899999999999996},"min":{"1min":0.26100000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2330000000000005,"5min":8.2330000000000005,"15min":8.2330000000000005},"last":2.4889999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.4890000000000001,"15min":0.82999999999999996},"min":{"1min":0.249,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":4.4189999999999996,"5min":24.097999999999999,"15min":24.097999999999999},"last":2.3570000000000002},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.5680000000000001,"15min":0.86199999999999999},"min":{"1min":0.35299999999999998,"5min":0.26600000000000001,"15min":0.24099999999999999},"max":{"1min":3.5859999999999999,"5min":24.132999999999999,"15min":24.132999999999999},"last":3.2810000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.1950000000000001,"15min":0.72399999999999998},"min":{"1min":0.29499999999999998,"5min":0.245,"15min":0.23599999999999999},"max":{"1min":3.831,"5min":5.2910000000000004,"15min":5.2910000000000004},"last":2.407},{"interface":"front","average":{"1min":1.9159999999999999,"5min":1.2,"15min":0.71599999999999997},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":10.369,"5min":10.369,"15min":10.369},"last":3.262}]},{"osd":4,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.2170000000000001,"15min":0.877},"min":{"1min":0.40600000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.976,"5min":5.1189999999999998,"15min":5.7960000000000003},"last":3.323},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3680000000000001,"15min":0.92800000000000005},"min":{"1min":0.379,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.1580000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"last":0.70699999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.405,"15min":0.96699999999999997},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.29699999999999999},"max":{"1min":5.117,"5min":5.9359999999999999,"15min":5.9359999999999999},"last":3.8420000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.373,"15min":0.92100000000000004},"min":{"1min":0.32400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.6360000000000001,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":2.4689999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.27,"15min":0.89600000000000002},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.762,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":2.387},{"interface":"front","average":{"1min":1.401,"5min":1.252,"15min":0.88100000000000001},"min":{"1min":0.32200000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.3760000000000003,"5min":5.0439999999999996,"15min":5.8259999999999996},"last":3.2989999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.3919999999999999,"15min":0.997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35299999999999998},"max":{"1min":4.351,"5min":7.7240000000000002,"15min":7.7240000000000002},"last":3.3450000000000002},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.423,"15min":0.98199999999999998},"min":{"1min":0.41299999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.2160000000000002,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.40000000000000002}]}]},{"osd":2,"up_from":133,"seq":571230650478,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2811968,"kb_used_data":2630264,"kb_used_omap":6,"kb_used_meta":181497,"kb_avail":90933184,"statfs":{"total":95995035648,"available":93115580416,"internally_reserved":0,"allocated":2693390336,"data_stored":2692370673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":185853429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.78800000000000003,"15min":0.54100000000000004},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":2.883,"5min":7.476,"15min":7.476},"last":5.0640000000000001},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.1399999999999999,"15min":0.63600000000000001},"min":{"1min":0.24099999999999999,"5min":0.182,"15min":0.182},"max":{"1min":11.369,"5min":11.369,"15min":11.369},"last":3.6669999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.099,"15min":0.747},"min":{"1min":0.221,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.613,"5min":5.8200000000000003,"15min":21.920000000000002},"last":1.9470000000000001},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.0920000000000001,"15min":0.73399999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":5.7439999999999998,"5min":13.054,"15min":21.573},"last":0.67800000000000005}]},{"osd":3,"last update":"Fri Jun 14 23:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.4119999999999999,"15min":0.83099999999999996},"min":{"1min":0.23999999999999999,"5min":0.219,"15min":0.219},"max":{"1min":11.24,"5min":11.24,"15min":21.66},"last":3.5699999999999998},{"interface":"front","average":{"1min":2.0099999999999998,"5min":1.3149999999999999,"15min":0.872},"min":{"1min":0.27100000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":11.271000000000001,"5min":11.271000000000001,"15min":22.341000000000001},"last":3.6469999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6139999999999999,"5min":1.4430000000000001,"15min":0.88400000000000001},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.318},"max":{"1min":14.438000000000001,"5min":14.438000000000001,"15min":14.438000000000001},"last":3.6000000000000001},{"interface":"front","average":{"1min":2.1629999999999998,"5min":1.333,"15min":0.83899999999999997},"min":{"1min":0.33100000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":14.423,"5min":14.423,"15min":14.423},"last":1.9830000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":1.4239999999999999,"15min":0.86399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":31.539999999999999,"5min":31.539999999999999,"15min":31.539999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":2.778,"5min":1.405,"15min":0.875},"min":{"1min":0.437,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":31.481999999999999,"5min":31.481999999999999,"15min":31.481999999999999},"last":3.6259999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":1.1819999999999999,"15min":0.78300000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":11.454000000000001,"5min":11.454000000000001,"15min":11.454000000000001},"last":2.0030000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.1599999999999999,"15min":0.77700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.30399999999999999},"max":{"1min":12.227,"5min":12.227,"15min":12.227},"last":0.89700000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:01:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3560000000000001,"15min":0.86499999999999999},"min":{"1min":0.36499999999999999,"5min":0.36199999999999999,"15min":0.35899999999999999},"max":{"1min":10.614000000000001,"5min":10.614000000000001,"15min":10.614000000000001},"last":2.0249999999999999},{"interface":"front","average":{"1min":1.02,"5min":1.038,"15min":0.752},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":3.1230000000000002,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":0.87}]}]},{"osd":1,"up_from":116,"seq":498216206495,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1773432,"kb_used_data":1330480,"kb_used_omap":5,"kb_used_meta":442938,"kb_avail":91971720,"statfs":{"total":95995035648,"available":94179041280,"internally_reserved":0,"allocated":1362411520,"data_stored":1361399201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":453568792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.385,"15min":1.712},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.6269999999999998,"5min":11.257999999999999,"15min":11.257999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.635,"5min":1.234,"15min":1.3520000000000001},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":10.426},"last":3.6800000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.4610000000000001,"15min":0.77700000000000002},"min":{"1min":0.22500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.6790000000000003,"5min":9.3759999999999994,"15min":9.3759999999999994},"last":1.163},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.054,"15min":0.60899999999999999},"min":{"1min":0.255,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":3.7799999999999998,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":3.0449999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.3380000000000001,"15min":0.89600000000000002},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.8929999999999998,"5min":8.2089999999999996,"15min":21.422000000000001},"last":2.3919999999999999},{"interface":"front","average":{"1min":0.88,"5min":1.462,"15min":0.94999999999999996},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.214},"max":{"1min":4.2450000000000001,"5min":9.4990000000000006,"15min":21.016999999999999},"last":0.33400000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.238,"15min":1.518},"min":{"1min":0.28899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.806,"5min":6.6360000000000001,"15min":8.8599999999999994},"last":2.0209999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.5840000000000001,"15min":1.6419999999999999},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.9589999999999996,"5min":9.4169999999999998,"15min":11.161},"last":0.40200000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.482,"15min":1.5580000000000001},"min":{"1min":0.38100000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.855,"5min":10.103,"15min":10.103},"last":3.089},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.2210000000000001,"15min":1.536},"min":{"1min":0.308,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.5,"5min":6.5730000000000004,"15min":8.125},"last":2.0640000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":4.5919999999999996,"5min":1.9910000000000001,"15min":1.7649999999999999},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":63.795999999999999,"5min":63.795999999999999,"15min":63.795999999999999},"last":3.7029999999999998},{"interface":"front","average":{"1min":4.2080000000000002,"5min":1.927,"15min":1.7569999999999999},"min":{"1min":0.35699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":63.811,"5min":63.811,"15min":63.811},"last":2.0910000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.641,"15min":1.6919999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.923,"5min":9.4429999999999996,"15min":9.4429999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.3520000000000001,"15min":1.45},"min":{"1min":0.433,"5min":0.371,"15min":0.36499999999999999},"max":{"1min":3.8340000000000001,"5min":6.3529999999999998,"15min":9.5449999999999999},"last":3.653}]}]},{"osd":0,"up_from":91,"seq":390842024150,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5008332,"kb_used_data":4339904,"kb_used_omap":5,"kb_used_meta":668218,"kb_avail":88736820,"statfs":{"total":95995035648,"available":90866503680,"internally_reserved":0,"allocated":4444061696,"data_stored":4443039350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":684255322},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.341,"15min":1.482},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":2.6000000000000001,"5min":15.547000000000001,"15min":16.305},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.454,"15min":1.7609999999999999},"min":{"1min":0.20499999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.9249999999999998,"5min":6.9889999999999999,"15min":16.184999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.24,"15min":0.73399999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.053000000000001,"5min":11.326000000000001,"15min":16.757000000000001},"last":0.223},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.4059999999999999,"15min":0.81299999999999994},"min":{"1min":0.24299999999999999,"5min":0.222,"15min":0.222},"max":{"1min":9.2769999999999992,"5min":11.519,"15min":16.055},"last":0.27500000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.3260000000000001,"15min":0.83599999999999997},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":4.2370000000000001,"5min":7.04,"15min":16.873999999999999},"last":2.0739999999999998},{"interface":"front","average":{"1min":2.0720000000000001,"5min":1.46,"15min":0.86099999999999999},"min":{"1min":0.27300000000000002,"5min":0.224,"15min":0.224},"max":{"1min":9.4760000000000009,"5min":12.308999999999999,"15min":16.085999999999999},"last":2.383}]},{"osd":4,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.546,"15min":1.456},"min":{"1min":0.42499999999999999,"5min":0.307,"15min":0.27600000000000002},"max":{"1min":12.436,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.0179999999999998,"5min":1.4890000000000001,"15min":1.3380000000000001},"min":{"1min":0.35499999999999998,"5min":0.33200000000000002,"15min":0.28599999999999998},"max":{"1min":4.8250000000000002,"5min":14.475,"15min":15.976000000000001},"last":2.1539999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.5489999999999999,"15min":1.452},"min":{"1min":0.32800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.9009999999999998,"5min":6.9640000000000004,"15min":16.108000000000001},"last":2.105},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.5529999999999999,"15min":1.5269999999999999},"min":{"1min":0.32500000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":9.2279999999999998,"5min":11.436,"15min":16.006},"last":0.54700000000000004}]},{"osd":6,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7969999999999999,"5min":1.5629999999999999,"15min":1.4219999999999999},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.8730000000000002,"5min":14.247,"15min":16.152000000000001},"last":2.1819999999999999},{"interface":"front","average":{"1min":2.3239999999999998,"5min":1.6619999999999999,"15min":1.5129999999999999},"min":{"1min":0.41299999999999998,"5min":0.312,"15min":0.29399999999999998},"max":{"1min":9.3130000000000006,"5min":14.523999999999999,"15min":16.227},"last":0.51900000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":2.1440000000000001,"5min":1.4950000000000001,"15min":1.46},"min":{"1min":0.434,"5min":0.35899999999999999,"15min":0.33100000000000002},"max":{"1min":9.2579999999999991,"5min":11.364000000000001,"15min":16.199999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.867,"5min":1.546,"15min":1.524},"min":{"1min":0.39900000000000002,"5min":0.36899999999999999,"15min":0.314},"max":{"1min":9.2940000000000005,"5min":12.361000000000001,"15min":16.259},"last":2.278}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4404809728,"data_stored":4404805656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1349386240,"data_stored":1349386240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2676162560,"data_stored":2676162560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2149515264,"data_stored":2149515264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3460370432,"data_stored":3460366360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2154233856,"data_stored":2154233856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3024818176,"data_stored":3024814104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1715732480,"data_stored":1715732480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:02:38.116 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-14T23:02:38.116 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:02:38.116 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-06-14T23:02:38.116 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:02:38.116 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:02:38.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-06-14T23:02:38.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:38.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:38.148 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:37 smithi137 ceph-mon[90634]: pgmap v820: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:02:38.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:38.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:38.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:38.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:38.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:38.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:38.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:38.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:38.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:38.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:38.571 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:02:38.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:38.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:38.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:38.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:38.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:38.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:38.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:38.968 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:38.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:38.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:39.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:39.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:39.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:39.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:39.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:39.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:39.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:39.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:39.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:39.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:39.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:39.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:39.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:39.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:39.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:39.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:39.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:39.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:39.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:39.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:39.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:39.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:39.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:39 smithi086 ceph-mon[97913]: pgmap v821: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-06-14T23:02:39.970 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:39 smithi086 ceph-mon[102754]: pgmap v821: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-06-14T23:02:39.979 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:40.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:40.077 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:40.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:40.105 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:39 smithi137 ceph-mon[90634]: pgmap v821: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-06-14T23:02:40.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:40.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:40.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:40.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:40.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:40.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:40.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:40.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:40.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:40.577 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:02:40.577 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-14T23:02:40.577 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:02:40.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:40.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:40.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:40.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:40.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:40.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:40.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:40.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:40.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:41.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:41.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:41.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:41.097 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:02:41.097 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:02:41.130 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:02:41.130 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:02:41.164 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:02:41.165 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:02:41.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:41.202 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:02:41.202 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:02:41.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:41.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:41.234 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:02:41.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:02:41.263 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:02:41.263 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:02:41.292 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:02:41.293 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:02:41.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:41.314 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:02:41.314 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:02:41.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:41.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:41.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:41.478 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:41.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:41.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:41.613 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:41.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:41.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:41.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:41.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:41.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:41.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:41.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:41.895 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:41 smithi137 ceph-mon[90634]: pgmap v822: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 931 op/s 2024-06-14T23:02:41.896 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3877193139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:41.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:42.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:42.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:41 smithi086 ceph-mon[102754]: pgmap v822: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 931 op/s 2024-06-14T23:02:42.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3877193139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:42.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:41 smithi086 ceph-mon[97913]: pgmap v822: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 931 op/s 2024-06-14T23:02:42.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3877193139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:42.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:42.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:42.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:42.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:42.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:42.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:42.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:42.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:42.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:42.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:42.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:42.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:42.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:42.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:42.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:42.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:42.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:42.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:42.869 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:42.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:42.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:43.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:43.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:43.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:43.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:43.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:43.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:43.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:43.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:43.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:43.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:43.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:43.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:43.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:43.568 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:43.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:43.660 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:02:43.661 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:02:43.661 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:43.661 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:43.661 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:43 smithi086 ceph-mon[102754]: pgmap v823: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 829 op/s 2024-06-14T23:02:43.661 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:43.661 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:43.662 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:43 smithi086 ceph-mon[97913]: pgmap v823: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 829 op/s 2024-06-14T23:02:43.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:43.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:43.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:43.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:43.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:43.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:43 smithi137 ceph-mon[90634]: pgmap v823: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 829 op/s 2024-06-14T23:02:43.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:43.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:43.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:43.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:43.984 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:43.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:44.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:44.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:44.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:44.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:44.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:44.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:44.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:02:44.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:44.425 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:44.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:44.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:44.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:44.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:44.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:44.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:44.828 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:02:44.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:44.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:44.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:45.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:45.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:45.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:45.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3374948028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:02:45.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:02:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:02:44] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.43.0" 2024-06-14T23:02:45.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3374948028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:02:45.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:45.166 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:45.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:45.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3374948028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:02:45.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:45.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:45.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:45.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:45.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:45.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:45.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:45.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:45.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:45.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:45.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:45.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:45.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:45.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:45.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:45.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:45.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:45.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:45.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:45.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-14T23:02:45.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:45.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:46.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:46.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:46.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:46.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:45 smithi137 ceph-mon[90634]: pgmap v824: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-06-14T23:02:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:45 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:02:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3374948028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:02:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:45 smithi137 ceph-mon[90634]: osdmap e208: 8 total, 8 up, 8 in 2024-06-14T23:02:46.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:46.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:46.283 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:02:46.283 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:02:46.283 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_5 2024-06-14T23:02:46.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:02:46.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:45 smithi086 ceph-mon[102754]: pgmap v824: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-06-14T23:02:46.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:45 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:02:46.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3374948028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:02:46.328 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:45 smithi086 ceph-mon[102754]: osdmap e208: 8 total, 8 up, 8 in 2024-06-14T23:02:46.328 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:45 smithi086 ceph-mon[97913]: pgmap v824: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-06-14T23:02:46.328 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:45 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:02:46.328 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3374948028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:02:46.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:45 smithi086 ceph-mon[97913]: osdmap e208: 8 total, 8 up, 8 in 2024-06-14T23:02:46.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:46.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:46.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:46.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:46.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:46.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:46.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:46.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:46.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:46.650 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:46.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:46.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:46.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:46.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:46.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:47.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:47.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:47.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:47.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:47.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:47.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:47.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:47.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:47.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:47.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:47.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:47.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:47.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:47.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:47.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:47.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:47.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:47.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:47.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:47.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:47.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:47.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:47.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:47.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:48.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:48.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:48.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:47 smithi086 ceph-mon[97913]: pgmap v826: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 940 op/s 2024-06-14T23:02:48.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:47 smithi086 ceph-mon[102754]: pgmap v826: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 940 op/s 2024-06-14T23:02:48.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:48.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:48.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:48.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:48.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:47 smithi137 ceph-mon[90634]: pgmap v826: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 940 op/s 2024-06-14T23:02:48.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:48.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:48.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:48.313 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:48.317 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:48.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:48.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:48.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:48.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:48.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:02:48.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:48.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:48.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:48.750 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:48.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:48.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:48.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:48.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:48.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:49.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:49.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:49.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:49.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:49.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:49.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:49.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:49.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:49.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:49.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:49.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:49.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:49.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:49.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:49.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:49.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:49.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:49 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:49.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:49 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:49.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:49 smithi137 ceph-mon[90634]: pgmap v827: 99 pgs: 99 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 938 op/s 2024-06-14T23:02:49.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:49.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:49 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:49.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:49 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:49.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:49 smithi086 ceph-mon[97913]: pgmap v827: 99 pgs: 99 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 938 op/s 2024-06-14T23:02:49.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:49 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:49.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:49 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:49.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:49 smithi086 ceph-mon[102754]: pgmap v827: 99 pgs: 99 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 938 op/s 2024-06-14T23:02:49.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:49.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:49.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:50.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:50.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:50.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:50.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:50.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:50.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:50.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:50.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:50.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:50.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:50.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:50.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:50.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:50.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:50.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:50.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:50.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:50.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:50.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:50.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:50.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:50.958 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:50.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:51.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:51.096 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:51.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:51.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:51.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:51.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:51.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:51.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:51.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:51.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:51.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:51.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:51.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:51.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:51.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:51.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:51.770 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:51.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:51.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:51.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:51.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:52.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:52.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:52.066 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:02:52.066 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:02:52.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:51 smithi086 ceph-mon[97913]: pgmap v828: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 934 op/s 2024-06-14T23:02:52.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:02:52.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:02:52.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:52.067 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:51 smithi086 ceph-mon[102754]: pgmap v828: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 934 op/s 2024-06-14T23:02:52.067 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:02:52.067 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:02:52.067 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:52.095 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:02:52.095 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:02:52.126 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:02:52.126 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:02:52.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:52.164 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:02:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:02:52.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:52.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:52.196 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:02:52.196 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:02:52.237 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:02:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:02:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:51 smithi137 ceph-mon[90634]: pgmap v828: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 934 op/s 2024-06-14T23:02:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:02:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:02:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:52.267 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:02:52.267 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:02:52.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:52.297 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:02:52.297 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:02:52.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:52.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:52.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:52.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:52.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:52.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:52.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:52.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:52.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:52.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:52.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:52.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:52.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:52.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:52.913 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:02:52.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:53.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:53.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:53.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/112513018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:53.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/112513018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:53.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:53.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:53.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:53.226 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/112513018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:02:53.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:53.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:53.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:53.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:53.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:53.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:53.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:53.582 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:53.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:53.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:53.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:53.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:53.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:53.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:53.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:53.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:53.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:53.991 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:53.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:53 smithi086 ceph-mon[97913]: pgmap v829: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:02:54.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:53 smithi086 ceph-mon[102754]: pgmap v829: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:02:54.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:54.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:54.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:54.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:54.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:54.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:53 smithi137 ceph-mon[90634]: pgmap v829: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:02:54.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:54.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:54.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:54.395 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:54.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:54.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:54.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:54.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:54.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:54.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:54.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:54.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:54.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:54.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:54.818 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:02:54.819 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:02:54.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:54.919 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:02:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:02:54] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.43.0" 2024-06-14T23:02:54.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:54.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:55.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:55.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:55.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:55.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:55.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:55.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:55.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:55.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:55.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:55.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:02:55.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:55.496 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:55.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:55.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:55.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:55.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:55.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:55.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:55.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:55.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:55.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:55.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:55.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:56.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:56.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:55 smithi086 ceph-mon[97913]: pgmap v830: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 949 op/s 2024-06-14T23:02:56.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:55 smithi086 ceph-mon[97913]: from='client.45983 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:02:56.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:55 smithi086 ceph-mon[102754]: pgmap v830: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 949 op/s 2024-06-14T23:02:56.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:55 smithi086 ceph-mon[102754]: from='client.45983 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:02:56.041 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:56.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:56.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:56.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:56.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:56.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:55 smithi137 ceph-mon[90634]: pgmap v830: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 949 op/s 2024-06-14T23:02:56.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:55 smithi137 ceph-mon[90634]: from='client.45983 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:02:56.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:56.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:56.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:56.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:56.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:56.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:56.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:02:56.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:56.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:02:56.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:56.733 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:56.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:56.794 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:02:56.794 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:02:56.794 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:02:56.794 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:02:56.794 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:02:56.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:56.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:56.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:56.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:57.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:57.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:57.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:57.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:57.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:57.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:57.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:57.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:57.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:57.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:57.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:57.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:57.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:57.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:57.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:57.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:57.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:57.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:57.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:57.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:57 smithi086 ceph-mon[97913]: pgmap v831: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-06-14T23:02:57.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:57.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:57.949 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:57 smithi086 ceph-mon[102754]: pgmap v831: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-06-14T23:02:57.949 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:57.949 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:57.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:57.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:57.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:58.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:58.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:58.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:57 smithi137 ceph-mon[90634]: pgmap v831: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-06-14T23:02:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:02:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:02:58.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:58.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:58.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:58.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:58.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:02:58.427 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:58.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:02:58.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:58.546 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":830,"stamp":"2024-06-14T23:02:54.732611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5735454686,"num_objects":87532,"num_object_clones":0,"num_object_copies":262596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174485,"num_write_kb":8381790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133350,"ondisk_log_size":133350,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22150176,"kb_used_data":16908640,"kb_used_omap":29,"kb_used_meta":5240866,"kb_avail":727811040,"statfs":{"total":767960285184,"available":745278504960,"internally_reserved":0,"allocated":17314447360,"data_stored":17306201766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":5366647461},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-622919680,"num_objects":-9505,"num_object_clones":0,"num_object_copies":-28515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9505,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007328"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":139,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:44.830668+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:02:44.830668+0000","last_peered":"2024-06-14T23:02:44.830668+0000","last_clean":"2024-06-14T23:02:44.830668+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:02:44.830668+0000","last_undegraded":"2024-06-14T23:02:44.830668+0000","last_fullsized":"2024-06-14T23:02:44.830668+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":171,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.844461+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T23:02:30.844461+0000","last_peered":"2024-06-14T23:02:30.844461+0000","last_clean":"2024-06-14T23:02:30.844461+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T23:02:30.844461+0000","last_undegraded":"2024-06-14T23:02:30.844461+0000","last_fullsized":"2024-06-14T23:02:30.844461+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":168,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837516+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:02:30.837516+0000","last_peered":"2024-06-14T23:02:30.837516+0000","last_clean":"2024-06-14T23:02:30.837516+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:02:30.837516+0000","last_undegraded":"2024-06-14T23:02:30.837516+0000","last_fullsized":"2024-06-14T23:02:30.837516+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":160,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.839613+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:02:30.839613+0000","last_peered":"2024-06-14T23:02:30.839613+0000","last_clean":"2024-06-14T23:02:30.839613+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:02:30.839613+0000","last_undegraded":"2024-06-14T23:02:30.839613+0000","last_fullsized":"2024-06-14T23:02:30.839613+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":161,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840527+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T23:02:30.840527+0000","last_peered":"2024-06-14T23:02:30.840527+0000","last_clean":"2024-06-14T23:02:30.840527+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T23:02:30.840527+0000","last_undegraded":"2024-06-14T23:02:30.840527+0000","last_fullsized":"2024-06-14T23:02:30.840527+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":75,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.839958+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T23:02:30.839958+0000","last_peered":"2024-06-14T23:02:30.839958+0000","last_clean":"2024-06-14T23:02:30.839958+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T23:02:30.839958+0000","last_undegraded":"2024-06-14T23:02:30.839958+0000","last_fullsized":"2024-06-14T23:02:30.839958+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":162,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.841811+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:02:30.841811+0000","last_peered":"2024-06-14T23:02:30.841811+0000","last_clean":"2024-06-14T23:02:30.841811+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:02:30.841811+0000","last_undegraded":"2024-06-14T23:02:30.841811+0000","last_fullsized":"2024-06-14T23:02:30.841811+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":177,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.835153+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T23:02:30.835153+0000","last_peered":"2024-06-14T23:02:30.835153+0000","last_clean":"2024-06-14T23:02:30.835153+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T23:02:30.835153+0000","last_undegraded":"2024-06-14T23:02:30.835153+0000","last_fullsized":"2024-06-14T23:02:30.835153+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"208'10934","reported_seq":10964,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:54.295553+0000","last_change":"2024-06-14T23:00:27.800011+0000","last_active":"2024-06-14T23:02:54.295553+0000","last_peered":"2024-06-14T23:02:54.295553+0000","last_clean":"2024-06-14T23:02:54.295553+0000","last_became_active":"2024-06-14T23:00:27.799716+0000","last_became_peered":"2024-06-14T23:00:27.799716+0000","last_unstale":"2024-06-14T23:02:54.295553+0000","last_undegraded":"2024-06-14T23:02:54.295553+0000","last_fullsized":"2024-06-14T23:02:54.295553+0000","mapping_epoch":186,"log_start":"191'1200","ondisk_log_start":"191'1200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9734,"log_dups_size":0,"ondisk_log_size":9734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:41:34.619251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351272960,"num_objects":5360,"num_object_clones":0,"num_object_copies":16080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10934,"num_write_kb":521408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":158,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.844914+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:02:30.844914+0000","last_peered":"2024-06-14T23:02:30.844914+0000","last_clean":"2024-06-14T23:02:30.844914+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:02:30.844914+0000","last_undegraded":"2024-06-14T23:02:30.844914+0000","last_fullsized":"2024-06-14T23:02:30.844914+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":151,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837315+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T23:02:30.837315+0000","last_peered":"2024-06-14T23:02:30.837315+0000","last_clean":"2024-06-14T23:02:30.837315+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T23:02:30.837315+0000","last_undegraded":"2024-06-14T23:02:30.837315+0000","last_fullsized":"2024-06-14T23:02:30.837315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"208'10956","reported_seq":10986,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:53.908554+0000","last_change":"2024-06-14T23:00:27.798857+0000","last_active":"2024-06-14T23:02:53.908554+0000","last_peered":"2024-06-14T23:02:53.908554+0000","last_clean":"2024-06-14T23:02:53.908554+0000","last_became_active":"2024-06-14T23:00:27.798566+0000","last_became_peered":"2024-06-14T23:00:27.798566+0000","last_unstale":"2024-06-14T23:02:53.908554+0000","last_undegraded":"2024-06-14T23:02:53.908554+0000","last_fullsized":"2024-06-14T23:02:53.908554+0000","mapping_epoch":186,"log_start":"192'3200","ondisk_log_start":"192'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7756,"log_dups_size":0,"ondisk_log_size":7756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:38.237668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355991552,"num_objects":5432,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10956,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":159,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.844819+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:02:30.844819+0000","last_peered":"2024-06-14T23:02:30.844819+0000","last_clean":"2024-06-14T23:02:30.844819+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:02:30.844819+0000","last_undegraded":"2024-06-14T23:02:30.844819+0000","last_fullsized":"2024-06-14T23:02:30.844819+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":185,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840056+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:02:30.840056+0000","last_peered":"2024-06-14T23:02:30.840056+0000","last_clean":"2024-06-14T23:02:30.840056+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:02:30.840056+0000","last_undegraded":"2024-06-14T23:02:30.840056+0000","last_fullsized":"2024-06-14T23:02:30.840056+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"208'10726","reported_seq":10758,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:52.036827+0000","last_change":"2024-06-14T23:00:27.797882+0000","last_active":"2024-06-14T23:02:52.036827+0000","last_peered":"2024-06-14T23:02:52.036827+0000","last_clean":"2024-06-14T23:02:52.036827+0000","last_became_active":"2024-06-14T23:00:27.797600+0000","last_became_peered":"2024-06-14T23:00:27.797600+0000","last_unstale":"2024-06-14T23:02:52.036827+0000","last_undegraded":"2024-06-14T23:02:52.036827+0000","last_fullsized":"2024-06-14T23:02:52.036827+0000","mapping_epoch":186,"log_start":"192'3000","ondisk_log_start":"192'3000","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7726,"log_dups_size":0,"ondisk_log_size":7726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:04:20.589709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359399424,"num_objects":5484,"num_object_clones":0,"num_object_copies":16452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10726,"num_write_kb":518720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":86,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837465+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T23:02:30.837465+0000","last_peered":"2024-06-14T23:02:30.837465+0000","last_clean":"2024-06-14T23:02:30.837465+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T23:02:30.837465+0000","last_undegraded":"2024-06-14T23:02:30.837465+0000","last_fullsized":"2024-06-14T23:02:30.837465+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":168,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.835154+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T23:02:30.835154+0000","last_peered":"2024-06-14T23:02:30.835154+0000","last_clean":"2024-06-14T23:02:30.835154+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T23:02:30.835154+0000","last_undegraded":"2024-06-14T23:02:30.835154+0000","last_fullsized":"2024-06-14T23:02:30.835154+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"208'11016","reported_seq":11048,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:50.364311+0000","last_change":"2024-06-14T23:00:27.794220+0000","last_active":"2024-06-14T23:02:50.364311+0000","last_peered":"2024-06-14T23:02:50.364311+0000","last_clean":"2024-06-14T23:02:50.364311+0000","last_became_active":"2024-06-14T23:00:27.793960+0000","last_became_peered":"2024-06-14T23:00:27.793960+0000","last_unstale":"2024-06-14T23:02:50.364311+0000","last_undegraded":"2024-06-14T23:02:50.364311+0000","last_fullsized":"2024-06-14T23:02:50.364311+0000","mapping_epoch":186,"log_start":"191'1000","ondisk_log_start":"191'1000","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:04:26.933537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378273792,"num_objects":5772,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11016,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":169,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837475+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T23:02:30.837475+0000","last_peered":"2024-06-14T23:02:30.837475+0000","last_clean":"2024-06-14T23:02:30.837475+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T23:02:30.837475+0000","last_undegraded":"2024-06-14T23:02:30.837475+0000","last_fullsized":"2024-06-14T23:02:30.837475+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":162,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:44.852539+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T23:02:44.852539+0000","last_peered":"2024-06-14T23:02:44.852539+0000","last_clean":"2024-06-14T23:02:44.852539+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T23:02:44.852539+0000","last_undegraded":"2024-06-14T23:02:44.852539+0000","last_fullsized":"2024-06-14T23:02:44.852539+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"208'10865","reported_seq":10896,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:50.060680+0000","last_change":"2024-06-14T23:00:27.797355+0000","last_active":"2024-06-14T23:02:50.060680+0000","last_peered":"2024-06-14T23:02:50.060680+0000","last_clean":"2024-06-14T23:02:50.060680+0000","last_became_active":"2024-06-14T23:00:27.796666+0000","last_became_peered":"2024-06-14T23:00:27.796666+0000","last_unstale":"2024-06-14T23:02:50.060680+0000","last_undegraded":"2024-06-14T23:02:50.060680+0000","last_fullsized":"2024-06-14T23:02:50.060680+0000","mapping_epoch":186,"log_start":"192'2500","ondisk_log_start":"192'2500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":8365,"log_dups_size":0,"ondisk_log_size":8365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:57:29.139721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371130368,"num_objects":5663,"num_object_clones":0,"num_object_copies":16989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10865,"num_write_kb":528896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":507,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.834500+0000","last_change":"2024-06-14T23:00:30.637455+0000","last_active":"2024-06-14T23:02:30.834500+0000","last_peered":"2024-06-14T23:02:30.834500+0000","last_clean":"2024-06-14T23:02:30.834500+0000","last_became_active":"2024-06-14T23:00:17.398732+0000","last_became_peered":"2024-06-14T23:00:17.398732+0000","last_unstale":"2024-06-14T23:02:30.834500+0000","last_undegraded":"2024-06-14T23:02:30.834500+0000","last_fullsized":"2024-06-14T23:02:30.834500+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.637373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.637373+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:10.033075+0000","scrub_duration":0.0016781470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":78,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:42.011009+0000","last_change":"2024-06-14T22:58:09.752985+0000","last_active":"2024-06-14T23:02:42.011009+0000","last_peered":"2024-06-14T23:02:42.011009+0000","last_clean":"2024-06-14T23:02:42.011009+0000","last_became_active":"2024-06-14T22:58:09.752630+0000","last_became_peered":"2024-06-14T22:58:09.752630+0000","last_unstale":"2024-06-14T23:02:42.011009+0000","last_undegraded":"2024-06-14T23:02:42.011009+0000","last_fullsized":"2024-06-14T23:02:42.011009+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:22.320986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":168,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840361+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T23:02:30.840361+0000","last_peered":"2024-06-14T23:02:30.840361+0000","last_clean":"2024-06-14T23:02:30.840361+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T23:02:30.840361+0000","last_undegraded":"2024-06-14T23:02:30.840361+0000","last_fullsized":"2024-06-14T23:02:30.840361+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"208'10774","reported_seq":10806,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:50.321581+0000","last_change":"2024-06-14T23:00:27.797468+0000","last_active":"2024-06-14T23:02:50.321581+0000","last_peered":"2024-06-14T23:02:50.321581+0000","last_clean":"2024-06-14T23:02:50.321581+0000","last_became_active":"2024-06-14T23:00:27.797180+0000","last_became_peered":"2024-06-14T23:00:27.797180+0000","last_unstale":"2024-06-14T23:02:50.321581+0000","last_undegraded":"2024-06-14T23:02:50.321581+0000","last_fullsized":"2024-06-14T23:02:50.321581+0000","mapping_epoch":186,"log_start":"191'700","ondisk_log_start":"191'700","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:56:13.872188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373948416,"num_objects":5706,"num_object_clones":0,"num_object_copies":17118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10774,"num_write_kb":527360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":505,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837223+0000","last_change":"2024-06-14T23:00:33.815761+0000","last_active":"2024-06-14T23:02:30.837223+0000","last_peered":"2024-06-14T23:02:30.837223+0000","last_clean":"2024-06-14T23:02:30.837223+0000","last_became_active":"2024-06-14T23:00:17.400781+0000","last_became_peered":"2024-06-14T23:00:17.400781+0000","last_unstale":"2024-06-14T23:02:30.837223+0000","last_undegraded":"2024-06-14T23:02:30.837223+0000","last_fullsized":"2024-06-14T23:02:30.837223+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:21:41.196504+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":169,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.835330+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T23:02:30.835330+0000","last_peered":"2024-06-14T23:02:30.835330+0000","last_clean":"2024-06-14T23:02:30.835330+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T23:02:30.835330+0000","last_undegraded":"2024-06-14T23:02:30.835330+0000","last_fullsized":"2024-06-14T23:02:30.835330+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":178,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840533+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T23:02:30.840533+0000","last_peered":"2024-06-14T23:02:30.840533+0000","last_clean":"2024-06-14T23:02:30.840533+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T23:02:30.840533+0000","last_undegraded":"2024-06-14T23:02:30.840533+0000","last_fullsized":"2024-06-14T23:02:30.840533+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"208'10946","reported_seq":10976,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:53.854458+0000","last_change":"2024-06-14T23:00:27.802274+0000","last_active":"2024-06-14T23:02:53.854458+0000","last_peered":"2024-06-14T23:02:53.854458+0000","last_clean":"2024-06-14T23:02:53.854458+0000","last_became_active":"2024-06-14T23:00:27.801956+0000","last_became_peered":"2024-06-14T23:00:27.801956+0000","last_unstale":"2024-06-14T23:02:53.854458+0000","last_undegraded":"2024-06-14T23:02:53.854458+0000","last_fullsized":"2024-06-14T23:02:53.854458+0000","mapping_epoch":186,"log_start":"192'3200","ondisk_log_start":"192'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7746,"log_dups_size":0,"ondisk_log_size":7746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:23.791512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354025472,"num_objects":5402,"num_object_clones":0,"num_object_copies":16206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10946,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":169,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840112+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T23:02:30.840112+0000","last_peered":"2024-06-14T23:02:30.840112+0000","last_clean":"2024-06-14T23:02:30.840112+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T23:02:30.840112+0000","last_undegraded":"2024-06-14T23:02:30.840112+0000","last_fullsized":"2024-06-14T23:02:30.840112+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":168,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840544+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T23:02:30.840544+0000","last_peered":"2024-06-14T23:02:30.840544+0000","last_clean":"2024-06-14T23:02:30.840544+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T23:02:30.840544+0000","last_undegraded":"2024-06-14T23:02:30.840544+0000","last_fullsized":"2024-06-14T23:02:30.840544+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"208'10962","reported_seq":10992,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:53.833014+0000","last_change":"2024-06-14T23:00:27.801110+0000","last_active":"2024-06-14T23:02:53.833014+0000","last_peered":"2024-06-14T23:02:53.833014+0000","last_clean":"2024-06-14T23:02:53.833014+0000","last_became_active":"2024-06-14T23:00:27.800817+0000","last_became_peered":"2024-06-14T23:00:27.800817+0000","last_unstale":"2024-06-14T23:02:53.833014+0000","last_undegraded":"2024-06-14T23:02:53.833014+0000","last_fullsized":"2024-06-14T23:02:53.833014+0000","mapping_epoch":186,"log_start":"194'4100","ondisk_log_start":"194'4100","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:48:04.422151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352976896,"num_objects":5386,"num_object_clones":0,"num_object_copies":16158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10962,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":505,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837260+0000","last_change":"2024-06-14T23:00:31.843436+0000","last_active":"2024-06-14T23:02:30.837260+0000","last_peered":"2024-06-14T23:02:30.837260+0000","last_clean":"2024-06-14T23:02:30.837260+0000","last_became_active":"2024-06-14T23:00:17.398181+0000","last_became_peered":"2024-06-14T23:00:17.398181+0000","last_unstale":"2024-06-14T23:02:30.837260+0000","last_undegraded":"2024-06-14T23:02:30.837260+0000","last_fullsized":"2024-06-14T23:02:30.837260+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:05:06.069424+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":164,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:44.612474+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:02:44.612474+0000","last_peered":"2024-06-14T23:02:44.612474+0000","last_clean":"2024-06-14T23:02:44.612474+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:02:44.612474+0000","last_undegraded":"2024-06-14T23:02:44.612474+0000","last_fullsized":"2024-06-14T23:02:44.612474+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":176,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.833042+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:02:30.833042+0000","last_peered":"2024-06-14T23:02:30.833042+0000","last_clean":"2024-06-14T23:02:30.833042+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:02:30.833042+0000","last_undegraded":"2024-06-14T23:02:30.833042+0000","last_fullsized":"2024-06-14T23:02:30.833042+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"208'11104","reported_seq":11134,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:53.905514+0000","last_change":"2024-06-14T23:00:27.797497+0000","last_active":"2024-06-14T23:02:53.905514+0000","last_peered":"2024-06-14T23:02:53.905514+0000","last_clean":"2024-06-14T23:02:53.905514+0000","last_became_active":"2024-06-14T23:00:27.797221+0000","last_became_peered":"2024-06-14T23:00:27.797221+0000","last_unstale":"2024-06-14T23:02:53.905514+0000","last_undegraded":"2024-06-14T23:02:53.905514+0000","last_fullsized":"2024-06-14T23:02:53.905514+0000","mapping_epoch":186,"log_start":"192'3400","ondisk_log_start":"192'3400","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7704,"log_dups_size":0,"ondisk_log_size":7704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:24:58.530131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357564416,"num_objects":5456,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11104,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":498,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840351+0000","last_change":"2024-06-14T23:00:30.610273+0000","last_active":"2024-06-14T23:02:30.840351+0000","last_peered":"2024-06-14T23:02:30.840351+0000","last_clean":"2024-06-14T23:02:30.840351+0000","last_became_active":"2024-06-14T23:00:09.766095+0000","last_became_peered":"2024-06-14T23:00:09.766095+0000","last_unstale":"2024-06-14T23:02:30.840351+0000","last_undegraded":"2024-06-14T23:02:30.840351+0000","last_fullsized":"2024-06-14T23:02:30.840351+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.610180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.610180+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:53.521891+0000","scrub_duration":0.00217396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":173,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.841822+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T23:02:30.841822+0000","last_peered":"2024-06-14T23:02:30.841822+0000","last_clean":"2024-06-14T23:02:30.841822+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T23:02:30.841822+0000","last_undegraded":"2024-06-14T23:02:30.841822+0000","last_fullsized":"2024-06-14T23:02:30.841822+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":112,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837345+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T23:02:30.837345+0000","last_peered":"2024-06-14T23:02:30.837345+0000","last_clean":"2024-06-14T23:02:30.837345+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T23:02:30.837345+0000","last_undegraded":"2024-06-14T23:02:30.837345+0000","last_fullsized":"2024-06-14T23:02:30.837345+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":151,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.835434+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T23:02:30.835434+0000","last_peered":"2024-06-14T23:02:30.835434+0000","last_clean":"2024-06-14T23:02:30.835434+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T23:02:30.835434+0000","last_undegraded":"2024-06-14T23:02:30.835434+0000","last_fullsized":"2024-06-14T23:02:30.835434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"208'10750","reported_seq":10782,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:52.043497+0000","last_change":"2024-06-14T23:00:27.797246+0000","last_active":"2024-06-14T23:02:52.043497+0000","last_peered":"2024-06-14T23:02:52.043497+0000","last_clean":"2024-06-14T23:02:52.043497+0000","last_became_active":"2024-06-14T23:00:27.796988+0000","last_became_peered":"2024-06-14T23:00:27.796988+0000","last_unstale":"2024-06-14T23:02:52.043497+0000","last_undegraded":"2024-06-14T23:02:52.043497+0000","last_fullsized":"2024-06-14T23:02:52.043497+0000","mapping_epoch":186,"log_start":"192'3000","ondisk_log_start":"192'3000","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:17.545715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358350848,"num_objects":5468,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10750,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":507,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.828041+0000","last_change":"2024-06-14T23:00:30.719758+0000","last_active":"2024-06-14T23:02:30.828041+0000","last_peered":"2024-06-14T23:02:30.828041+0000","last_clean":"2024-06-14T23:02:30.828041+0000","last_became_active":"2024-06-14T23:00:17.403817+0000","last_became_peered":"2024-06-14T23:00:17.403817+0000","last_unstale":"2024-06-14T23:02:30.828041+0000","last_undegraded":"2024-06-14T23:02:30.828041+0000","last_fullsized":"2024-06-14T23:02:30.828041+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:03:47.728089+0000","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":167,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.828097+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:02:30.828097+0000","last_peered":"2024-06-14T23:02:30.828097+0000","last_clean":"2024-06-14T23:02:30.828097+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:02:30.828097+0000","last_undegraded":"2024-06-14T23:02:30.828097+0000","last_fullsized":"2024-06-14T23:02:30.828097+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":161,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.839776+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T23:02:30.839776+0000","last_peered":"2024-06-14T23:02:30.839776+0000","last_clean":"2024-06-14T23:02:30.839776+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T23:02:30.839776+0000","last_undegraded":"2024-06-14T23:02:30.839776+0000","last_fullsized":"2024-06-14T23:02:30.839776+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":175,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840755+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T23:02:30.840755+0000","last_peered":"2024-06-14T23:02:30.840755+0000","last_clean":"2024-06-14T23:02:30.840755+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T23:02:30.840755+0000","last_undegraded":"2024-06-14T23:02:30.840755+0000","last_fullsized":"2024-06-14T23:02:30.840755+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"208'10770","reported_seq":10800,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:54.367007+0000","last_change":"2024-06-14T23:00:27.801838+0000","last_active":"2024-06-14T23:02:54.367007+0000","last_peered":"2024-06-14T23:02:54.367007+0000","last_clean":"2024-06-14T23:02:54.367007+0000","last_became_active":"2024-06-14T23:00:27.801477+0000","last_became_peered":"2024-06-14T23:00:27.801477+0000","last_unstale":"2024-06-14T23:02:54.367007+0000","last_undegraded":"2024-06-14T23:02:54.367007+0000","last_fullsized":"2024-06-14T23:02:54.367007+0000","mapping_epoch":186,"log_start":"191'1000","ondisk_log_start":"191'1000","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:59:17.239200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351928320,"num_objects":5370,"num_object_clones":0,"num_object_copies":16110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10770,"num_write_kb":516480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":515,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.839531+0000","last_change":"2024-06-14T23:00:31.643024+0000","last_active":"2024-06-14T23:02:30.839531+0000","last_peered":"2024-06-14T23:02:30.839531+0000","last_clean":"2024-06-14T23:02:30.839531+0000","last_became_active":"2024-06-14T23:00:17.400819+0000","last_became_peered":"2024-06-14T23:00:17.400819+0000","last_unstale":"2024-06-14T23:02:30.839531+0000","last_undegraded":"2024-06-14T23:02:30.839531+0000","last_fullsized":"2024-06-14T23:02:30.839531+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.642940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.642940+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:52:45.475940+0000","scrub_duration":0.001647253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":162,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.834807+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T23:02:30.834807+0000","last_peered":"2024-06-14T23:02:30.834807+0000","last_clean":"2024-06-14T23:02:30.834807+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T23:02:30.834807+0000","last_undegraded":"2024-06-14T23:02:30.834807+0000","last_fullsized":"2024-06-14T23:02:30.834807+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":177,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.828316+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T23:02:30.828316+0000","last_peered":"2024-06-14T23:02:30.828316+0000","last_clean":"2024-06-14T23:02:30.828316+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T23:02:30.828316+0000","last_undegraded":"2024-06-14T23:02:30.828316+0000","last_fullsized":"2024-06-14T23:02:30.828316+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":162,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.832890+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:02:30.832890+0000","last_peered":"2024-06-14T23:02:30.832890+0000","last_clean":"2024-06-14T23:02:30.832890+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:02:30.832890+0000","last_undegraded":"2024-06-14T23:02:30.832890+0000","last_fullsized":"2024-06-14T23:02:30.832890+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"208'10930","reported_seq":10959,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:53.885547+0000","last_change":"2024-06-14T23:00:27.799968+0000","last_active":"2024-06-14T23:02:53.885547+0000","last_peered":"2024-06-14T23:02:53.885547+0000","last_clean":"2024-06-14T23:02:53.885547+0000","last_became_active":"2024-06-14T23:00:27.799651+0000","last_became_peered":"2024-06-14T23:00:27.799651+0000","last_unstale":"2024-06-14T23:02:53.885547+0000","last_undegraded":"2024-06-14T23:02:53.885547+0000","last_fullsized":"2024-06-14T23:02:53.885547+0000","mapping_epoch":186,"log_start":"192'3200","ondisk_log_start":"192'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7730,"log_dups_size":0,"ondisk_log_size":7730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:23:57.581265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357695488,"num_objects":5458,"num_object_clones":0,"num_object_copies":16374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10930,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":504,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837293+0000","last_change":"2024-06-14T23:00:32.818134+0000","last_active":"2024-06-14T23:02:30.837293+0000","last_peered":"2024-06-14T23:02:30.837293+0000","last_clean":"2024-06-14T23:02:30.837293+0000","last_became_active":"2024-06-14T23:00:17.399852+0000","last_became_peered":"2024-06-14T23:00:17.399852+0000","last_unstale":"2024-06-14T23:02:30.837293+0000","last_undegraded":"2024-06-14T23:02:30.837293+0000","last_fullsized":"2024-06-14T23:02:30.837293+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.818076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.818076+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:12:39.666043+0000","scrub_duration":0.0013568390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":173,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837425+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T23:02:30.837425+0000","last_peered":"2024-06-14T23:02:30.837425+0000","last_clean":"2024-06-14T23:02:30.837425+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T23:02:30.837425+0000","last_undegraded":"2024-06-14T23:02:30.837425+0000","last_fullsized":"2024-06-14T23:02:30.837425+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":169,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840229+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T23:02:30.840229+0000","last_peered":"2024-06-14T23:02:30.840229+0000","last_clean":"2024-06-14T23:02:30.840229+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T23:02:30.840229+0000","last_undegraded":"2024-06-14T23:02:30.840229+0000","last_fullsized":"2024-06-14T23:02:30.840229+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":192,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.832216+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T23:02:30.832216+0000","last_peered":"2024-06-14T23:02:30.832216+0000","last_clean":"2024-06-14T23:02:30.832216+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T23:02:30.832216+0000","last_undegraded":"2024-06-14T23:02:30.832216+0000","last_fullsized":"2024-06-14T23:02:30.832216+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"208'10958","reported_seq":10988,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:53.993685+0000","last_change":"2024-06-14T23:00:27.803439+0000","last_active":"2024-06-14T23:02:53.993685+0000","last_peered":"2024-06-14T23:02:53.993685+0000","last_clean":"2024-06-14T23:02:53.993685+0000","last_became_active":"2024-06-14T23:00:27.802783+0000","last_became_peered":"2024-06-14T23:00:27.802783+0000","last_unstale":"2024-06-14T23:02:53.993685+0000","last_undegraded":"2024-06-14T23:02:53.993685+0000","last_fullsized":"2024-06-14T23:02:53.993685+0000","mapping_epoch":186,"log_start":"192'3000","ondisk_log_start":"192'3000","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:35:39.179455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355074048,"num_objects":5418,"num_object_clones":0,"num_object_copies":16254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10958,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":458,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.835429+0000","last_change":"2024-06-14T23:00:42.661659+0000","last_active":"2024-06-14T23:02:30.835429+0000","last_peered":"2024-06-14T23:02:30.835429+0000","last_clean":"2024-06-14T23:02:30.835429+0000","last_became_active":"2024-06-14T23:00:17.398852+0000","last_became_peered":"2024-06-14T23:00:17.398852+0000","last_unstale":"2024-06-14T23:02:30.835429+0000","last_undegraded":"2024-06-14T23:02:30.835429+0000","last_fullsized":"2024-06-14T23:02:30.835429+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:47:14.444805+0000","scrub_duration":0.0053445919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":186,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837592+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T23:02:30.837592+0000","last_peered":"2024-06-14T23:02:30.837592+0000","last_clean":"2024-06-14T23:02:30.837592+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T23:02:30.837592+0000","last_undegraded":"2024-06-14T23:02:30.837592+0000","last_fullsized":"2024-06-14T23:02:30.837592+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":167,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.832372+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:02:30.832372+0000","last_peered":"2024-06-14T23:02:30.832372+0000","last_clean":"2024-06-14T23:02:30.832372+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:02:30.832372+0000","last_undegraded":"2024-06-14T23:02:30.832372+0000","last_fullsized":"2024-06-14T23:02:30.832372+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":162,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.828551+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:02:30.828551+0000","last_peered":"2024-06-14T23:02:30.828551+0000","last_clean":"2024-06-14T23:02:30.828551+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:02:30.828551+0000","last_undegraded":"2024-06-14T23:02:30.828551+0000","last_fullsized":"2024-06-14T23:02:30.828551+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"208'11016","reported_seq":11047,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:53.882470+0000","last_change":"2024-06-14T23:00:27.800097+0000","last_active":"2024-06-14T23:02:53.882470+0000","last_peered":"2024-06-14T23:02:53.882470+0000","last_clean":"2024-06-14T23:02:53.882470+0000","last_became_active":"2024-06-14T23:00:27.799790+0000","last_became_peered":"2024-06-14T23:00:27.799790+0000","last_unstale":"2024-06-14T23:02:53.882470+0000","last_undegraded":"2024-06-14T23:02:53.882470+0000","last_fullsized":"2024-06-14T23:02:53.882470+0000","mapping_epoch":186,"log_start":"192'3300","ondisk_log_start":"192'3300","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:21:01.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352911384,"num_objects":5386,"num_object_clones":0,"num_object_copies":16158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5386,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11016,"num_write_kb":524801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":458,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.835244+0000","last_change":"2024-06-14T23:00:31.555325+0000","last_active":"2024-06-14T23:02:30.835244+0000","last_peered":"2024-06-14T23:02:30.835244+0000","last_clean":"2024-06-14T23:02:30.835244+0000","last_became_active":"2024-06-14T23:00:17.400871+0000","last_became_peered":"2024-06-14T23:00:17.400871+0000","last_unstale":"2024-06-14T23:02:30.835244+0000","last_undegraded":"2024-06-14T23:02:30.835244+0000","last_fullsized":"2024-06-14T23:02:30.835244+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:58.191436+0000","scrub_duration":0.001665094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":177,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.832306+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T23:02:30.832306+0000","last_peered":"2024-06-14T23:02:30.832306+0000","last_clean":"2024-06-14T23:02:30.832306+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T23:02:30.832306+0000","last_undegraded":"2024-06-14T23:02:30.832306+0000","last_fullsized":"2024-06-14T23:02:30.832306+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":173,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.844609+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:02:30.844609+0000","last_peered":"2024-06-14T23:02:30.844609+0000","last_clean":"2024-06-14T23:02:30.844609+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:02:30.844609+0000","last_undegraded":"2024-06-14T23:02:30.844609+0000","last_fullsized":"2024-06-14T23:02:30.844609+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":147,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840293+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T23:02:30.840293+0000","last_peered":"2024-06-14T23:02:30.840293+0000","last_clean":"2024-06-14T23:02:30.840293+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T23:02:30.840293+0000","last_undegraded":"2024-06-14T23:02:30.840293+0000","last_fullsized":"2024-06-14T23:02:30.840293+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"208'11028","reported_seq":11058,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:53.832827+0000","last_change":"2024-06-14T23:00:27.797841+0000","last_active":"2024-06-14T23:02:53.832827+0000","last_peered":"2024-06-14T23:02:53.832827+0000","last_clean":"2024-06-14T23:02:53.832827+0000","last_became_active":"2024-06-14T23:00:27.797256+0000","last_became_peered":"2024-06-14T23:00:27.797256+0000","last_unstale":"2024-06-14T23:02:53.832827+0000","last_undegraded":"2024-06-14T23:02:53.832827+0000","last_fullsized":"2024-06-14T23:02:53.832827+0000","mapping_epoch":186,"log_start":"194'4200","ondisk_log_start":"194'4200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:52:24.873796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359268352,"num_objects":5482,"num_object_clones":0,"num_object_copies":16446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11028,"num_write_kb":528320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":505,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840441+0000","last_change":"2024-06-14T23:00:32.174187+0000","last_active":"2024-06-14T23:02:30.840441+0000","last_peered":"2024-06-14T23:02:30.840441+0000","last_clean":"2024-06-14T23:02:30.840441+0000","last_became_active":"2024-06-14T23:00:17.395654+0000","last_became_peered":"2024-06-14T23:00:17.395654+0000","last_unstale":"2024-06-14T23:02:30.840441+0000","last_undegraded":"2024-06-14T23:02:30.840441+0000","last_fullsized":"2024-06-14T23:02:30.840441+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.174132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.174132+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:59.252795+0000","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":69,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:31.985911+0000","last_change":"2024-06-14T22:58:09.752729+0000","last_active":"2024-06-14T23:02:31.985911+0000","last_peered":"2024-06-14T23:02:31.985911+0000","last_clean":"2024-06-14T23:02:31.985911+0000","last_became_active":"2024-06-14T22:58:09.752393+0000","last_became_peered":"2024-06-14T22:58:09.752393+0000","last_unstale":"2024-06-14T23:02:31.985911+0000","last_undegraded":"2024-06-14T23:02:31.985911+0000","last_fullsized":"2024-06-14T23:02:31.985911+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:21:58.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":161,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.832766+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T23:02:30.832766+0000","last_peered":"2024-06-14T23:02:30.832766+0000","last_clean":"2024-06-14T23:02:30.832766+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T23:02:30.832766+0000","last_undegraded":"2024-06-14T23:02:30.832766+0000","last_fullsized":"2024-06-14T23:02:30.832766+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":153,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.844097+0000","last_change":"2024-06-14T22:58:32.754079+0000","last_active":"2024-06-14T23:02:30.844097+0000","last_peered":"2024-06-14T23:02:30.844097+0000","last_clean":"2024-06-14T23:02:30.844097+0000","last_became_active":"2024-06-14T22:58:32.753814+0000","last_became_peered":"2024-06-14T22:58:32.753814+0000","last_unstale":"2024-06-14T23:02:30.844097+0000","last_undegraded":"2024-06-14T23:02:30.844097+0000","last_fullsized":"2024-06-14T23:02:30.844097+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:37:07.414370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"208'10689","reported_seq":10719,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:53.903196+0000","last_change":"2024-06-14T23:00:27.798887+0000","last_active":"2024-06-14T23:02:53.903196+0000","last_peered":"2024-06-14T23:02:53.903196+0000","last_clean":"2024-06-14T23:02:53.903196+0000","last_became_active":"2024-06-14T23:00:27.798590+0000","last_became_peered":"2024-06-14T23:00:27.798590+0000","last_unstale":"2024-06-14T23:02:53.903196+0000","last_undegraded":"2024-06-14T23:02:53.903196+0000","last_fullsized":"2024-06-14T23:02:53.903196+0000","mapping_epoch":186,"log_start":"192'2900","ondisk_log_start":"192'2900","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7789,"log_dups_size":0,"ondisk_log_size":7789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:02:33.396409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345047040,"num_objects":5265,"num_object_clones":0,"num_object_copies":15795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10689,"num_write_kb":510528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":507,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.836575+0000","last_change":"2024-06-14T23:00:32.716013+0000","last_active":"2024-06-14T23:02:30.836575+0000","last_peered":"2024-06-14T23:02:30.836575+0000","last_clean":"2024-06-14T23:02:30.836575+0000","last_became_active":"2024-06-14T23:00:17.398349+0000","last_became_peered":"2024-06-14T23:00:17.398349+0000","last_unstale":"2024-06-14T23:02:30.836575+0000","last_undegraded":"2024-06-14T23:02:30.836575+0000","last_fullsized":"2024-06-14T23:02:30.836575+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.715907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.715907+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:23:40.212866+0000","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":165,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840606+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T23:02:30.840606+0000","last_peered":"2024-06-14T23:02:30.840606+0000","last_clean":"2024-06-14T23:02:30.840606+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T23:02:30.840606+0000","last_undegraded":"2024-06-14T23:02:30.840606+0000","last_fullsized":"2024-06-14T23:02:30.840606+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":173,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.835541+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T23:02:30.835541+0000","last_peered":"2024-06-14T23:02:30.835541+0000","last_clean":"2024-06-14T23:02:30.835541+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T23:02:30.835541+0000","last_undegraded":"2024-06-14T23:02:30.835541+0000","last_fullsized":"2024-06-14T23:02:30.835541+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":151,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.839961+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T23:02:30.839961+0000","last_peered":"2024-06-14T23:02:30.839961+0000","last_clean":"2024-06-14T23:02:30.839961+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T23:02:30.839961+0000","last_undegraded":"2024-06-14T23:02:30.839961+0000","last_fullsized":"2024-06-14T23:02:30.839961+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":168,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840194+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T23:02:30.840194+0000","last_peered":"2024-06-14T23:02:30.840194+0000","last_clean":"2024-06-14T23:02:30.840194+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T23:02:30.840194+0000","last_undegraded":"2024-06-14T23:02:30.840194+0000","last_fullsized":"2024-06-14T23:02:30.840194+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":174,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840394+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T23:02:30.840394+0000","last_peered":"2024-06-14T23:02:30.840394+0000","last_clean":"2024-06-14T23:02:30.840394+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T23:02:30.840394+0000","last_undegraded":"2024-06-14T23:02:30.840394+0000","last_fullsized":"2024-06-14T23:02:30.840394+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":176,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.827741+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T23:02:30.827741+0000","last_peered":"2024-06-14T23:02:30.827741+0000","last_clean":"2024-06-14T23:02:30.827741+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T23:02:30.827741+0000","last_undegraded":"2024-06-14T23:02:30.827741+0000","last_fullsized":"2024-06-14T23:02:30.827741+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":87,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.838957+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:02:30.838957+0000","last_peered":"2024-06-14T23:02:30.838957+0000","last_clean":"2024-06-14T23:02:30.838957+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:02:30.838957+0000","last_undegraded":"2024-06-14T23:02:30.838957+0000","last_fullsized":"2024-06-14T23:02:30.838957+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":169,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837635+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:02:30.837635+0000","last_peered":"2024-06-14T23:02:30.837635+0000","last_clean":"2024-06-14T23:02:30.837635+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:02:30.837635+0000","last_undegraded":"2024-06-14T23:02:30.837635+0000","last_fullsized":"2024-06-14T23:02:30.837635+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":112,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.834982+0000","last_change":"2024-06-14T22:58:09.753103+0000","last_active":"2024-06-14T23:02:30.834982+0000","last_peered":"2024-06-14T23:02:30.834982+0000","last_clean":"2024-06-14T23:02:30.834982+0000","last_became_active":"2024-06-14T22:58:09.752761+0000","last_became_peered":"2024-06-14T22:58:09.752761+0000","last_unstale":"2024-06-14T23:02:30.834982+0000","last_undegraded":"2024-06-14T23:02:30.834982+0000","last_fullsized":"2024-06-14T23:02:30.834982+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:20:20.619331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":160,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.841068+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T23:02:30.841068+0000","last_peered":"2024-06-14T23:02:30.841068+0000","last_clean":"2024-06-14T23:02:30.841068+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T23:02:30.841068+0000","last_undegraded":"2024-06-14T23:02:30.841068+0000","last_fullsized":"2024-06-14T23:02:30.841068+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":162,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.836005+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T23:02:30.836005+0000","last_peered":"2024-06-14T23:02:30.836005+0000","last_clean":"2024-06-14T23:02:30.836005+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T23:02:30.836005+0000","last_undegraded":"2024-06-14T23:02:30.836005+0000","last_fullsized":"2024-06-14T23:02:30.836005+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":168,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.835470+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T23:02:30.835470+0000","last_peered":"2024-06-14T23:02:30.835470+0000","last_clean":"2024-06-14T23:02:30.835470+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T23:02:30.835470+0000","last_undegraded":"2024-06-14T23:02:30.835470+0000","last_fullsized":"2024-06-14T23:02:30.835470+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":156,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840274+0000","last_change":"2024-06-14T22:58:09.785074+0000","last_active":"2024-06-14T23:02:30.840274+0000","last_peered":"2024-06-14T23:02:30.840274+0000","last_clean":"2024-06-14T23:02:30.840274+0000","last_became_active":"2024-06-14T22:58:09.784309+0000","last_became_peered":"2024-06-14T22:58:09.784309+0000","last_unstale":"2024-06-14T23:02:30.840274+0000","last_undegraded":"2024-06-14T23:02:30.840274+0000","last_fullsized":"2024-06-14T23:02:30.840274+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:23.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":163,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-06-14T23:02:44.853097+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T23:02:44.853097+0000","last_peered":"2024-06-14T23:02:44.853097+0000","last_clean":"2024-06-14T23:02:44.853097+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T23:02:44.853097+0000","last_undegraded":"2024-06-14T23:02:44.853097+0000","last_fullsized":"2024-06-14T23:02:44.853097+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":176,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.827315+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:02:30.827315+0000","last_peered":"2024-06-14T23:02:30.827315+0000","last_clean":"2024-06-14T23:02:30.827315+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:02:30.827315+0000","last_undegraded":"2024-06-14T23:02:30.827315+0000","last_fullsized":"2024-06-14T23:02:30.827315+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":164,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840791+0000","last_change":"2024-06-14T22:58:09.756040+0000","last_active":"2024-06-14T23:02:30.840791+0000","last_peered":"2024-06-14T23:02:30.840791+0000","last_clean":"2024-06-14T23:02:30.840791+0000","last_became_active":"2024-06-14T22:58:09.755659+0000","last_became_peered":"2024-06-14T22:58:09.755659+0000","last_unstale":"2024-06-14T23:02:30.840791+0000","last_undegraded":"2024-06-14T23:02:30.840791+0000","last_fullsized":"2024-06-14T23:02:30.840791+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:40.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":187,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840217+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T23:02:30.840217+0000","last_peered":"2024-06-14T23:02:30.840217+0000","last_clean":"2024-06-14T23:02:30.840217+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T23:02:30.840217+0000","last_undegraded":"2024-06-14T23:02:30.840217+0000","last_fullsized":"2024-06-14T23:02:30.840217+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":181,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.834468+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T23:02:30.834468+0000","last_peered":"2024-06-14T23:02:30.834468+0000","last_clean":"2024-06-14T23:02:30.834468+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T23:02:30.834468+0000","last_undegraded":"2024-06-14T23:02:30.834468+0000","last_fullsized":"2024-06-14T23:02:30.834468+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":149,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840203+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T23:02:30.840203+0000","last_peered":"2024-06-14T23:02:30.840203+0000","last_clean":"2024-06-14T23:02:30.840203+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T23:02:30.840203+0000","last_undegraded":"2024-06-14T23:02:30.840203+0000","last_fullsized":"2024-06-14T23:02:30.840203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":185,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840144+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:02:30.840144+0000","last_peered":"2024-06-14T23:02:30.840144+0000","last_clean":"2024-06-14T23:02:30.840144+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:02:30.840144+0000","last_undegraded":"2024-06-14T23:02:30.840144+0000","last_fullsized":"2024-06-14T23:02:30.840144+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":171,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837459+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:02:30.837459+0000","last_peered":"2024-06-14T23:02:30.837459+0000","last_clean":"2024-06-14T23:02:30.837459+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:02:30.837459+0000","last_undegraded":"2024-06-14T23:02:30.837459+0000","last_fullsized":"2024-06-14T23:02:30.837459+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":185,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840174+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:02:30.840174+0000","last_peered":"2024-06-14T23:02:30.840174+0000","last_clean":"2024-06-14T23:02:30.840174+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:02:30.840174+0000","last_undegraded":"2024-06-14T23:02:30.840174+0000","last_fullsized":"2024-06-14T23:02:30.840174+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":154,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.837545+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T23:02:30.837545+0000","last_peered":"2024-06-14T23:02:30.837545+0000","last_clean":"2024-06-14T23:02:30.837545+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T23:02:30.837545+0000","last_undegraded":"2024-06-14T23:02:30.837545+0000","last_fullsized":"2024-06-14T23:02:30.837545+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":171,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.834689+0000","last_change":"2024-06-14T22:58:09.752281+0000","last_active":"2024-06-14T23:02:30.834689+0000","last_peered":"2024-06-14T23:02:30.834689+0000","last_clean":"2024-06-14T23:02:30.834689+0000","last_became_active":"2024-06-14T22:58:09.751952+0000","last_became_peered":"2024-06-14T22:58:09.751952+0000","last_unstale":"2024-06-14T23:02:30.834689+0000","last_undegraded":"2024-06-14T23:02:30.834689+0000","last_fullsized":"2024-06-14T23:02:30.834689+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:39:19.836734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":67,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.835329+0000","last_change":"2024-06-14T22:58:09.755622+0000","last_active":"2024-06-14T23:02:30.835329+0000","last_peered":"2024-06-14T23:02:30.835329+0000","last_clean":"2024-06-14T23:02:30.835329+0000","last_became_active":"2024-06-14T22:58:09.755383+0000","last_became_peered":"2024-06-14T22:58:09.755383+0000","last_unstale":"2024-06-14T23:02:30.835329+0000","last_undegraded":"2024-06-14T23:02:30.835329+0000","last_fullsized":"2024-06-14T23:02:30.835329+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:13.884346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":168,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.839846+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T23:02:30.839846+0000","last_peered":"2024-06-14T23:02:30.839846+0000","last_clean":"2024-06-14T23:02:30.839846+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T23:02:30.839846+0000","last_undegraded":"2024-06-14T23:02:30.839846+0000","last_fullsized":"2024-06-14T23:02:30.839846+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":165,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-06-14T23:02:30.840990+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T23:02:30.840990+0000","last_peered":"2024-06-14T23:02:30.840990+0000","last_clean":"2024-06-14T23:02:30.840990+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T23:02:30.840990+0000","last_undegraded":"2024-06-14T23:02:30.840990+0000","last_fullsized":"2024-06-14T23:02:30.840990+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5734858776,"num_objects":87508,"num_object_clones":0,"num_object_copies":262524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87508,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174424,"num_write_kb":8381761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17226084352,"data_stored":17226072136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131524,"ondisk_log_size":131524,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1771,"ondisk_log_size":1771,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234346,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2287276,"kb_used_data":1384228,"kb_used_omap":1,"kb_used_meta":903038,"kb_avail":91457876,"statfs":{"total":95995035648,"available":93652865024,"internally_reserved":0,"allocated":1417449472,"data_stored":1416405545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924711256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.105,"15min":1.0840000000000001},"min":{"1min":0.224,"5min":0.221,"15min":0.20200000000000001},"max":{"1min":2.351,"5min":6.9539999999999997,"15min":9.3919999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.661,"5min":1.242,"15min":1.24},"min":{"1min":0.29099999999999998,"5min":0.26800000000000002,"15min":0.20300000000000001},"max":{"1min":4.806,"5min":5.2130000000000001,"15min":10.416},"last":6.2220000000000004}]},{"osd":1,"last update":"Fri Jun 14 23:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.4339999999999999,"15min":1.7669999999999999},"min":{"1min":0.308,"5min":0.29199999999999998,"15min":0.28100000000000003},"max":{"1min":6.3090000000000002,"5min":8.5340000000000007,"15min":83.295000000000002},"last":6.3090000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":1.2949999999999999,"15min":1.768},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.223},"max":{"1min":2.5550000000000002,"5min":6.8620000000000001,"15min":82.972999999999999},"last":0.90400000000000003}]},{"osd":2,"last update":"Fri Jun 14 23:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.196,"15min":0.84899999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.468,"5min":7.468,"15min":7.468},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.246,"15min":0.89800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.0469999999999997,"5min":6.9779999999999998,"15min":8.1150000000000002},"last":0.86499999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.651,"5min":1.4199999999999999,"15min":1.0669999999999999},"min":{"1min":0.34899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":6.2220000000000004,"5min":12.992000000000001,"15min":55.683999999999997},"last":6.7329999999999997},{"interface":"front","average":{"1min":1.401,"5min":1.3180000000000001,"15min":1.0920000000000001},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.528,"5min":9.5559999999999992,"15min":55.533999999999999},"last":6.2569999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.002,"15min":0.73899999999999999},"min":{"1min":0.34200000000000003,"5min":0.26200000000000001,"15min":0.245},"max":{"1min":3.1480000000000001,"5min":6.7610000000000001,"15min":6.7610000000000001},"last":6.1619999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.0840000000000001,"15min":0.79400000000000004},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":6.1600000000000001,"5min":16.879000000000001,"15min":16.879000000000001},"last":6.7939999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.331,"15min":0.91500000000000004},"min":{"1min":0.317,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.4320000000000004,"5min":6.9000000000000004,"15min":6.9000000000000004},"last":3.613},{"interface":"front","average":{"1min":1.216,"5min":1.137,"15min":0.90700000000000003},"min":{"1min":0.33200000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.9380000000000002,"5min":6.8529999999999998,"15min":6.8529999999999998},"last":6.2809999999999997}]},{"osd":6,"last update":"Fri Jun 14 23:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.0920000000000001,"15min":0.84199999999999997},"min":{"1min":0.35799999999999998,"5min":0.307,"15min":0.28100000000000003},"max":{"1min":2.9689999999999999,"5min":5.5,"15min":5.806},"last":6.1950000000000003},{"interface":"front","average":{"1min":1.518,"5min":1.099,"15min":0.84899999999999998},"min":{"1min":0.38700000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.1980000000000004,"5min":12.951000000000001,"15min":12.951000000000001},"last":6.7690000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495982,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3336224,"kb_used_data":2443096,"kb_used_omap":1,"kb_used_meta":893118,"kb_avail":90408928,"statfs":{"total":95995035648,"available":92578742272,"internally_reserved":0,"allocated":2501730304,"data_stored":2500698220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914553176},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:14 2024","interfaces":[{"interface":"back","average":{"1min":2.431,"5min":1.4059999999999999,"15min":1.351},"min":{"1min":0.36399999999999999,"5min":0.23699999999999999,"15min":0.218},"max":{"1min":10.348000000000001,"5min":10.348000000000001,"15min":31.358000000000001},"last":3.48},{"interface":"front","average":{"1min":1.859,"5min":1.119,"15min":1.2929999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.21299999999999999},"max":{"1min":10.348000000000001,"5min":10.348000000000001,"15min":29.399000000000001},"last":1.522}]},{"osd":1,"last update":"Fri Jun 14 23:02:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":1.1899999999999999,"15min":1.1799999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.2360000000000002,"5min":6.6680000000000001,"15min":9.9559999999999995},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":1.254,"15min":1.1990000000000001},"min":{"1min":0.27600000000000002,"5min":0.25,"15min":0.247},"max":{"1min":2.1280000000000001,"5min":6.7519999999999998,"15min":9.5510000000000002},"last":1.046}]},{"osd":2,"last update":"Fri Jun 14 23:02:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0800000000000001,"15min":0.79200000000000004},"min":{"1min":0.31,"5min":0.29799999999999999,"15min":0.27800000000000002},"max":{"1min":2.8479999999999999,"5min":9.9600000000000009,"15min":9.9600000000000009},"last":2.0510000000000002},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.2869999999999999,"15min":0.88},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":10.823,"5min":10.823,"15min":10.823},"last":7.4820000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:02:14 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.3149999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.28599999999999998,"15min":0.27300000000000002},"max":{"1min":3.1560000000000001,"5min":9.8970000000000002,"15min":29.152000000000001},"last":3.54},{"interface":"front","average":{"1min":1.081,"5min":1.3069999999999999,"15min":0.98999999999999999},"min":{"1min":0.379,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.621,"5min":6.3040000000000003,"15min":25.864000000000001},"last":1.2190000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:02:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.98699999999999999,"15min":0.92400000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.264,"5min":7.7709999999999999,"15min":47.698},"last":1.5469999999999999},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.93600000000000005,"15min":0.877},"min":{"1min":0.28899999999999998,"5min":0.27600000000000002,"15min":0.24299999999999999},"max":{"1min":2.3999999999999999,"5min":9.9610000000000003,"15min":48.197000000000003},"last":3.5129999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1459999999999999,"15min":0.88400000000000001},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.26500000000000001},"max":{"1min":5.0010000000000003,"5min":7.3540000000000001,"15min":8.6500000000000004},"last":2.306},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.103,"15min":0.90300000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":2.714,"5min":7.3769999999999998,"15min":8.6349999999999998},"last":0.47299999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:02:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.224,"15min":0.89800000000000002},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.29299999999999998},"max":{"1min":4.4720000000000004,"5min":7.8579999999999997,"15min":8.6950000000000003},"last":2.5920000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.147,"15min":0.92900000000000005},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.28899999999999998},"max":{"1min":3.6869999999999998,"5min":7.8120000000000003,"15min":8.673},"last":2.6240000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757621,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2628888,"kb_used_data":1733392,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":91116264,"statfs":{"total":95995035648,"available":93303054336,"internally_reserved":0,"allocated":1774993408,"data_stored":1773965850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:37 2024","interfaces":[{"interface":"back","average":{"1min":2.605,"5min":1.4890000000000001,"15min":1.2769999999999999},"min":{"1min":0.35599999999999998,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":9.8079999999999998,"5min":9.8079999999999998,"15min":17.766999999999999},"last":1.244},{"interface":"front","average":{"1min":2.3420000000000001,"5min":1.389,"15min":1.2430000000000001},"min":{"1min":0.25,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":5.1660000000000004,"5min":5.1950000000000003,"15min":17.806000000000001},"last":1.1599999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.92800000000000005,"15min":1.1180000000000001},"min":{"1min":0.24399999999999999,"5min":0.23999999999999999,"15min":0.23300000000000001},"max":{"1min":3.4049999999999998,"5min":5.6429999999999998,"15min":7.694},"last":0.308},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.252,"15min":1.1930000000000001},"min":{"1min":0.27600000000000002,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"last":0.85499999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":1.274,"15min":0.86499999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.2489999999999997,"5min":7.2489999999999997,"15min":7.2489999999999997},"last":0.83099999999999996},{"interface":"front","average":{"1min":2.1400000000000001,"5min":1.381,"15min":0.89500000000000002},"min":{"1min":0.32400000000000001,"5min":0.28100000000000003,"15min":0.27800000000000002},"max":{"1min":7.4500000000000002,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":0.90100000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:02:19 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.534,"15min":1.1319999999999999},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.4580000000000002,"5min":10.358000000000001,"15min":45.744},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.845,"5min":1.5900000000000001,"15min":1.222},"min":{"1min":0.378,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.4199999999999999,"5min":10.385,"15min":45.838000000000001},"last":0.88200000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:02:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.872,"15min":0.66200000000000003},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.214},"max":{"1min":2.5299999999999998,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.312,"5min":0.28299999999999997,"15min":0.22700000000000001},"max":{"1min":4.4359999999999999,"5min":4.657,"15min":5.8109999999999999},"last":1.0740000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:02:41 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.1559999999999999,"15min":0.88300000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.23000000000000001},"max":{"1min":5.5609999999999999,"5min":6.931,"15min":9.6020000000000003},"last":1.135},{"interface":"front","average":{"1min":1.377,"5min":1.1080000000000001,"15min":0.84999999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.25},"max":{"1min":5.625,"5min":5.625,"15min":12.339},"last":2.5339999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:02:05 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.1040000000000001,"15min":0.83299999999999996},"min":{"1min":0.30499999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.4369999999999998,"5min":5.3129999999999997,"15min":5.3129999999999997},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.1220000000000001,"15min":0.83899999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":5.1820000000000004,"5min":5.6189999999999998,"15min":10.638999999999999},"last":0.501}]}]},{"osd":4,"up_from":30,"seq":128849019257,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3666728,"kb_used_data":2779872,"kb_used_omap":1,"kb_used_meta":886590,"kb_avail":90078424,"statfs":{"total":95995035648,"available":92240306176,"internally_reserved":0,"allocated":2846588928,"data_stored":2845548786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907868504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4220000000000002,"5min":1.3140000000000001,"15min":1.125},"min":{"1min":0.25800000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":10.567,"5min":10.567,"15min":10.567},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.2110000000000001,"15min":1.139},"min":{"1min":0.23999999999999999,"5min":0.22700000000000001,"15min":0.221},"max":{"1min":5.9189999999999996,"5min":8.3390000000000004,"15min":9.1470000000000002},"last":0.68999999999999995}]},{"osd":1,"last update":"Fri Jun 14 23:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":2.044,"15min":1.395},"min":{"1min":0.27000000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.6929999999999996,"5min":56.917000000000002,"15min":56.917000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":2.1280000000000001,"5min":2.0470000000000002,"15min":1.4890000000000001},"min":{"1min":0.28799999999999998,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":10.539999999999999,"5min":56.789999999999999,"15min":56.789999999999999},"last":0.25700000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.0780000000000001,"15min":0.748},"min":{"1min":0.26500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.0039999999999996,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.5369999999999999,"5min":1.238,"15min":0.83899999999999997},"min":{"1min":0.41699999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":10.478,"5min":10.478,"15min":10.478},"last":0.373}]},{"osd":3,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.268,"15min":0.86399999999999999},"min":{"1min":0.28599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.327,"5min":6.617,"15min":6.617},"last":0.59099999999999997},{"interface":"front","average":{"1min":2.1269999999999998,"5min":1.3819999999999999,"15min":0.92700000000000005},"min":{"1min":0.33100000000000002,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":10.510999999999999,"5min":10.510999999999999,"15min":10.510999999999999},"last":0.314}]},{"osd":5,"last update":"Fri Jun 14 23:02:07 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":2.6579999999999999,"15min":1.3240000000000001},"min":{"1min":0.32100000000000001,"5min":0.253,"15min":0.22500000000000001},"max":{"1min":5.0629999999999997,"5min":139.13200000000001,"15min":139.13200000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.0069999999999999,"5min":2.7759999999999998,"15min":1.353},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.23699999999999999},"max":{"1min":2.3069999999999999,"5min":138.86000000000001,"15min":138.86000000000001},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0629999999999999,"15min":0.75700000000000001},"min":{"1min":0.32600000000000001,"5min":0.28899999999999998,"15min":0.254},"max":{"1min":4.6269999999999998,"5min":5.1210000000000004,"15min":5.1210000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.1120000000000001,"15min":0.748},"min":{"1min":0.35899999999999999,"5min":0.27900000000000003,"15min":0.254},"max":{"1min":5.8680000000000003,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.45200000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:01:57 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.056,"15min":0.755},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.0009999999999999,"5min":8.9489999999999998,"15min":8.9489999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.2410000000000001,"15min":0.871},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.308},"max":{"1min":2.9409999999999998,"5min":7.25,"15min":8.4730000000000008},"last":0.438}]}]},{"osd":3,"up_from":138,"seq":592705486956,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2049560,"kb_used_data":1730960,"kb_used_omap":4,"kb_used_meta":318523,"kb_avail":91695592,"statfs":{"total":95995035648,"available":93896286208,"internally_reserved":0,"allocated":1772503040,"data_stored":1771475255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":326168030},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.4570000000000001,"15min":0.78100000000000003},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.184},"max":{"1min":5.8499999999999996,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.502},{"interface":"front","average":{"1min":2.0009999999999999,"5min":1.2569999999999999,"15min":0.70899999999999996},"min":{"1min":0.26100000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2330000000000005,"5min":8.2330000000000005,"15min":8.2330000000000005},"last":0.56399999999999995}]},{"osd":1,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.4890000000000001,"15min":0.82999999999999996},"min":{"1min":0.249,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":4.4189999999999996,"5min":24.097999999999999,"15min":24.097999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.5680000000000001,"15min":0.86199999999999999},"min":{"1min":0.35299999999999998,"5min":0.26600000000000001,"15min":0.24099999999999999},"max":{"1min":3.5859999999999999,"5min":24.132999999999999,"15min":24.132999999999999},"last":0.49199999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.1950000000000001,"15min":0.72399999999999998},"min":{"1min":0.29499999999999998,"5min":0.245,"15min":0.23599999999999999},"max":{"1min":3.831,"5min":5.2910000000000004,"15min":5.2910000000000004},"last":1.877},{"interface":"front","average":{"1min":1.9159999999999999,"5min":1.2,"15min":0.71599999999999997},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":10.369,"5min":10.369,"15min":10.369},"last":1.016}]},{"osd":4,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.2170000000000001,"15min":0.877},"min":{"1min":0.40600000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.976,"5min":5.1189999999999998,"15min":5.7960000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3680000000000001,"15min":0.92800000000000005},"min":{"1min":0.379,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.1580000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.405,"15min":0.96699999999999997},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.29699999999999999},"max":{"1min":5.117,"5min":5.9359999999999999,"15min":5.9359999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.427,"5min":1.373,"15min":0.92100000000000004},"min":{"1min":0.32400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.6360000000000001,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.52200000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.27,"15min":0.89600000000000002},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.762,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.401,"5min":1.252,"15min":0.88100000000000001},"min":{"1min":0.32200000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.3760000000000003,"5min":5.0439999999999996,"15min":5.8259999999999996},"last":0.376}]},{"osd":7,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.3919999999999999,"15min":0.997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35299999999999998},"max":{"1min":4.351,"5min":7.7240000000000002,"15min":7.7240000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.423,"15min":0.98199999999999998},"min":{"1min":0.41299999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.2160000000000002,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.55500000000000005}]}]},{"osd":2,"up_from":133,"seq":571230650482,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2380312,"kb_used_data":2188944,"kb_used_omap":6,"kb_used_meta":191353,"kb_avail":91364840,"statfs":{"total":95995035648,"available":93557596160,"internally_reserved":0,"allocated":2241478656,"data_stored":2240451085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":195945973},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.85599999999999998,"15min":0.58299999999999996},"min":{"1min":0.19,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.0640000000000001,"5min":7.476,"15min":7.476},"last":1.2470000000000001},{"interface":"front","average":{"1min":2.1139999999999999,"5min":1.4370000000000001,"15min":0.753},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.182},"max":{"1min":8.0180000000000007,"5min":11.369,"15min":11.369},"last":0.71799999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.073,"15min":0.80500000000000005},"min":{"1min":0.221,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.726,"5min":5.8200000000000003,"15min":21.920000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.872,"5min":0.98299999999999998,"15min":0.76800000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":3.9940000000000002,"5min":13.054,"15min":21.573},"last":0.217}]},{"osd":3,"last update":"Fri Jun 14 23:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.4119999999999999,"15min":0.83099999999999996},"min":{"1min":0.23999999999999999,"5min":0.219,"15min":0.219},"max":{"1min":11.24,"5min":11.24,"15min":21.66},"last":0.77700000000000002},{"interface":"front","average":{"1min":2.0099999999999998,"5min":1.3149999999999999,"15min":0.872},"min":{"1min":0.27100000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":11.271000000000001,"5min":11.271000000000001,"15min":22.341000000000001},"last":0.79900000000000004}]},{"osd":4,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2349999999999999,"5min":1.708,"15min":0.99399999999999999},"min":{"1min":0.41999999999999998,"5min":0.36499999999999999,"15min":0.318},"max":{"1min":4.9189999999999996,"5min":14.438000000000001,"15min":14.438000000000001},"last":0.751},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.5629999999999999,"15min":0.94299999999999995},"min":{"1min":0.308,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.9290000000000003,"5min":14.423,"15min":14.423},"last":0.39600000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5900000000000001,"15min":0.93000000000000005},"min":{"1min":0.29599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.9409999999999998,"5min":31.539999999999999,"15min":31.539999999999999},"last":4.173},{"interface":"front","average":{"1min":1.663,"5min":1.619,"15min":0.94699999999999995},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":4.1760000000000002,"5min":31.481999999999999,"15min":31.481999999999999},"last":0.68799999999999994}]},{"osd":6,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":2.0049999999999999,"5min":1.468,"15min":0.879},"min":{"1min":0.375,"5min":0.309,"15min":0.309},"max":{"1min":8.9730000000000008,"5min":11.454000000000001,"15min":11.454000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1659999999999999,"15min":0.81799999999999995},"min":{"1min":0.372,"5min":0.32800000000000001,"15min":0.30399999999999999},"max":{"1min":3.8980000000000001,"5min":12.227,"15min":12.227},"last":1.343}]},{"osd":7,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":1.4830000000000001,"15min":0.93999999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.782,"5min":10.614000000000001,"15min":10.614000000000001},"last":0.374},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1140000000000001,"15min":0.78700000000000003},"min":{"1min":0.35999999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":3.968,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":0.48699999999999999}]}]},{"osd":1,"up_from":116,"seq":498216206499,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1555536,"kb_used_data":1098632,"kb_used_omap":5,"kb_used_meta":456890,"kb_avail":92189616,"statfs":{"total":95995035648,"available":94402166784,"internally_reserved":0,"allocated":1124999168,"data_stored":1123978941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":467855640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.385,"15min":1.712},"min":{"1min":0.22600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.6269999999999998,"5min":11.257999999999999,"15min":11.257999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.635,"5min":1.234,"15min":1.3520000000000001},"min":{"1min":0.20000000000000001,"5min":0.186,"15min":0.186},"max":{"1min":5.7539999999999996,"5min":5.7539999999999996,"15min":10.426},"last":1.0469999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.4610000000000001,"15min":0.77700000000000002},"min":{"1min":0.22500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":6.6790000000000003,"5min":9.3759999999999994,"15min":9.3759999999999994},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.054,"15min":0.60899999999999999},"min":{"1min":0.255,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":3.7799999999999998,"5min":5.5119999999999996,"15min":5.5119999999999996},"last":1.0800000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.3380000000000001,"15min":0.89600000000000002},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.8929999999999998,"5min":8.2089999999999996,"15min":21.422000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":0.88,"5min":1.462,"15min":0.94999999999999996},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.214},"max":{"1min":4.2450000000000001,"5min":9.4990000000000006,"15min":21.016999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.238,"15min":1.518},"min":{"1min":0.28899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.806,"5min":6.6360000000000001,"15min":8.8599999999999994},"last":1.28},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.5840000000000001,"15min":1.6419999999999999},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.9589999999999996,"5min":9.4169999999999998,"15min":11.161},"last":0.47899999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.482,"15min":1.5580000000000001},"min":{"1min":0.38100000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.855,"5min":10.103,"15min":10.103},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.2210000000000001,"15min":1.536},"min":{"1min":0.308,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.5,"5min":6.5730000000000004,"15min":8.125},"last":1.2030000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":4.5919999999999996,"5min":1.9910000000000001,"15min":1.7649999999999999},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":63.795999999999999,"5min":63.795999999999999,"15min":63.795999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":4.2080000000000002,"5min":1.927,"15min":1.7569999999999999},"min":{"1min":0.35699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":63.811,"5min":63.811,"15min":63.811},"last":1.3180000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:01:49 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.641,"15min":1.6919999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.923,"5min":9.4429999999999996,"15min":9.4429999999999996},"last":0.501},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.3520000000000001,"15min":1.45},"min":{"1min":0.433,"5min":0.371,"15min":0.36499999999999999},"max":{"1min":3.8340000000000001,"5min":6.3529999999999998,"15min":9.5449999999999999},"last":1.0129999999999999}]}]},{"osd":0,"up_from":91,"seq":390842024154,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4245652,"kb_used_data":3549516,"kb_used_omap":5,"kb_used_meta":696122,"kb_avail":89499500,"statfs":{"total":95995035648,"available":91647488000,"internally_reserved":0,"allocated":3634704384,"data_stored":3633678084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":712829018},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.3129999999999999,"15min":1.488},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":9.4380000000000006,"5min":15.547000000000001,"15min":16.305},"last":0.33900000000000002},{"interface":"front","average":{"1min":1.395,"5min":1.2629999999999999,"15min":1.6890000000000001},"min":{"1min":0.20899999999999999,"5min":0.20499999999999999,"15min":0.17899999999999999},"max":{"1min":4.9329999999999998,"5min":6.9889999999999999,"15min":16.184999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:01:55 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.24,"15min":0.73399999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.053000000000001,"5min":11.326000000000001,"15min":16.757000000000001},"last":0.745},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.4059999999999999,"15min":0.81299999999999994},"min":{"1min":0.24299999999999999,"5min":0.222,"15min":0.222},"max":{"1min":9.2769999999999992,"5min":11.519,"15min":16.055},"last":0.77800000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.3260000000000001,"15min":0.83599999999999997},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":4.2370000000000001,"5min":7.04,"15min":16.873999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":2.0720000000000001,"5min":1.46,"15min":0.86099999999999999},"min":{"1min":0.27300000000000002,"5min":0.224,"15min":0.224},"max":{"1min":9.4760000000000009,"5min":12.308999999999999,"15min":16.085999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.546,"15min":1.456},"min":{"1min":0.42499999999999999,"5min":0.307,"15min":0.27600000000000002},"max":{"1min":12.436,"5min":16.527000000000001,"15min":16.527000000000001},"last":2.6850000000000001},{"interface":"front","average":{"1min":2.0179999999999998,"5min":1.4890000000000001,"15min":1.3380000000000001},"min":{"1min":0.35499999999999998,"5min":0.33200000000000002,"15min":0.28599999999999998},"max":{"1min":4.8250000000000002,"5min":14.475,"15min":15.976000000000001},"last":1.573}]},{"osd":5,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.5489999999999999,"15min":1.452},"min":{"1min":0.32800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.9009999999999998,"5min":6.9640000000000004,"15min":16.108000000000001},"last":4.774},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.5529999999999999,"15min":1.5269999999999999},"min":{"1min":0.32500000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":9.2279999999999998,"5min":11.436,"15min":16.006},"last":0.83099999999999996}]},{"osd":6,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7969999999999999,"5min":1.5629999999999999,"15min":1.4219999999999999},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.8730000000000002,"5min":14.247,"15min":16.152000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":2.3239999999999998,"5min":1.6619999999999999,"15min":1.5129999999999999},"min":{"1min":0.41299999999999998,"5min":0.312,"15min":0.29399999999999998},"max":{"1min":9.3130000000000006,"5min":14.523999999999999,"15min":16.227},"last":5.234}]},{"osd":7,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":2.1440000000000001,"5min":1.4950000000000001,"15min":1.46},"min":{"1min":0.434,"5min":0.35899999999999999,"15min":0.33100000000000002},"max":{"1min":9.2579999999999991,"5min":11.364000000000001,"15min":16.199999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.867,"5min":1.546,"15min":1.524},"min":{"1min":0.39900000000000002,"5min":0.36899999999999999,"15min":0.314},"max":{"1min":9.2940000000000005,"5min":12.361000000000001,"15min":16.259},"last":4.3949999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3627421696,"data_stored":3627417624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1111752704,"data_stored":1111752704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2218196992,"data_stored":2218196992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1765933056,"data_stored":1765933056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2836336640,"data_stored":2836332568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1771634688,"data_stored":1771634688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2482442240,"data_stored":2482438168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1412366336,"data_stored":1412366336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:02:58.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set unique_pool_5 pgp_num 16 2024-06-14T23:02:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:58.569 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:58.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:58.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:02:58.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:58.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:02:58.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:58.820 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:02:58.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:58.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:58.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:58.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:58.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:59.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3934838168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:59.081 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:58 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:59.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3934838168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:59.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:58 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:59.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:02:59.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-14T23:02:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:59.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:02:59.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3934838168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:59.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:58 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:02:59.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:02:59.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:59.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:02:59.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:59.419 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:59.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:59.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:02:59.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:59.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:02:59.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:02:59.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:02:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:59.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:02:59.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:59.840 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:59.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:02:59.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:02:59.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:02:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:02:59.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:00.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:00.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:59 smithi086 ceph-mon[97913]: pgmap v832: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 833 op/s 2024-06-14T23:03:00.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:59 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:00.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:59 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:03:00.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:02:59 smithi086 ceph-mon[97913]: osdmap e209: 8 total, 8 up, 8 in 2024-06-14T23:03:00.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:59 smithi086 ceph-mon[102754]: pgmap v832: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 833 op/s 2024-06-14T23:03:00.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:59 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:00.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:59 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:03:00.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:02:59 smithi086 ceph-mon[102754]: osdmap e209: 8 total, 8 up, 8 in 2024-06-14T23:03:00.098 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:00.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:00.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:00.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:00.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:00.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:59 smithi137 ceph-mon[90634]: pgmap v832: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 833 op/s 2024-06-14T23:03:00.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:59 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:00.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:59 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:03:00.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:02:59 smithi137 ceph-mon[90634]: osdmap e209: 8 total, 8 up, 8 in 2024-06-14T23:03:00.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:00.375 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:00.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:00.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:00.507 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:00.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:00.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:00.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:00.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:00.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:00.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:00.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:00.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:00.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:00.910 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:00.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:01.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:01.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:01.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:01.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:01.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:01.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:01.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:01.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:01.342 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:01.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:01.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:01.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:01.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:01.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:01.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:01.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:01.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:01.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:01.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:01.876 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:01.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:01.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:02.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:02.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:02.051 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:01 smithi086 ceph-mon[102754]: pgmap v834: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 931 op/s 2024-06-14T23:03:02.051 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:01 smithi086 ceph-mon[97913]: pgmap v834: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 931 op/s 2024-06-14T23:03:02.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:02.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:02.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:02.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:02.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:01 smithi137 ceph-mon[90634]: pgmap v834: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 931 op/s 2024-06-14T23:03:02.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:02.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:02.371 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:02.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:02.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:02.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:02.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:02.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:02.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:02.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:02.696 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:02.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:02.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:02.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:02.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:02.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:02.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:02.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:03.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:03.104 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:03.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:03.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:03.241 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:03.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:03.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:03:03.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:03.399 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:03.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:03.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:03.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:03.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:03.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:03.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:03.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:03.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:03.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:03.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:03.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:03.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:03.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:04.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:04.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:03 smithi086 ceph-mon[97913]: pgmap v835: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-06-14T23:03:04.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:03 smithi086 ceph-mon[102754]: pgmap v835: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-06-14T23:03:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:04.102 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:04.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:04.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:04.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:04.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:03 smithi137 ceph-mon[90634]: pgmap v835: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 932 op/s 2024-06-14T23:03:04.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:04.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:04.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:04.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:04.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:04.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:04.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:04.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:04.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:04.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:04.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:04.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:05.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:05.074 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:05.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:05.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:05.212 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:05.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:05.265 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:03:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:03:04] "GET /metrics HTTP/1.1" 200 36590 "" "Prometheus/2.43.0" 2024-06-14T23:03:05.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:05.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:05.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:05.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:05.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:05.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:05.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:05.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:05.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:05.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:05.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:05.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:05.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:05.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:06.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:06.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:06.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:06.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:06.171 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:05 smithi086 ceph-mon[97913]: pgmap v836: 99 pgs: 99 active+clean; 4.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 917 op/s 2024-06-14T23:03:06.171 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:05 smithi086 ceph-mon[102754]: pgmap v836: 99 pgs: 99 active+clean; 4.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 917 op/s 2024-06-14T23:03:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:06.202 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:06.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:06.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:05 smithi137 ceph-mon[90634]: pgmap v836: 99 pgs: 99 active+clean; 4.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 917 op/s 2024-06-14T23:03:06.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:06.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:06.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:06.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:06.478 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:06.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:06.574 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:03:06.574 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:03:06.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:06.609 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:03:06.610 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:03:06.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:06.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:06.645 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:03:06.645 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:03:06.689 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:03:06.689 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:03:06.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:06.721 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:03:06.721 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:03:06.758 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:03:06.758 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:03:06.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:06.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:06.789 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:03:06.789 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:03:06.830 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:03:06.830 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:03:06.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:06.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:06.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:07.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:07.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:07.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2469659274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:07.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2469659274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:07.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:07.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:07.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:07.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2469659274' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:07.275 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:07.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:07.317 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:07.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:07.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:07.423 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:03:07.423 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:03:07.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:07.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:07.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:07.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:07.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:07.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:07.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:07.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:07.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:07.868 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 10 pgp_num to 16 2024-06-14T23:03:07.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:07.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:07.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:08.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:08.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:08.078 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:07 smithi086 ceph-mon[102754]: pgmap v837: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 924 op/s 2024-06-14T23:03:08.078 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/840953557' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:03:08.079 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:07 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:03:08.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:07 smithi086 ceph-mon[97913]: pgmap v837: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 924 op/s 2024-06-14T23:03:08.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/840953557' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:03:08.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:07 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:03:08.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:08.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:08.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:08.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:07 smithi137 ceph-mon[90634]: pgmap v837: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 924 op/s 2024-06-14T23:03:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/840953557' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:03:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:07 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:03:08.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:08.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:08.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:08.408 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:08.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:08.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:08.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:08.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:08.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:08.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:08.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:08.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:08.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:08.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:08.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:08.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:08.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:09.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:09.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:09.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:09.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:09.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:08 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:03:09.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:08 smithi137 ceph-mon[90634]: osdmap e210: 8 total, 8 up, 8 in 2024-06-14T23:03:09.246 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:09.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:09.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:09.273 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:08 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:03:09.273 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:08 smithi086 ceph-mon[102754]: osdmap e210: 8 total, 8 up, 8 in 2024-06-14T23:03:09.273 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:08 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:03:09.273 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:08 smithi086 ceph-mon[97913]: osdmap e210: 8 total, 8 up, 8 in 2024-06-14T23:03:09.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:09.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:09.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:09.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:09.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:09.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:09.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:09.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:09.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:09.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:09.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:09.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:09.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-14T23:03:09.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:09.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:10.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:10.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:09 smithi086 ceph-mon[97913]: pgmap v839: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 939 op/s 2024-06-14T23:03:10.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:09 smithi086 ceph-mon[102754]: pgmap v839: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 939 op/s 2024-06-14T23:03:10.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:10.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:10.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:03:10.145 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:09 smithi137 ceph-mon[90634]: pgmap v839: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 939 op/s 2024-06-14T23:03:10.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:10.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:10.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:10.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:10.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:10.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:10.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:10.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:10.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:10.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:10.697 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:10.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:10.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:10.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:10.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:10.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:10.966 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:10.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:11.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:11.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:11.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:11.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:11.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:11.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:11.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:11.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:11.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:11.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:11.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:11.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:11.542 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:11.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:11.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:11.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:11.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:11.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:11.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:11.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:11.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:11.907 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:11 smithi086 ceph-mon[97913]: pgmap v840: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 958 op/s 2024-06-14T23:03:11.907 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:11 smithi086 ceph-mon[102754]: pgmap v840: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 958 op/s 2024-06-14T23:03:11.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:11.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:12.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:12.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:12.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:12.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:11 smithi137 ceph-mon[90634]: pgmap v840: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 958 op/s 2024-06-14T23:03:12.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:12.212 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:12.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:12.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:12.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:12.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:12.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:12.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:12.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:12.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:12.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:12.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:12.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:12.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:12.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:12.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:03:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:12.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:12.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:13.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:13.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:13.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:13.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:13.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:13.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:13.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:13.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3745194707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:03:13.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:13.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:13.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:13.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:13.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3745194707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:03:13.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:13.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3745194707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:03:13.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:13.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:13.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:13.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:13.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:13.584 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:13.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:13.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:13.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:13.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:13.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:13.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:13.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:13.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:13.931 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:03:13.931 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:03:13.931 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-06-14T23:03:13.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:03:13.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:13.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:14.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:14.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:14.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:14.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:14.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:14.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:14.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:13 smithi086 ceph-mon[102754]: pgmap v841: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 961 op/s 2024-06-14T23:03:14.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:13 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:14.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3745194707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:03:14.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:13 smithi086 ceph-mon[102754]: osdmap e211: 8 total, 8 up, 8 in 2024-06-14T23:03:14.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:13 smithi086 ceph-mon[97913]: pgmap v841: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 961 op/s 2024-06-14T23:03:14.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:13 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:14.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3745194707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:03:14.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:13 smithi086 ceph-mon[97913]: osdmap e211: 8 total, 8 up, 8 in 2024-06-14T23:03:14.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:13 smithi137 ceph-mon[90634]: pgmap v841: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 961 op/s 2024-06-14T23:03:14.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:13 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:14.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3745194707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:03:14.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:13 smithi137 ceph-mon[90634]: osdmap e211: 8 total, 8 up, 8 in 2024-06-14T23:03:14.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:14.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:14.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:14.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:14.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:14.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:14.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:14.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:14.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:14.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:14.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:14.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:14.811 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:14.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:14.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:14.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:14.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:15.042 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:03:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:03:14] "GET /metrics HTTP/1.1" 200 36593 "" "Prometheus/2.43.0" 2024-06-14T23:03:15.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:15.090 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:15.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:15.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:15.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:15.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:15.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:15.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:15.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:15.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:15.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:15.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:15.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:15.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:15.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:15.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:15.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:15.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:15.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:15.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:15.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:15.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:16.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:16.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:16.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:16.133 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:03:16.133 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:03:16.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:16.146 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:15 smithi086 ceph-mon[97913]: pgmap v843: 99 pgs: 99 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T23:03:16.146 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:15 smithi086 ceph-mon[102754]: pgmap v843: 99 pgs: 99 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T23:03:16.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:16.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:16.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:15 smithi137 ceph-mon[90634]: pgmap v843: 99 pgs: 99 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-06-14T23:03:16.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:16.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:16.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:16.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:16.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:16.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:16.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:16.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:16.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:16.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:16.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:16.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:03:16.710 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:16.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:16.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:16.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:16.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:16.947 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:03:16.947 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:03:16.947 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:03:16.948 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:03:16.949 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:03:16.949 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:03:16.949 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:03:16.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2501691350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:16.949 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:16 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2501691350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:16.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:16.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:16.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:17.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:17.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:17.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:17.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:16 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2501691350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:17.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:17.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:17.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:17.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:17.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:17.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:17.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:17.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:17.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:17.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:17.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:17.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:17.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:17.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:17.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:17.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:17.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:18.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:18.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:18.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:18.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:17 smithi086 ceph-mon[97913]: pgmap v844: 99 pgs: 99 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-06-14T23:03:18.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:17 smithi086 ceph-mon[102754]: pgmap v844: 99 pgs: 99 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-06-14T23:03:18.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:18.193 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:18.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:18.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:17 smithi137 ceph-mon[90634]: pgmap v844: 99 pgs: 99 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-06-14T23:03:18.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:18.329 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:18.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:18.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:18.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:18.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:18.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:18.598 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:18.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:18.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-14T23:03:18.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:18.722 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:18.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:18.732 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:18.734 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:18.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:18.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:18.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:18.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:19.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:19.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:19.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:19.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:19.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:19.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:19.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:19.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:19.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:19.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:19.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:19.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:19.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:19.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:19.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:19.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:19.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:19.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:19.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:19.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:03:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:19.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:19.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:19.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:19.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:20.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:20.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:20.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:20.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:20.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:19 smithi137 ceph-mon[90634]: pgmap v845: 99 pgs: 99 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 951 op/s 2024-06-14T23:03:20.252 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:20.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:20.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:19 smithi086 ceph-mon[102754]: pgmap v845: 99 pgs: 99 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 951 op/s 2024-06-14T23:03:20.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:19 smithi086 ceph-mon[97913]: pgmap v845: 99 pgs: 99 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 951 op/s 2024-06-14T23:03:20.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:20.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:20.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:20.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:20.516 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:20.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:20.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:20.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:20.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:20.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:20.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:20.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:20.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:20.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:20.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:21.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:21.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:21.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:21.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:21.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:21.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:21.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:20 smithi137 ceph-mon[90634]: pgmap v846: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 942 op/s 2024-06-14T23:03:21.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:21.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:20 smithi086 ceph-mon[102754]: pgmap v846: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 942 op/s 2024-06-14T23:03:21.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:20 smithi086 ceph-mon[97913]: pgmap v846: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 942 op/s 2024-06-14T23:03:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:21.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:21.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:21.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:21.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:21.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:21.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:21.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:21.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:21.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:21.732 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:21.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:21.830 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:21.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:21.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:21.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:21.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:21.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:22.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:22.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:22.111 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:22.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:22.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:22.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:22.273 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:22.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:22.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:22.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:22.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:22.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:22.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:22.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:22.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:22.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:22.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:22.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:22.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:22.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:22.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:22.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:22.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:22.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:23.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:23.084 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:23.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:23.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:23.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:23.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:23.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:23.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:23.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:23.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:23.490 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:23.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:23.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:23.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:23.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:23.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:23.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:23.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:23.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:23.818 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:03:23.827 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:23 smithi086 ceph-mon[102754]: pgmap v847: 99 pgs: 99 active+clean; 3.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-06-14T23:03:23.828 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2343541038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:03:23.828 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:03:23.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:23 smithi086 ceph-mon[97913]: pgmap v847: 99 pgs: 99 active+clean; 3.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-06-14T23:03:23.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2343541038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:03:23.833 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:03:23.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:23.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:23.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:23.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:23.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:24.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:24.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:24.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:24.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:24.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:24.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:23 smithi137 ceph-mon[90634]: pgmap v847: 99 pgs: 99 active+clean; 3.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-06-14T23:03:24.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2343541038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:03:24.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:24.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:24.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:24.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:24.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:24.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:24.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:24.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:24.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:24.697 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:24.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:24.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:24.840 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:24.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:24.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:24.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:24.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:24.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:25.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:25.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:03:25.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:03:25.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[97913]: from='client.36078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:03:25.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2343541038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:03:25.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[97913]: osdmap e212: 8 total, 8 up, 8 in 2024-06-14T23:03:25.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:03:24] "GET /metrics HTTP/1.1" 200 36593 "" "Prometheus/2.43.0" 2024-06-14T23:03:25.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:03:25.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:03:25.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[102754]: from='client.36078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:03:25.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2343541038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:03:25.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:24 smithi086 ceph-mon[102754]: osdmap e212: 8 total, 8 up, 8 in 2024-06-14T23:03:25.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:25.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:25.146 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":847,"stamp":"2024-06-14T23:03:22.743863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4093056990,"num_objects":62471,"num_object_clones":0,"num_object_copies":187413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199546,"num_write_kb":8381790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133311,"ondisk_log_size":133311,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17357952,"kb_used_data":12078336,"kb_used_omap":29,"kb_used_meta":5279394,"kb_avail":732603264,"statfs":{"total":767960285184,"available":750185742336,"internally_reserved":0,"allocated":12368216064,"data_stored":12359878294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":5406100133},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,6,12,7,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-621477888,"num_objects":-9483,"num_object_clones":0,"num_object_copies":-28449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004392"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":142,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866562+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:03:07.866562+0000","last_peered":"2024-06-14T23:03:07.866562+0000","last_clean":"2024-06-14T23:03:07.866562+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:03:07.866562+0000","last_undegraded":"2024-06-14T23:03:07.866562+0000","last_fullsized":"2024-06-14T23:03:07.866562+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":174,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.874458+0000","last_change":"2024-06-14T22:53:32.497040+0000","last_active":"2024-06-14T23:03:07.874458+0000","last_peered":"2024-06-14T23:03:07.874458+0000","last_clean":"2024-06-14T23:03:07.874458+0000","last_became_active":"2024-06-14T22:53:32.496662+0000","last_became_peered":"2024-06-14T22:53:32.496662+0000","last_unstale":"2024-06-14T23:03:07.874458+0000","last_undegraded":"2024-06-14T23:03:07.874458+0000","last_fullsized":"2024-06-14T23:03:07.874458+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:21:06.876581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":171,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867591+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:03:07.867591+0000","last_peered":"2024-06-14T23:03:07.867591+0000","last_clean":"2024-06-14T23:03:07.867591+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:03:07.867591+0000","last_undegraded":"2024-06-14T23:03:07.867591+0000","last_fullsized":"2024-06-14T23:03:07.867591+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":163,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876963+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:03:07.876963+0000","last_peered":"2024-06-14T23:03:07.876963+0000","last_clean":"2024-06-14T23:03:07.876963+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:03:07.876963+0000","last_undegraded":"2024-06-14T23:03:07.876963+0000","last_fullsized":"2024-06-14T23:03:07.876963+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":164,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.877322+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T23:03:07.877322+0000","last_peered":"2024-06-14T23:03:07.877322+0000","last_clean":"2024-06-14T23:03:07.877322+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T23:03:07.877322+0000","last_undegraded":"2024-06-14T23:03:07.877322+0000","last_fullsized":"2024-06-14T23:03:07.877322+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":78,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.871391+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T23:03:07.871391+0000","last_peered":"2024-06-14T23:03:07.871391+0000","last_clean":"2024-06-14T23:03:07.871391+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T23:03:07.871391+0000","last_undegraded":"2024-06-14T23:03:07.871391+0000","last_fullsized":"2024-06-14T23:03:07.871391+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":165,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.898870+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:03:07.898870+0000","last_peered":"2024-06-14T23:03:07.898870+0000","last_clean":"2024-06-14T23:03:07.898870+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:03:07.898870+0000","last_undegraded":"2024-06-14T23:03:07.898870+0000","last_fullsized":"2024-06-14T23:03:07.898870+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":180,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.883114+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T23:03:07.883114+0000","last_peered":"2024-06-14T23:03:07.883114+0000","last_clean":"2024-06-14T23:03:07.883114+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T23:03:07.883114+0000","last_undegraded":"2024-06-14T23:03:07.883114+0000","last_fullsized":"2024-06-14T23:03:07.883114+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"211'12421","reported_seq":12454,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:19.387723+0000","last_change":"2024-06-14T23:00:27.800011+0000","last_active":"2024-06-14T23:03:19.387723+0000","last_peered":"2024-06-14T23:03:19.387723+0000","last_clean":"2024-06-14T23:03:19.387723+0000","last_became_active":"2024-06-14T23:00:27.799716+0000","last_became_peered":"2024-06-14T23:00:27.799716+0000","last_unstale":"2024-06-14T23:03:19.387723+0000","last_undegraded":"2024-06-14T23:03:19.387723+0000","last_fullsized":"2024-06-14T23:03:19.387723+0000","mapping_epoch":186,"log_start":"192'2700","ondisk_log_start":"192'2700","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9721,"log_dups_size":0,"ondisk_log_size":9721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:41:34.619251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253820928,"num_objects":3873,"num_object_clones":0,"num_object_copies":11619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12421,"num_write_kb":521408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":161,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.874824+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:03:07.874824+0000","last_peered":"2024-06-14T23:03:07.874824+0000","last_clean":"2024-06-14T23:03:07.874824+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:03:07.874824+0000","last_undegraded":"2024-06-14T23:03:07.874824+0000","last_fullsized":"2024-06-14T23:03:07.874824+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":154,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866945+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T23:03:07.866945+0000","last_peered":"2024-06-14T23:03:07.866945+0000","last_clean":"2024-06-14T23:03:07.866945+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T23:03:07.866945+0000","last_undegraded":"2024-06-14T23:03:07.866945+0000","last_fullsized":"2024-06-14T23:03:07.866945+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"211'12444","reported_seq":12477,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:18.936845+0000","last_change":"2024-06-14T23:00:27.798857+0000","last_active":"2024-06-14T23:03:18.936845+0000","last_peered":"2024-06-14T23:03:18.936845+0000","last_clean":"2024-06-14T23:03:18.936845+0000","last_became_active":"2024-06-14T23:00:27.798566+0000","last_became_peered":"2024-06-14T23:00:27.798566+0000","last_unstale":"2024-06-14T23:03:18.936845+0000","last_undegraded":"2024-06-14T23:03:18.936845+0000","last_fullsized":"2024-06-14T23:03:18.936845+0000","mapping_epoch":186,"log_start":"195'4700","ondisk_log_start":"195'4700","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7744,"log_dups_size":0,"ondisk_log_size":7744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:38.237668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258473984,"num_objects":3944,"num_object_clones":0,"num_object_copies":11832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12444,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":162,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.875018+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:03:07.875018+0000","last_peered":"2024-06-14T23:03:07.875018+0000","last_clean":"2024-06-14T23:03:07.875018+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:03:07.875018+0000","last_undegraded":"2024-06-14T23:03:07.875018+0000","last_fullsized":"2024-06-14T23:03:07.875018+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":188,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.872756+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:03:07.872756+0000","last_peered":"2024-06-14T23:03:07.872756+0000","last_clean":"2024-06-14T23:03:07.872756+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:03:07.872756+0000","last_undegraded":"2024-06-14T23:03:07.872756+0000","last_fullsized":"2024-06-14T23:03:07.872756+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"211'12555","reported_seq":12590,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:22.112000+0000","last_change":"2024-06-14T23:00:27.797882+0000","last_active":"2024-06-14T23:03:22.112000+0000","last_peered":"2024-06-14T23:03:22.112000+0000","last_clean":"2024-06-14T23:03:22.112000+0000","last_became_active":"2024-06-14T23:00:27.797600+0000","last_became_peered":"2024-06-14T23:00:27.797600+0000","last_unstale":"2024-06-14T23:03:22.112000+0000","last_undegraded":"2024-06-14T23:03:22.112000+0000","last_fullsized":"2024-06-14T23:03:22.112000+0000","mapping_epoch":186,"log_start":"195'4800","ondisk_log_start":"195'4800","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7755,"log_dups_size":0,"ondisk_log_size":7755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:04:20.589709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239534080,"num_objects":3655,"num_object_clones":0,"num_object_copies":10965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12555,"num_write_kb":518720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":89,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867529+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T23:03:07.867529+0000","last_peered":"2024-06-14T23:03:07.867529+0000","last_clean":"2024-06-14T23:03:07.867529+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T23:03:07.867529+0000","last_undegraded":"2024-06-14T23:03:07.867529+0000","last_fullsized":"2024-06-14T23:03:07.867529+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":171,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.879249+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T23:03:07.879249+0000","last_peered":"2024-06-14T23:03:07.879249+0000","last_clean":"2024-06-14T23:03:07.879249+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T23:03:07.879249+0000","last_undegraded":"2024-06-14T23:03:07.879249+0000","last_fullsized":"2024-06-14T23:03:07.879249+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"211'12778","reported_seq":12813,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:20.363124+0000","last_change":"2024-06-14T23:00:27.794220+0000","last_active":"2024-06-14T23:03:20.363124+0000","last_peered":"2024-06-14T23:03:20.363124+0000","last_clean":"2024-06-14T23:03:20.363124+0000","last_became_active":"2024-06-14T23:00:27.793960+0000","last_became_peered":"2024-06-14T23:00:27.793960+0000","last_unstale":"2024-06-14T23:03:20.363124+0000","last_undegraded":"2024-06-14T23:03:20.363124+0000","last_fullsized":"2024-06-14T23:03:20.363124+0000","mapping_epoch":186,"log_start":"192'2700","ondisk_log_start":"192'2700","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:04:26.933537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262799360,"num_objects":4010,"num_object_clones":0,"num_object_copies":12030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12778,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":172,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867399+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T23:03:07.867399+0000","last_peered":"2024-06-14T23:03:07.867399+0000","last_clean":"2024-06-14T23:03:07.867399+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T23:03:07.867399+0000","last_undegraded":"2024-06-14T23:03:07.867399+0000","last_fullsized":"2024-06-14T23:03:07.867399+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":165,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876903+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T23:03:07.876903+0000","last_peered":"2024-06-14T23:03:07.876903+0000","last_clean":"2024-06-14T23:03:07.876903+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T23:03:07.876903+0000","last_undegraded":"2024-06-14T23:03:07.876903+0000","last_fullsized":"2024-06-14T23:03:07.876903+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"211'12675","reported_seq":12709,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:20.067722+0000","last_change":"2024-06-14T23:00:27.797355+0000","last_active":"2024-06-14T23:03:20.067722+0000","last_peered":"2024-06-14T23:03:20.067722+0000","last_clean":"2024-06-14T23:03:20.067722+0000","last_became_active":"2024-06-14T23:00:27.796666+0000","last_became_peered":"2024-06-14T23:00:27.796666+0000","last_unstale":"2024-06-14T23:03:20.067722+0000","last_undegraded":"2024-06-14T23:03:20.067722+0000","last_fullsized":"2024-06-14T23:03:20.067722+0000","mapping_epoch":186,"log_start":"194'4300","ondisk_log_start":"194'4300","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:57:29.139721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252510208,"num_objects":3853,"num_object_clones":0,"num_object_copies":11559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12675,"num_write_kb":528896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":510,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.874946+0000","last_change":"2024-06-14T23:00:30.637455+0000","last_active":"2024-06-14T23:03:07.874946+0000","last_peered":"2024-06-14T23:03:07.874946+0000","last_clean":"2024-06-14T23:03:07.874946+0000","last_became_active":"2024-06-14T23:00:17.398732+0000","last_became_peered":"2024-06-14T23:00:17.398732+0000","last_unstale":"2024-06-14T23:03:07.874946+0000","last_undegraded":"2024-06-14T23:03:07.874946+0000","last_fullsized":"2024-06-14T23:03:07.874946+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.637373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.637373+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:10.033075+0000","scrub_duration":0.0016781470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":81,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876732+0000","last_change":"2024-06-14T22:58:09.752985+0000","last_active":"2024-06-14T23:03:07.876732+0000","last_peered":"2024-06-14T23:03:07.876732+0000","last_clean":"2024-06-14T23:03:07.876732+0000","last_became_active":"2024-06-14T22:58:09.752630+0000","last_became_peered":"2024-06-14T22:58:09.752630+0000","last_unstale":"2024-06-14T23:03:07.876732+0000","last_undegraded":"2024-06-14T23:03:07.876732+0000","last_fullsized":"2024-06-14T23:03:07.876732+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:22.320986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":171,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.871786+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T23:03:07.871786+0000","last_peered":"2024-06-14T23:03:07.871786+0000","last_clean":"2024-06-14T23:03:07.871786+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T23:03:07.871786+0000","last_undegraded":"2024-06-14T23:03:07.871786+0000","last_fullsized":"2024-06-14T23:03:07.871786+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"211'12550","reported_seq":12585,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:20.373571+0000","last_change":"2024-06-14T23:00:27.797468+0000","last_active":"2024-06-14T23:03:20.373571+0000","last_peered":"2024-06-14T23:03:20.373571+0000","last_clean":"2024-06-14T23:03:20.373571+0000","last_became_active":"2024-06-14T23:00:27.797180+0000","last_became_peered":"2024-06-14T23:00:27.797180+0000","last_unstale":"2024-06-14T23:03:20.373571+0000","last_undegraded":"2024-06-14T23:03:20.373571+0000","last_fullsized":"2024-06-14T23:03:20.373571+0000","mapping_epoch":186,"log_start":"192'2500","ondisk_log_start":"192'2500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:56:13.872188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257556480,"num_objects":3930,"num_object_clones":0,"num_object_copies":11790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12550,"num_write_kb":527360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":508,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866798+0000","last_change":"2024-06-14T23:00:33.815761+0000","last_active":"2024-06-14T23:03:07.866798+0000","last_peered":"2024-06-14T23:03:07.866798+0000","last_clean":"2024-06-14T23:03:07.866798+0000","last_became_active":"2024-06-14T23:00:17.400781+0000","last_became_peered":"2024-06-14T23:00:17.400781+0000","last_unstale":"2024-06-14T23:03:07.866798+0000","last_undegraded":"2024-06-14T23:03:07.866798+0000","last_fullsized":"2024-06-14T23:03:07.866798+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:21:41.196504+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":172,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.879367+0000","last_change":"2024-06-14T22:49:25.495522+0000","last_active":"2024-06-14T23:03:07.879367+0000","last_peered":"2024-06-14T23:03:07.879367+0000","last_clean":"2024-06-14T23:03:07.879367+0000","last_became_active":"2024-06-14T22:49:25.495035+0000","last_became_peered":"2024-06-14T22:49:25.495035+0000","last_unstale":"2024-06-14T23:03:07.879367+0000","last_undegraded":"2024-06-14T23:03:07.879367+0000","last_fullsized":"2024-06-14T23:03:07.879367+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:51.777221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":181,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.869529+0000","last_change":"2024-06-14T22:53:32.496346+0000","last_active":"2024-06-14T23:03:07.869529+0000","last_peered":"2024-06-14T23:03:07.869529+0000","last_clean":"2024-06-14T23:03:07.869529+0000","last_became_active":"2024-06-14T22:53:32.495986+0000","last_became_peered":"2024-06-14T22:53:32.495986+0000","last_unstale":"2024-06-14T23:03:07.869529+0000","last_undegraded":"2024-06-14T23:03:07.869529+0000","last_fullsized":"2024-06-14T23:03:07.869529+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:16.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"211'12429","reported_seq":12462,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:18.932611+0000","last_change":"2024-06-14T23:00:27.802274+0000","last_active":"2024-06-14T23:03:18.932611+0000","last_peered":"2024-06-14T23:03:18.932611+0000","last_clean":"2024-06-14T23:03:18.932611+0000","last_became_active":"2024-06-14T23:00:27.801956+0000","last_became_peered":"2024-06-14T23:00:27.801956+0000","last_unstale":"2024-06-14T23:03:18.932611+0000","last_undegraded":"2024-06-14T23:03:18.932611+0000","last_fullsized":"2024-06-14T23:03:18.932611+0000","mapping_epoch":186,"log_start":"195'4700","ondisk_log_start":"195'4700","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:23.791512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256835584,"num_objects":3919,"num_object_clones":0,"num_object_copies":11757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12429,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":172,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.872828+0000","last_change":"2024-06-14T22:53:32.494949+0000","last_active":"2024-06-14T23:03:07.872828+0000","last_peered":"2024-06-14T23:03:07.872828+0000","last_clean":"2024-06-14T23:03:07.872828+0000","last_became_active":"2024-06-14T22:53:32.494603+0000","last_became_peered":"2024-06-14T22:53:32.494603+0000","last_unstale":"2024-06-14T23:03:07.872828+0000","last_undegraded":"2024-06-14T23:03:07.872828+0000","last_fullsized":"2024-06-14T23:03:07.872828+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:42.895860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":171,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.870651+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T23:03:07.870651+0000","last_peered":"2024-06-14T23:03:07.870651+0000","last_clean":"2024-06-14T23:03:07.870651+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T23:03:07.870651+0000","last_undegraded":"2024-06-14T23:03:07.870651+0000","last_fullsized":"2024-06-14T23:03:07.870651+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"211'12494","reported_seq":12527,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:18.918138+0000","last_change":"2024-06-14T23:00:27.801110+0000","last_active":"2024-06-14T23:03:18.918138+0000","last_peered":"2024-06-14T23:03:18.918138+0000","last_clean":"2024-06-14T23:03:18.918138+0000","last_became_active":"2024-06-14T23:00:27.800817+0000","last_became_peered":"2024-06-14T23:00:27.800817+0000","last_unstale":"2024-06-14T23:03:18.918138+0000","last_undegraded":"2024-06-14T23:03:18.918138+0000","last_fullsized":"2024-06-14T23:03:18.918138+0000","mapping_epoch":186,"log_start":"197'5600","ondisk_log_start":"197'5600","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":6894,"log_dups_size":0,"ondisk_log_size":6894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:48:04.422151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252575744,"num_objects":3854,"num_object_clones":0,"num_object_copies":11562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12494,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":508,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867968+0000","last_change":"2024-06-14T23:00:31.843436+0000","last_active":"2024-06-14T23:03:07.867968+0000","last_peered":"2024-06-14T23:03:07.867968+0000","last_clean":"2024-06-14T23:03:07.867968+0000","last_became_active":"2024-06-14T23:00:17.398181+0000","last_became_peered":"2024-06-14T23:00:17.398181+0000","last_unstale":"2024-06-14T23:03:07.867968+0000","last_undegraded":"2024-06-14T23:03:07.867968+0000","last_fullsized":"2024-06-14T23:03:07.867968+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:05:06.069424+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":167,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.881146+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:03:07.881146+0000","last_peered":"2024-06-14T23:03:07.881146+0000","last_clean":"2024-06-14T23:03:07.881146+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:03:07.881146+0000","last_undegraded":"2024-06-14T23:03:07.881146+0000","last_fullsized":"2024-06-14T23:03:07.881146+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":180,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867038+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:03:07.867038+0000","last_peered":"2024-06-14T23:03:07.867038+0000","last_clean":"2024-06-14T23:03:07.867038+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:03:07.867038+0000","last_undegraded":"2024-06-14T23:03:07.867038+0000","last_fullsized":"2024-06-14T23:03:07.867038+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"211'12545","reported_seq":12578,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:18.928424+0000","last_change":"2024-06-14T23:00:27.797497+0000","last_active":"2024-06-14T23:03:18.928424+0000","last_peered":"2024-06-14T23:03:18.928424+0000","last_clean":"2024-06-14T23:03:18.928424+0000","last_became_active":"2024-06-14T23:00:27.797221+0000","last_became_peered":"2024-06-14T23:00:27.797221+0000","last_unstale":"2024-06-14T23:03:18.928424+0000","last_undegraded":"2024-06-14T23:03:18.928424+0000","last_fullsized":"2024-06-14T23:03:18.928424+0000","mapping_epoch":186,"log_start":"195'4800","ondisk_log_start":"195'4800","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:24:58.530131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263127040,"num_objects":4015,"num_object_clones":0,"num_object_copies":12045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12545,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":501,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876806+0000","last_change":"2024-06-14T23:00:30.610273+0000","last_active":"2024-06-14T23:03:07.876806+0000","last_peered":"2024-06-14T23:03:07.876806+0000","last_clean":"2024-06-14T23:03:07.876806+0000","last_became_active":"2024-06-14T23:00:09.766095+0000","last_became_peered":"2024-06-14T23:00:09.766095+0000","last_unstale":"2024-06-14T23:03:07.876806+0000","last_undegraded":"2024-06-14T23:03:07.876806+0000","last_fullsized":"2024-06-14T23:03:07.876806+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.610180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.610180+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:53.521891+0000","scrub_duration":0.00217396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":176,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.875521+0000","last_change":"2024-06-14T22:53:32.495475+0000","last_active":"2024-06-14T23:03:07.875521+0000","last_peered":"2024-06-14T23:03:07.875521+0000","last_clean":"2024-06-14T23:03:07.875521+0000","last_became_active":"2024-06-14T22:53:32.495078+0000","last_became_peered":"2024-06-14T22:53:32.495078+0000","last_unstale":"2024-06-14T23:03:07.875521+0000","last_undegraded":"2024-06-14T23:03:07.875521+0000","last_fullsized":"2024-06-14T23:03:07.875521+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:43.819301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":115,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867043+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T23:03:07.867043+0000","last_peered":"2024-06-14T23:03:07.867043+0000","last_clean":"2024-06-14T23:03:07.867043+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T23:03:07.867043+0000","last_undegraded":"2024-06-14T23:03:07.867043+0000","last_fullsized":"2024-06-14T23:03:07.867043+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":154,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.880393+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T23:03:07.880393+0000","last_peered":"2024-06-14T23:03:07.880393+0000","last_clean":"2024-06-14T23:03:07.880393+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T23:03:07.880393+0000","last_undegraded":"2024-06-14T23:03:07.880393+0000","last_fullsized":"2024-06-14T23:03:07.880393+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"211'12529","reported_seq":12564,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:22.136626+0000","last_change":"2024-06-14T23:00:27.797246+0000","last_active":"2024-06-14T23:03:22.136626+0000","last_peered":"2024-06-14T23:03:22.136626+0000","last_clean":"2024-06-14T23:03:22.136626+0000","last_became_active":"2024-06-14T23:00:27.796988+0000","last_became_peered":"2024-06-14T23:00:27.796988+0000","last_unstale":"2024-06-14T23:03:22.136626+0000","last_undegraded":"2024-06-14T23:03:22.136626+0000","last_fullsized":"2024-06-14T23:03:22.136626+0000","mapping_epoch":186,"log_start":"195'4800","ondisk_log_start":"195'4800","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:17.545715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241762304,"num_objects":3689,"num_object_clones":0,"num_object_copies":11067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12529,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":510,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867192+0000","last_change":"2024-06-14T23:00:30.719758+0000","last_active":"2024-06-14T23:03:07.867192+0000","last_peered":"2024-06-14T23:03:07.867192+0000","last_clean":"2024-06-14T23:03:07.867192+0000","last_became_active":"2024-06-14T23:00:17.403817+0000","last_became_peered":"2024-06-14T23:00:17.403817+0000","last_unstale":"2024-06-14T23:03:07.867192+0000","last_undegraded":"2024-06-14T23:03:07.867192+0000","last_fullsized":"2024-06-14T23:03:07.867192+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:03:47.728089+0000","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":170,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866179+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:03:07.866179+0000","last_peered":"2024-06-14T23:03:07.866179+0000","last_clean":"2024-06-14T23:03:07.866179+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:03:07.866179+0000","last_undegraded":"2024-06-14T23:03:07.866179+0000","last_fullsized":"2024-06-14T23:03:07.866179+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":164,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.877960+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T23:03:07.877960+0000","last_peered":"2024-06-14T23:03:07.877960+0000","last_clean":"2024-06-14T23:03:07.877960+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T23:03:07.877960+0000","last_undegraded":"2024-06-14T23:03:07.877960+0000","last_fullsized":"2024-06-14T23:03:07.877960+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":178,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.877396+0000","last_change":"2024-06-14T22:53:32.497018+0000","last_active":"2024-06-14T23:03:07.877396+0000","last_peered":"2024-06-14T23:03:07.877396+0000","last_clean":"2024-06-14T23:03:07.877396+0000","last_became_active":"2024-06-14T22:53:32.496693+0000","last_became_peered":"2024-06-14T22:53:32.496693+0000","last_unstale":"2024-06-14T23:03:07.877396+0000","last_undegraded":"2024-06-14T23:03:07.877396+0000","last_fullsized":"2024-06-14T23:03:07.877396+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:08.556909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"211'12264","reported_seq":12297,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:19.388523+0000","last_change":"2024-06-14T23:00:27.801838+0000","last_active":"2024-06-14T23:03:19.388523+0000","last_peered":"2024-06-14T23:03:19.388523+0000","last_clean":"2024-06-14T23:03:19.388523+0000","last_became_active":"2024-06-14T23:00:27.801477+0000","last_became_peered":"2024-06-14T23:00:27.801477+0000","last_unstale":"2024-06-14T23:03:19.388523+0000","last_undegraded":"2024-06-14T23:03:19.388523+0000","last_fullsized":"2024-06-14T23:03:19.388523+0000","mapping_epoch":186,"log_start":"192'2500","ondisk_log_start":"192'2500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":9764,"log_dups_size":0,"ondisk_log_size":9764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:59:17.239200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254017536,"num_objects":3876,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":516480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":518,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876937+0000","last_change":"2024-06-14T23:00:31.643024+0000","last_active":"2024-06-14T23:03:07.876937+0000","last_peered":"2024-06-14T23:03:07.876937+0000","last_clean":"2024-06-14T23:03:07.876937+0000","last_became_active":"2024-06-14T23:00:17.400819+0000","last_became_peered":"2024-06-14T23:00:17.400819+0000","last_unstale":"2024-06-14T23:03:07.876937+0000","last_undegraded":"2024-06-14T23:03:07.876937+0000","last_fullsized":"2024-06-14T23:03:07.876937+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.642940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.642940+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:52:45.475940+0000","scrub_duration":0.001647253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":165,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.875194+0000","last_change":"2024-06-14T22:54:06.275775+0000","last_active":"2024-06-14T23:03:07.875194+0000","last_peered":"2024-06-14T23:03:07.875194+0000","last_clean":"2024-06-14T23:03:07.875194+0000","last_became_active":"2024-06-14T22:54:06.275451+0000","last_became_peered":"2024-06-14T22:54:06.275451+0000","last_unstale":"2024-06-14T23:03:07.875194+0000","last_undegraded":"2024-06-14T23:03:07.875194+0000","last_fullsized":"2024-06-14T23:03:07.875194+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:20:32.798453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":180,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866294+0000","last_change":"2024-06-14T22:53:32.493488+0000","last_active":"2024-06-14T23:03:07.866294+0000","last_peered":"2024-06-14T23:03:07.866294+0000","last_clean":"2024-06-14T23:03:07.866294+0000","last_became_active":"2024-06-14T22:53:32.493171+0000","last_became_peered":"2024-06-14T22:53:32.493171+0000","last_unstale":"2024-06-14T23:03:07.866294+0000","last_undegraded":"2024-06-14T23:03:07.866294+0000","last_fullsized":"2024-06-14T23:03:07.866294+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:49.038464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":165,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866913+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:03:07.866913+0000","last_peered":"2024-06-14T23:03:07.866913+0000","last_clean":"2024-06-14T23:03:07.866913+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:03:07.866913+0000","last_undegraded":"2024-06-14T23:03:07.866913+0000","last_fullsized":"2024-06-14T23:03:07.866913+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"211'12400","reported_seq":12433,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:18.928305+0000","last_change":"2024-06-14T23:00:27.799968+0000","last_active":"2024-06-14T23:03:18.928305+0000","last_peered":"2024-06-14T23:03:18.928305+0000","last_clean":"2024-06-14T23:03:18.928305+0000","last_became_active":"2024-06-14T23:00:27.799651+0000","last_became_peered":"2024-06-14T23:00:27.799651+0000","last_unstale":"2024-06-14T23:03:18.928305+0000","last_undegraded":"2024-06-14T23:03:18.928305+0000","last_fullsized":"2024-06-14T23:03:18.928305+0000","mapping_epoch":186,"log_start":"195'4700","ondisk_log_start":"195'4700","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:23:57.581265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261357568,"num_objects":3988,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12400,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":507,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.869782+0000","last_change":"2024-06-14T23:00:32.818134+0000","last_active":"2024-06-14T23:03:07.869782+0000","last_peered":"2024-06-14T23:03:07.869782+0000","last_clean":"2024-06-14T23:03:07.869782+0000","last_became_active":"2024-06-14T23:00:17.399852+0000","last_became_peered":"2024-06-14T23:00:17.399852+0000","last_unstale":"2024-06-14T23:03:07.869782+0000","last_undegraded":"2024-06-14T23:03:07.869782+0000","last_fullsized":"2024-06-14T23:03:07.869782+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.818076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.818076+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:12:39.666043+0000","scrub_duration":0.0013568390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":176,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867189+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T23:03:07.867189+0000","last_peered":"2024-06-14T23:03:07.867189+0000","last_clean":"2024-06-14T23:03:07.867189+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T23:03:07.867189+0000","last_undegraded":"2024-06-14T23:03:07.867189+0000","last_fullsized":"2024-06-14T23:03:07.867189+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":172,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.869984+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T23:03:07.869984+0000","last_peered":"2024-06-14T23:03:07.869984+0000","last_clean":"2024-06-14T23:03:07.869984+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T23:03:07.869984+0000","last_undegraded":"2024-06-14T23:03:07.869984+0000","last_fullsized":"2024-06-14T23:03:07.869984+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":195,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866357+0000","last_change":"2024-06-14T22:54:06.280880+0000","last_active":"2024-06-14T23:03:07.866357+0000","last_peered":"2024-06-14T23:03:07.866357+0000","last_clean":"2024-06-14T23:03:07.866357+0000","last_became_active":"2024-06-14T22:54:06.280562+0000","last_became_peered":"2024-06-14T22:54:06.280562+0000","last_unstale":"2024-06-14T23:03:07.866357+0000","last_undegraded":"2024-06-14T23:03:07.866357+0000","last_fullsized":"2024-06-14T23:03:07.866357+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:07.871499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"211'12417","reported_seq":12450,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:19.056897+0000","last_change":"2024-06-14T23:00:27.803439+0000","last_active":"2024-06-14T23:03:19.056897+0000","last_peered":"2024-06-14T23:03:19.056897+0000","last_clean":"2024-06-14T23:03:19.056897+0000","last_became_active":"2024-06-14T23:00:27.802783+0000","last_became_peered":"2024-06-14T23:00:27.802783+0000","last_unstale":"2024-06-14T23:03:19.056897+0000","last_undegraded":"2024-06-14T23:03:19.056897+0000","last_fullsized":"2024-06-14T23:03:19.056897+0000","mapping_epoch":186,"log_start":"195'4500","ondisk_log_start":"195'4500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:35:39.179455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259457024,"num_objects":3959,"num_object_clones":0,"num_object_copies":11877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12417,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":461,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.878971+0000","last_change":"2024-06-14T23:00:42.661659+0000","last_active":"2024-06-14T23:03:07.878971+0000","last_peered":"2024-06-14T23:03:07.878971+0000","last_clean":"2024-06-14T23:03:07.878971+0000","last_became_active":"2024-06-14T23:00:17.398852+0000","last_became_peered":"2024-06-14T23:00:17.398852+0000","last_unstale":"2024-06-14T23:03:07.878971+0000","last_undegraded":"2024-06-14T23:03:07.878971+0000","last_fullsized":"2024-06-14T23:03:07.878971+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:47:14.444805+0000","scrub_duration":0.0053445919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":189,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867717+0000","last_change":"2024-06-14T22:54:06.280346+0000","last_active":"2024-06-14T23:03:07.867717+0000","last_peered":"2024-06-14T23:03:07.867717+0000","last_clean":"2024-06-14T23:03:07.867717+0000","last_became_active":"2024-06-14T22:54:06.280011+0000","last_became_peered":"2024-06-14T22:54:06.280011+0000","last_unstale":"2024-06-14T23:03:07.867717+0000","last_undegraded":"2024-06-14T23:03:07.867717+0000","last_fullsized":"2024-06-14T23:03:07.867717+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:18:41.143637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":170,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866499+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:03:07.866499+0000","last_peered":"2024-06-14T23:03:07.866499+0000","last_clean":"2024-06-14T23:03:07.866499+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:03:07.866499+0000","last_undegraded":"2024-06-14T23:03:07.866499+0000","last_fullsized":"2024-06-14T23:03:07.866499+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":165,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866755+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:03:07.866755+0000","last_peered":"2024-06-14T23:03:07.866755+0000","last_clean":"2024-06-14T23:03:07.866755+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:03:07.866755+0000","last_undegraded":"2024-06-14T23:03:07.866755+0000","last_fullsized":"2024-06-14T23:03:07.866755+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"211'12413","reported_seq":12447,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:18.902235+0000","last_change":"2024-06-14T23:00:27.800097+0000","last_active":"2024-06-14T23:03:18.902235+0000","last_peered":"2024-06-14T23:03:18.902235+0000","last_clean":"2024-06-14T23:03:18.902235+0000","last_became_active":"2024-06-14T23:00:27.799790+0000","last_became_peered":"2024-06-14T23:00:27.799790+0000","last_unstale":"2024-06-14T23:03:18.902235+0000","last_undegraded":"2024-06-14T23:03:18.902235+0000","last_fullsized":"2024-06-14T23:03:18.902235+0000","mapping_epoch":186,"log_start":"195'4700","ondisk_log_start":"195'4700","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:21:01.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261357592,"num_objects":3989,"num_object_clones":0,"num_object_copies":11967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12413,"num_write_kb":524801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":461,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.878557+0000","last_change":"2024-06-14T23:00:31.555325+0000","last_active":"2024-06-14T23:03:07.878557+0000","last_peered":"2024-06-14T23:03:07.878557+0000","last_clean":"2024-06-14T23:03:07.878557+0000","last_became_active":"2024-06-14T23:00:17.400871+0000","last_became_peered":"2024-06-14T23:00:17.400871+0000","last_unstale":"2024-06-14T23:03:07.878557+0000","last_undegraded":"2024-06-14T23:03:07.878557+0000","last_fullsized":"2024-06-14T23:03:07.878557+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:58.191436+0000","scrub_duration":0.001665094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":180,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866431+0000","last_change":"2024-06-14T22:53:32.495753+0000","last_active":"2024-06-14T23:03:07.866431+0000","last_peered":"2024-06-14T23:03:07.866431+0000","last_clean":"2024-06-14T23:03:07.866431+0000","last_became_active":"2024-06-14T22:53:32.495461+0000","last_became_peered":"2024-06-14T22:53:32.495461+0000","last_unstale":"2024-06-14T23:03:07.866431+0000","last_undegraded":"2024-06-14T23:03:07.866431+0000","last_fullsized":"2024-06-14T23:03:07.866431+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:05.799728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":176,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.874779+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:03:07.874779+0000","last_peered":"2024-06-14T23:03:07.874779+0000","last_clean":"2024-06-14T23:03:07.874779+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:03:07.874779+0000","last_undegraded":"2024-06-14T23:03:07.874779+0000","last_fullsized":"2024-06-14T23:03:07.874779+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":150,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876889+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T23:03:07.876889+0000","last_peered":"2024-06-14T23:03:07.876889+0000","last_clean":"2024-06-14T23:03:07.876889+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T23:03:07.876889+0000","last_undegraded":"2024-06-14T23:03:07.876889+0000","last_fullsized":"2024-06-14T23:03:07.876889+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"211'12478","reported_seq":12511,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:18.904952+0000","last_change":"2024-06-14T23:00:27.797841+0000","last_active":"2024-06-14T23:03:18.904952+0000","last_peered":"2024-06-14T23:03:18.904952+0000","last_clean":"2024-06-14T23:03:18.904952+0000","last_became_active":"2024-06-14T23:00:27.797256+0000","last_became_peered":"2024-06-14T23:00:27.797256+0000","last_unstale":"2024-06-14T23:03:18.904952+0000","last_undegraded":"2024-06-14T23:03:18.904952+0000","last_fullsized":"2024-06-14T23:03:18.904952+0000","mapping_epoch":186,"log_start":"197'5600","ondisk_log_start":"197'5600","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:52:24.873796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264241152,"num_objects":4032,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12478,"num_write_kb":528320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":508,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.869447+0000","last_change":"2024-06-14T23:00:32.174187+0000","last_active":"2024-06-14T23:03:07.869447+0000","last_peered":"2024-06-14T23:03:07.869447+0000","last_clean":"2024-06-14T23:03:07.869447+0000","last_became_active":"2024-06-14T23:00:17.395654+0000","last_became_peered":"2024-06-14T23:00:17.395654+0000","last_unstale":"2024-06-14T23:03:07.869447+0000","last_undegraded":"2024-06-14T23:03:07.869447+0000","last_fullsized":"2024-06-14T23:03:07.869447+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.174132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.174132+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:59.252795+0000","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":72,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876826+0000","last_change":"2024-06-14T22:58:09.752729+0000","last_active":"2024-06-14T23:03:07.876826+0000","last_peered":"2024-06-14T23:03:07.876826+0000","last_clean":"2024-06-14T23:03:07.876826+0000","last_became_active":"2024-06-14T22:58:09.752393+0000","last_became_peered":"2024-06-14T22:58:09.752393+0000","last_unstale":"2024-06-14T23:03:07.876826+0000","last_undegraded":"2024-06-14T23:03:07.876826+0000","last_fullsized":"2024-06-14T23:03:07.876826+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:21:58.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":164,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867037+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T23:03:07.867037+0000","last_peered":"2024-06-14T23:03:07.867037+0000","last_clean":"2024-06-14T23:03:07.867037+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T23:03:07.867037+0000","last_undegraded":"2024-06-14T23:03:07.867037+0000","last_fullsized":"2024-06-14T23:03:07.867037+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":156,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.875010+0000","last_change":"2024-06-14T22:58:32.754079+0000","last_active":"2024-06-14T23:03:07.875010+0000","last_peered":"2024-06-14T23:03:07.875010+0000","last_clean":"2024-06-14T23:03:07.875010+0000","last_became_active":"2024-06-14T22:58:32.753814+0000","last_became_peered":"2024-06-14T22:58:32.753814+0000","last_unstale":"2024-06-14T23:03:07.875010+0000","last_undegraded":"2024-06-14T23:03:07.875010+0000","last_fullsized":"2024-06-14T23:03:07.875010+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:37:07.414370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"211'12093","reported_seq":12126,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-06-14T23:03:18.932785+0000","last_change":"2024-06-14T23:00:27.798887+0000","last_active":"2024-06-14T23:03:18.932785+0000","last_peered":"2024-06-14T23:03:18.932785+0000","last_clean":"2024-06-14T23:03:18.932785+0000","last_became_active":"2024-06-14T23:00:27.798590+0000","last_became_peered":"2024-06-14T23:00:27.798590+0000","last_unstale":"2024-06-14T23:03:18.932785+0000","last_undegraded":"2024-06-14T23:03:18.932785+0000","last_fullsized":"2024-06-14T23:03:18.932785+0000","mapping_epoch":186,"log_start":"194'4400","ondisk_log_start":"194'4400","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:00:26.778365+0000","last_clean_scrub_stamp":"2024-06-14T23:00:26.778365+0000","objects_scrubbed":0,"log_size":7693,"log_dups_size":0,"ondisk_log_size":7693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:02:33.396409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253034496,"num_objects":3861,"num_object_clones":0,"num_object_copies":11583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12093,"num_write_kb":510528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":510,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.882036+0000","last_change":"2024-06-14T23:00:32.716013+0000","last_active":"2024-06-14T23:03:07.882036+0000","last_peered":"2024-06-14T23:03:07.882036+0000","last_clean":"2024-06-14T23:03:07.882036+0000","last_became_active":"2024-06-14T23:00:17.398349+0000","last_became_peered":"2024-06-14T23:00:17.398349+0000","last_unstale":"2024-06-14T23:03:07.882036+0000","last_undegraded":"2024-06-14T23:03:07.882036+0000","last_fullsized":"2024-06-14T23:03:07.882036+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.715907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.715907+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:23:40.212866+0000","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":168,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.877311+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T23:03:07.877311+0000","last_peered":"2024-06-14T23:03:07.877311+0000","last_clean":"2024-06-14T23:03:07.877311+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T23:03:07.877311+0000","last_undegraded":"2024-06-14T23:03:07.877311+0000","last_fullsized":"2024-06-14T23:03:07.877311+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":176,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.879077+0000","last_change":"2024-06-14T22:53:32.496209+0000","last_active":"2024-06-14T23:03:07.879077+0000","last_peered":"2024-06-14T23:03:07.879077+0000","last_clean":"2024-06-14T23:03:07.879077+0000","last_became_active":"2024-06-14T22:53:32.495631+0000","last_became_peered":"2024-06-14T22:53:32.495631+0000","last_unstale":"2024-06-14T23:03:07.879077+0000","last_undegraded":"2024-06-14T23:03:07.879077+0000","last_fullsized":"2024-06-14T23:03:07.879077+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:15.522372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":154,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.871465+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T23:03:07.871465+0000","last_peered":"2024-06-14T23:03:07.871465+0000","last_clean":"2024-06-14T23:03:07.871465+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T23:03:07.871465+0000","last_undegraded":"2024-06-14T23:03:07.871465+0000","last_fullsized":"2024-06-14T23:03:07.871465+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":171,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.869839+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T23:03:07.869839+0000","last_peered":"2024-06-14T23:03:07.869839+0000","last_clean":"2024-06-14T23:03:07.869839+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T23:03:07.869839+0000","last_undegraded":"2024-06-14T23:03:07.869839+0000","last_fullsized":"2024-06-14T23:03:07.869839+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":177,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.871870+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T23:03:07.871870+0000","last_peered":"2024-06-14T23:03:07.871870+0000","last_clean":"2024-06-14T23:03:07.871870+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T23:03:07.871870+0000","last_undegraded":"2024-06-14T23:03:07.871870+0000","last_fullsized":"2024-06-14T23:03:07.871870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":179,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866643+0000","last_change":"2024-06-14T22:53:32.495449+0000","last_active":"2024-06-14T23:03:07.866643+0000","last_peered":"2024-06-14T23:03:07.866643+0000","last_clean":"2024-06-14T23:03:07.866643+0000","last_became_active":"2024-06-14T22:53:32.495134+0000","last_became_peered":"2024-06-14T22:53:32.495134+0000","last_unstale":"2024-06-14T23:03:07.866643+0000","last_undegraded":"2024-06-14T23:03:07.866643+0000","last_fullsized":"2024-06-14T23:03:07.866643+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:23.794434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":91,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:09.633145+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:03:09.633145+0000","last_peered":"2024-06-14T23:03:09.633145+0000","last_clean":"2024-06-14T23:03:09.633145+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:03:09.633145+0000","last_undegraded":"2024-06-14T23:03:09.633145+0000","last_fullsized":"2024-06-14T23:03:09.633145+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":172,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867405+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:03:07.867405+0000","last_peered":"2024-06-14T23:03:07.867405+0000","last_clean":"2024-06-14T23:03:07.867405+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:03:07.867405+0000","last_undegraded":"2024-06-14T23:03:07.867405+0000","last_fullsized":"2024-06-14T23:03:07.867405+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":115,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.878564+0000","last_change":"2024-06-14T22:58:09.753103+0000","last_active":"2024-06-14T23:03:07.878564+0000","last_peered":"2024-06-14T23:03:07.878564+0000","last_clean":"2024-06-14T23:03:07.878564+0000","last_became_active":"2024-06-14T22:58:09.752761+0000","last_became_peered":"2024-06-14T22:58:09.752761+0000","last_unstale":"2024-06-14T23:03:07.878564+0000","last_undegraded":"2024-06-14T23:03:07.878564+0000","last_fullsized":"2024-06-14T23:03:07.878564+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:20:20.619331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":163,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876982+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T23:03:07.876982+0000","last_peered":"2024-06-14T23:03:07.876982+0000","last_clean":"2024-06-14T23:03:07.876982+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T23:03:07.876982+0000","last_undegraded":"2024-06-14T23:03:07.876982+0000","last_fullsized":"2024-06-14T23:03:07.876982+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":165,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.880973+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T23:03:07.880973+0000","last_peered":"2024-06-14T23:03:07.880973+0000","last_clean":"2024-06-14T23:03:07.880973+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T23:03:07.880973+0000","last_undegraded":"2024-06-14T23:03:07.880973+0000","last_fullsized":"2024-06-14T23:03:07.880973+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":171,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.879010+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T23:03:07.879010+0000","last_peered":"2024-06-14T23:03:07.879010+0000","last_clean":"2024-06-14T23:03:07.879010+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T23:03:07.879010+0000","last_undegraded":"2024-06-14T23:03:07.879010+0000","last_fullsized":"2024-06-14T23:03:07.879010+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":160,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876725+0000","last_change":"2024-06-14T22:58:09.785074+0000","last_active":"2024-06-14T23:03:07.876725+0000","last_peered":"2024-06-14T23:03:07.876725+0000","last_clean":"2024-06-14T23:03:07.876725+0000","last_became_active":"2024-06-14T22:58:09.784309+0000","last_became_peered":"2024-06-14T22:58:09.784309+0000","last_unstale":"2024-06-14T23:03:07.876725+0000","last_undegraded":"2024-06-14T23:03:07.876725+0000","last_fullsized":"2024-06-14T23:03:07.876725+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:23.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":166,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.877387+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T23:03:07.877387+0000","last_peered":"2024-06-14T23:03:07.877387+0000","last_clean":"2024-06-14T23:03:07.877387+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T23:03:07.877387+0000","last_undegraded":"2024-06-14T23:03:07.877387+0000","last_fullsized":"2024-06-14T23:03:07.877387+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":179,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.866272+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:03:07.866272+0000","last_peered":"2024-06-14T23:03:07.866272+0000","last_clean":"2024-06-14T23:03:07.866272+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:03:07.866272+0000","last_undegraded":"2024-06-14T23:03:07.866272+0000","last_fullsized":"2024-06-14T23:03:07.866272+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":168,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:10.383398+0000","last_change":"2024-06-14T22:58:09.756040+0000","last_active":"2024-06-14T23:03:10.383398+0000","last_peered":"2024-06-14T23:03:10.383398+0000","last_clean":"2024-06-14T23:03:10.383398+0000","last_became_active":"2024-06-14T22:58:09.755659+0000","last_became_peered":"2024-06-14T22:58:09.755659+0000","last_unstale":"2024-06-14T23:03:10.383398+0000","last_undegraded":"2024-06-14T23:03:10.383398+0000","last_fullsized":"2024-06-14T23:03:10.383398+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:40.759248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":190,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.868764+0000","last_change":"2024-06-14T22:53:32.496671+0000","last_active":"2024-06-14T23:03:07.868764+0000","last_peered":"2024-06-14T23:03:07.868764+0000","last_clean":"2024-06-14T23:03:07.868764+0000","last_became_active":"2024-06-14T22:53:32.496327+0000","last_became_peered":"2024-06-14T22:53:32.496327+0000","last_unstale":"2024-06-14T23:03:07.868764+0000","last_undegraded":"2024-06-14T23:03:07.868764+0000","last_fullsized":"2024-06-14T23:03:07.868764+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:38:12.333887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":184,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.875098+0000","last_change":"2024-06-14T22:54:06.277110+0000","last_active":"2024-06-14T23:03:07.875098+0000","last_peered":"2024-06-14T23:03:07.875098+0000","last_clean":"2024-06-14T23:03:07.875098+0000","last_became_active":"2024-06-14T22:54:06.276807+0000","last_became_peered":"2024-06-14T22:54:06.276807+0000","last_unstale":"2024-06-14T23:03:07.875098+0000","last_undegraded":"2024-06-14T23:03:07.875098+0000","last_fullsized":"2024-06-14T23:03:07.875098+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:37.586503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":152,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876811+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T23:03:07.876811+0000","last_peered":"2024-06-14T23:03:07.876811+0000","last_clean":"2024-06-14T23:03:07.876811+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T23:03:07.876811+0000","last_undegraded":"2024-06-14T23:03:07.876811+0000","last_fullsized":"2024-06-14T23:03:07.876811+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":188,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.869772+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:03:07.869772+0000","last_peered":"2024-06-14T23:03:07.869772+0000","last_clean":"2024-06-14T23:03:07.869772+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:03:07.869772+0000","last_undegraded":"2024-06-14T23:03:07.869772+0000","last_fullsized":"2024-06-14T23:03:07.869772+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":174,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867327+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:03:07.867327+0000","last_peered":"2024-06-14T23:03:07.867327+0000","last_clean":"2024-06-14T23:03:07.867327+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:03:07.867327+0000","last_undegraded":"2024-06-14T23:03:07.867327+0000","last_fullsized":"2024-06-14T23:03:07.867327+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":188,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.868690+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:03:07.868690+0000","last_peered":"2024-06-14T23:03:07.868690+0000","last_clean":"2024-06-14T23:03:07.868690+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:03:07.868690+0000","last_undegraded":"2024-06-14T23:03:07.868690+0000","last_fullsized":"2024-06-14T23:03:07.868690+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":157,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.867660+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T23:03:07.867660+0000","last_peered":"2024-06-14T23:03:07.867660+0000","last_clean":"2024-06-14T23:03:07.867660+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T23:03:07.867660+0000","last_undegraded":"2024-06-14T23:03:07.867660+0000","last_fullsized":"2024-06-14T23:03:07.867660+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":175,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.875128+0000","last_change":"2024-06-14T22:58:09.752281+0000","last_active":"2024-06-14T23:03:07.875128+0000","last_peered":"2024-06-14T23:03:07.875128+0000","last_clean":"2024-06-14T23:03:07.875128+0000","last_became_active":"2024-06-14T22:58:09.751952+0000","last_became_peered":"2024-06-14T22:58:09.751952+0000","last_unstale":"2024-06-14T23:03:07.875128+0000","last_undegraded":"2024-06-14T23:03:07.875128+0000","last_fullsized":"2024-06-14T23:03:07.875128+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:39:19.836734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":70,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.878643+0000","last_change":"2024-06-14T22:58:09.755622+0000","last_active":"2024-06-14T23:03:07.878643+0000","last_peered":"2024-06-14T23:03:07.878643+0000","last_clean":"2024-06-14T23:03:07.878643+0000","last_became_active":"2024-06-14T22:58:09.755383+0000","last_became_peered":"2024-06-14T22:58:09.755383+0000","last_unstale":"2024-06-14T23:03:07.878643+0000","last_undegraded":"2024-06-14T23:03:07.878643+0000","last_fullsized":"2024-06-14T23:03:07.878643+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:13.884346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":171,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.868481+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T23:03:07.868481+0000","last_peered":"2024-06-14T23:03:07.868481+0000","last_clean":"2024-06-14T23:03:07.868481+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T23:03:07.868481+0000","last_undegraded":"2024-06-14T23:03:07.868481+0000","last_fullsized":"2024-06-14T23:03:07.868481+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":168,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-06-14T23:03:07.876680+0000","last_change":"2024-06-14T22:53:32.496455+0000","last_active":"2024-06-14T23:03:07.876680+0000","last_peered":"2024-06-14T23:03:07.876680+0000","last_clean":"2024-06-14T23:03:07.876680+0000","last_became_active":"2024-06-14T22:53:32.496053+0000","last_became_peered":"2024-06-14T22:53:32.496053+0000","last_unstale":"2024-06-14T23:03:07.876680+0000","last_undegraded":"2024-06-14T23:03:07.876680+0000","last_fullsized":"2024-06-14T23:03:07.876680+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:25:47.732066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4092461080,"num_objects":62447,"num_object_clones":0,"num_object_copies":187341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199485,"num_write_kb":8381761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12216971264,"data_stored":12216959048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131485,"ondisk_log_size":131485,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1771,"ondisk_log_size":1771,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234351,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1917328,"kb_used_data":1006088,"kb_used_omap":1,"kb_used_meta":911230,"kb_avail":91827824,"statfs":{"total":95995035648,"available":94031691776,"internally_reserved":0,"allocated":1030234112,"data_stored":1029178663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":933099864},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.135,"15min":1.034},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.1699999999999999,"5min":6.9539999999999997,"15min":9.3919999999999995},"last":3.0299999999999998},{"interface":"front","average":{"1min":2.2509999999999999,"5min":1.4219999999999999,"15min":1.226},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20300000000000001},"max":{"1min":6.9550000000000001,"5min":6.9550000000000001,"15min":10.416},"last":1.9990000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.4339999999999999,"15min":1.7669999999999999},"min":{"1min":0.308,"5min":0.29199999999999998,"15min":0.28100000000000003},"max":{"1min":6.3090000000000002,"5min":8.5340000000000007,"15min":83.295000000000002},"last":1.645},{"interface":"front","average":{"1min":0.73099999999999998,"5min":1.2949999999999999,"15min":1.768},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.223},"max":{"1min":2.5550000000000002,"5min":6.8620000000000001,"15min":82.972999999999999},"last":6.2279999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.196,"15min":0.84899999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.468,"5min":7.468,"15min":7.468},"last":3.2749999999999999},{"interface":"front","average":{"1min":1.288,"5min":1.246,"15min":0.89800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.0469999999999997,"5min":6.9779999999999998,"15min":8.1150000000000002},"last":3.1200000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":1.3009999999999999,"15min":1.1539999999999999},"min":{"1min":0.29099999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.8609999999999998,"5min":7.8609999999999998,"15min":55.683999999999997},"last":0.312},{"interface":"front","average":{"1min":1.9850000000000001,"5min":1.3240000000000001,"15min":1.1839999999999999},"min":{"1min":0.35099999999999998,"5min":0.30099999999999999,"15min":0.29799999999999999},"max":{"1min":7.7670000000000003,"5min":7.7670000000000003,"15min":55.533999999999999},"last":1.6759999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:03:08 2024","interfaces":[{"interface":"back","average":{"1min":2.294,"5min":1.2709999999999999,"15min":0.86399999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":11.342000000000001,"5min":11.342000000000001,"15min":11.342000000000001},"last":1.71},{"interface":"front","average":{"1min":1.905,"5min":1.135,"15min":0.88800000000000001},"min":{"1min":0.253,"5min":0.247,"15min":0.247},"max":{"1min":12.651,"5min":12.651,"15min":16.879000000000001},"last":3.145}]},{"osd":5,"last update":"Fri Jun 14 23:03:08 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3859999999999999,"15min":0.98099999999999998},"min":{"1min":0.27800000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.6619999999999999,"5min":6.9000000000000004,"15min":6.9000000000000004},"last":3.0579999999999998},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.1870000000000001,"15min":0.97599999999999998},"min":{"1min":0.27100000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.2809999999999997,"5min":6.2809999999999997,"15min":6.8529999999999998},"last":1.732}]},{"osd":6,"last update":"Fri Jun 14 23:03:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":1.3380000000000001,"15min":0.94499999999999995},"min":{"1min":0.32600000000000001,"5min":0.307,"15min":0.28100000000000003},"max":{"1min":8.952,"5min":8.952,"15min":8.952},"last":1.798},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.181,"15min":0.93799999999999994},"min":{"1min":0.29599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.7690000000000001,"5min":6.7690000000000001,"15min":12.951000000000001},"last":1.508}]}]},{"osd":6,"up_from":46,"seq":197568495987,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2690436,"kb_used_data":1793148,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":91054716,"statfs":{"total":95995035648,"available":93240029184,"internally_reserved":0,"allocated":1836183552,"data_stored":1835139946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:03:14 2024","interfaces":[{"interface":"back","average":{"1min":4.5179999999999998,"5min":2.0129999999999999,"15min":1.5149999999999999},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.218},"max":{"1min":45.579999999999998,"5min":45.579999999999998,"15min":45.579999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":4.0030000000000001,"5min":1.7210000000000001,"15min":1.375},"min":{"1min":0.34300000000000003,"5min":0.246,"15min":0.21299999999999999},"max":{"1min":44.918999999999997,"5min":44.918999999999997,"15min":44.918999999999997},"last":2.7370000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:02:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":1.1899999999999999,"15min":1.1799999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.2360000000000002,"5min":6.6680000000000001,"15min":9.9559999999999995},"last":2.762},{"interface":"front","average":{"1min":0.76700000000000002,"5min":1.254,"15min":1.1990000000000001},"min":{"1min":0.27600000000000002,"5min":0.25,"15min":0.247},"max":{"1min":2.1280000000000001,"5min":6.7519999999999998,"15min":9.5510000000000002},"last":0.628}]},{"osd":2,"last update":"Fri Jun 14 23:02:45 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0800000000000001,"15min":0.79200000000000004},"min":{"1min":0.31,"5min":0.29799999999999999,"15min":0.27800000000000002},"max":{"1min":2.8479999999999999,"5min":9.9600000000000009,"15min":9.9600000000000009},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.2869999999999999,"15min":0.88},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":10.823,"5min":10.823,"15min":10.823},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.22,"15min":1.0309999999999999},"min":{"1min":0.32000000000000001,"5min":0.28599999999999998,"15min":0.27300000000000002},"max":{"1min":5.0949999999999998,"5min":6.5449999999999999,"15min":29.152000000000001},"last":1.202},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.1819999999999999,"15min":1.036},"min":{"1min":0.372,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.331,"5min":6.3040000000000003,"15min":25.864000000000001},"last":2.8500000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:02:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.98699999999999999,"15min":0.92400000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.264,"5min":7.7709999999999999,"15min":47.698},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.93600000000000005,"15min":0.877},"min":{"1min":0.28899999999999998,"5min":0.27600000000000002,"15min":0.24299999999999999},"max":{"1min":2.3999999999999999,"5min":9.9610000000000003,"15min":48.197000000000003},"last":0.77500000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1459999999999999,"15min":0.88400000000000001},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.26500000000000001},"max":{"1min":5.0010000000000003,"5min":7.3540000000000001,"15min":8.6500000000000004},"last":2.7040000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.103,"15min":0.90300000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":2.714,"5min":7.3769999999999998,"15min":8.6349999999999998},"last":0.58399999999999996}]},{"osd":7,"last update":"Fri Jun 14 23:03:06 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.095,"15min":0.90400000000000003},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.29299999999999998},"max":{"1min":3.1680000000000001,"5min":7.8579999999999997,"15min":8.6950000000000003},"last":2.7839999999999998},{"interface":"front","average":{"1min":0.81799999999999995,"5min":1.129,"15min":0.93600000000000005},"min":{"1min":0.34899999999999998,"5min":0.34699999999999998,"15min":0.28899999999999998},"max":{"1min":2.6240000000000001,"5min":7.8120000000000003,"15min":8.673},"last":0.60699999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757626,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2166460,"kb_used_data":1275252,"kb_used_omap":1,"kb_used_meta":891134,"kb_avail":91578692,"statfs":{"total":95995035648,"available":93776580608,"internally_reserved":0,"allocated":1305858048,"data_stored":1304818968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912521560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:37 2024","interfaces":[{"interface":"back","average":{"1min":2.605,"5min":1.4890000000000001,"15min":1.2769999999999999},"min":{"1min":0.35599999999999998,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":9.8079999999999998,"5min":9.8079999999999998,"15min":17.766999999999999},"last":1.286},{"interface":"front","average":{"1min":2.3420000000000001,"5min":1.389,"15min":1.2430000000000001},"min":{"1min":0.25,"5min":0.24299999999999999,"15min":0.221},"max":{"1min":5.1660000000000004,"5min":5.1950000000000003,"15min":17.806000000000001},"last":0.77700000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.92800000000000005,"15min":1.1180000000000001},"min":{"1min":0.24399999999999999,"5min":0.23999999999999999,"15min":0.23300000000000001},"max":{"1min":3.4049999999999998,"5min":5.6429999999999998,"15min":7.694},"last":1.274},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.252,"15min":1.1930000000000001},"min":{"1min":0.27600000000000002,"5min":0.247,"15min":0.23400000000000001},"max":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"last":1.2629999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":1.274,"15min":0.86499999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.2489999999999997,"5min":7.2489999999999997,"15min":7.2489999999999997},"last":3.7919999999999998},{"interface":"front","average":{"1min":2.1400000000000001,"5min":1.381,"15min":0.89500000000000002},"min":{"1min":0.32400000000000001,"5min":0.28100000000000003,"15min":0.27800000000000002},"max":{"1min":7.4500000000000002,"5min":7.9000000000000004,"15min":7.9000000000000004},"last":1.2410000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:02:19 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.534,"15min":1.1319999999999999},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.4580000000000002,"5min":10.358000000000001,"15min":45.744},"last":2.5379999999999998},{"interface":"front","average":{"1min":1.845,"5min":1.5900000000000001,"15min":1.222},"min":{"1min":0.378,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":6.4199999999999999,"5min":10.385,"15min":45.838000000000001},"last":3.7570000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:03:12 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":0.94499999999999995,"15min":0.71499999999999997},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.214},"max":{"1min":5.4000000000000004,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.254,"15min":0.86799999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.22700000000000001},"max":{"1min":9.2420000000000009,"5min":9.2420000000000009,"15min":9.2420000000000009},"last":0.75}]},{"osd":6,"last update":"Fri Jun 14 23:02:41 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.1559999999999999,"15min":0.88300000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.23000000000000001},"max":{"1min":5.5609999999999999,"5min":6.931,"15min":9.6020000000000003},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.377,"5min":1.1080000000000001,"15min":0.84999999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.25},"max":{"1min":5.625,"5min":5.625,"15min":12.339},"last":1.2090000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:03:08 2024","interfaces":[{"interface":"back","average":{"1min":2.3210000000000002,"5min":1.3720000000000001,"15min":0.95699999999999996},"min":{"1min":0.33600000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":14.818,"5min":14.818,"15min":14.818},"last":2.1110000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.0660000000000001,"15min":0.85799999999999998},"min":{"1min":0.30299999999999999,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":3.331,"5min":5.6189999999999998,"15min":10.638999999999999},"last":0.50900000000000001}]}]},{"osd":4,"up_from":30,"seq":128849019262,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2917004,"kb_used_data":2018116,"kb_used_omap":1,"kb_used_meta":898878,"kb_avail":90828148,"statfs":{"total":95995035648,"available":93008023552,"internally_reserved":0,"allocated":2066550784,"data_stored":2065499120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920451416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4220000000000002,"5min":1.3140000000000001,"15min":1.125},"min":{"1min":0.25800000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":10.567,"5min":10.567,"15min":10.567},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.2110000000000001,"15min":1.139},"min":{"1min":0.23999999999999999,"5min":0.22700000000000001,"15min":0.221},"max":{"1min":5.9189999999999996,"5min":8.3390000000000004,"15min":9.1470000000000002},"last":3.798}]},{"osd":1,"last update":"Fri Jun 14 23:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":2.044,"15min":1.395},"min":{"1min":0.27000000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.6929999999999996,"5min":56.917000000000002,"15min":56.917000000000002},"last":4.4489999999999998},{"interface":"front","average":{"1min":2.1280000000000001,"5min":2.0470000000000002,"15min":1.4890000000000001},"min":{"1min":0.28799999999999998,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":10.539999999999999,"5min":56.789999999999999,"15min":56.789999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.667,"5min":1.0780000000000001,"15min":0.748},"min":{"1min":0.26500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.0039999999999996,"5min":8.9250000000000007,"15min":8.9250000000000007},"last":4.4279999999999999},{"interface":"front","average":{"1min":2.5369999999999999,"5min":1.238,"15min":0.83899999999999997},"min":{"1min":0.41699999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":10.478,"5min":10.478,"15min":10.478},"last":1.591}]},{"osd":3,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.268,"15min":0.86399999999999999},"min":{"1min":0.28599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.327,"5min":6.617,"15min":6.617},"last":3.9009999999999998},{"interface":"front","average":{"1min":2.1269999999999998,"5min":1.3819999999999999,"15min":0.92700000000000005},"min":{"1min":0.33100000000000002,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":10.510999999999999,"5min":10.510999999999999,"15min":10.510999999999999},"last":4.1609999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:03:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":2.754,"15min":1.405},"min":{"1min":0.29499999999999998,"5min":0.253,"15min":0.22500000000000001},"max":{"1min":5.2859999999999996,"5min":139.13200000000001,"15min":139.13200000000001},"last":3.7599999999999998},{"interface":"front","average":{"1min":2.089,"5min":2.8879999999999999,"15min":1.452},"min":{"1min":0.29799999999999999,"5min":0.28000000000000003,"15min":0.23699999999999999},"max":{"1min":10.419,"5min":138.86000000000001,"15min":138.86000000000001},"last":0.93899999999999995}]},{"osd":6,"last update":"Fri Jun 14 23:02:32 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0629999999999999,"15min":0.75700000000000001},"min":{"1min":0.32600000000000001,"5min":0.28899999999999998,"15min":0.254},"max":{"1min":4.6269999999999998,"5min":5.1210000000000004,"15min":5.1210000000000004},"last":3.8210000000000002},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.1120000000000001,"15min":0.748},"min":{"1min":0.35899999999999999,"5min":0.27900000000000003,"15min":0.254},"max":{"1min":5.8680000000000003,"5min":8.8819999999999997,"15min":8.8819999999999997},"last":0.40600000000000003}]},{"osd":7,"last update":"Fri Jun 14 23:02:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.141,"15min":0.80900000000000005},"min":{"1min":0.30099999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.0800000000000001,"5min":8.9489999999999998,"15min":8.9489999999999998},"last":4.4000000000000004},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.284,"15min":0.92800000000000005},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.3479999999999999,"5min":7.25,"15min":8.4730000000000008},"last":3.8439999999999999}]}]},{"osd":3,"up_from":138,"seq":592705486961,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1584060,"kb_used_data":1273716,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":92161092,"statfs":{"total":95995035648,"available":94372958208,"internally_reserved":0,"allocated":1304285184,"data_stored":1303245877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":317779422},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.9850000000000001,"5min":1.4570000000000001,"15min":0.78100000000000003},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.184},"max":{"1min":5.8499999999999996,"5min":6.3360000000000003,"15min":6.3360000000000003},"last":0.86399999999999999},{"interface":"front","average":{"1min":2.0009999999999999,"5min":1.2569999999999999,"15min":0.70899999999999996},"min":{"1min":0.26100000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":8.2330000000000005,"5min":8.2330000000000005,"15min":8.2330000000000005},"last":0.26900000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.4890000000000001,"15min":0.82999999999999996},"min":{"1min":0.249,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":4.4189999999999996,"5min":24.097999999999999,"15min":24.097999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.5680000000000001,"15min":0.86199999999999999},"min":{"1min":0.35299999999999998,"5min":0.26600000000000001,"15min":0.24099999999999999},"max":{"1min":3.5859999999999999,"5min":24.132999999999999,"15min":24.132999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.1950000000000001,"15min":0.72399999999999998},"min":{"1min":0.29499999999999998,"5min":0.245,"15min":0.23599999999999999},"max":{"1min":3.831,"5min":5.2910000000000004,"15min":5.2910000000000004},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.9159999999999999,"5min":1.2,"15min":0.71599999999999997},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":10.369,"5min":10.369,"15min":10.369},"last":0.39200000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.2170000000000001,"15min":0.877},"min":{"1min":0.40600000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.976,"5min":5.1189999999999998,"15min":5.7960000000000003},"last":1.724},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3680000000000001,"15min":0.92800000000000005},"min":{"1min":0.379,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.1580000000000004,"5min":5.6660000000000004,"15min":5.6660000000000004},"last":0.81200000000000006}]},{"osd":5,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.405,"15min":0.96699999999999997},"min":{"1min":0.34799999999999998,"5min":0.30599999999999999,"15min":0.29699999999999999},"max":{"1min":5.117,"5min":5.9359999999999999,"15min":5.9359999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.427,"5min":1.373,"15min":0.92100000000000004},"min":{"1min":0.32400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.6360000000000001,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":1.528}]},{"osd":6,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.27,"15min":0.89600000000000002},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.762,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.252,"15min":0.88100000000000001},"min":{"1min":0.32200000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.3760000000000003,"5min":5.0439999999999996,"15min":5.8259999999999996},"last":2.1640000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:02:20 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.3919999999999999,"15min":0.997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35299999999999998},"max":{"1min":4.351,"5min":7.7240000000000002,"15min":7.7240000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.423,"15min":0.98199999999999998},"min":{"1min":0.41299999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":5.2160000000000002,"5min":5.8300000000000001,"15min":5.8300000000000001},"last":0.83899999999999997}]}]},{"osd":2,"up_from":133,"seq":571230650488,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1728828,"kb_used_data":1511220,"kb_used_omap":6,"kb_used_meta":217593,"kb_avail":92016324,"statfs":{"total":95995035648,"available":94224715776,"internally_reserved":0,"allocated":1547489280,"data_stored":1546450187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":222815733},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.85599999999999998,"15min":0.58299999999999996},"min":{"1min":0.19,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":5.0640000000000001,"5min":7.476,"15min":7.476},"last":0.97899999999999998},{"interface":"front","average":{"1min":2.1139999999999999,"5min":1.4370000000000001,"15min":0.753},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.182},"max":{"1min":8.0180000000000007,"5min":11.369,"15min":11.369},"last":0.42999999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.073,"15min":0.80500000000000005},"min":{"1min":0.221,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.726,"5min":5.8200000000000003,"15min":21.920000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.872,"5min":0.98299999999999998,"15min":0.76800000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":3.9940000000000002,"5min":13.054,"15min":21.573},"last":0.53700000000000003}]},{"osd":3,"last update":"Fri Jun 14 23:02:21 2024","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.4119999999999999,"15min":0.83099999999999996},"min":{"1min":0.23999999999999999,"5min":0.219,"15min":0.219},"max":{"1min":11.24,"5min":11.24,"15min":21.66},"last":0.48299999999999998},{"interface":"front","average":{"1min":2.0099999999999998,"5min":1.3149999999999999,"15min":0.872},"min":{"1min":0.27100000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":11.271000000000001,"5min":11.271000000000001,"15min":22.341000000000001},"last":0.81000000000000005}]},{"osd":4,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2349999999999999,"5min":1.708,"15min":0.99399999999999999},"min":{"1min":0.41999999999999998,"5min":0.36499999999999999,"15min":0.318},"max":{"1min":4.9189999999999996,"5min":14.438000000000001,"15min":14.438000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.5629999999999999,"15min":0.94299999999999995},"min":{"1min":0.308,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.9290000000000003,"5min":14.423,"15min":14.423},"last":0.498}]},{"osd":5,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5900000000000001,"15min":0.93000000000000005},"min":{"1min":0.29599999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.9409999999999998,"5min":31.539999999999999,"15min":31.539999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.663,"5min":1.619,"15min":0.94699999999999995},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":4.1760000000000002,"5min":31.481999999999999,"15min":31.481999999999999},"last":1.1279999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":2.0049999999999999,"5min":1.468,"15min":0.879},"min":{"1min":0.375,"5min":0.309,"15min":0.309},"max":{"1min":8.9730000000000008,"5min":11.454000000000001,"15min":11.454000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1659999999999999,"15min":0.81799999999999995},"min":{"1min":0.372,"5min":0.32800000000000001,"15min":0.30399999999999999},"max":{"1min":3.8980000000000001,"5min":12.227,"15min":12.227},"last":0.87}]},{"osd":7,"last update":"Fri Jun 14 23:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":1.4830000000000001,"15min":0.93999999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.782,"5min":10.614000000000001,"15min":10.614000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1140000000000001,"15min":0.78700000000000003},"min":{"1min":0.35999999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":3.968,"5min":7.6260000000000003,"15min":7.6260000000000003},"last":0.73199999999999998}]}]},{"osd":1,"up_from":116,"seq":498216206505,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1218100,"kb_used_data":757100,"kb_used_omap":5,"kb_used_meta":460986,"kb_avail":92527052,"statfs":{"total":95995035648,"available":94747701248,"internally_reserved":0,"allocated":775270400,"data_stored":774238651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":472049944},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.3999999999999999,"15min":1.6319999999999999},"min":{"1min":0.34699999999999998,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":6.3929999999999998,"5min":11.257999999999999,"15min":11.257999999999999},"last":0.193},{"interface":"front","average":{"1min":1.371,"5min":1.206,"15min":1.3160000000000001},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":4.4349999999999996,"5min":5.7539999999999996,"15min":10.426},"last":5.9249999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:02:52 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.6599999999999999,"15min":0.92600000000000005},"min":{"1min":0.28599999999999998,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":7.4930000000000003,"5min":9.3759999999999994,"15min":9.3759999999999994},"last":9.5470000000000006},{"interface":"front","average":{"1min":1.919,"5min":1.1459999999999999,"15min":0.71499999999999997},"min":{"1min":0.23599999999999999,"5min":0.23200000000000001,"15min":0.23000000000000001},"max":{"1min":8.0060000000000002,"5min":8.0060000000000002,"15min":8.0060000000000002},"last":6.0289999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:03:19 2024","interfaces":[{"interface":"back","average":{"1min":2.4359999999999999,"5min":1.349,"15min":1.0369999999999999},"min":{"1min":0.27500000000000002,"5min":0.22600000000000001,"15min":0.20499999999999999},"max":{"1min":6.2060000000000004,"5min":7.782,"15min":21.422000000000001},"last":2.6320000000000001},{"interface":"front","average":{"1min":2.2959999999999998,"5min":1.5429999999999999,"15min":1.0780000000000001},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":8.1219999999999999,"5min":9.4990000000000006,"15min":21.016999999999999},"last":2.4220000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8160000000000001,"5min":1.3149999999999999,"15min":1.488},"min":{"1min":0.309,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.5350000000000001,"5min":6.6360000000000001,"15min":8.8599999999999994},"last":2.5430000000000001},{"interface":"front","average":{"1min":1.794,"5min":1.603,"15min":1.621},"min":{"1min":0.40200000000000002,"5min":0.29399999999999998,"15min":0.29199999999999998},"max":{"1min":6.2960000000000003,"5min":9.4169999999999998,"15min":11.161},"last":0.73499999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.4970000000000001,"15min":1.538},"min":{"1min":0.41099999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.3179999999999996,"5min":10.103,"15min":10.103},"last":6.0049999999999999},{"interface":"front","average":{"1min":1.546,"5min":1.1910000000000001,"15min":1.482},"min":{"1min":0.34300000000000003,"5min":0.308,"15min":0.28599999999999998},"max":{"1min":5.8250000000000002,"5min":6.5730000000000004,"15min":8.125},"last":2.5819999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":2.044,"15min":1.7410000000000001},"min":{"1min":0.45800000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.4080000000000004,"5min":63.795999999999999,"15min":63.795999999999999},"last":5.8719999999999999},{"interface":"front","average":{"1min":2.0600000000000001,"5min":2.0710000000000002,"15min":1.7370000000000001},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.28499999999999998},"max":{"1min":5.8019999999999996,"5min":63.811,"15min":63.811},"last":2.6070000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:02:52 2024","interfaces":[{"interface":"back","average":{"1min":1.792,"5min":1.6870000000000001,"15min":1.6559999999999999},"min":{"1min":0.42399999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":6.3369999999999997,"5min":9.4429999999999996,"15min":9.4429999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.452,"5min":1.3440000000000001,"15min":1.429},"min":{"1min":0.41199999999999998,"5min":0.371,"15min":0.36499999999999999},"max":{"1min":4.4610000000000003,"5min":6.3529999999999998,"15min":9.5449999999999999},"last":5.9580000000000002}]}]},{"osd":0,"up_from":91,"seq":390842024160,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3135736,"kb_used_data":2443696,"kb_used_omap":5,"kb_used_meta":692026,"kb_avail":90609416,"statfs":{"total":95995035648,"available":92784041984,"internally_reserved":0,"allocated":2502344704,"data_stored":2501306882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":708634714},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.3129999999999999,"15min":1.488},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":9.4380000000000006,"5min":15.547000000000001,"15min":16.305},"last":0.254},{"interface":"front","average":{"1min":1.395,"5min":1.2629999999999999,"15min":1.6890000000000001},"min":{"1min":0.20899999999999999,"5min":0.20499999999999999,"15min":0.17899999999999999},"max":{"1min":4.9329999999999998,"5min":6.9889999999999999,"15min":16.184999999999999},"last":0.45200000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:02:57 2024","interfaces":[{"interface":"back","average":{"1min":2.0459999999999998,"5min":1.3819999999999999,"15min":0.84599999999999997},"min":{"1min":0.223,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.0020000000000007,"5min":11.326000000000001,"15min":16.757000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.677,"5min":1.444,"15min":0.89900000000000002},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":7.9800000000000004,"5min":11.519,"15min":16.055},"last":0.76400000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":1.3260000000000001,"15min":0.83599999999999997},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":4.2370000000000001,"5min":7.04,"15min":16.873999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":2.0720000000000001,"5min":1.46,"15min":0.86099999999999999},"min":{"1min":0.27300000000000002,"5min":0.224,"15min":0.224},"max":{"1min":9.4760000000000009,"5min":12.308999999999999,"15min":16.085999999999999},"last":0.72299999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.546,"15min":1.456},"min":{"1min":0.42499999999999999,"5min":0.307,"15min":0.27600000000000002},"max":{"1min":12.436,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.0179999999999998,"5min":1.4890000000000001,"15min":1.3380000000000001},"min":{"1min":0.35499999999999998,"5min":0.33200000000000002,"15min":0.28599999999999998},"max":{"1min":4.8250000000000002,"5min":14.475,"15min":15.976000000000001},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.5489999999999999,"15min":1.452},"min":{"1min":0.32800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.9009999999999998,"5min":6.9640000000000004,"15min":16.108000000000001},"last":1.494},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.5529999999999999,"15min":1.5269999999999999},"min":{"1min":0.32500000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":9.2279999999999998,"5min":11.436,"15min":16.006},"last":2.2069999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7969999999999999,"5min":1.5629999999999999,"15min":1.4219999999999999},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.8730000000000002,"5min":14.247,"15min":16.152000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":2.3239999999999998,"5min":1.6619999999999999,"15min":1.5129999999999999},"min":{"1min":0.41299999999999998,"5min":0.312,"15min":0.29399999999999998},"max":{"1min":9.3130000000000006,"5min":14.523999999999999,"15min":16.227},"last":1.6759999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:02:28 2024","interfaces":[{"interface":"back","average":{"1min":2.1440000000000001,"5min":1.4950000000000001,"15min":1.46},"min":{"1min":0.434,"5min":0.35899999999999999,"15min":0.33100000000000002},"max":{"1min":9.2579999999999991,"5min":11.364000000000001,"15min":16.199999999999999},"last":1.478},{"interface":"front","average":{"1min":1.867,"5min":1.546,"15min":1.524},"min":{"1min":0.39900000000000002,"5min":0.36899999999999999,"15min":0.314},"max":{"1min":9.2940000000000005,"5min":12.361000000000001,"15min":16.259},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2462257152,"data_stored":2462253080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":763625472,"data_stored":763625472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1524760576,"data_stored":1524760576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1281228800,"data_stored":1281228800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2059603968,"data_stored":2059599896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1285029888,"data_stored":1285029888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1818300416,"data_stored":1818296344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1022164992,"data_stored":1022164992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:03:25.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-06-14T23:03:25.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:25.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:24 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:03:25.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:24 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:03:25.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:24 smithi137 ceph-mon[90634]: from='client.36078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:03:25.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:24 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2343541038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:03:25.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:24 smithi137 ceph-mon[90634]: osdmap e212: 8 total, 8 up, 8 in 2024-06-14T23:03:25.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:25.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:25.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:25.384 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:25.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:25.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:25.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:25.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:25.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:25.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:25.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:25.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:25.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:25.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:25.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:25.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:25.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:26.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:26.084 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:26.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:26.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:25 smithi086 ceph-mon[97913]: pgmap v849: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 943 op/s 2024-06-14T23:03:26.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:25 smithi086 ceph-mon[102754]: pgmap v849: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 943 op/s 2024-06-14T23:03:26.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:26.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:26.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:26.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:25 smithi137 ceph-mon[90634]: pgmap v849: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 943 op/s 2024-06-14T23:03:26.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:26.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:26.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:26.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:26.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:26.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:26.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:26.624 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:26.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:26.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:26.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:26.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:26.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:26.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:27.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:27.041 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:27.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:27.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:27.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:27.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:27.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:27.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:27.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:27.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:27.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:27.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:27.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:27.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:27.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:27.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:27.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:27.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:27.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:27.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:27.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:27.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:27.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:27.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:27.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:27.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:28.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:27 smithi086 ceph-mon[102754]: pgmap v850: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 945 op/s 2024-06-14T23:03:28.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:03:28.026 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:28.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:27 smithi086 ceph-mon[97913]: pgmap v850: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 945 op/s 2024-06-14T23:03:28.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:03:28.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:28.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:28.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:28.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:28.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:03:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:27 smithi137 ceph-mon[90634]: pgmap v850: 99 pgs: 99 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 945 op/s 2024-06-14T23:03:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:03:28.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:28.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:28.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:28.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:28.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:28.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:28.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:28.468 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:28.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:28.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:28.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:28.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:28.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:28.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:28.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:28.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:28.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:28.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:28.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:29.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:29.104 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:03:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:03:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:03:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:03:29.104 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:03:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:29.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:29.231 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:29.261 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:29.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:29.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:29.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:03:29.448 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:03:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:03:29.451 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:03:29.451 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:03:29.487 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:03:29.487 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:03:29.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:29.544 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:29.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:29.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:29.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:29.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:29.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:29.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:29.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:29.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:29.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:29.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:30.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:30.065 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:29 smithi086 ceph-mon[102754]: pgmap v851: 99 pgs: 99 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 940 op/s 2024-06-14T23:03:30.065 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:29 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1768456460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:30.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:29 smithi086 ceph-mon[97913]: pgmap v851: 99 pgs: 99 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 940 op/s 2024-06-14T23:03:30.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:29 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1768456460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:30.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:30.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:30.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:30.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:30.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:30.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:30.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:29 smithi137 ceph-mon[90634]: pgmap v851: 99 pgs: 99 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 940 op/s 2024-06-14T23:03:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:29 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1768456460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:30.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:30.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:30.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:30.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:30.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:30.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:30.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:30.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:30.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:30.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:30.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:30.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:30.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:30.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:30.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:31.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:31.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:31.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:31.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:31.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:31.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:31.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:31.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:31.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:31.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:31.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:31.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:31.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:31.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:31.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:31.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:31.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:31.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:31.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:31.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:31.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-14T23:03:31.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:32.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:32.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:32.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:31 smithi086 ceph-mon[102754]: pgmap v852: 99 pgs: 99 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:03:32.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:31 smithi086 ceph-mon[97913]: pgmap v852: 99 pgs: 99 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:03:32.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:32.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:32.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:31 smithi137 ceph-mon[90634]: pgmap v852: 99 pgs: 99 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:03:32.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:32.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:32.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:32.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:32.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:32.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:32.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:32.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:32.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:32.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:32.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:32.730 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:32.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:32.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:32.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:32.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:32.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:33.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:33.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:33.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:33.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:33.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:33.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:33.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:33.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:33.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:33.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:33.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:33.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:33.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:33.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:33.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:33.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:33.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:33.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:33.913 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:33.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:33.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:33.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:33.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:34.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:34.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:34.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:34.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:33 smithi086 ceph-mon[97913]: pgmap v853: 99 pgs: 99 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 926 op/s 2024-06-14T23:03:34.113 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:33 smithi086 ceph-mon[102754]: pgmap v853: 99 pgs: 99 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 926 op/s 2024-06-14T23:03:34.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:34.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:34.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:34.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:33 smithi137 ceph-mon[90634]: pgmap v853: 99 pgs: 99 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 926 op/s 2024-06-14T23:03:34.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:34.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:34.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:34.501 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:34.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:34.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:34.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:34.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:34.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:34.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:34.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:34.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:34.845 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 1 pgp_num to 11 2024-06-14T23:03:34.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:34.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:34.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:35.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:35.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2780512929' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:03:35.040 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:34 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:03:35.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2780512929' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:03:35.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:34 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:03:35.040 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:03:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:03:34] "GET /metrics HTTP/1.1" 200 36593 "" "Prometheus/2.43.0" 2024-06-14T23:03:35.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:03:35.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:35.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:35.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:35.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:35.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:35.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2780512929' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:03:35.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:34 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-06-14T23:03:35.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:35.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:35.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:35.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:35.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:35.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:35.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:35.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:35.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:35.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:35.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:35.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:35.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:35.869 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:35.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:35.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:36.006 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:36.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:36.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:35 smithi086 ceph-mon[97913]: pgmap v854: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T23:03:36.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:35 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-06-14T23:03:36.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:35 smithi086 ceph-mon[97913]: osdmap e213: 8 total, 8 up, 8 in 2024-06-14T23:03:36.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:35 smithi086 ceph-mon[102754]: pgmap v854: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T23:03:36.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:35 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-06-14T23:03:36.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:35 smithi086 ceph-mon[102754]: osdmap e213: 8 total, 8 up, 8 in 2024-06-14T23:03:36.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:36.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:35 smithi137 ceph-mon[90634]: pgmap v854: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T23:03:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:35 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-06-14T23:03:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:35 smithi137 ceph-mon[90634]: osdmap e213: 8 total, 8 up, 8 in 2024-06-14T23:03:36.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:36.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:36.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:36.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:36.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:36.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:36.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:36.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:36.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:36.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:36.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:36.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:36.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:36.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:36.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:36.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:36.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:37.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:37.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:37.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:37.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:37.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:37.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:37.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:37.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:37.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:37.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:37.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:37.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:37.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:37.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:37.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:37.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:37.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:37.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:37.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:37.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:37.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:37.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:38.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:38.080 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:38.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:38.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:38.216 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:38.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:38.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:37 smithi137 ceph-mon[90634]: pgmap v856: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-06-14T23:03:38.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:38.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:37 smithi086 ceph-mon[97913]: pgmap v856: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-06-14T23:03:38.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:37 smithi086 ceph-mon[102754]: pgmap v856: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-06-14T23:03:38.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:38.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:38.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:38.491 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:38.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:38.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:38.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:38.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:38.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:38.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:38.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:38.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:38.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:38.871 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:03:38.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:38.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:38.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:39.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:39.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:39.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:39.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:39.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:39.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:39.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:38 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1106960113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:03:39.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:38 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:03:39.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:39.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:39.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:39.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:38 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1106960113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:03:39.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:38 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:03:39.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:38 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1106960113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:03:39.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:38 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:03:39.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:39.427 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:39.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:39.527 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:03:39.528 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:03:39.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:39.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:39.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:39.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:39.703 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:39.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:39.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:39.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:39.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:39.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:39.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:39.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:40.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:40.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:39 smithi086 ceph-mon[97913]: pgmap v857: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:03:40.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:39 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:40.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:39 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:03:40.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:39 smithi086 ceph-mon[97913]: osdmap e214: 8 total, 8 up, 8 in 2024-06-14T23:03:40.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:39 smithi086 ceph-mon[102754]: pgmap v857: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:03:40.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:39 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:40.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:39 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:03:40.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:39 smithi086 ceph-mon[102754]: osdmap e214: 8 total, 8 up, 8 in 2024-06-14T23:03:40.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:40.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:40.146 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:39 smithi137 ceph-mon[90634]: pgmap v857: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 935 op/s 2024-06-14T23:03:40.146 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:39 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:40.146 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:39 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:03:40.146 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:39 smithi137 ceph-mon[90634]: osdmap e214: 8 total, 8 up, 8 in 2024-06-14T23:03:40.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:40.238 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:40.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:40.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:40.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:40.371 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:40.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:40.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:40.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:40.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:40.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:40.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:40.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:40.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:40.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:40.779 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:40.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:40.824 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:03:40.824 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:03:40.824 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-06-14T23:03:40.824 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:03:40.824 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:03:40.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-06-14T23:03:40.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:40.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:40.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:41.000 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:41.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:41.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-14T23:03:41.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:41.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:41.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:41.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:41.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:41.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:41.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:41.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:41.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:41.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:41.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:41.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:41.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:41.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:41.645 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:03:41.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:41.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:41.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:41.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:41.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:41.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:41.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:42.024 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:42.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:42.046 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:41 smithi137 ceph-mon[90634]: pgmap v859: 99 pgs: 99 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-06-14T23:03:42.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:42.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:03:42.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:42.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:42.164 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:41 smithi086 ceph-mon[97913]: pgmap v859: 99 pgs: 99 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-06-14T23:03:42.164 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:41 smithi086 ceph-mon[102754]: pgmap v859: 99 pgs: 99 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-06-14T23:03:42.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:42.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:42.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:42.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:42.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:42.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:42.450 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:42.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:42.563 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:42.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:42.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:42.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:42.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:42.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:42.845 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:42.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:42.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:42.975 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:42.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:43.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:43.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:43.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:43.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:43.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:43.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:43.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:43.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:43.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:43.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:43.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:43.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:43.583 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:43.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:43.642 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:03:43.642 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:43.642 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:43 smithi137 ceph-mon[90634]: pgmap v860: 99 pgs: 99 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 787 op/s 2024-06-14T23:03:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:43.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:43.707 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:03:43.708 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:43.708 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:43 smithi086 ceph-mon[97913]: pgmap v860: 99 pgs: 99 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 787 op/s 2024-06-14T23:03:43.708 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:03:43.708 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:43.708 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:43 smithi086 ceph-mon[102754]: pgmap v860: 99 pgs: 99 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 787 op/s 2024-06-14T23:03:43.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:43.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:43.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:43.884 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:03:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:03:43.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:43.919 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:03:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:03:43.946 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:43.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:43.949 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:03:43.949 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:03:43.985 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:03:43.985 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:03:44.019 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:03:44.020 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:03:44.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:44.051 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:03:44.051 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:03:44.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:44.093 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:03:44.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:44.094 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:03:44.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:44.107 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:222347 10.6 e214/186 10:697c78f8:::benchmark_data_smithi086_189745_object91379:head v 214'13819, mlcod=214'13816)", 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:44.101046+0000", 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0052861130000000003, 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053133390000000003, 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101046+0000", 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:44.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101046+0000", 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101045+0000", 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101053+0000", 2024-06-14T23:03:44.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0120000000000005e-06 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101073+0000", 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9862000000000001e-05 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101078+0000", 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5570000000000003e-06 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101863+0000", 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00078495099999999999 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101905+0000", 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1566000000000001e-05 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:222349 10.e 10:7d5fd435:::benchmark_data_smithi086_189745_object91381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:44.101723+0000", 2024-06-14T23:03:44.111 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0046089740000000001, 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046995589999999999, 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 222349 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101723+0000", 2024-06-14T23:03:44.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101723+0000", 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101722+0000", 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101727+0000", 2024-06-14T23:03:44.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3820000000000003e-06 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101740+0000", 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2809999999999999e-05 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.101745+0000", 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6550000000000001e-06 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.102924+0000", 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011781490000000001 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.103248+0000", 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032402400000000001 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.103659+0000", 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00041119000000000003 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:44.115 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:222337 10.6 e214/186 10:6c9ccfd9:::benchmark_data_smithi086_189745_object91369:head v 214'13818, mlcod=214'13816)", 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:44.085819+0000", 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.020512671999999999, 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020672392000000001, 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.085819+0000", 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:44.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.085819+0000", 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.085819+0000", 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.085824+0000", 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8150000000000002e-06 2024-06-14T23:03:44.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.086023+0000", 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019901000000000001 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.086029+0000", 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5459999999999999e-06 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:44.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.086168+0000", 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013924200000000001 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:44.086238+0000", 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9746000000000006e-05 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 3 2024-06-14T23:03:44.119 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:03:44.136 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:03:44.136 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:03:44.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:44.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:44.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:44.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:44.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:44.370 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:44.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:44.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:44.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:44.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:44.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:44.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:44.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:44.661 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/775738281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:44.661 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/775738281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:44.744 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/775738281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:44.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:44.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:44.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:44.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:44.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:44.926 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:44.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:44.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-14T23:03:45.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:45.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:45.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:45.067 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:03:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:03:44] "GET /metrics HTTP/1.1" 200 36592 "" "Prometheus/2.43.0" 2024-06-14T23:03:45.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:45.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:45.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:45.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:45.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:45.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:45.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:45.453 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:03:45.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:45.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:45.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:45.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:45.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:45.660 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:45 smithi086 ceph-mon[102754]: pgmap v861: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 960 op/s 2024-06-14T23:03:45.660 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:45 smithi086 ceph-mon[97913]: pgmap v861: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 960 op/s 2024-06-14T23:03:45.679 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:45 smithi137 ceph-mon[90634]: pgmap v861: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 960 op/s 2024-06-14T23:03:45.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:45.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:45.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:45.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:45.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:45.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:46.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:46.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:46.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:46.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:46.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:46.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:46.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:46.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:46.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:46.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:46.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:46.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:46.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:46.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:46.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:46.754 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:46.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:46.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:46.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:46.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:47.005 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:03:47.005 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-14T23:03:47.005 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:03:47.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:47.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:47.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:47.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:47.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:47.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:47.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:47.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:47.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:47.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:47.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:47.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:47.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:47.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:47.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:47.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:47.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:47.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:47.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:47.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:47.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:47.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:47.948 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:47 smithi137 ceph-mon[90634]: pgmap v862: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 948 op/s 2024-06-14T23:03:47.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:48.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:48.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:48.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:47 smithi086 ceph-mon[102754]: pgmap v862: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 948 op/s 2024-06-14T23:03:48.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:47 smithi086 ceph-mon[97913]: pgmap v862: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 948 op/s 2024-06-14T23:03:48.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:48.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:48.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:48.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:48.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:48.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:48.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:48.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:48.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:48.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:48.561 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:48.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:48.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:48.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:48.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:48.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:48.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:48.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:48.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:48.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:48.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:49.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:03:49.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:49.071 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:03:49.071 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:03:49.071 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:03:49.071 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:03:49.071 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:03:49.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:49.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:49.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:49.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:49.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:49.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:49.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:49.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:49.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:49.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:49.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:49.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:49.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:49.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:49.682 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:49.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:49.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-14T23:03:49.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:49.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:49.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:49.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:49.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:49.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:49.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:49 smithi137 ceph-mon[90634]: pgmap v863: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-06-14T23:03:50.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:49 smithi086 ceph-mon[102754]: pgmap v863: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-06-14T23:03:50.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:49 smithi086 ceph-mon[97913]: pgmap v863: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-06-14T23:03:50.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:50.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:50.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:50.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:50.194 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:03:50.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:50.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:50.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:50.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:50.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:50.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:50.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:50.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:50.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:50.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:50.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:50.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:50.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:50.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:50.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:50.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:50.839 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:03:50.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:50.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:50.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:50 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2961566156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:03:50.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:50 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:03:51.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:51.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:51.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:51.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:50 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2961566156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:03:51.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:50 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:03:51.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:50 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2961566156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:03:51.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:50 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:03:51.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:51.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:51.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:51.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:51.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:51.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:51.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:51.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:51.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:51.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:51.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:51.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:51.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:51.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:51.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:51.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:51.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:51.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:51.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:51 smithi137 ceph-mon[90634]: pgmap v864: 99 pgs: 99 active+clean; 2.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T23:03:51.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:51 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:51.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:03:51.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:51 smithi137 ceph-mon[90634]: osdmap e215: 8 total, 8 up, 8 in 2024-06-14T23:03:51.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:03:52.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:52.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:52.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:52.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[97913]: pgmap v864: 99 pgs: 99 active+clean; 2.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T23:03:52.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:52.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:03:52.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[97913]: osdmap e215: 8 total, 8 up, 8 in 2024-06-14T23:03:52.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:03:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[102754]: pgmap v864: 99 pgs: 99 active+clean; 2.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-06-14T23:03:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:03:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:03:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[102754]: osdmap e215: 8 total, 8 up, 8 in 2024-06-14T23:03:52.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:03:52.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:52.154 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:24732 9.e 9:7eaedcc6:::benchmark_data_smithi086_152062_object24731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:55:52.672402+0000", 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 479.477623494, 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.000701877, 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:03:52.155 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 24732 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672402+0000", 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672402+0000", 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672400+0000", 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672569+0000", 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000168104 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672584+0000", 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5814000000000001e-05 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672592+0000", 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4630000000000004e-06 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672659+0000", 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7403000000000005e-05 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.672668+0000", 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5459999999999992e-06 2024-06-14T23:03:52.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.665640+0000", 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99297187200000003 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.666460+0000", 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082062499999999996 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.666642+0000", 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018165999999999999 2024-06-14T23:03:52.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672317+0000", 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056752010000000004 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672900+0000", 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058328000000000004 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672946+0000", 2024-06-14T23:03:52.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5955999999999998e-05 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.673063+0000", 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116419 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.673104+0000", 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1393999999999997e-05 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.161 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:24769 9.e 9:7ad8e74e:::benchmark_data_smithi086_152062_object24768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:55:52.688040+0000", 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 479.461985266, 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98628556000000001, 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 24769 2024-06-14T23:03:52.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688040+0000", 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688040+0000", 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688032+0000", 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999914 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688071+0000", 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9511999999999997e-05 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688090+0000", 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8975e-05 2024-06-14T23:03:52.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688099+0000", 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7900000000000005e-06 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688164+0000", 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4812000000000002e-05 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.688174+0000", 2024-06-14T23:03:52.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7259999999999997e-06 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.668016+0000", 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97984286799999998 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.668178+0000", 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000161087 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:03:52.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.668379+0000", 2024-06-14T23:03:52.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000201475 2024-06-14T23:03:52.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672327+0000", 2024-06-14T23:03:52.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039483599999999997 2024-06-14T23:03:52.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.673146+0000", 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00081893899999999997 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674250+0000", 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011034510000000001 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674285+0000", 2024-06-14T23:03:52.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5465e-05 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674326+0000", 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0779999999999999e-05 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:24787 9.e 9:7afcc34a:::benchmark_data_smithi086_152062_object24786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:55:52.701009+0000", 2024-06-14T23:03:52.169 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 479.44901722700001, 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97348232199999996, 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 24787 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701009+0000", 2024-06-14T23:03:52.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701009+0000", 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701007+0000", 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701112+0000", 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104862 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701129+0000", 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6694e-05 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701134+0000", 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7769999999999999e-06 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701188+0000", 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3649000000000001e-05 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.701196+0000", 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7729999999999999e-06 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.669515+0000", 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.968319708 2024-06-14T23:03:52.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.669670+0000", 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015500299999999999 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.669830+0000", 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015971 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.672380+0000", 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025496999999999998 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674225+0000", 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018455019999999999 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674392+0000", 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000166751 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674447+0000", 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.49e-05 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674491+0000", 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3803000000000001e-05 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:24789 9.e 9:760cab86:::benchmark_data_smithi086_152062_object24788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:55:52.702965+0000", 2024-06-14T23:03:52.176 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 479.44706030499998, 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97231838199999998, 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 24789 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.702965+0000", 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.702965+0000", 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.702965+0000", 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703013+0000", 2024-06-14T23:03:52.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7966000000000001e-05 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703024+0000", 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1094000000000001e-05 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703028+0000", 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5750000000000002e-06 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703068+0000", 2024-06-14T23:03:52.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9561000000000003e-05 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:52.703075+0000", 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8410000000000002e-06 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.670935+0000", 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96786069299999999 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.671087+0000", 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000151366 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.671240+0000", 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015322999999999999 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.673153+0000", 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001913465 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.674539+0000", 2024-06-14T23:03:52.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013852230000000001 2024-06-14T23:03:52.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.675086+0000", 2024-06-14T23:03:52.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00054754700000000001 2024-06-14T23:03:52.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.675170+0000", 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4192999999999998e-05 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:55:53.675284+0000", 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113508 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.183 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84914 9.e 9:7e1d350b:::benchmark_data_smithi086_152062_object84913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:30.093445+0000", 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 442.05658107900001, 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.105775729, 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84914 2024-06-14T23:03:52.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093445+0000", 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093445+0000", 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093444+0000", 2024-06-14T23:03:52.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093508+0000", 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3757000000000006e-05 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093519+0000", 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0881e-05 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.093524+0000", 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1619999999999999e-06 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.096870+0000", 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033461860000000001 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.097013+0000", 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014290700000000001 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.097189+0000", 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000175835 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.100394+0000", 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032054779999999999 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.103706+0000", 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033121270000000002 2024-06-14T23:03:52.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.199150+0000", 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095443745999999996 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.199190+0000", 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0014e-05 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.199220+0000", 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0354000000000001e-05 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84933 9.e 9:7994e0bc:::benchmark_data_smithi086_152062_object84932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:30.112138+0000", 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 442.03788735299997, 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.092303347999999993, 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.190 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84933 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112138+0000", 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112138+0000", 2024-06-14T23:03:52.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112137+0000", 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112208+0000", 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1501999999999994e-05 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112224+0000", 2024-06-14T23:03:52.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6008999999999999e-05 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112232+0000", 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6790000000000005e-06 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112301+0000", 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9287000000000003e-05 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112439+0000", 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013751200000000001 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.112628+0000", 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018896599999999999 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.115361+0000", 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002733564 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.115839+0000", 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047746600000000001 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.204341+0000", 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.088502239999999996 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.204397+0000", 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5859000000000002e-05 2024-06-14T23:03:52.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:30.204442+0000", 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5028000000000003e-05 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:117226 10.6 e202/186 10:65a499ab:::benchmark_data_smithi086_189745_object117225:head v 202'7278, mlcod=202'7276)", 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.020013+0000", 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.130013189, 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.102547216, 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.196 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.020013+0000", 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.020013+0000", 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.020012+0000", 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.020038+0000", 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5799000000000002e-05 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.020049+0000", 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.117e-05 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.020054+0000", 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4670000000000003e-06 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.020149+0000", 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4381000000000003e-05 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.020166+0000", 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7598999999999999e-05 2024-06-14T23:03:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122525+0000", 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10235865399999999 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122560+0000", 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4926999999999999e-05 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:117232 10.e 10:723bdcab:::benchmark_data_smithi086_189745_object117231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.022215+0000", 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.12781028800001, 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.103852031, 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117232 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.022215+0000", 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.022215+0000", 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.022214+0000", 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.022278+0000", 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4120000000000003e-05 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.022291+0000", 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3344e-05 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.022297+0000", 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9719999999999997e-06 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.022338+0000", 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1415999999999998e-05 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.022448+0000", 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010942600000000001 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.022579+0000", 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131302 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031177+0000", 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0085978760000000008 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122515+0000", 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.091337574000000005 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.125978+0000", 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034632159999999999 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.126027+0000", 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9438999999999999e-05 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.126068+0000", 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0185999999999998e-05 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:117233 10.e 10:733fe19a:::benchmark_data_smithi086_189745_object117232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:03:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.024480+0000", 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.125545543, 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.108642816, 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117233 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024480+0000", 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024480+0000", 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024478+0000", 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024504+0000", 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5913000000000001e-05 2024-06-14T23:03:52.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024517+0000", 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3409e-05 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024523+0000", 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.947e-06 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.210 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:52.211 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:52.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026253+0000", 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017299800000000001 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026395+0000", 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141648 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026528+0000", 2024-06-14T23:03:52.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013276999999999999 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.125950+0000", 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.099422650000000001 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.126111+0000", 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160555 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.133029+0000", 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0069178759999999999 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.133083+0000", 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4388999999999997e-05 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.133123+0000", 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9734999999999998e-05 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.214 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.45821.0:117232 10.e e202/186 ondisk, result = 0)", 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.026761+0000", 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.123264517, 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095802332000000004, 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026761+0000", 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026761+0000", 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026760+0000", 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026771+0000", 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0543000000000001e-05 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026780+0000", 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2010000000000005e-06 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026785+0000", 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0080000000000002e-06 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031206+0000", 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044210930000000001 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122500+0000", 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.091293689999999997 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122564+0000", 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3646999999999996e-05 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.218 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:117237 10.2 e202/186 10:4e671e84:::benchmark_data_smithi086_189745_object117236:head v 202'7437, mlcod=202'7435)", 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.027680+0000", 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.122345606, 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.094970902999999995, 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.027680+0000", 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.027680+0000", 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.027679+0000", 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.027708+0000", 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.885e-05 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.027720+0000", 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1603000000000001e-05 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.027725+0000", 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.135e-06 2024-06-14T23:03:52.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.027765+0000", 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0244000000000003e-05 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.027784+0000", 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9072999999999998e-05 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122589+0000", 2024-06-14T23:03:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.094804248999999993 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122651+0000", 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2528999999999997e-05 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:117239 10.e 10:73df0b96:::benchmark_data_smithi086_189745_object117238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.029598+0000", 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.120427589, 2024-06-14T23:03:52.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.104880501, 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117239 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029598+0000", 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029598+0000", 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029596+0000", 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029646+0000", 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0021000000000001e-05 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029668+0000", 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2027000000000001e-05 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029675+0000", 2024-06-14T23:03:52.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.426e-06 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122576+0000", 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.092900782000000001 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122706+0000", 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013087900000000001 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122869+0000", 2024-06-14T23:03:52.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016291799999999999 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.129414+0000", 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065442110000000003 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.133165+0000", 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037517330000000002 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.134397+0000", 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012313109999999999 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.134444+0000", 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6993999999999998e-05 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.134479+0000", 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5061000000000002e-05 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:117240 10.0 e202/186 10:034e9ce1:::benchmark_data_smithi086_189745_object117239:head v 202'7311, mlcod=202'7309)", 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.031664+0000", 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.118362054, 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10380703500000001, 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031664+0000", 2024-06-14T23:03:52.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031664+0000", 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031662+0000", 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031687+0000", 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4324e-05 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122508+0000", 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.090821608999999998 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122516+0000", 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7789999999999996e-06 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122563+0000", 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7265999999999998e-05 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122586+0000", 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2565000000000002e-05 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.135404+0000", 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012818120000000001 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.135471+0000", 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6860999999999996e-05 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.45821.0:117233 10.e e202/186 ondisk, result = 0)", 2024-06-14T23:03:52.234 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.031852+0000", 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.118174205, 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.094100699999999995, 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031852+0000", 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031852+0000", 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031851+0000", 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031857+0000", 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7350000000000001e-06 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122480+0000", 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.090623932000000004 2024-06-14T23:03:52.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122496+0000", 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5062000000000001e-05 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.125938+0000", 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034422839999999999 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.125944+0000", 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2570000000000004e-06 2024-06-14T23:03:52.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.125952+0000", 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1610000000000003e-06 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:117241 10.2 e202/186 10:40bacbc2:::benchmark_data_smithi086_189745_object117240:head v 202'7438, mlcod=202'7435)", 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.031858+0000", 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.118167755, 2024-06-14T23:03:52.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095395129999999995, 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031858+0000", 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031858+0000", 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031857+0000", 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031958+0000", 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010021 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122499+0000", 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.090541663999999994 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122508+0000", 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4169999999999992e-06 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122672+0000", 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016484099999999999 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122691+0000", 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8595999999999999e-05 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.127217+0000", 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045259330000000002 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.127253+0000", 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6118000000000002e-05 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:175484 10.6 e208/186 10:6ce43a59:::benchmark_data_smithi086_189745_object44516:head v 208'10930, mlcod=208'10928)", 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:02:54.216970+0000", 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 57.933056139999998, 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.077682176000000006, 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.216970+0000", 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.216970+0000", 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.216969+0000", 2024-06-14T23:03:52.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.216975+0000", 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4930000000000003e-06 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.273180+0000", 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.056204847000000002 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.273214+0000", 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3785000000000003e-05 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.273419+0000", 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020547600000000001 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.273434+0000", 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4756000000000001e-05 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.294545+0000", 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021110356 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:02:54.294652+0000", 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107247 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:180736 10.e 10:722d59ea:::benchmark_data_smithi086_189745_object49768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:00.023008+0000", 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.127017991999999, 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095050751000000003, 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 180736 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023008+0000", 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023008+0000", 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023006+0000", 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:03:52.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023014+0000", 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7440000000000007e-06 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023027+0000", 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.323e-05 2024-06-14T23:03:52.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023034+0000", 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4840000000000001e-06 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023244+0000", 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021080599999999999 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023356+0000", 2024-06-14T23:03:52.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111958 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023468+0000", 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011124300000000001 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.035301+0000", 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011833576 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.035485+0000", 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018402499999999999 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.117976+0000", 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.082490771000000004 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.118020+0000", 2024-06-14T23:03:52.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4435999999999999e-05 2024-06-14T23:03:52.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.118059+0000", 2024-06-14T23:03:52.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8034999999999998e-05 2024-06-14T23:03:52.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.255 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.255 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:180737 10.e 10:72b8b4ce:::benchmark_data_smithi086_189745_object49769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-14T23:03:52.255 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:00.023367+0000", 2024-06-14T23:03:52.256 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.126659050999997, 2024-06-14T23:03:52.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.094826647999999999, 2024-06-14T23:03:52.256 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.256 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.257 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:03:52.257 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:03:52.257 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 180737 2024-06-14T23:03:52.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.258 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023367+0000", 2024-06-14T23:03:52.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023367+0000", 2024-06-14T23:03:52.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023366+0000", 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023371+0000", 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.583e-06 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023380+0000", 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9539999999999999e-06 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.023386+0000", 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2599999999999996e-06 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.026841+0000", 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034547969999999999 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.026963+0000", 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122815 2024-06-14T23:03:52.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.027082+0000", 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011879699999999999 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.035308+0000", 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082255009999999996 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.040961+0000", 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056531610000000003 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.118111+0000", 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.077150344999999995 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.118157+0000", 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5942999999999997e-05 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.118193+0000", 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6235999999999997e-05 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:180739 10.e 10:7b749aef:::benchmark_data_smithi086_189745_object49771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:00.026586+0000", 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.123439802999997, 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.091718346000000006, 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.264 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 180739 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.026586+0000", 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.026586+0000", 2024-06-14T23:03:52.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.026584+0000", 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.026591+0000", 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4670000000000001e-06 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.026607+0000", 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6175999999999999e-05 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.026613+0000", 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0569999999999998e-06 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.031069+0000", 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044563529999999997 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.031180+0000", 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110414 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:03:52.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.031307+0000", 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127379 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.035460+0000", 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041527430000000004 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.041005+0000", 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055451600000000004 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.118229+0000", 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.077223490000000006 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.118272+0000", 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3223999999999998e-05 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.118304+0000", 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2434000000000002e-05 2024-06-14T23:03:52.270 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:180745 10.e 10:7e910d14:::benchmark_data_smithi086_189745_object49777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:00.032139+0000", 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.117886360999996, 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.093837069999999995, 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:03:52.271 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 180745 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.032139+0000", 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.032139+0000", 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.032138+0000", 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.032144+0000", 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0499999999999997e-06 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.032161+0000", 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7229999999999999e-05 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.032253+0000", 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1636000000000002e-05 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.035506+0000", 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00325268 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:03:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.035603+0000", 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7226999999999996e-05 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.035701+0000", 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8513999999999999e-05 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.039263+0000", 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035617840000000001 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.041060+0000", 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017973500000000001 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.125883+0000", 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.084822930000000005 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.125941+0000", 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7157999999999997e-05 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:03:52.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:03:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:03:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.125976+0000", 2024-06-14T23:03:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5939000000000003e-05 2024-06-14T23:03:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:03:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:03:52.277 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:03:52.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:52.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:52.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:52.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:52.448 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:03:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:03:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:52.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:52.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:52.487 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:03:52.487 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:03:52.522 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:03:52.522 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:03:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:03:52.566 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:03:52.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:52.597 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:03:52.597 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:03:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:52.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:52.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:52.636 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:03:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:03:52.671 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:03:52.671 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:03:52.708 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:03:52.708 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:03:52.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:52.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:52.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:52.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:52.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:52.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:52.905 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2166414773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:52.906 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2166414773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:53.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:53.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:53.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:53.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:53.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:53.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2166414773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:03:53.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:53.327 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:53.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:53.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:53.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:53.466 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:53.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:53.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:53.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:53.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:53.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:53.773 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:53.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:53.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:03:53.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:53.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:53.944 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:53.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:53.948 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:03:53.948 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:03:53.948 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-06-14T23:03:53.948 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:03:53.948 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:03:53.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-06-14T23:03:53.983 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:53 smithi137 ceph-mon[90634]: pgmap v866: 99 pgs: 99 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 930 op/s 2024-06-14T23:03:54.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T23:03:54.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:54.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:54.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:54.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:54.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:54.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:54.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:54.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:53 smithi086 ceph-mon[97913]: pgmap v866: 99 pgs: 99 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 930 op/s 2024-06-14T23:03:54.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:53 smithi086 ceph-mon[102754]: pgmap v866: 99 pgs: 99 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 930 op/s 2024-06-14T23:03:54.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:54.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:54.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:54.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:54.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:54.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:54.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:54.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:54.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:54.720 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:54.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:54.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:54.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:54.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:54.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:54.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:55.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:55.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:55.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:55.144 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:03:55.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:55.165 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:03:55.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:55.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:55.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:55.335 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:03:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:03:54] "GET /metrics HTTP/1.1" 200 36592 "" "Prometheus/2.43.0" 2024-06-14T23:03:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:55.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:55.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:55.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:03:55.481 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:55.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:03:55.583 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:55.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:55.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:55.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:55.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:55.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:55.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:55.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:55.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:55.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:55.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:03:55.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:56.011 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:55 smithi137 ceph-mon[90634]: pgmap v867: 99 pgs: 99 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 920 op/s 2024-06-14T23:03:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:56.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:56.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:56.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:56.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:56.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:56.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:56.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:56.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:56.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:55 smithi086 ceph-mon[97913]: pgmap v867: 99 pgs: 99 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 920 op/s 2024-06-14T23:03:56.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:55 smithi086 ceph-mon[102754]: pgmap v867: 99 pgs: 99 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 920 op/s 2024-06-14T23:03:56.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:56.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:56.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:56.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:56.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:56.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:56.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:56.755 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:56.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:56.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:56.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:56.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:57.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:57.053 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:57.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:57.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:57.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:57.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:57.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:57.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:57.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:57.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:57.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:57.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:57.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:57.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:57.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:57.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:57.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:57.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:57.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:57.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:57.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:57.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:57.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:57.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:57.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:58.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:58.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:58.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:58.044 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:57 smithi086 ceph-mon[97913]: pgmap v868: 99 pgs: 99 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 920 op/s 2024-06-14T23:03:58.044 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:58.044 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:57 smithi086 ceph-mon[102754]: pgmap v868: 99 pgs: 99 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 920 op/s 2024-06-14T23:03:58.044 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:58.066 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:03:58.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:58.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:58.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:58.198 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:57 smithi137 ceph-mon[90634]: pgmap v868: 99 pgs: 99 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 920 op/s 2024-06-14T23:03:58.198 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:03:58.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:03:58.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:58.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:03:58.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:58.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:58.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:58.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:03:58.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:58.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:03:58.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:58.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:58.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:58.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:58.834 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:58.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:58.879 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:03:58.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:58.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:58.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:59.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:03:59.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/756303518' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:03:59.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/756303518' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:03:59.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:59.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:59.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:03:59.210 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/756303518' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:03:59.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:03:59.251 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:03:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:59.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:03:59.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:59.405 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:59.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:03:59.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:59.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:59.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:59.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:03:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:59.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:03:59.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:03:59.795 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:03:59.795 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:03:59.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:59.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:59.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:03:59.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:03:59.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:03:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:03:59.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:00.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:00.076 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:00.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:00.102 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:04:00.102 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "" 2024-06-14T23:04:00.102 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:04:00.102 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:59 smithi137 ceph-mon[90634]: pgmap v869: 99 pgs: 99 active+clean; 1.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 913 op/s 2024-06-14T23:04:00.103 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:59 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:00.103 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:59 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/756303518' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:04:00.103 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:03:59 smithi137 ceph-mon[90634]: osdmap e216: 8 total, 8 up, 8 in 2024-06-14T23:04:00.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:00.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:00.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:00.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:00.242 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:59 smithi086 ceph-mon[97913]: pgmap v869: 99 pgs: 99 active+clean; 1.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 913 op/s 2024-06-14T23:04:00.242 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:59 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:00.242 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:59 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/756303518' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:04:00.242 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:03:59 smithi086 ceph-mon[97913]: osdmap e216: 8 total, 8 up, 8 in 2024-06-14T23:04:00.244 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:59 smithi086 ceph-mon[102754]: pgmap v869: 99 pgs: 99 active+clean; 1.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 913 op/s 2024-06-14T23:04:00.244 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:59 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:00.244 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:59 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/756303518' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:04:00.245 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:03:59 smithi086 ceph-mon[102754]: osdmap e216: 8 total, 8 up, 8 in 2024-06-14T23:04:00.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:00.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:00.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:00.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:00.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:00.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:00.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:00.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:00.628 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:00.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:00.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:00.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:00.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:00.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:00.914 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:00.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:01.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:01.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:01.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:01.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:01.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:01.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:01.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:01.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:01.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:01.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:01.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:01.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:01.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:01.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:01.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:01.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:01.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:01.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-14T23:04:01.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:01.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:01.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:01.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:01.978 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:04:01.979 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:04:02.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:04:02.011 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:04:02.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:02.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:02.046 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:04:02.046 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:04:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:04:02.076 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:04:02.076 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:01 smithi086 ceph-mon[97913]: pgmap v871: 99 pgs: 99 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-06-14T23:04:02.077 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:01 smithi086 ceph-mon[102754]: pgmap v871: 99 pgs: 99 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-06-14T23:04:02.115 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:04:02.115 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:04:02.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:04:02.149 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:04:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:04:02.186 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:04:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:02.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:02.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:02.206 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:01 smithi137 ceph-mon[90634]: pgmap v871: 99 pgs: 99 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-06-14T23:04:02.220 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:04:02.220 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:04:02.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:02.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:02.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:02.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:02.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:02.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:02.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:02.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:02.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:02.643 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:02.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:02.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:02.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:02.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:02.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:02.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:03.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:03.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:03.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:03.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:03.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:03.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:03.216 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2834904491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:03.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:03.326 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2834904491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:03.326 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2834904491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:03.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:03.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:03.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:03.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:03.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:03.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:03.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:03.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:03.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:03.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:03.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:04:03.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:03.750 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:03.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:03.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:03.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:03.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:04.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:04.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:04.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:04.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:04.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:03 smithi086 ceph-mon[97913]: pgmap v872: 99 pgs: 99 active+clean; 1.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 876 op/s 2024-06-14T23:04:04.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:03 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:04.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:03 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:04.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:03 smithi086 ceph-mon[102754]: pgmap v872: 99 pgs: 99 active+clean; 1.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 876 op/s 2024-06-14T23:04:04.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:03 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:04.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:03 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:04.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:04.182 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:04.200 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:03 smithi137 ceph-mon[90634]: pgmap v872: 99 pgs: 99 active+clean; 1.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 876 op/s 2024-06-14T23:04:04.200 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:03 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:04.200 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:03 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:04.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:04.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:04.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:04.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:04.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:04.501 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:04.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:04.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:04.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:04.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:04.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:04.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:04.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:04.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:04.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:05.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:05.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:05.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:05.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:04:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:04:04] "GET /metrics HTTP/1.1" 200 36592 "" "Prometheus/2.43.0" 2024-06-14T23:04:05.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:05.180 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:05.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:05.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:05.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:05.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:04:05.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:05.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:05.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:05.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:05.561 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:04:05.562 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:04:05.562 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:04:05.562 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:04:05.562 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:04:05.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:05.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:05.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:05.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:05.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:05.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:05.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:05.867 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:04:05.867 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:04:05.867 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:04:05.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd out 2 2024-06-14T23:04:05.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:05.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:05.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[97913]: pgmap v873: 99 pgs: 99 active+clean; 1.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-06-14T23:04:05.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:05.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:05.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:04:05.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:04:05.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:05.964 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[102754]: pgmap v873: 99 pgs: 99 active+clean; 1.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-06-14T23:04:05.964 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:05.965 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:05.965 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:04:05.965 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:04:05.965 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:05 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:05.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:06.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:06.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:06.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:06.146 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:06.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:06.222 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:05 smithi137 ceph-mon[90634]: pgmap v873: 99 pgs: 99 active+clean; 1.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-06-14T23:04:06.222 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:06.223 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:06.223 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:04:06.223 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:04:06.223 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:05 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:06.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:06.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-14T23:04:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:06.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:06.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:06.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:06.395 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:06.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:06.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:06.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:06.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:06.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:06.683 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:06.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:06.724 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:06.766 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:06.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:06.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:06.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:06.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:06.921 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:04:07.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:07.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:07.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:07.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1780837979' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:07.082 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:06 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:07.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:07.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:07.205 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1780837979' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:07.205 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:06 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:07.205 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1780837979' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:07.205 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:06 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:07.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:07.320 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:07.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:07.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:07.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:07.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:07.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:07.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:07.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:07.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:07.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:07.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:07.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:07.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:07.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:07.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:07.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:08.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:08.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:08.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:08.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:08.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:08.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:07 smithi137 ceph-mon[90634]: pgmap v874: 99 pgs: 99 active+clean; 1.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 870 op/s 2024-06-14T23:04:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:07 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:04:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:07 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:04:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:07 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:04:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:07 smithi137 ceph-mon[90634]: osdmap e217: 8 total, 8 up, 8 in 2024-06-14T23:04:08.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:08.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:08.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[97913]: pgmap v874: 99 pgs: 99 active+clean; 1.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 870 op/s 2024-06-14T23:04:08.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:04:08.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:04:08.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:04:08.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[97913]: osdmap e217: 8 total, 8 up, 8 in 2024-06-14T23:04:08.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[102754]: pgmap v874: 99 pgs: 99 active+clean; 1.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 870 op/s 2024-06-14T23:04:08.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:04:08.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:04:08.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:04:08.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:07 smithi086 ceph-mon[102754]: osdmap e217: 8 total, 8 up, 8 in 2024-06-14T23:04:08.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:08.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:08.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:08.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:08.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:08.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:08.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:08.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:08.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:08.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:08.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:08.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:08.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:08.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:08.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:09.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:09.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:09.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:09.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:09.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:09.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:09.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:09.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:09.369 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:09.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:09.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:09.505 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:09.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:09.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:09 smithi086 ceph-mon[97913]: pgmap v876: 99 pgs: 99 active+clean; 1.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 881 op/s 2024-06-14T23:04:09.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:09 smithi086 ceph-mon[102754]: pgmap v876: 99 pgs: 99 active+clean; 1.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 881 op/s 2024-06-14T23:04:09.590 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:09 smithi137 ceph-mon[90634]: pgmap v876: 99 pgs: 99 active+clean; 1.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 881 op/s 2024-06-14T23:04:09.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:09.636 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:09.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:09.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:09.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:09.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:09.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:09.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:09.909 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:09.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:10.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:10.050 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:10.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:10.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:10.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:10.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:10.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:10.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:10.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:10.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:10.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:10.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:10.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:10.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:10.646 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:04:10.646 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:04:10.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:04:10.646 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:04:10.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.646 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84001 9.7 9:ec42c003:::benchmark_data_smithi086_152062_object84000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:04:10.646 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:29.109679+0000", 2024-06-14T23:04:10.646 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 461.01319227699997, 2024-06-14T23:04:10.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14871410800000001, 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84001 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.109679+0000", 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.109679+0000", 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.109678+0000", 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.109745+0000", 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6929999999999998e-05 2024-06-14T23:04:10.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.109756+0000", 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1489e-05 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.109761+0000", 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8489999999999997e-06 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.109797+0000", 2024-06-14T23:04:10.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5596999999999998e-05 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.109930+0000", 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000132924 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.110068+0000", 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013795499999999999 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.112626+0000", 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002558472 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.112729+0000", 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102718 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.258169+0000", 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14544012000000001 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.258221+0000", 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2043999999999998e-05 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.258393+0000", 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017192100000000001 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84158 9.f 9:fccd6e2b:::benchmark_data_smithi086_152062_object84157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:29.372082+0000", 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 460.75078898499999, 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13950326699999999, 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84158 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.372082+0000", 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.372082+0000", 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.372080+0000", 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.372147+0000", 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6265000000000003e-05 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.372159+0000", 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2343e-05 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.372164+0000", 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7790000000000003e-06 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.375087+0000", 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029227200000000002 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.375202+0000", 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114982 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3", 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.375328+0000", 2024-06-14T23:04:10.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126976 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.384338+0000", 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090100130000000007 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.388068+0000", 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003729843 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.511499+0000", 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.123430945 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.511547+0000", 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7318999999999999e-05 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.511585+0000", 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8866999999999999e-05 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.658 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84166 9.f 9:f1d04dd5:::benchmark_data_smithi086_152062_object84165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:29.378292+0000", 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 460.74457868899998, 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13355194000000001, 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84166 2024-06-14T23:04:10.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.378292+0000", 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.378292+0000", 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.378291+0000", 2024-06-14T23:04:10.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.378362+0000", 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1259000000000003e-05 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.378377+0000", 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4253000000000001e-05 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.378382+0000", 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0150000000000003e-06 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.378422+0000", 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0521e-05 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.378587+0000", 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016490600000000001 2024-06-14T23:04:10.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3", 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.378773+0000", 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018622199999999999 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.384348+0000", 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055742949999999999 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.388089+0000", 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003741439 2024-06-14T23:04:10.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.511641+0000", 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.123551628 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.511686+0000", 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5377000000000003e-05 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.511844+0000", 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000158367 2024-06-14T23:04:10.664 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84172 9.f 9:f86621d4:::benchmark_data_smithi086_152062_object84171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:29.380105+0000", 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 460.74276644700001, 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13190748499999999, 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84172 2024-06-14T23:04:10.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.380105+0000", 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.380105+0000", 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.380103+0000", 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.380161+0000", 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7435999999999997e-05 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.380174+0000", 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3185e-05 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.380179+0000", 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.994e-06 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.381472+0000", 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012932600000000001 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.381589+0000", 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116858 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3", 2024-06-14T23:04:10.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.381719+0000", 2024-06-14T23:04:10.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012947800000000001 2024-06-14T23:04:10.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.387445+0000", 2024-06-14T23:04:10.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057266239999999996 2024-06-14T23:04:10.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.388121+0000", 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067582799999999995 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.511952+0000", 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12383106200000001 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.511992+0000", 2024-06-14T23:04:10.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9579000000000002e-05 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.512012+0000", 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.054e-05 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84174 9.f 9:f08b5acf:::benchmark_data_smithi086_152062_object84173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:29.381607+0000", 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 460.74126421400001, 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13055863700000001, 2024-06-14T23:04:10.671 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84174 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.381607+0000", 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.381607+0000", 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.381606+0000", 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.381657+0000", 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1479e-05 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.381669+0000", 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1506e-05 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.381675+0000", 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.976e-06 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.384366+0000", 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026911130000000002 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.384498+0000", 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013162100000000001 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3", 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.384652+0000", 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015427400000000001 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.387547+0000", 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028955299999999999 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.388138+0000", 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00059028200000000002 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.512100+0000", 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12396225 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.512128+0000", 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7942000000000001e-05 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.512166+0000", 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7812999999999999e-05 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:84179 9.7 9:edbbe4a3:::benchmark_data_smithi086_152062_object84178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:56:29.386034+0000", 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 460.73683672499999, 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12624389699999999, 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.677 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 84179 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.386034+0000", 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.386034+0000", 2024-06-14T23:04:10.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.386033+0000", 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.386080+0000", 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7134e-05 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.386092+0000", 2024-06-14T23:04:10.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1525000000000001e-05 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.386098+0000", 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3110000000000002e-06 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.387458+0000", 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001359344 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.387584+0000", 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125988 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.387726+0000", 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014286 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.390095+0000", 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023689800000000001 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.390671+0000", 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00057532199999999996 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.512202+0000", 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.121531346 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.512243+0000", 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0822000000000003e-05 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:56:29.512278+0000", 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5456000000000001e-05 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189315 9.6 9:64c260e8:::benchmark_data_smithi086_152062_object56833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.739313+0000", 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.38355770599998, 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13235851200000001, 2024-06-14T23:04:10.683 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189315 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.739313+0000", 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.739313+0000", 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.739312+0000", 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.739318+0000", 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3650000000000003e-06 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.739329+0000", 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0896e-05 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.739334+0000", 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2039999999999996e-06 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.739775+0000", 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000441383 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.739869+0000", 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3516000000000004e-05 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.739977+0000", 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010804200000000001 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751069+0000", 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011092585 2024-06-14T23:04:10.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.865861+0000", 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.114792011 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.871400+0000", 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055383330000000003 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.871549+0000", 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014925999999999999 2024-06-14T23:04:10.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.871672+0000", 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123027 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189319 9.6 9:60d183a1:::benchmark_data_smithi086_152062_object56837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.743565+0000", 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.37930601099998, 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12856694499999999, 2024-06-14T23:04:10.689 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189319 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.743565+0000", 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.743565+0000", 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.743565+0000", 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.743567+0000", 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5959999999999998e-06 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.743574+0000", 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5769999999999999e-06 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.743578+0000", 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5820000000000002e-06 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.744951+0000", 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001373293 2024-06-14T23:04:10.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.745057+0000", 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106564 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.745761+0000", 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00070334399999999995 2024-06-14T23:04:10.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.865830+0000", 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12006904 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.865929+0000", 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9073999999999994e-05 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.872057+0000", 2024-06-14T23:04:10.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061284629999999998 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.872099+0000", 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1199999999999999e-05 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.872132+0000", 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3606000000000003e-05 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189321 9.f 9:f5f12ca3:::benchmark_data_smithi086_152062_object56839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.745789+0000", 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.37708212899997, 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.124224509, 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189321 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.745789+0000", 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.745789+0000", 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.745788+0000", 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.745793+0000", 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5969999999999999e-06 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.745805+0000", 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1406999999999999e-05 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.745810+0000", 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2249999999999999e-06 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.746268+0000", 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000457847 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.746372+0000", 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104051 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3", 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.746474+0000", 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010168899999999999 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.869724+0000", 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12325037 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.869861+0000", 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013724899999999999 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.869904+0000", 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2636000000000003e-05 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.869964+0000", 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0433999999999998e-05 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.870014+0000", 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9234999999999999e-05 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:189320 9.9 e165/140 9:9724c2fa:::benchmark_data_smithi086_152062_object56838:head v 165'11747, mlcod=165'11744)", 2024-06-14T23:04:10.701 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.748788+0000", 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.37408352199998, 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.116139482, 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.748788+0000", 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.748788+0000", 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.748787+0000", 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.748792+0000", 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3990000000000003e-06 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.748801+0000", 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8310000000000007e-06 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.748806+0000", 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3599999999999998e-06 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.748853+0000", 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7729999999999999e-05 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.748870+0000", 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7041000000000002e-05 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864885+0000", 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.116014271 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864927+0000", 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2525999999999999e-05 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189324 9.f 9:ff980870:::benchmark_data_smithi086_152062_object56842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.750812+0000", 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.37205949899999, 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13115876000000001, 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189324 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750812+0000", 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750812+0000", 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750810+0000", 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750816+0000", 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8839999999999999e-06 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750829+0000", 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2675e-05 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750834+0000", 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0749999999999997e-06 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751523+0000", 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00068893000000000003 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751628+0000", 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105164 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3", 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864799+0000", 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.113171324 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.870092+0000", 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052931219999999999 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.870156+0000", 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4148999999999999e-05 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.880475+0000", 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010318709000000001 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.880579+0000", 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103491 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.881970+0000", 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00139182 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:189323 9.d e165/145 9:bf1fdd81:::benchmark_data_smithi086_152062_object56841:head v 165'11886, mlcod=165'11882)", 2024-06-14T23:04:10.711 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.750869+0000", 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.37200264900002, 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.122542127, 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750869+0000", 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750869+0000", 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750868+0000", 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750872+0000", 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0300000000000004e-06 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750878+0000", 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2449999999999998e-06 2024-06-14T23:04:10.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750880+0000", 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9210000000000001e-06 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750939+0000", 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8977000000000001e-05 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.750956+0000", 2024-06-14T23:04:10.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6954999999999999e-05 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.873378+0000", 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12242133500000001 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.873411+0000", 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2923999999999999e-05 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.715 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189326 9.6 9:6842f614:::benchmark_data_smithi086_152062_object56844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.751260+0000", 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.37161090799998, 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.120949917, 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189326 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751260+0000", 2024-06-14T23:04:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751260+0000", 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751259+0000", 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751264+0000", 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3449999999999999e-06 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751273+0000", 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5140000000000007e-06 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751278+0000", 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9139999999999999e-06 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751312+0000", 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4122000000000003e-05 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751424+0000", 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011144199999999999 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751534+0000", 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110184 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.865892+0000", 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.114357496 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.865958+0000", 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6519999999999993e-05 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.872139+0000", 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061811510000000002 2024-06-14T23:04:10.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.872177+0000", 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7672000000000001e-05 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.872210+0000", 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3349000000000002e-05 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.721 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189329 9.a 9:5e86d06f:::benchmark_data_smithi086_152062_object56847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.751617+0000", 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.37125440099999, 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12715058900000001, 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189329 2024-06-14T23:04:10.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751617+0000", 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751617+0000", 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751616+0000", 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751621+0000", 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3020000000000002e-06 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751631+0000", 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3200000000000006e-06 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751635+0000", 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7190000000000001e-06 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.751663+0000", 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7237e-05 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864781+0000", 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.113118417 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.866646+0000", 2024-06-14T23:04:10.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018648339999999999 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.873455+0000", 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068092919999999998 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.874534+0000", 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010784519999999999 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.878665+0000", 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041311079999999997 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.878712+0000", 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7311999999999997e-05 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.878767+0000", 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5315999999999997e-05 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35580.0:189328 9.b e165/140 9:ded1c998:::benchmark_data_smithi086_152062_object56846:head v 165'12015, mlcod=165'12013)", 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.752057+0000", 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.37081415699998, 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14397589599999999, 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.752057+0000", 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.752057+0000", 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.752056+0000", 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.752061+0000", 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6809999999999997e-06 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864874+0000", 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.112813286 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864883+0000", 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2849999999999995e-06 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864928+0000", 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4873999999999998e-05 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864949+0000", 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1285999999999999e-05 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.896002+0000", 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031053035999999999 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.896033+0000", 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1083e-05 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35580.0:189337 9.6 9:639fd726:::benchmark_data_smithi086_152062_object56855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T22:58:18.765053+0000", 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 351.35781800699999, 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.115047202, 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.732 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35580", 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189337 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765053+0000", 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765053+0000", 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765052+0000", 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.765060+0000", 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9619999999999993e-06 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864942+0000", 2024-06-14T23:04:10.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.099881841999999998 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.864948+0000", 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8800000000000002e-06 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.865983+0000", 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001034812 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.866149+0000", 2024-06-14T23:04:10.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000165907 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.866277+0000", 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127658 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.871815+0000", 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055383029999999996 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.872045+0000", 2024-06-14T23:04:10.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023028 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.880011+0000", 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079658369999999999 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.880062+0000", 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0779999999999998e-05 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T22:58:18.880100+0000", 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8343999999999997e-05 2024-06-14T23:04:10.737 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:117226 10.6 10:65a499ab:::benchmark_data_smithi086_189745_object117225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.016769+0000", 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 137.106102021, 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10652734799999999, 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:10.738 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117226 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.016769+0000", 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.016769+0000", 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.016768+0000", 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.016833+0000", 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5123000000000007e-05 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.016844+0000", 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0706e-05 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.016849+0000", 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3800000000000002e-06 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.017114+0000", 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026519800000000002 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.017521+0000", 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000406303 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.017993+0000", 2024-06-14T23:04:10.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047259299999999999 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024462+0000", 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064686530000000004 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.027657+0000", 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031951560000000002 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.123175+0000", 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095518351000000001 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.123259+0000", 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4034000000000002e-05 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.123297+0000", 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7081e-05 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.743 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:202573 10.6 10:67aafbef:::benchmark_data_smithi086_189745_object71605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:22.898415+0000", 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.224456471000003, 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10999808699999999, 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202573 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.744 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898415+0000", 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898415+0000", 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898413+0000", 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:10.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898419+0000", 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8270000000000003e-06 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898431+0000", 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1831000000000001e-05 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898436+0000", 2024-06-14T23:04:10.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2839999999999997e-06 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.909978+0000", 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01154174 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.910090+0000", 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111508 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.910201+0000", 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110943 2024-06-14T23:04:10.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.925078+0000", 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014877223 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.929351+0000", 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042730329999999999 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:23.008329+0000", 2024-06-14T23:04:10.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.078977884999999998 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:23.008374+0000", 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5328999999999999e-05 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:23.008413+0000", 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8724999999999999e-05 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:202574 10.6 10:666a2f87:::benchmark_data_smithi086_189745_object71606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:22.898739+0000", 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.22413169, 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10983515200000001, 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202574 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898739+0000", 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898739+0000", 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898739+0000", 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:10.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898743+0000", 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3630000000000004e-06 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898751+0000", 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8739999999999995e-06 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.898756+0000", 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6249999999999998e-06 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.915110+0000", 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016354345999999999 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.916442+0000", 2024-06-14T23:04:10.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013320420000000001 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.916580+0000", 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013757900000000001 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.929385+0000", 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012805429 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.933892+0000", 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045065690000000002 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:23.008466+0000", 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074574915000000006 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:23.008533+0000", 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6191000000000004e-05 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:23.008575+0000", 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2001e-05 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:202582 10.6 10:6c523175:::benchmark_data_smithi086_189745_object71614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:22.905732+0000", 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.217139682999999, 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.103004294, 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:10.756 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202582 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.905732+0000", 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.905732+0000", 2024-06-14T23:04:10.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.905730+0000", 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.905737+0000", 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0709999999999997e-06 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:10.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.905748+0000", 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0594999999999999e-05 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.905753+0000", 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2259999999999996e-06 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.921205+0000", 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01545205 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.921353+0000", 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000147725 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.921483+0000", 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013016300000000001 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.932995+0000", 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011512333 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:22.933938+0000", 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00094267400000000001 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:23.008610+0000", 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074672344000000002 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:23.008698+0000", 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7367000000000006e-05 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:10.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:10.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:23.008736+0000", 2024-06-14T23:04:10.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8152000000000003e-05 2024-06-14T23:04:10.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.762 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:10.762 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:10.762 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:04:10.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:10.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:10.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:10.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:04:10.887 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:04:10.903 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:04:10.903 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:04:10.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:10.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:10.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:10.953 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:04:10.953 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:04:10.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1559166045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:10.998 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:04:10.999 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:04:11.037 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:04:11.038 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:04:11.038 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1559166045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:11.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1559166045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:11.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:11.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:11.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:11.084 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:04:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:04:11.117 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:04:11.117 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:04:11.150 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:04:11.151 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:04:11.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:11.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:11.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:11.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:11.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:11 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:04:11.068+0000 7fe0161db640 -1 mon.a@0(leader).osd e217 definitely_dead 0 2024-06-14T23:04:11.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:11.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:11.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:11.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:11.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:11.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:11.628 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:11.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:11.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T23:04:11.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:11.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:11.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:11.862 INFO:teuthology.orchestra.run.smithi086.stderr:marked out osd.2. 2024-06-14T23:04:11.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:11.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:04:11.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:11.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:11.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:11.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:11 smithi137 ceph-mon[90634]: pgmap v877: 99 pgs: 99 active+clean; 1.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 877 op/s 2024-06-14T23:04:11.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2831825474' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:04:11.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:11 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:04:11.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:11 smithi137 ceph-mon[90634]: Client client.admin marked osd.2 out, while it was still marked up 2024-06-14T23:04:12.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:12.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:12.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:12.038 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:12.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:12.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:12.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:12.226 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:12.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:12.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:12.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:12.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:11 smithi086 ceph-mon[97913]: pgmap v877: 99 pgs: 99 active+clean; 1.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 877 op/s 2024-06-14T23:04:12.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2831825474' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:04:12.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:11 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:04:12.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:11 smithi086 ceph-mon[97913]: Client client.admin marked osd.2 out, while it was still marked up 2024-06-14T23:04:12.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:11 smithi086 ceph-mon[102754]: pgmap v877: 99 pgs: 99 active+clean; 1.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 877 op/s 2024-06-14T23:04:12.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2831825474' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:04:12.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:11 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:04:12.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:11 smithi086 ceph-mon[102754]: Client client.admin marked osd.2 out, while it was still marked up 2024-06-14T23:04:12.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:12.399 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:12.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:12.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:12.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:12.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:12.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:12.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:12.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:12.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:12.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:12.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:12.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:12.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:13.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:13.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:13.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:13.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:13.145 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:13.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:13.197 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:12 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-06-14T23:04:13.198 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:12 smithi137 ceph-mon[90634]: osdmap e218: 8 total, 8 up, 7 in 2024-06-14T23:04:13.198 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:13.198 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:13.198 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:12 smithi137 ceph-mon[90634]: osdmap e219: 8 total, 8 up, 7 in 2024-06-14T23:04:13.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:13.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:13.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:13.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-06-14T23:04:13.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[102754]: osdmap e218: 8 total, 8 up, 7 in 2024-06-14T23:04:13.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:13.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:13.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[102754]: osdmap e219: 8 total, 8 up, 7 in 2024-06-14T23:04:13.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-06-14T23:04:13.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[97913]: osdmap e218: 8 total, 8 up, 7 in 2024-06-14T23:04:13.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:13.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:13.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:12 smithi086 ceph-mon[97913]: osdmap e219: 8 total, 8 up, 7 in 2024-06-14T23:04:13.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:13.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:13.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:13.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:13.516 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:13.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:13.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:13.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:13.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:13.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:13.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:13.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:13.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:13.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:13.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:13.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:13.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:13.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:04:14.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:14.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:14.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:14.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:14.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:14.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:14.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:14.217 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:13 smithi137 ceph-mon[90634]: pgmap v879: 99 pgs: 1 unknown, 5 peering, 93 active+clean; 1.0 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T23:04:14.217 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:13 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:04:14.217 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:13 smithi137 ceph-mon[90634]: osdmap e220: 8 total, 8 up, 7 in 2024-06-14T23:04:14.218 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:13 smithi137 ceph-mon[90634]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 217'15296 2024-06-14T23:04:14.218 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:13 smithi137 ceph-mon[90634]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 217'15842 2024-06-14T23:04:14.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:14.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:14.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:14.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[97913]: pgmap v879: 99 pgs: 1 unknown, 5 peering, 93 active+clean; 1.0 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T23:04:14.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:04:14.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[97913]: osdmap e220: 8 total, 8 up, 7 in 2024-06-14T23:04:14.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[97913]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 217'15296 2024-06-14T23:04:14.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[97913]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 217'15842 2024-06-14T23:04:14.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[102754]: pgmap v879: 99 pgs: 1 unknown, 5 peering, 93 active+clean; 1.0 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 717 op/s 2024-06-14T23:04:14.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:04:14.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[102754]: osdmap e220: 8 total, 8 up, 7 in 2024-06-14T23:04:14.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[102754]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 217'15296 2024-06-14T23:04:14.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:13 smithi086 ceph-mon[102754]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 217'15842 2024-06-14T23:04:14.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:14.419 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:04:14.419 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:04:14.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:14.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:14.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:14.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:14.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:14.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:14.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:14.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:14.830 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:14.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:14.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:14.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:14.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:15.024 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:15.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:15.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:14 smithi086 ceph-mon[102754]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 217'15540 2024-06-14T23:04:15.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:14 smithi086 ceph-mon[102754]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 217'15227 2024-06-14T23:04:15.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:14 smithi086 ceph-mon[102754]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 217'15580 2024-06-14T23:04:15.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:14 smithi086 ceph-mon[102754]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 217'15068 2024-06-14T23:04:15.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:14 smithi086 ceph-mon[97913]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 217'15540 2024-06-14T23:04:15.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:14 smithi086 ceph-mon[97913]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 217'15227 2024-06-14T23:04:15.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:14 smithi086 ceph-mon[97913]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 217'15580 2024-06-14T23:04:15.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:14 smithi086 ceph-mon[97913]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 217'15068 2024-06-14T23:04:15.035 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:04:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:04:14] "GET /metrics HTTP/1.1" 200 36585 "" "Prometheus/2.43.0" 2024-06-14T23:04:15.107 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:14 smithi137 ceph-mon[90634]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 217'15540 2024-06-14T23:04:15.107 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:14 smithi137 ceph-mon[90634]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 217'15227 2024-06-14T23:04:15.107 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:14 smithi137 ceph-mon[90634]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 217'15580 2024-06-14T23:04:15.107 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:14 smithi137 ceph-mon[90634]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 217'15068 2024-06-14T23:04:15.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:15.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-14T23:04:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:15.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:15.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:15.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:15.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:15.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:15.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:15.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:15.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:15.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:15.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:15.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:15.678 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:15.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:15.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:15.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:15.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:15.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:15.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:15.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:04:15.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:16.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[97913]: pgmap v882: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 4 active+undersized+remapped, 1 unknown, 1 active+remapped, 5 peering, 85 active+clean; 932 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 1/44787 objects degraded (0.002%); 3620/44787 objects misplaced (8.083%); 267 KiB/s, 1 keys/s, 4 objects/s recovering 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[97913]: Health check failed: Degraded data redundancy: 1/44787 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[97913]: osdmap e221: 8 total, 8 up, 7 in 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3871337908' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[102754]: pgmap v882: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 4 active+undersized+remapped, 1 unknown, 1 active+remapped, 5 peering, 85 active+clean; 932 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 1/44787 objects degraded (0.002%); 3620/44787 objects misplaced (8.083%); 267 KiB/s, 1 keys/s, 4 objects/s recovering 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[102754]: Health check failed: Degraded data redundancy: 1/44787 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[102754]: osdmap e221: 8 total, 8 up, 7 in 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3871337908' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:04:16.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:15 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:04:16.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:16.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:16.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:16.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:16.195 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:15 smithi137 ceph-mon[90634]: pgmap v882: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 4 active+undersized+remapped, 1 unknown, 1 active+remapped, 5 peering, 85 active+clean; 932 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 1/44787 objects degraded (0.002%); 3620/44787 objects misplaced (8.083%); 267 KiB/s, 1 keys/s, 4 objects/s recovering 2024-06-14T23:04:16.195 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:15 smithi137 ceph-mon[90634]: Health check failed: Degraded data redundancy: 1/44787 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-06-14T23:04:16.195 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:15 smithi137 ceph-mon[90634]: osdmap e221: 8 total, 8 up, 7 in 2024-06-14T23:04:16.195 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3871337908' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:04:16.195 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:15 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:04:16.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:16.291 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:16.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:16.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:16.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:16.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:16.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:16.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:16.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:16.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:16.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:16.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:16.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:16.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:16.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:16.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:16.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:16.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:16 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:16.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:16 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:04:16.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:16 smithi137 ceph-mon[90634]: osdmap e222: 8 total, 8 up, 7 in 2024-06-14T23:04:17.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:17.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:17.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:17.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:17.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:17.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:17.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:16 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:16 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:04:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:16 smithi086 ceph-mon[97913]: osdmap e222: 8 total, 8 up, 7 in 2024-06-14T23:04:17.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:16 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:17.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:16 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:04:17.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:16 smithi086 ceph-mon[102754]: osdmap e222: 8 total, 8 up, 7 in 2024-06-14T23:04:17.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:17.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:17.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:17.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:17.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:17.522 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:17.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:17.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:17.670 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:17.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:17.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:17.719 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:04:17.720 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:04:17.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:17.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:17.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:17.899 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:04:17.899 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:04:17.899 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-06-14T23:04:17.899 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:04:17.899 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:04:17.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-06-14T23:04:17.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:17.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:17 smithi137 ceph-mon[90634]: pgmap v885: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 4 active+undersized+remapped, 1 unknown, 1 active+remapped, 5 peering, 85 active+clean; 923 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 511 op/s; 1/44367 objects degraded (0.002%); 3620/44367 objects misplaced (8.159%); 400 KiB/s, 2 keys/s, 6 objects/s recovering 2024-06-14T23:04:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:18.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:18.067 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:17 smithi086 ceph-mon[102754]: pgmap v885: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 4 active+undersized+remapped, 1 unknown, 1 active+remapped, 5 peering, 85 active+clean; 923 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 511 op/s; 1/44367 objects degraded (0.002%); 3620/44367 objects misplaced (8.159%); 400 KiB/s, 2 keys/s, 6 objects/s recovering 2024-06-14T23:04:18.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:17 smithi086 ceph-mon[97913]: pgmap v885: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 4 active+undersized+remapped, 1 unknown, 1 active+remapped, 5 peering, 85 active+clean; 923 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 511 op/s; 1/44367 objects degraded (0.002%); 3620/44367 objects misplaced (8.159%); 400 KiB/s, 2 keys/s, 6 objects/s recovering 2024-06-14T23:04:18.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:18.139 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:18.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:18.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:18.149 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:04:18.149 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:04:18.186 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:04:18.186 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:04:18.224 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:04:18.224 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:04:18.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:18.259 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:04:18.260 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:04:18.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:18.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:18.292 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:04:18.292 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:04:18.328 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:04:18.329 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:04:18.344 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:18.369 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:04:18.370 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:04:18.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:18.406 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:04:18.406 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:04:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:18.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:18.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:18.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-14T23:04:18.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:18.549 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:18.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:18.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:18.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:18.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:18.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:18.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:18.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:18.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:18.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:18.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:18.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:18.971 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:19.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:19.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:19.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:19.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:19.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1123185940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:19.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:19.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1123185940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:19.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1123185940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:19.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:19.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:19.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:19.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:19.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:19.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:04:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:19.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:19.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:19.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:19.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:19.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:19.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:19.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:19.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:19.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:19.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:20.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:20.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:20.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:20.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:20.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:20.195 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:20.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:20.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:19 smithi137 ceph-mon[90634]: pgmap v886: 99 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 4 active+undersized+remapped, 1 active+remapped, 89 active+clean; 892 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 430 op/s; 1/42867 objects degraded (0.002%); 4740/42867 objects misplaced (11.057%); 5.0 MiB/s, 1 keys/s, 80 objects/s recovering 2024-06-14T23:04:20.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:19 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T23:04:20.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:20.297 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:19 smithi086 ceph-mon[97913]: pgmap v886: 99 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 4 active+undersized+remapped, 1 active+remapped, 89 active+clean; 892 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 430 op/s; 1/42867 objects degraded (0.002%); 4740/42867 objects misplaced (11.057%); 5.0 MiB/s, 1 keys/s, 80 objects/s recovering 2024-06-14T23:04:20.297 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:19 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T23:04:20.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:19 smithi086 ceph-mon[102754]: pgmap v886: 99 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 4 active+undersized+remapped, 1 active+remapped, 89 active+clean; 892 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 430 op/s; 1/42867 objects degraded (0.002%); 4740/42867 objects misplaced (11.057%); 5.0 MiB/s, 1 keys/s, 80 objects/s recovering 2024-06-14T23:04:20.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:19 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T23:04:20.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:20.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:20.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:20.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:20.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:20.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:20.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:20.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:20.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:20.704 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:20.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:20.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:20.833 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:20.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:20.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:20.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:20 smithi086 ceph-mon[97913]: osdmap e223: 8 total, 8 up, 7 in 2024-06-14T23:04:20.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:20 smithi086 ceph-mon[97913]: osdmap e224: 8 total, 8 up, 7 in 2024-06-14T23:04:20.961 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:20.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:21.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:21.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:21.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:21.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:21.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:21.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:21.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:20 smithi137 ceph-mon[90634]: osdmap e223: 8 total, 8 up, 7 in 2024-06-14T23:04:21.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:20 smithi137 ceph-mon[90634]: osdmap e224: 8 total, 8 up, 7 in 2024-06-14T23:04:21.270 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:20 smithi086 ceph-mon[102754]: osdmap e223: 8 total, 8 up, 7 in 2024-06-14T23:04:21.270 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:20 smithi086 ceph-mon[102754]: osdmap e224: 8 total, 8 up, 7 in 2024-06-14T23:04:21.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:21.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:21.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:21.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:21.476 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:21.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:21.551 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:04:21.551 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-14T23:04:21.551 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:04:21.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:21.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:21.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:21.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:21.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:21.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:21.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:21.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:21.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:21.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:21.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:21.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:21.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:21.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:22.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:22.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:22.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:22.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:04:22.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:22.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:21 smithi137 ceph-mon[90634]: pgmap v888: 99 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 93 active+clean; 673 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 691 op/s; 3313/32328 objects misplaced (10.248%); 9.0 MiB/s, 144 objects/s recovering 2024-06-14T23:04:22.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:21 smithi137 ceph-mon[90634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/42867 objects degraded (0.002%), 1 pg degraded) 2024-06-14T23:04:22.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:21 smithi137 ceph-mon[90634]: osdmap e225: 8 total, 8 up, 7 in 2024-06-14T23:04:22.226 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:22.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:22.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:22.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:21 smithi086 ceph-mon[97913]: pgmap v888: 99 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 93 active+clean; 673 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 691 op/s; 3313/32328 objects misplaced (10.248%); 9.0 MiB/s, 144 objects/s recovering 2024-06-14T23:04:22.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:21 smithi086 ceph-mon[97913]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/42867 objects degraded (0.002%), 1 pg degraded) 2024-06-14T23:04:22.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:21 smithi086 ceph-mon[97913]: osdmap e225: 8 total, 8 up, 7 in 2024-06-14T23:04:22.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:21 smithi086 ceph-mon[102754]: pgmap v888: 99 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_wait, 93 active+clean; 673 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 691 op/s; 3313/32328 objects misplaced (10.248%); 9.0 MiB/s, 144 objects/s recovering 2024-06-14T23:04:22.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:21 smithi086 ceph-mon[102754]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/42867 objects degraded (0.002%), 1 pg degraded) 2024-06-14T23:04:22.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:21 smithi086 ceph-mon[102754]: osdmap e225: 8 total, 8 up, 7 in 2024-06-14T23:04:22.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:22.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:22.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:22.475 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:22.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:22.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:22.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:22.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:22.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:22.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:22.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:22.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:22.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:22.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:22.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:22.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:22.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:23.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:23.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:23.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:23.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:23.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:23.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:22 smithi137 ceph-mon[90634]: pgmap v891: 99 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 94 active+clean; 640 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 2193/30768 objects misplaced (7.128%); 19 MiB/s, 300 objects/s recovering 2024-06-14T23:04:23.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:22 smithi137 ceph-mon[90634]: osdmap e226: 8 total, 8 up, 7 in 2024-06-14T23:04:23.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:23.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:22 smithi086 ceph-mon[97913]: pgmap v891: 99 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 94 active+clean; 640 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 2193/30768 objects misplaced (7.128%); 19 MiB/s, 300 objects/s recovering 2024-06-14T23:04:23.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:22 smithi086 ceph-mon[97913]: osdmap e226: 8 total, 8 up, 7 in 2024-06-14T23:04:23.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:22 smithi086 ceph-mon[102754]: pgmap v891: 99 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 94 active+clean; 640 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 2193/30768 objects misplaced (7.128%); 19 MiB/s, 300 objects/s recovering 2024-06-14T23:04:23.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:22 smithi086 ceph-mon[102754]: osdmap e226: 8 total, 8 up, 7 in 2024-06-14T23:04:23.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:23.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:23.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:23.471 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:23.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:117240 10.0 10:034e9ce1:::benchmark_data_smithi086_189745_object117239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.029612+0000", 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 150.50951803000001, 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10616302, 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117240 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029612+0000", 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029612+0000", 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029611+0000", 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029647+0000", 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6857999999999998e-05 2024-06-14T23:04:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029658+0000", 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.065e-05 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029663+0000", 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1850000000000003e-06 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029705+0000", 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1727000000000002e-05 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029809+0000", 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104215 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029934+0000", 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012450199999999999 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.037189+0000", 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072549939999999999 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.040251+0000", 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003062646 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.135661+0000", 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095410160999999993 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.135739+0000", 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7656000000000003e-05 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.135775+0000", 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5499e-05 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:117242 10.0 10:057c22a3:::benchmark_data_smithi086_189745_object117241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:04:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.031608+0000", 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 150.50752146600001, 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.109624362, 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117242 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031608+0000", 2024-06-14T23:04:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031608+0000", 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031607+0000", 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031657+0000", 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0504000000000003e-05 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031667+0000", 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4259999999999992e-06 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.031671+0000", 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.634e-06 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.033231+0000", 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001559726 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.033375+0000", 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014343900000000001 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.033521+0000", 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000146883 2024-06-14T23:04:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.037320+0000", 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037986980000000001 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.040301+0000", 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029806910000000002 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.141120+0000", 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.100819679 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.141193+0000", 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2294999999999998e-05 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.141233+0000", 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9771999999999998e-05 2024-06-14T23:04:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:117254 10.0 10:0f818e93:::benchmark_data_smithi086_189745_object117253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.046716+0000", 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 150.492413989, 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.094660921999999995, 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117254 2024-06-14T23:04:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.046716+0000", 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.046716+0000", 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.046715+0000", 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.046755+0000", 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9762000000000003e-05 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.046763+0000", 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1790000000000008e-06 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.046767+0000", 2024-06-14T23:04:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2910000000000001e-06 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.046822+0000", 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4642999999999998e-05 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.046933+0000", 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110839 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.047049+0000", 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011613 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.050813+0000", 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003763972 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.053420+0000", 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026067310000000002 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.141297+0000", 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.087877046 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.141347+0000", 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0207999999999998e-05 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.141377+0000", 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9768999999999999e-05 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:180728 10.f 10:f758b55e:::benchmark_data_smithi086_189745_object49760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:03:00.018168+0000", 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 83.520961286000002, 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.088104851999999997, 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 180728 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.018168+0000", 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.018168+0000", 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.018166+0000", 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.018175+0000", 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3469999999999999e-06 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.018195+0000", 2024-06-14T23:04:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9901000000000001e-05 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.018201+0000", 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7519999999999999e-06 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.019207+0000", 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010062109999999999 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.019306+0000", 2024-06-14T23:04:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8561999999999996e-05 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.019411+0000", 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105424 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.024324+0000", 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0049129830000000001 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.024603+0000", 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027875799999999999 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.106174+0000", 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081570590999999998 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.106234+0000", 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9908000000000003e-05 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:03:00.106273+0000", 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9533999999999998e-05 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:244152 10.b 10:d619d6ac:::benchmark_data_smithi086_189745_object113184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:08.594002+0000", 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.945127527, 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13457730700000001, 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 244152 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.594002+0000", 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.594002+0000", 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.594001+0000", 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.594006+0000", 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2170000000000002e-06 2024-06-14T23:04:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646264+0000", 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.052258279999999997 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646284+0000", 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9093000000000002e-05 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646336+0000", 2024-06-14T23:04:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2658999999999998e-05 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646464+0000", 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012728300000000001 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646587+0000", 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012330699999999999 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.656046+0000", 2024-06-14T23:04:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094588580000000005 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.656081+0000", 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5385000000000001e-05 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.728497+0000", 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.072416407000000002 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.728539+0000", 2024-06-14T23:04:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1372000000000003e-05 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.728579+0000", 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0602000000000002e-05 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:244157 10.a 10:54e72af8:::benchmark_data_smithi086_189745_object113189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:08.646343+0000", 2024-06-14T23:04:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.892786569, 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.089732518999999997, 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 244157 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646343+0000", 2024-06-14T23:04:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646343+0000", 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646343+0000", 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646346+0000", 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8559999999999998e-06 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646350+0000", 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7169999999999999e-06 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646353+0000", 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9079999999999999e-06 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646600+0000", 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024690999999999998 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646706+0000", 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106006 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646821+0000", 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011434000000000001 2024-06-14T23:04:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.656907+0000", 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010086322999999999 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.656921+0000", 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4039999999999999e-05 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.735998+0000", 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.079076792000000007 2024-06-14T23:04:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736040+0000", 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2688999999999997e-05 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736076+0000", 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5228999999999998e-05 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:244166 10.a 10:589c74f0:::benchmark_data_smithi086_189745_object113198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:08.646432+0000", 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.892697247999999, 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.089737394999999998, 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 244166 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646432+0000", 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646432+0000", 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646432+0000", 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646434+0000", 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.832e-06 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646438+0000", 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3529999999999997e-06 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646441+0000", 2024-06-14T23:04:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0979999999999998e-06 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.651625+0000", 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051839199999999998 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.651724+0000", 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9049000000000007e-05 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.651822+0000", 2024-06-14T23:04:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7968999999999997e-05 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.656954+0000", 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051315309999999999 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.662178+0000", 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052236330000000001 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736112+0000", 2024-06-14T23:04:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.073934435000000007 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736143+0000", 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0596000000000003e-05 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736170+0000", 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7240999999999999e-05 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.586 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:244167 10.a 10:5de433b9:::benchmark_data_smithi086_189745_object113199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:08.646463+0000", 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.892667121000001, 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.089790134999999993, 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 244167 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.587 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646463+0000", 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646463+0000", 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646462+0000", 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:23.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646464+0000", 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0899999999999999e-06 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646470+0000", 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2870000000000004e-06 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.646473+0000", 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2600000000000001e-06 2024-06-14T23:04:23.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.656973+0000", 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010500508 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.657088+0000", 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011462400000000001 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.657204+0000", 2024-06-14T23:04:23.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116165 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.661579+0000", 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004375131 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736083+0000", 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074503933999999994 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736196+0000", 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113131 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736226+0000", 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9266999999999999e-05 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736253+0000", 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7046000000000002e-05 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.592 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:244176 10.a 10:51cacdfb:::benchmark_data_smithi086_189745_object113208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:08.663203+0000", 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.875926707, 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.087058265999999995, 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 244176 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.593 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.663203+0000", 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.663203+0000", 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.663202+0000", 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.663210+0000", 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4710000000000001e-06 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.663221+0000", 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1199999999999999e-05 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.663226+0000", 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.305e-06 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.663271+0000", 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4419000000000003e-05 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.728468+0000", 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.065197292000000004 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:04:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.728591+0000", 2024-06-14T23:04:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123418 2024-06-14T23:04:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.736284+0000", 2024-06-14T23:04:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076925969999999998 2024-06-14T23:04:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:23.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.750098+0000", 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013814659 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.750174+0000", 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5014999999999999e-05 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.750219+0000", 2024-06-14T23:04:23.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5685000000000001e-05 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:08.750261+0000", 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2030000000000002e-05 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252320 10.b 10:d99e9314:::benchmark_data_smithi086_189745_object121352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.026998+0000", 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.5121314630000002, 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.094222763000000001, 2024-06-14T23:04:23.600 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252320 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026998+0000", 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026998+0000", 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026997+0000", 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.027002+0000", 2024-06-14T23:04:23.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2809999999999999e-06 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.027014+0000", 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1204e-05 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.027019+0000", 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0939999999999998e-06 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.027061+0000", 2024-06-14T23:04:23.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2020000000000001e-05 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.027135+0000", 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4297999999999999e-05 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.116886+0000", 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.089751422999999997 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.120829+0000", 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039425470000000002 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.120835+0000", 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.964e-06 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121124+0000", 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000289449 2024-06-14T23:04:23.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121171+0000", 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7074999999999999e-05 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121221+0000", 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9567999999999997e-05 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252321 10.f 10:fe0bdbfa:::benchmark_data_smithi086_189745_object121353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.029616+0000", 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.5095131889999998, 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.093090397000000005, 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252321 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.029616+0000", 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.029616+0000", 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.029615+0000", 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:04:23.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.029626+0000", 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1749999999999999e-05 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.117946+0000", 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.088319534000000005 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.117958+0000", 2024-06-14T23:04:23.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2473999999999999e-05 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.118014+0000", 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5581000000000002e-05 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.118152+0000", 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013826800000000001 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3,4", 2024-06-14T23:04:23.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.118285+0000", 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013301900000000001 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.122507+0000", 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042221660000000003 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.122514+0000", 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.302e-06 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.122540+0000", 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6465000000000002e-05 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.122568+0000", 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7789000000000001e-05 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.122614+0000", 2024-06-14T23:04:23.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5534000000000001e-05 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.122707+0000", 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3305000000000002e-05 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252324 10.a 10:5688e3e5:::benchmark_data_smithi086_189745_object121356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.121320+0000", 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.417809691, 2024-06-14T23:04:23.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.143980313, 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252324 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121320+0000", 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121320+0000", 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121318+0000", 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121325+0000", 2024-06-14T23:04:23.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6109999999999999e-06 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121337+0000", 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2096000000000001e-05 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121343+0000", 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1920000000000002e-06 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121390+0000", 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6638999999999998e-05 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121500+0000", 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010990899999999999 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121608+0000", 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010861000000000001 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.128821+0000", 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072121140000000004 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.128829+0000", 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6710000000000005e-06 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136485+0000", 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076560040000000001 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265260+0000", 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12877435700000001 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265300+0000", 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0717999999999999e-05 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.619 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252325 10.a 10:5177a615:::benchmark_data_smithi086_189745_object121357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.121659+0000", 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.4174705350000001, 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.143766909, 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.620 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252325 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121659+0000", 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121659+0000", 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121658+0000", 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121663+0000", 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1760000000000003e-06 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121670+0000", 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8089999999999993e-06 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.121676+0000", 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3260000000000003e-06 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.125185+0000", 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035087650000000001 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.125285+0000", 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100757 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.125388+0000", 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010282500000000001 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.129893+0000", 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045044639999999997 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132140+0000", 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002247626 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265341+0000", 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13320047400000001 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265393+0000", 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2797000000000003e-05 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265426+0000", 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2543000000000003e-05 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.625 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252329 10.7 10:ef772d32:::benchmark_data_smithi086_189745_object121361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.125495+0000", 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.413634295, 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14390391399999999, 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.626 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252329 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.125495+0000", 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.125495+0000", 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.125495+0000", 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.125497+0000", 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5790000000000002e-06 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.125503+0000", 2024-06-14T23:04:23.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6420000000000004e-06 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.125506+0000", 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3299999999999999e-06 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131395+0000", 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058880570000000004 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131489+0000", 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4041000000000003e-05 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131585+0000", 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6354999999999995e-05 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136342+0000", 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047575780000000002 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265330+0000", 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.128987034 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.269325+0000", 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039956890000000002 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.269367+0000", 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1613000000000003e-05 2024-06-14T23:04:23.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.269399+0000", 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2453999999999998e-05 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252331 10.d 10:b279c64c:::benchmark_data_smithi086_189745_object121363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.131251+0000", 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.4078783769999998, 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.137810236, 2024-06-14T23:04:23.632 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252331 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131251+0000", 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131251+0000", 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131250+0000", 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131256+0000", 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7900000000000002e-06 2024-06-14T23:04:23.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131270+0000", 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3105000000000001e-05 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131276+0000", 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2380000000000002e-06 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131329+0000", 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3696999999999998e-05 2024-06-14T23:04:23.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131452+0000", 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012256800000000001 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.131585+0000", 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013302 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:23.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265269+0000", 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13368360400000001 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265302+0000", 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3019999999999999e-05 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.268976+0000", 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036743280000000001 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.269023+0000", 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7281999999999999e-05 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.269062+0000", 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8243e-05 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.45821.0:252324 10.a e224/186 ondisk, result = 0)", 2024-06-14T23:04:23.638 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.136412+0000", 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.4027179840000001, 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.128892067, 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136412+0000", 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136412+0000", 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136411+0000", 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136419+0000", 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9789999999999993e-06 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136428+0000", 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8160000000000005e-06 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136434+0000", 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4040000000000003e-06 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136472+0000", 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8009999999999997e-05 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136479+0000", 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.605e-06 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265304+0000", 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12882433600000001 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.642 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.45821.0:252325 10.a e224/186 ondisk, result = 0)", 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.136448+0000", 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.4026821350000001, 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.128984034, 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136448+0000", 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136448+0000", 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136447+0000", 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136450+0000", 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8459999999999999e-06 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136456+0000", 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0249999999999999e-06 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136460+0000", 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4350000000000001e-06 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265328+0000", 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12886846299999999 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265334+0000", 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5099999999999998e-06 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265432+0000", 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8027000000000002e-05 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:252334 10.e e224/219 10:703c40e1:::benchmark_data_smithi086_189745_object121366:head v 224'15885, mlcod=223'15883)", 2024-06-14T23:04:23.646 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.136813+0000", 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.4023162579999999, 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.147872484, 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136813+0000", 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136813+0000", 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136813+0000", 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.136819+0000", 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1489999999999999e-06 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265218+0000", 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12839888699999999 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265228+0000", 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0104e-05 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265286+0000", 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8505999999999999e-05 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265311+0000", 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5037000000000001e-05 2024-06-14T23:04:23.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.284661+0000", 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019349321999999999 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.284686+0000", 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5162000000000001e-05 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.45821.0:252329 10.7 e224/186 ondisk, result = 0)", 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.137552+0000", 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.4015780420000001, 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12778114400000001, 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137552+0000", 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137552+0000", 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137551+0000", 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137555+0000", 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.425e-06 2024-06-14T23:04:23.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265213+0000", 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12765797300000001 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265223+0000", 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.183e-06 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265318+0000", 2024-06-14T23:04:23.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5624999999999993e-05 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265324+0000", 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6659999999999996e-06 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265333+0000", 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8480000000000007e-06 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.654 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.45821.0:252331 10.d e224/186 ondisk, result = 0)", 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.139774+0000", 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.3993551699999998, 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12552904200000001, 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139774+0000", 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139774+0000", 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139773+0000", 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139781+0000", 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8760000000000005e-06 2024-06-14T23:04:23.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265248+0000", 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12546685499999999 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265257+0000", 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4610000000000008e-06 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265289+0000", 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1948e-05 2024-06-14T23:04:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265296+0000", 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.294e-06 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.265304+0000", 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6109999999999996e-06 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.658 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:23.659 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:23.659 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:04:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:23.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:23.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:23.808 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:23.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:23.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:23.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:23.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:24.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:24.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:24.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:24.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:24.201 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:24.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:24.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:24.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:24.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:24.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:24.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:24.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-14T23:04:24.492 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:24.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:24.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:24.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:24.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:24.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:24.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:24.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:24.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:24.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:24.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:24.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:24.958 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:24.970 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:04:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:04:24] "GET /metrics HTTP/1.1" 200 36585 "" "Prometheus/2.43.0" 2024-06-14T23:04:25.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:25.033 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:25.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:25.063 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:25.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:25.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:25.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:25.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:25.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:24 smithi137 ceph-mon[90634]: osdmap e227: 8 total, 8 up, 7 in 2024-06-14T23:04:25.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:24 smithi137 ceph-mon[90634]: pgmap v894: 99 pgs: 1 peering, 98 active+clean; 569 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 415 op/s; 46 MiB/s, 730 objects/s recovering 2024-06-14T23:04:25.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:25.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:25.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:25.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:24 smithi086 ceph-mon[97913]: osdmap e227: 8 total, 8 up, 7 in 2024-06-14T23:04:25.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:24 smithi086 ceph-mon[97913]: pgmap v894: 99 pgs: 1 peering, 98 active+clean; 569 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 415 op/s; 46 MiB/s, 730 objects/s recovering 2024-06-14T23:04:25.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:24 smithi086 ceph-mon[102754]: osdmap e227: 8 total, 8 up, 7 in 2024-06-14T23:04:25.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:24 smithi086 ceph-mon[102754]: pgmap v894: 99 pgs: 1 peering, 98 active+clean; 569 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 415 op/s; 46 MiB/s, 730 objects/s recovering 2024-06-14T23:04:25.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:25.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:25.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:25.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:25.549 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:25.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:25.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:25.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:25.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:25.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:25.816 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:25.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:25.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:25.944 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:25.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:26.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:26.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:26.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:26.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:26.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:26.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:26.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:26.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:26.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:26.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:26.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:26.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:26.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:26.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:26.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:26.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:26.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:26.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:26.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:26.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:26.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:26.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:27.008 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:27.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:27.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:27.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:27.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:27.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:27.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:27.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:27.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:27.414 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:27.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:27.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:27.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:27.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:27.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:27.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:27.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:27.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:27.814 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:27.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:27.833 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:04:27.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:27.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:27.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:28.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:28.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[102754]: pgmap v895: 99 pgs: 1 peering, 98 active+clean; 561 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 304 op/s; 31 MiB/s, 500 objects/s recovering 2024-06-14T23:04:28.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2708959967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:28.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:28.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:28.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[97913]: pgmap v895: 99 pgs: 1 peering, 98 active+clean; 561 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 304 op/s; 31 MiB/s, 500 objects/s recovering 2024-06-14T23:04:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2708959967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:28.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:28.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:28.189 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:27 smithi137 ceph-mon[90634]: pgmap v895: 99 pgs: 1 peering, 98 active+clean; 561 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 304 op/s; 31 MiB/s, 500 objects/s recovering 2024-06-14T23:04:28.189 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2708959967' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:28.189 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:28.190 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:28.190 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:28.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:28.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:28.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:28.228 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:04:28.229 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:04:28.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.229 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:256611 10.c 10:36c7b28a:::benchmark_data_smithi086_189745_object125643:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-06-14T23:04:28.229 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:28.189111+0000", 2024-06-14T23:04:28.229 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.037932644000000001, 2024-06-14T23:04:28.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037961808, 2024-06-14T23:04:28.230 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:28.230 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:04:28.230 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:04:28.230 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:04:28.231 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:04:28.231 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 256611 2024-06-14T23:04:28.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.231 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:28.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:28.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.189111+0000", 2024-06-14T23:04:28.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:28.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:28.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.189111+0000", 2024-06-14T23:04:28.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:28.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:28.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.189108+0000", 2024-06-14T23:04:28.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999976 2024-06-14T23:04:28.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:28.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.189126+0000", 2024-06-14T23:04:28.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8025000000000001e-05 2024-06-14T23:04:28.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:28.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.189146+0000", 2024-06-14T23:04:28.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9565999999999999e-05 2024-06-14T23:04:28.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:28.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.189153+0000", 2024-06-14T23:04:28.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4309999999999997e-06 2024-06-14T23:04:28.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:28.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.190442+0000", 2024-06-14T23:04:28.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001288389 2024-06-14T23:04:28.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:28.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.190548+0000", 2024-06-14T23:04:28.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010609 2024-06-14T23:04:28.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:04:28.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.190666+0000", 2024-06-14T23:04:28.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117828 2024-06-14T23:04:28.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:04:28.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.199475+0000", 2024-06-14T23:04:28.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008809496 2024-06-14T23:04:28.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:04:28.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.202578+0000", 2024-06-14T23:04:28.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031029930000000001 2024-06-14T23:04:28.241 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:28.241 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:28.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:28.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.242 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:256634 10.9 e228/224 10:9cf3147c:::benchmark_data_smithi086_189745_object125666:head v 228'15815, mlcod=228'15813)", 2024-06-14T23:04:28.242 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:28.221945+0000", 2024-06-14T23:04:28.242 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0050982789999999998, 2024-06-14T23:04:28.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052121190000000003, 2024-06-14T23:04:28.243 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:28.243 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:04:28.243 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:28.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.243 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:28.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.221945+0000", 2024-06-14T23:04:28.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:28.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.221945+0000", 2024-06-14T23:04:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:28.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.221944+0000", 2024-06-14T23:04:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:04:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.221953+0000", 2024-06-14T23:04:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1619999999999997e-06 2024-06-14T23:04:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.221973+0000", 2024-06-14T23:04:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0458e-05 2024-06-14T23:04:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.221980+0000", 2024-06-14T23:04:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8419999999999999e-06 2024-06-14T23:04:28.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.222129+0000", 2024-06-14T23:04:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014894800000000001 2024-06-14T23:04:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.222174+0000", 2024-06-14T23:04:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4716999999999999e-05 2024-06-14T23:04:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:28.250 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:28.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.251 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:256640 10.1 e228/186 10:8528966f:::benchmark_data_smithi086_189745_object125672:head v 228'16054, mlcod=228'16052)", 2024-06-14T23:04:28.251 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:28.226078+0000", 2024-06-14T23:04:28.251 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.000965769, 2024-06-14T23:04:28.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001132547, 2024-06-14T23:04:28.252 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:04:28.252 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:04:28.252 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:04:28.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:04:28.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.226078+0000", 2024-06-14T23:04:28.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:28.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:04:28.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.226078+0000", 2024-06-14T23:04:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:04:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:04:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.226076+0000", 2024-06-14T23:04:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:04:28.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:04:28.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.226085+0000", 2024-06-14T23:04:28.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4970000000000001e-06 2024-06-14T23:04:28.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:04:28.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.226104+0000", 2024-06-14T23:04:28.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8640000000000001e-05 2024-06-14T23:04:28.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:04:28.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.226110+0000", 2024-06-14T23:04:28.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6259999999999997e-06 2024-06-14T23:04:28.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:04:28.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.226173+0000", 2024-06-14T23:04:28.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2785000000000003e-05 2024-06-14T23:04:28.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:04:28.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:04:28.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:04:28.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:28.226195+0000", 2024-06-14T23:04:28.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2175e-05 2024-06-14T23:04:28.259 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:28.259 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:04:28.259 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:28.259 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:04:28.259 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T23:04:28.260 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 3 2024-06-14T23:04:28.260 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:04:28.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:28.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:28.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:28.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:28.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:28.504 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:28.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:28.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:28.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:28.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:28.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:28.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:28.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:28.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:28.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:28.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:28.886 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:04:28.886 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:04:28.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:28.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:28.919 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:04:28.919 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:04:28.919 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:28 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:28.919 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2708959967' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:04:28.920 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:28 smithi086 ceph-mon[102754]: osdmap e228: 8 total, 8 up, 7 in 2024-06-14T23:04:28.920 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:28 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:28.920 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2708959967' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:04:28.920 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:28 smithi086 ceph-mon[97913]: osdmap e228: 8 total, 8 up, 7 in 2024-06-14T23:04:28.951 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:04:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:04:28.976 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:04:28.976 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:04:28.980 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:28 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:28.980 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2708959967' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:04:28.980 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:28 smithi137 ceph-mon[90634]: osdmap e228: 8 total, 8 up, 7 in 2024-06-14T23:04:29.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:29.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-14T23:04:29.032 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:04:29.033 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:04:29.048 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:29.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:29.065 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:04:29.065 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:04:29.095 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:04:29.095 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:04:29.116 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:04:29.116 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:04:29.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:29.170 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:04:29.170 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:04:29.171 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-06-14T23:04:29.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd in 2 2024-06-14T23:04:29.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:29.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:29.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:29.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:29.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:29.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:29.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:29.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:29.488 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:29.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:29.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:29.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:29.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:29.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:29.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:29.870 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:29.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:29.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:29.990 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:29 smithi137 ceph-mon[90634]: pgmap v897: 99 pgs: 99 active+clean; 522 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 314 op/s; 21 MiB/s, 331 objects/s recovering 2024-06-14T23:04:29.991 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:29 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3110128189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:29.997 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:29.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:30.011 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:29 smithi086 ceph-mon[97913]: pgmap v897: 99 pgs: 99 active+clean; 522 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 314 op/s; 21 MiB/s, 331 objects/s recovering 2024-06-14T23:04:30.011 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:29 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3110128189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:30.012 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:29 smithi086 ceph-mon[102754]: pgmap v897: 99 pgs: 99 active+clean; 522 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 314 op/s; 21 MiB/s, 331 objects/s recovering 2024-06-14T23:04:30.012 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:29 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3110128189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:30.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:30.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:30.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:30.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:30.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:30.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:30.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:30.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:30.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:30.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:30.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:30.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:30.588 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:30.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:30.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:30.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:30.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:30.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:30.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:30.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:30.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:30.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:30.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:31.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:31.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:31.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:31.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:31.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:31.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:31.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:31.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:31.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:31.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:04:31.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:31.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:31.532 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:04:31.532 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:04:31.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:31.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:31.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:31.732 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:31.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:31.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:31.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:31.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:32.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:32.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:32.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:32.146 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:31 smithi086 ceph-mon[97913]: pgmap v898: 99 pgs: 99 active+clean; 291 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 16 MiB/s, 254 objects/s recovering 2024-06-14T23:04:32.147 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:31 smithi086 ceph-mon[102754]: pgmap v898: 99 pgs: 99 active+clean; 291 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 16 MiB/s, 254 objects/s recovering 2024-06-14T23:04:32.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:32.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:32.183 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:31 smithi137 ceph-mon[90634]: pgmap v898: 99 pgs: 99 active+clean; 291 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 16 MiB/s, 254 objects/s recovering 2024-06-14T23:04:32.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-14T23:04:32.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:32.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:32.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:32.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:32.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:32.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:32.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:32.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:04:32.551 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:32.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:32.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:32.689 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:32.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:32.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:32.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:32.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:32.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:32.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:33.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:33.048 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:33.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:33.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:32 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:04:32.919+0000 7fe0161db640 -1 mon.a@0(leader).osd e228 definitely_dead 0 2024-06-14T23:04:33.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:33.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:33.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:33.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:33.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:33.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:33.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:33.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:33.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:33.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:33.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:33.470 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:33.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:33.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:33.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:33.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:33.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:33.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:33.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:33.873 INFO:teuthology.orchestra.run.smithi086.stderr:marked in osd.2. 2024-06-14T23:04:33.892 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:33.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:33.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:33.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:33 smithi137 ceph-mon[90634]: pgmap v899: 99 pgs: 99 active+clean; 237 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 731 op/s; 14 MiB/s, 225 objects/s recovering 2024-06-14T23:04:33.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/738351485' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-14T23:04:33.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:33 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-14T23:04:34.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:34.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:34.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:34.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:34.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:34.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:34.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:34.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:34.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:34.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:34.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:33 smithi086 ceph-mon[97913]: pgmap v899: 99 pgs: 99 active+clean; 237 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 731 op/s; 14 MiB/s, 225 objects/s recovering 2024-06-14T23:04:34.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/738351485' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-14T23:04:34.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:33 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-14T23:04:34.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:33 smithi086 ceph-mon[102754]: pgmap v899: 99 pgs: 99 active+clean; 237 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 731 op/s; 14 MiB/s, 225 objects/s recovering 2024-06-14T23:04:34.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/738351485' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-14T23:04:34.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:33 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-14T23:04:34.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:34.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:34.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:34.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:34.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:34.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:34.533 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:04:34.533 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:04:34.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:34.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:34.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:34.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:34.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:34.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:34.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:34.896 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:34.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:34.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:34 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-06-14T23:04:34.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:34 smithi086 ceph-mon[97913]: osdmap e229: 8 total, 8 up, 8 in 2024-06-14T23:04:34.930 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:04:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:04:34] "GET /metrics HTTP/1.1" 200 36614 "" "Prometheus/2.43.0" 2024-06-14T23:04:34.931 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:34 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-06-14T23:04:34.931 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:34 smithi086 ceph-mon[102754]: osdmap e229: 8 total, 8 up, 8 in 2024-06-14T23:04:34.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:35.018 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:35.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:35.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:35.133 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:34 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-06-14T23:04:35.133 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:34 smithi137 ceph-mon[90634]: osdmap e229: 8 total, 8 up, 8 in 2024-06-14T23:04:35.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:35.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:35.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-14T23:04:35.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:35.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:35.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:35.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:35.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:35.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:35.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:35.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:35.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:35.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:35.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:35.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:35.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:35.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:35.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:35.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:35.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:35.879 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:04:35.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:35.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:35.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:36.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:36.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:36.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:36.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:36.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:36.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:36.195 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 130965 objects 2024-06-14T23:04:36.196 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :148.961 2024-06-14T23:04:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:35 smithi137 ceph-mon[90634]: pgmap v901: 99 pgs: 15 peering, 84 active+clean; 50 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-06-14T23:04:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:35 smithi137 ceph-mon[90634]: osdmap e230: 8 total, 8 up, 8 in 2024-06-14T23:04:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/610220822' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:04:36.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:36.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:36.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:35 smithi086 ceph-mon[97913]: pgmap v901: 99 pgs: 15 peering, 84 active+clean; 50 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-06-14T23:04:36.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:35 smithi086 ceph-mon[97913]: osdmap e230: 8 total, 8 up, 8 in 2024-06-14T23:04:36.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/610220822' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:04:36.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:35 smithi086 ceph-mon[102754]: pgmap v901: 99 pgs: 15 peering, 84 active+clean; 50 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-06-14T23:04:36.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:35 smithi086 ceph-mon[102754]: osdmap e230: 8 total, 8 up, 8 in 2024-06-14T23:04:36.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/610220822' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:04:36.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:36.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:36.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:36.464 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-06-14T23:04:36.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:36.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:36.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:36.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:36.635 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:36.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:36.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:36.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:36.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:36.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:36.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:37.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:37.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:37.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:37.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[97913]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16478 2024-06-14T23:04:37.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[97913]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 228'16507 2024-06-14T23:04:37.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/610220822' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:04:37.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[97913]: osdmap e231: 8 total, 8 up, 8 in 2024-06-14T23:04:37.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[97913]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16215 2024-06-14T23:04:37.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[97913]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16786 2024-06-14T23:04:37.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[97913]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 228'15954 2024-06-14T23:04:37.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[97913]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16137 2024-06-14T23:04:37.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:37.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[102754]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16478 2024-06-14T23:04:37.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[102754]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 228'16507 2024-06-14T23:04:37.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/610220822' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:04:37.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[102754]: osdmap e231: 8 total, 8 up, 8 in 2024-06-14T23:04:37.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[102754]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16215 2024-06-14T23:04:37.089 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[102754]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16786 2024-06-14T23:04:37.089 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[102754]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 228'15954 2024-06-14T23:04:37.089 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:36 smithi086 ceph-mon[102754]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16137 2024-06-14T23:04:37.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:37.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:37.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:37.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:36 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:36 smithi137 ceph-mon[90634]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16478 2024-06-14T23:04:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:36 smithi137 ceph-mon[90634]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 228'16507 2024-06-14T23:04:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/610220822' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:04:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:36 smithi137 ceph-mon[90634]: osdmap e231: 8 total, 8 up, 8 in 2024-06-14T23:04:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:36 smithi137 ceph-mon[90634]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16215 2024-06-14T23:04:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:36 smithi137 ceph-mon[90634]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16786 2024-06-14T23:04:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:36 smithi137 ceph-mon[90634]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 228'15954 2024-06-14T23:04:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:36 smithi137 ceph-mon[90634]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 228'16137 2024-06-14T23:04:37.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:37.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:37.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:37.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:37.399 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:37.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:37.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:37.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:37.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:37.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:37.638 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:04:37.639 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:04:37.639 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:04:37.639 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:04:37.648 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:04:37.648 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:04:37.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:37.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:37.665 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:04:37.665 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:04:37.681 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:04:37.681 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:04:37.697 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:04:37.698 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:04:37.714 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:04:37.714 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:04:37.730 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:04:37.730 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:04:37.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:37.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:37.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:37.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:37.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:37.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:37.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:38.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:38.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:38.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:38.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:38.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:37 smithi137 ceph-mon[90634]: pgmap v904: 99 pgs: 15 peering, 84 active+clean; 35 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 683 op/s 2024-06-14T23:04:38.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:37 smithi137 ceph-mon[90634]: osdmap e232: 8 total, 8 up, 8 in 2024-06-14T23:04:38.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/517566123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:38.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:37 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:38.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:38.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:38.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:37 smithi086 ceph-mon[97913]: pgmap v904: 99 pgs: 15 peering, 84 active+clean; 35 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 683 op/s 2024-06-14T23:04:38.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:37 smithi086 ceph-mon[97913]: osdmap e232: 8 total, 8 up, 8 in 2024-06-14T23:04:38.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/517566123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:38.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:37 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:38.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:37 smithi086 ceph-mon[102754]: pgmap v904: 99 pgs: 15 peering, 84 active+clean; 35 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 683 op/s 2024-06-14T23:04:38.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:37 smithi086 ceph-mon[102754]: osdmap e232: 8 total, 8 up, 8 in 2024-06-14T23:04:38.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/517566123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:38.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:37 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:38.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:38.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:38.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:38.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:38.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:38.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:38.540 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:38.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:38.635 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:38.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:38.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:38.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:38.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:38.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:38.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:38.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:38.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:39.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:39.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:39.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:39.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:39.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:39.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:04:39.197 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:38 smithi086 ceph-mon[97913]: osdmap e233: 8 total, 8 up, 8 in 2024-06-14T23:04:39.198 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:38 smithi086 ceph-mon[102754]: osdmap e233: 8 total, 8 up, 8 in 2024-06-14T23:04:39.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:39.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:38 smithi137 ceph-mon[90634]: osdmap e233: 8 total, 8 up, 8 in 2024-06-14T23:04:39.250 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:39.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:39.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:39.375 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:39.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:39.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:39.499 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:39.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:39.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:39.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:39.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:39.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:39.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:39.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:39.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:39.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:39.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:39.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:39.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:39.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:39.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:40.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:39 smithi086 ceph-mon[97913]: pgmap v907: 99 pgs: 16 peering, 83 active+clean; 23 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 107 op/s; 142 B/s, 0 objects/s recovering 2024-06-14T23:04:40.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:39 smithi086 ceph-mon[102754]: pgmap v907: 99 pgs: 16 peering, 83 active+clean; 23 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 107 op/s; 142 B/s, 0 objects/s recovering 2024-06-14T23:04:40.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:40.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:40.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:40.160 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:39 smithi137 ceph-mon[90634]: pgmap v907: 99 pgs: 16 peering, 83 active+clean; 23 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 107 op/s; 142 B/s, 0 objects/s recovering 2024-06-14T23:04:40.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:40.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:40.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:40.336 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:04:40.337 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:04:40.337 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-06-14T23:04:40.337 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:04:40.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:40.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:40.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:40.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:40.492 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:40.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:40.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:04:40.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:40.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:40.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:40.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:40.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:40.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:40.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:40.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:40.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:40.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:40.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:40.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:41.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-14T23:04:41.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:41.088 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:41.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:41.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:41.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:41.220 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:40 smithi086 ceph-mon[97913]: 1.9 scrub starts 2024-06-14T23:04:41.220 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:40 smithi086 ceph-mon[97913]: 1.9 scrub ok 2024-06-14T23:04:41.220 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:40 smithi086 ceph-mon[102754]: 1.9 scrub starts 2024-06-14T23:04:41.220 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:40 smithi086 ceph-mon[102754]: 1.9 scrub ok 2024-06-14T23:04:41.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:41.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:41.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:40 smithi137 ceph-mon[90634]: 1.9 scrub starts 2024-06-14T23:04:41.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:40 smithi137 ceph-mon[90634]: 1.9 scrub ok 2024-06-14T23:04:41.329 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-06-14T23:04:41.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-06-14T23:04:41.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:41.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:41.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:41.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:41.464 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:04:41.464 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:04:41.465 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-06-14T23:04:41.465 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:04:41.465 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:04:41.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-06-14T23:04:41.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:41.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:41.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:41.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:41.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:41.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:41.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:41.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:41.734 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:41.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:41.827 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:41.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:41.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:41.859 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:41.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:41.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:41.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:42.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:42.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:42.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:42.146 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:41 smithi137 ceph-mon[90634]: pgmap v908: 99 pgs: 1 peering, 98 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 133 op/s; 65 KiB/s, 1 keys/s, 2 objects/s recovering 2024-06-14T23:04:42.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:42.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:41 smithi086 ceph-mon[102754]: pgmap v908: 99 pgs: 1 peering, 98 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 133 op/s; 65 KiB/s, 1 keys/s, 2 objects/s recovering 2024-06-14T23:04:42.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:41 smithi086 ceph-mon[97913]: pgmap v908: 99 pgs: 1 peering, 98 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 133 op/s; 65 KiB/s, 1 keys/s, 2 objects/s recovering 2024-06-14T23:04:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:42.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:42.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:42.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:42.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:42.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:42.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:42.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:42.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:42.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:42.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:42.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:42.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:42.712 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:42.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:42.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:42.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:42.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:42.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:42.956 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:42.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:42.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:43.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:43.079 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:43.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:43.154 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:43.154 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:43.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:43.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:43.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:43.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:43.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:43.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:43.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:43.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:43.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:43.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:43.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:43.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:43.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:43.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:43.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:43.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:43.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:43.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:43.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:43.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:43.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:43.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:43.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:43.969 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:04:43.969 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "" 2024-06-14T23:04:43.969 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:04:44.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:44.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:44.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:44.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:44.170 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:43 smithi086 ceph-mon[97913]: pgmap v909: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 79 op/s; 56 KiB/s, 1 keys/s, 1 objects/s recovering 2024-06-14T23:04:44.170 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:43 smithi086 ceph-mon[102754]: pgmap v909: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 79 op/s; 56 KiB/s, 1 keys/s, 1 objects/s recovering 2024-06-14T23:04:44.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:44.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:44.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:43 smithi137 ceph-mon[90634]: pgmap v909: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 79 op/s; 56 KiB/s, 1 keys/s, 1 objects/s recovering 2024-06-14T23:04:44.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:44.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:44.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:44.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:44.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:44.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:44.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:44.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:44.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:44.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:44.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:44.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:44.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:44.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:44.753 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:44.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:44.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:44.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:44.921 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:04:44.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:44.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:45.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:45.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:45.150 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3480301734' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:45.150 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3480301734' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:45.150 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:04:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:04:44] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.43.0" 2024-06-14T23:04:45.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:45.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:45.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:45.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3480301734' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:04:45.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:45.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:45.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:45.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:45.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:45.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:45.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:45.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:45.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:45.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:45.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:45.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:45.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:45.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:45.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:45.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:46.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:46.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:46.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:46.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[97913]: pgmap v910: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 68 op/s; 48 KiB/s, 1 keys/s, 1 objects/s recovering 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3480301734' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[97913]: osdmap e234: 8 total, 8 up, 8 in 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[102754]: pgmap v910: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 68 op/s; 48 KiB/s, 1 keys/s, 1 objects/s recovering 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3480301734' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:04:46.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:45 smithi086 ceph-mon[102754]: osdmap e234: 8 total, 8 up, 8 in 2024-06-14T23:04:46.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:46.167 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:46.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:45 smithi137 ceph-mon[90634]: pgmap v910: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 68 op/s; 48 KiB/s, 1 keys/s, 1 objects/s recovering 2024-06-14T23:04:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:45 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:04:46.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:45 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:04:46.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3480301734' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:04:46.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:45 smithi137 ceph-mon[90634]: osdmap e234: 8 total, 8 up, 8 in 2024-06-14T23:04:46.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:46.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:46.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:46.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:46.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:46.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:46.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:46.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:46.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:46.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:46.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:46.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:46.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:46.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:46.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:46.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:46.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:46.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:46.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:47.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:47.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:47.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:47.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:47.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:47.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:47.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:47.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:47.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:47.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:47.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:47.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:47.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:47.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:47.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:47.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:47.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:47.673 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:47.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:47.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:47.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:47.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:47.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:47.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:47.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:48.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:48.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:48.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:48.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:47 smithi086 ceph-mon[102754]: pgmap v912: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 40 op/s; 43 KiB/s, 1 keys/s, 1 objects/s recovering 2024-06-14T23:04:48.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:47 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:48.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:47 smithi086 ceph-mon[97913]: pgmap v912: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 40 op/s; 43 KiB/s, 1 keys/s, 1 objects/s recovering 2024-06-14T23:04:48.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:47 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:48.119 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:04:48.119 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:04:48.136 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:04:48.136 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:04:48.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:48.152 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:04:48.152 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:04:48.169 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:04:48.169 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:04:48.171 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:48.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:48.186 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:04:48.186 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:04:48.202 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:04:48.203 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:04:48.218 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:04:48.219 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:04:48.235 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:04:48.235 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:04:48.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:47 smithi137 ceph-mon[90634]: pgmap v912: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 40 op/s; 43 KiB/s, 1 keys/s, 1 objects/s recovering 2024-06-14T23:04:48.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:47 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:04:48.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:48.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:48.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:48.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:48.345 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:04:48.345 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:04:48.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:48.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:48.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:48.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:04:48.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:48.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:48.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:48.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:48.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:48.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:48.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:48.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:48.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:48.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:48.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:49.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:49.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:49.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:49.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:49.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:49.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:49.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/417980826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:49.225 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/417980826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:49.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/417980826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:04:49.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-14T23:04:49.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:49.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:49.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:49.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:49.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:49.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:49.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:49.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:49.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:49.640 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:04:49.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:04:49.640 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-06-14T23:04:49.641 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:04:49.641 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:04:49.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-06-14T23:04:49.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:49.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:49.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:49.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:49.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:49.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:49.907 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:04:49.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:49.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:49.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:50.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:50.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:50.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:50.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:50.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:50.202 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:49 smithi086 ceph-mon[97913]: pgmap v913: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 35 op/s; 38 KiB/s, 0 keys/s, 0 objects/s recovering 2024-06-14T23:04:50.202 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:49 smithi086 ceph-mon[102754]: pgmap v913: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 35 op/s; 38 KiB/s, 0 keys/s, 0 objects/s recovering 2024-06-14T23:04:50.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:50.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:50.222 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:50.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:49 smithi137 ceph-mon[90634]: pgmap v913: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 35 op/s; 38 KiB/s, 0 keys/s, 0 objects/s recovering 2024-06-14T23:04:50.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:50.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:50.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:50.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:50.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:50.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:50.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:50.579 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:50.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:50.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:50.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:50.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:50.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:50.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:50.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:50.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:50.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:51.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:51.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:51.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:51.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:51.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:51.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:51.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:51.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:51.372 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:51.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:51.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:51.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:51.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:51.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:51.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:51.592 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:04:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-14T23:04:51.592 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:04:51.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:51.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:51.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:51.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:51.811 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:51.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:51.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:51.932 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_5' removed 2024-06-14T23:04:51.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:51.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:52.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:52.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:52.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:52.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:52.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:52.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:52.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:04:52.205 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:51 smithi137 ceph-mon[90634]: pgmap v914: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:52.206 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1367755695' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:04:52.206 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:04:52.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:51 smithi086 ceph-mon[97913]: pgmap v914: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:52.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1367755695' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:04:52.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:04:52.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:51 smithi086 ceph-mon[102754]: pgmap v914: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:52.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1367755695' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:04:52.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:04:52.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:52.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:52.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:52.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:52.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:52.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:52.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:52.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:52.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:52.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:52.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:52.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:52.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:52.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:52.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:52.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:52.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:52.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:53.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:53.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:53.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:53.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:52 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:04:53.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:52 smithi086 ceph-mon[102754]: osdmap e235: 8 total, 8 up, 8 in 2024-06-14T23:04:53.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:52 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:04:53.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:52 smithi086 ceph-mon[97913]: osdmap e235: 8 total, 8 up, 8 in 2024-06-14T23:04:53.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:53.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:53.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:53.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:52 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:04:53.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:52 smithi137 ceph-mon[90634]: osdmap e235: 8 total, 8 up, 8 in 2024-06-14T23:04:53.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:53.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:53.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:53.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:53.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:53.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:53.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:53.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:53.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:53.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:53.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:53.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:53.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:53.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:53.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:53.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:53.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:53.953 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:04:54.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:54.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:54.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:54.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:53 smithi086 ceph-mon[97913]: pgmap v916: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:54.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4284751579' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:04:54.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:53 smithi086 ceph-mon[102754]: pgmap v916: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:54.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4284751579' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:04:54.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:54.174 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:54.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:54.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:53 smithi137 ceph-mon[90634]: pgmap v916: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:54.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4284751579' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:04:54.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:54.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:54.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:54.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:54.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:54.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:54.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:54.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:54.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:54.631 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:04:54.631 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:04:54.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:54.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:54.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:54.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:54.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:54.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:54.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:54.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:54.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:55.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:55.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:55.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:55.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:55.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:55.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:55.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:54 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:55.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:54 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4284751579' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:04:55.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:54 smithi137 ceph-mon[90634]: osdmap e236: 8 total, 8 up, 8 in 2024-06-14T23:04:55.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:55.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:55.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:55.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:04:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:04:54] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.43.0" 2024-06-14T23:04:55.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:54 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:55.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4284751579' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:04:55.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:54 smithi086 ceph-mon[102754]: osdmap e236: 8 total, 8 up, 8 in 2024-06-14T23:04:55.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:54 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:04:55.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4284751579' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:04:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:54 smithi086 ceph-mon[97913]: osdmap e236: 8 total, 8 up, 8 in 2024-06-14T23:04:55.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:55.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:55.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:04:55.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:55.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:55.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:55.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:55.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:55.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:55.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:55.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:55.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:55.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:55.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:55.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:56.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:56.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:56.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:56.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:55 smithi086 ceph-mon[97913]: pgmap v918: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:56.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:55 smithi086 ceph-mon[102754]: pgmap v918: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:56.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:56.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:56.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:56.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:55 smithi137 ceph-mon[90634]: pgmap v918: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:56.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:56.290 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T23:04:56.291 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T23:04:56.291 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-06-14T23:04:56.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-06-14T23:04:56.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:56.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:56.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:56.439 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:56.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:56.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:56.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:56.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:56.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:56.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:56.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:56.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:56.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:56.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:56.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:56.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:56.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:57.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:57.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:57.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:57.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:57.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:57.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:57.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:57.231 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:04:57.231 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:04:57.231 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T23:04:57.231 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.774572 2024-06-14T23:04:57.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 6 0.7745720774826873 2024-06-14T23:04:57.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:57.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:57.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:57.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:57.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-14T23:04:57.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:57.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:57.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:57.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:57.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:57.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:57.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:04:57.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:57.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:04:57.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:57.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:57.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:57.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:04:57.908 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:57.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:57.963 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:57.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:57.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:58.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:58.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:58.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:58.157 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:57 smithi086 ceph-mon[102754]: pgmap v919: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:58.158 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:58.158 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:57 smithi086 ceph-mon[97913]: pgmap v919: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:58.158 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:58.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:58.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:58.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:57 smithi137 ceph-mon[90634]: pgmap v919: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:04:58.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:58.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:58.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:58.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:58.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:58.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:58.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:58.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:58.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:58.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:58.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:58.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:04:58.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:58.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:04:58.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:58.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:59.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:59.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:59.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:59.118 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:04:59.118 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:04:59.148 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:04:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:04:59.165 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:04:59.165 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:04:59.180 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:04:59.185 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:04:59.185 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:04:59.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:04:59.208 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:04:59.208 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:04:59.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:59.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:04:59.229 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:04:59.229 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:04:59.246 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:04:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:04:59.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:04:59.262 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:04:59.263 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:04:59 smithi086 ceph-mon[97913]: pgmap v920: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:59.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:04:59 smithi086 ceph-mon[102754]: pgmap v920: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:59.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:04:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:59.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:04:59.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:59.462 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:59.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:59.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:04:59 smithi137 ceph-mon[90634]: pgmap v920: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:04:59.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:04:59.588 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:59.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:04:59.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:04:59.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:59.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:04:59.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:04:59.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:04:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:59.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:04:59.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:04:59.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:04:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:04:59.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:00.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:00.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:00.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:00.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:00.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:00.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:00.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:00.334 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:00.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:00.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/706145573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:00.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/706145573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:00.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:00.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:00.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:00.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/706145573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:00.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:00.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:00.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:00.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:00.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:00.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:00.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:00.840 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:00.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:00.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:00.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:00.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:01.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:01.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:01.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:01.175 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:01.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:01.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:01.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:01.285 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:01 smithi086 ceph-mon[97913]: pgmap v921: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:01.285 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:01 smithi086 ceph-mon[102754]: pgmap v921: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:01.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:01.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:01.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:01.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:01.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:01.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:01.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:01 smithi137 ceph-mon[90634]: pgmap v921: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:01.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:01.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:01.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:01.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:01.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:01.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:01.852 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:01.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:01.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:01.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:01.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:01.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:02.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:02.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:02.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:02.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:02.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:02.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:02.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:02.353 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:02.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:02.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:02.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:02.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:02.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:02.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:02.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:02.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:02.726 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:02.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:02.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:02.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:05:02.755 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4009741937' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-14T23:05:02.755 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:02 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-14T23:05:02.755 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4009741937' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-14T23:05:02.755 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:02 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-14T23:05:02.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:02.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:02.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:02.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:02.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:02.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:02.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:02.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4009741937' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-14T23:05:02.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:02 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-14T23:05:03.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:03.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:03.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:03.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:03.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:03.353 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:03.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:03.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:03.477 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:03.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:03.482 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_6' created 2024-06-14T23:05:03.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:03.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:03.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:03.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:03.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:03.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:03.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:03.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:03.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:03.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:03.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:03.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:03 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-06-14T23:05:03.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:03 smithi086 ceph-mon[97913]: osdmap e237: 8 total, 8 up, 8 in 2024-06-14T23:05:03.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:03 smithi086 ceph-mon[97913]: pgmap v923: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:03.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:03 smithi086 ceph-mon[97913]: osdmap e238: 8 total, 8 up, 8 in 2024-06-14T23:05:03.971 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:03 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-06-14T23:05:03.971 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:03 smithi086 ceph-mon[102754]: osdmap e237: 8 total, 8 up, 8 in 2024-06-14T23:05:03.971 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:03 smithi086 ceph-mon[102754]: pgmap v923: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:03.971 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:03 smithi086 ceph-mon[102754]: osdmap e238: 8 total, 8 up, 8 in 2024-06-14T23:05:03.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:03.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:03.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:03 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-06-14T23:05:03.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:03 smithi137 ceph-mon[90634]: osdmap e237: 8 total, 8 up, 8 in 2024-06-14T23:05:03.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:03 smithi137 ceph-mon[90634]: pgmap v923: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:03.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:03 smithi137 ceph-mon[90634]: osdmap e238: 8 total, 8 up, 8 in 2024-06-14T23:05:04.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:04.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:04.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:04.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:04.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:04.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:04.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:04.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:04.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:04.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-06-14T23:05:04.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:04.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:04.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:04.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:04.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:04.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:04.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:04.618 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:05:04.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:04.739 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:04.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:04.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:04.865 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:04.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:04.870 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[102754]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:05:04.870 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2095148559' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:05:04.870 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:05:04.870 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2095148559' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:05:04.870 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[102754]: osdmap e239: 8 total, 8 up, 8 in 2024-06-14T23:05:04.870 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[97913]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:05:04.870 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2095148559' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:05:04.870 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:05:04.870 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2095148559' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:05:04.871 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-mon[97913]: osdmap e239: 8 total, 8 up, 8 in 2024-06-14T23:05:04.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:04.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:04.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:04.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:04 smithi137 ceph-mon[90634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:05:04.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:04 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2095148559' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:05:04.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:04 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:05:04.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:04 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2095148559' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:05:04.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:04 smithi137 ceph-mon[90634]: osdmap e239: 8 total, 8 up, 8 in 2024-06-14T23:05:05.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:05.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:05.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:05.186 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:05:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:05:04] "GET /metrics HTTP/1.1" 200 34195 "" "Prometheus/2.43.0" 2024-06-14T23:05:05.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:05.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:05.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:05.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:05.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:05.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:05.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:05.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:05.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:05.618 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:05.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:05.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:05.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:05.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:05.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:05.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:05 smithi086 ceph-mon[102754]: pgmap v926: 99 pgs: 7 unknown, 92 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:05.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:05 smithi086 ceph-mon[97913]: pgmap v926: 99 pgs: 7 unknown, 92 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:05.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:05.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:05.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:05.987 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:05.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:05.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:05 smithi137 ceph-mon[90634]: pgmap v926: 99 pgs: 7 unknown, 92 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:06.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:06.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:06.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:06.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:06.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:06.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:06.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:06.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:06.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:06.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:06.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:06.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:06.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:06.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:06.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:06.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:06.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:06.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:06.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:06.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:06.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:06.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:06.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:06.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:05:06.994 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:05:06.995 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:05:07.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:07.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:07.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:07.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:07.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:07.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:07.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:07.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:07.340 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:05:07.340 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:05:07.341 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-06-14T23:05:07.341 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:05:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:07.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:07.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:07.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:07.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:07.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:07.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:05:07.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:07.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:07.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:07.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:07.712 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:07.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:07.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:07.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:07.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:07.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:07.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:07.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:07.976 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:07 smithi137 ceph-mon[90634]: pgmap v927: 99 pgs: 1 unknown, 98 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:08.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:08.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:08.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:08.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:07 smithi086 ceph-mon[97913]: pgmap v927: 99 pgs: 1 unknown, 98 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:08.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:07 smithi086 ceph-mon[102754]: pgmap v927: 99 pgs: 1 unknown, 98 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:08.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:08.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:08.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:08.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:08.331 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:08.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:08.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:08.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:08.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:08.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:08.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:08.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:08.653 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.6 primary-affinity to 0.774572 (8507622) 2024-06-14T23:05:08.665 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2010641575' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.7745720774826873}]: dispatch 2024-06-14T23:05:08.665 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:08 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.7745720774826873}]: dispatch 2024-06-14T23:05:08.665 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2010641575' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.7745720774826873}]: dispatch 2024-06-14T23:05:08.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:08 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.7745720774826873}]: dispatch 2024-06-14T23:05:08.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:08.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:08.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:08.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:08.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:08.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:08.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:08.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:08.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:08.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2010641575' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.7745720774826873}]: dispatch 2024-06-14T23:05:08.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:08 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.7745720774826873}]: dispatch 2024-06-14T23:05:09.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:09.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:09.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:09.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:09.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:09.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:09.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:09.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:09.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:09.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:09.439 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:09.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:09.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:09.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:09.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:09.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:09.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:09.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:09.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:09.807 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:09 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.7745720774826873}]': finished 2024-06-14T23:05:09.807 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:09 smithi086 ceph-mon[102754]: osdmap e240: 8 total, 8 up, 8 in 2024-06-14T23:05:09.807 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:09 smithi086 ceph-mon[102754]: pgmap v929: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:09.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:09 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.7745720774826873}]': finished 2024-06-14T23:05:09.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:09 smithi086 ceph-mon[97913]: osdmap e240: 8 total, 8 up, 8 in 2024-06-14T23:05:09.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:09 smithi086 ceph-mon[97913]: pgmap v929: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:09.809 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:09.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:09.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:09.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:09.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:09.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:09 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.7745720774826873}]': finished 2024-06-14T23:05:09.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:09 smithi137 ceph-mon[90634]: osdmap e240: 8 total, 8 up, 8 in 2024-06-14T23:05:09.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:09 smithi137 ceph-mon[90634]: pgmap v929: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:10.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:10.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:10.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:10.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:10.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:10.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:10.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-14T23:05:10.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:10.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:10.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:10.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:10.424 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:10.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:10.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:10.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:10.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:10.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:10.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:10.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:10.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:10.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:10 smithi086 ceph-mon[97913]: osdmap e241: 8 total, 8 up, 8 in 2024-06-14T23:05:10.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:10 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:10.779 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:10 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:10.779 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:10 smithi086 ceph-mon[102754]: osdmap e241: 8 total, 8 up, 8 in 2024-06-14T23:05:10.779 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:10 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:10.779 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:10 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:10.788 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:05:10.788 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:05:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:10.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:10.805 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:05:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:05:10.821 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:05:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:05:10.838 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:05:10.838 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:05:10.854 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:05:10.854 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:05:10.870 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:05:10.870 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:05:10.886 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:05:10.886 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:05:10.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:10.903 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:05:10.903 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:05:10.914 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:10 smithi137 ceph-mon[90634]: osdmap e241: 8 total, 8 up, 8 in 2024-06-14T23:05:10.915 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:10 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:10.915 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:10 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:10.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:10.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:11.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:11.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:11.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:11.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:11.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:11.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:11.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:11.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:11.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:11.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:11.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:11.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:11.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:11.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:11.534 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:11.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:11.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:11.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:11.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:11.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:11.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:11.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:11.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:11.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2076066693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:11.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:11 smithi086 ceph-mon[97913]: pgmap v931: 99 pgs: 2 peering, 97 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:11.828 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2076066693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:11.828 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:11 smithi086 ceph-mon[102754]: pgmap v931: 99 pgs: 2 peering, 97 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:11.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:11.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:11.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:11.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2076066693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:11.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:11 smithi137 ceph-mon[90634]: pgmap v931: 99 pgs: 2 peering, 97 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:12.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:12.027 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:12.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:12.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:12.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:12.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:12.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:12.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:12.280 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:12.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:12.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:12.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:12.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:12.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:12.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:12.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:12.649 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:12.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:12.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:12.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:12.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:12.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:12 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:05:12.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:12.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:12.783 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:12 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:05:12.783 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:12.783 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:12.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:12.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:12.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:12 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:05:12.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:12.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:12.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:13.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:13.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:13.130 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:13.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:13.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:13.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:13.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:13.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:13.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:13.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:13.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:13.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:13.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:13.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:13.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:13.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:13.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:13.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:13.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:13.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:13.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:13.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:13.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:13 smithi086 ceph-mon[97913]: pgmap v932: 99 pgs: 2 peering, 97 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:13 smithi086 ceph-mon[102754]: pgmap v932: 99 pgs: 2 peering, 97 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:13.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:13.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:13.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:13.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:13.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:13.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:13 smithi137 ceph-mon[90634]: pgmap v932: 99 pgs: 2 peering, 97 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:14.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:14.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:14.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:14.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:14.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:14.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:14.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:05:14.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:14.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:14.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:14.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:14.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:14.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:14.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:14.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:14.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:14.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:14.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:14.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:14.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:14.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:14.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:14.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:14.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:14.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:15.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:15.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:15.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:15.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:15.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:15.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:15.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:14 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T23:05:15.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:15.273 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:14 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T23:05:15.273 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:14 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T23:05:15.273 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:05:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:05:14] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.43.0" 2024-06-14T23:05:15.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:15.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:15.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:15.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:15.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:15.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:15.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:15.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:15.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:15.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:15.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:15.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:15.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:15.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:15.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:15.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:15.893 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-06-14T23:05:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:15.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:16.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:16.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:16.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:16.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:15 smithi086 ceph-mon[97913]: pgmap v933: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:16.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1627338567' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:05:16.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:15 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:05:16.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:15 smithi086 ceph-mon[102754]: pgmap v933: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:16.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1627338567' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:05:16.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:15 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:05:16.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:16.161 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:16.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:16.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:15 smithi137 ceph-mon[90634]: pgmap v933: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:16.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1627338567' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:05:16.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:15 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:05:16.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:16.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:16.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:16.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:16.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:16.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:16.493 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:05:16.493 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:05:16.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:16.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:16.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:16.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:16.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:16.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:16.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:16.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:16.785 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:16.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:16.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:16.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:16.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:16.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:16.996 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:17.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:17.041 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:17.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:17.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:17.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:17.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:16 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:05:17.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:16 smithi137 ceph-mon[90634]: osdmap e242: 8 total, 8 up, 8 in 2024-06-14T23:05:17.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:17.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:17.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:16 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:05:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:16 smithi086 ceph-mon[97913]: osdmap e242: 8 total, 8 up, 8 in 2024-06-14T23:05:17.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:16 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:05:17.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:16 smithi086 ceph-mon[102754]: osdmap e242: 8 total, 8 up, 8 in 2024-06-14T23:05:17.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:17.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:17.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:17.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:17.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:17.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:17.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:17.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:17.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:17.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:17.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:17.779 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:17.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:17.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:17.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:17.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:17.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:18.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:18.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:18.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:18.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:18.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:17 smithi086 ceph-mon[97913]: pgmap v935: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:18.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:17 smithi086 ceph-mon[97913]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:05:18.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:17 smithi086 ceph-mon[102754]: pgmap v935: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:18.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:17 smithi086 ceph-mon[102754]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:05:18.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:18.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:18.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:18.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:17 smithi137 ceph-mon[90634]: pgmap v935: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:18.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:17 smithi137 ceph-mon[90634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:05:18.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:18.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:18.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:18.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:18.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:18.409 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:18.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:18.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:18.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:18.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:18.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:18.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:18.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:18.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:18.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:18.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:18.910 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:18.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:18.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:19.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:19.034 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:19.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:19.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:19.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:19.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:19.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:19.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:19.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:19.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:19.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:19.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:19.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:19.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:19.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:19.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:19.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:19.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:19.783 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:19.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:19.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:19.910 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:05:19.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:19.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:20.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:20.016 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:19 smithi086 ceph-mon[97913]: pgmap v936: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:20.016 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:19 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4075168030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:05:20.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:19 smithi086 ceph-mon[102754]: pgmap v936: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:20.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:19 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4075168030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:05:20.041 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:20.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:20.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:20.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:20.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:20.216 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-06-14T23:05:20.216 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_6 2024-06-14T23:05:20.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:05:20.219 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T23:05:20.219 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T23:05:20.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:19 smithi137 ceph-mon[90634]: pgmap v936: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-06-14T23:05:20.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:19 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4075168030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:05:20.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T23:05:20.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T23:05:20.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_226760 2024-06-14T23:05:20.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:05:20.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T23:05:20.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:20.320 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:20.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:20.366 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:05:20.366 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:05:20.406 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:05:20.406 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:05:20.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:20.442 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:05:20.442 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:05:20.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:20.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:20.476 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:05:20.476 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:05:20.510 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:05:20.510 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:05:20.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:05:20.544 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:05:20.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:20.583 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:05:20.583 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:05:20.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:20.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:20.618 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:05:20.618 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:05:20.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:20.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:20.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:20.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:20.876 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:20.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:20.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:21.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:21.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:21.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:21.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:21.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:21.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:21.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:20 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:05:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4075168030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:05:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:20 smithi137 ceph-mon[90634]: osdmap e243: 8 total, 8 up, 8 in 2024-06-14T23:05:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/492263990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:21.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:21.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:20 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:05:21.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4075168030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:05:21.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:20 smithi086 ceph-mon[97913]: osdmap e243: 8 total, 8 up, 8 in 2024-06-14T23:05:21.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/492263990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:21.262 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:20 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:05:21.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4075168030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:05:21.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:20 smithi086 ceph-mon[102754]: osdmap e243: 8 total, 8 up, 8 in 2024-06-14T23:05:21.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/492263990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:21.266 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 3911 3895 243.424 243.438 0.00319981 0.00409352 2024-06-14T23:05:21.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:21.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:21.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:21.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:21.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:21.560 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:21.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:21.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:21.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:21.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:21.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:21.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:21.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:21.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:21.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:22.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:22.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:22.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:22.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:22.214 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:21 smithi137 ceph-mon[90634]: pgmap v938: 99 pgs: 99 active+clean; 7.4 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 880 KiB/s wr, 13 op/s 2024-06-14T23:05:22.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:22.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:22.261 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:21 smithi086 ceph-mon[97913]: pgmap v938: 99 pgs: 99 active+clean; 7.4 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 880 KiB/s wr, 13 op/s 2024-06-14T23:05:22.261 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:21 smithi086 ceph-mon[102754]: pgmap v938: 99 pgs: 99 active+clean; 7.4 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 880 KiB/s wr, 13 op/s 2024-06-14T23:05:22.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 16 7378 7362 230.024 216.688 0.00516363 0.00433772 2024-06-14T23:05:22.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:22.384 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:22.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:22.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:22.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:22.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:22.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:22.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:22.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:22.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:22.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:22.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:22.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:22.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:22.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:22.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:22.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:23.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:23.065 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:23.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:23.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:23.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:23.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:23.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:23.267 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 16 10884 10868 226.384 219.125 0.00323826 0.00440955 2024-06-14T23:05:23.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:23.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:23.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:23.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:23.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:23.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:23.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:23.610 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:23.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:23.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:23.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:23.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:23.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:23.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:23.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:23.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:23.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:23.994 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:23 smithi137 ceph-mon[90634]: pgmap v939: 99 pgs: 99 active+clean; 38 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 4.7 MiB/s wr, 75 op/s 2024-06-14T23:05:24.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:24.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:24.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:24.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:23 smithi086 ceph-mon[97913]: pgmap v939: 99 pgs: 99 active+clean; 38 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 4.7 MiB/s wr, 75 op/s 2024-06-14T23:05:24.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:24.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:23 smithi086 ceph-mon[102754]: pgmap v939: 99 pgs: 99 active+clean; 38 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 4.7 MiB/s wr, 75 op/s 2024-06-14T23:05:24.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:24.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:24.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:24.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:05:24.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:24.268 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 16 13899 13883 216.826 188.438 0.00523537 0.00460359 2024-06-14T23:05:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:24.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:24.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:24.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:24.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:24.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:24.644 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:24.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:24.689 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:05:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:05:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:05:24.689 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:05:24.689 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:05:24.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:24.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:24.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:05:24.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:24.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:24.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:24.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:24.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:24.997 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:25.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:25.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:25.082 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:05:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:05:24] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.43.0" 2024-06-14T23:05:25.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:25.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:25.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:25.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:25.209 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:25.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:25.268 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 16 16318 16302 203.682 151.188 0.0071904 0.00490453 2024-06-14T23:05:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:25.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:25.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:25.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:25.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[97913]: pgmap v940: 99 pgs: 99 active+clean; 530 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s 2024-06-14T23:05:25.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:05:25.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:05:25.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:25.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:25.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:25.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[102754]: pgmap v940: 99 pgs: 99 active+clean; 530 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s 2024-06-14T23:05:25.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:05:25.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:05:25.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:25.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:25.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:25.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:25.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:25.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:25.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:25.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:25.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:25.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:25.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:25.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:25.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:25 smithi137 ceph-mon[90634]: pgmap v940: 99 pgs: 99 active+clean; 530 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s 2024-06-14T23:05:25.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:05:25.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:05:25.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:25.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:25.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:25.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:25.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:25.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:25.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:26.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:26.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:26.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-14T23:05:26.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:26.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:26.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:26.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 19284 19268 200.629 185.375 0.00682943 0.00497929 2024-06-14T23:05:26.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:26.384 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:26.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:26.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:26.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:26.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:26.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:26.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:26.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:26.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:26.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:26.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:26.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:26.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:26.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:26.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:26.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:27.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:27.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:27.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:27.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:27.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:27.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:27.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 16 21879 21863 195.137 162.188 0.00674833 0.00511955 2024-06-14T23:05:27.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:27.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:27.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:27.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:27.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:27.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:27.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:27.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:27.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:27.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:27.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:27.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:27.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:27.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:27.871 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:27.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:27.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:28.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:28.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:28.111 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:28.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:28.337 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 24281 24265 189.51 150.125 0.00450056 0.0052614 2024-06-14T23:05:28.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:27 smithi086 ceph-mon[102754]: pgmap v941: 99 pgs: 99 active+clean; 900 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-14T23:05:28.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:28.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:28.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:27 smithi086 ceph-mon[97913]: pgmap v941: 99 pgs: 99 active+clean; 900 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-14T23:05:28.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:28.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:28.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:28.339 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:27 smithi137 ceph-mon[90634]: pgmap v941: 99 pgs: 99 active+clean; 900 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-14T23:05:28.339 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:28.339 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:28.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:28.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:28.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:28.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:28.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:28.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:28.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:28.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:28.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:28.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:28.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:28.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:29.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:29.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:29.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:29.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:29.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:29.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:29.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 26466 26450 183.627 136.562 0.00374873 0.00544088 2024-06-14T23:05:29.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:29.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:29.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:29.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:29.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:29.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:29.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:29.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:29.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:29.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:29.730 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:29.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:29.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:29.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:29.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:29.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:30.004 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:30.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:30.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:30.129 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:30.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:30.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:29 smithi137 ceph-mon[90634]: pgmap v942: 99 pgs: 99 active+clean; 957 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-14T23:05:30.264 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:30.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:30.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 16 28547 28531 178.27 130.062 0.00678087 0.00560393 2024-06-14T23:05:30.331 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:29 smithi086 ceph-mon[102754]: pgmap v942: 99 pgs: 99 active+clean; 957 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-14T23:05:30.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:29 smithi086 ceph-mon[97913]: pgmap v942: 99 pgs: 99 active+clean; 957 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-14T23:05:30.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:30.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:30.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:30.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:30.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:30.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:30.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:30.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:30.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:30.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:30.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:30.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:30.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:30.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:30.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:31.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:31.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:31.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:31.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:31.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:31.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:31.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 30908 30892 175.479 147.562 0.00271307 0.00569374 2024-06-14T23:05:31.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:31.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:31.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:31.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:31.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:31.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:31.509 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:31.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:31.646 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:31.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:31.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:31.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:31.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:31.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:31.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:31.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:32.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:32.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:32.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:32.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:31 smithi086 ceph-mon[97913]: pgmap v943: 99 pgs: 99 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 155 MiB/s wr, 2.47k op/s 2024-06-14T23:05:32.074 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:31 smithi086 ceph-mon[102754]: pgmap v943: 99 pgs: 99 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 155 MiB/s wr, 2.47k op/s 2024-06-14T23:05:32.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:32.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:32.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:31 smithi137 ceph-mon[90634]: pgmap v943: 99 pgs: 99 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 155 MiB/s wr, 2.47k op/s 2024-06-14T23:05:32.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 16 33095 33079 172.245 136.688 0.00187537 0.00578167 2024-06-14T23:05:32.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:32.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:32.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:32.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:32.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:32.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:32.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:32.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:05:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:05:32.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:32.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:32.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:32.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:32.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:32.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:32.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:32.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:33.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:33.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:33.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:33.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:33.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:33.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:33.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 34946 34930 167.895 115.688 0.0109424 0.00595148 2024-06-14T23:05:33.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:33.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:33.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:33.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:33.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:33.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:33.568 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:33.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:33.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:33.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:33.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:33.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:33.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:33.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:33.880 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:05:33.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:33.982 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":943,"stamp":"2024-06-14T23:05:30.788012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1763579846,"num_objects":26925,"num_object_clones":0,"num_object_copies":80775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26962,"num_write_kb":1721693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":526584,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28727,"ondisk_log_size":28727,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10385428,"kb_used_data":4893332,"kb_used_omap":29,"kb_used_meta":5491938,"kb_avail":739575788,"statfs":{"total":767960285184,"available":757325606912,"internally_reserved":0,"allocated":5010771968,"data_stored":5001182014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":5623745189},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,7,11,6,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1762983936,"num_objects":26901,"num_object_clones":0,"num_object_copies":80703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26901,"num_write_kb":1721664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.880083"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":175,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891478+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:05:15.891478+0000","last_peered":"2024-06-14T23:05:15.891478+0000","last_clean":"2024-06-14T23:05:15.891478+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:05:15.891478+0000","last_undegraded":"2024-06-14T23:05:15.891478+0000","last_fullsized":"2024-06-14T23:05:15.891478+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":238,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.230077+0000","last_change":"2024-06-14T23:04:34.895560+0000","last_active":"2024-06-14T23:05:16.230077+0000","last_peered":"2024-06-14T23:05:16.230077+0000","last_clean":"2024-06-14T23:05:16.230077+0000","last_became_active":"2024-06-14T23:04:34.887055+0000","last_became_peered":"2024-06-14T23:04:34.887055+0000","last_unstale":"2024-06-14T23:05:16.230077+0000","last_undegraded":"2024-06-14T23:05:16.230077+0000","last_fullsized":"2024-06-14T23:05:16.230077+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:26:36.259241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.623203+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:05:16.623203+0000","last_peered":"2024-06-14T23:05:16.623203+0000","last_clean":"2024-06-14T23:05:16.623203+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:05:16.623203+0000","last_undegraded":"2024-06-14T23:05:16.623203+0000","last_fullsized":"2024-06-14T23:05:16.623203+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":196,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971401+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:05:15.971401+0000","last_peered":"2024-06-14T23:05:15.971401+0000","last_clean":"2024-06-14T23:05:15.971401+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:05:15.971401+0000","last_undegraded":"2024-06-14T23:05:15.971401+0000","last_fullsized":"2024-06-14T23:05:15.971401+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":197,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971556+0000","last_change":"2024-06-14T22:54:06.277765+0000","last_active":"2024-06-14T23:05:15.971556+0000","last_peered":"2024-06-14T23:05:15.971556+0000","last_clean":"2024-06-14T23:05:15.971556+0000","last_became_active":"2024-06-14T22:54:06.277465+0000","last_became_peered":"2024-06-14T22:54:06.277465+0000","last_unstale":"2024-06-14T23:05:15.971556+0000","last_undegraded":"2024-06-14T23:05:15.971556+0000","last_fullsized":"2024-06-14T23:05:15.971556+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:39.168581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":110,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.888854+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T23:05:15.888854+0000","last_peered":"2024-06-14T23:05:15.888854+0000","last_clean":"2024-06-14T23:05:15.888854+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T23:05:15.888854+0000","last_undegraded":"2024-06-14T23:05:15.888854+0000","last_fullsized":"2024-06-14T23:05:15.888854+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":198,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.892348+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:05:15.892348+0000","last_peered":"2024-06-14T23:05:15.892348+0000","last_clean":"2024-06-14T23:05:15.892348+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:05:15.892348+0000","last_undegraded":"2024-06-14T23:05:15.892348+0000","last_fullsized":"2024-06-14T23:05:15.892348+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":212,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.220222+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T23:05:16.220222+0000","last_peered":"2024-06-14T23:05:16.220222+0000","last_clean":"2024-06-14T23:05:16.220222+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T23:05:16.220222+0000","last_undegraded":"2024-06-14T23:05:16.220222+0000","last_fullsized":"2024-06-14T23:05:16.220222+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"243'1693","reported_seq":1707,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:29.501439+0000","last_change":"2024-06-14T23:05:03.621489+0000","last_active":"2024-06-14T23:05:29.501439+0000","last_peered":"2024-06-14T23:05:29.501439+0000","last_clean":"2024-06-14T23:05:29.501439+0000","last_became_active":"2024-06-14T23:05:03.621114+0000","last_became_peered":"2024-06-14T23:05:03.621114+0000","last_unstale":"2024-06-14T23:05:29.501439+0000","last_undegraded":"2024-06-14T23:05:29.501439+0000","last_fullsized":"2024-06-14T23:05:29.501439+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1693,"log_dups_size":0,"ondisk_log_size":1693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:33.811591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110952448,"num_objects":1693,"num_object_clones":0,"num_object_copies":5079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1693,"num_write_kb":108352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":193,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.230465+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:05:16.230465+0000","last_peered":"2024-06-14T23:05:16.230465+0000","last_clean":"2024-06-14T23:05:16.230465+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:05:16.230465+0000","last_undegraded":"2024-06-14T23:05:16.230465+0000","last_fullsized":"2024-06-14T23:05:16.230465+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":186,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.622993+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T23:05:16.622993+0000","last_peered":"2024-06-14T23:05:16.622993+0000","last_clean":"2024-06-14T23:05:16.622993+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T23:05:16.622993+0000","last_undegraded":"2024-06-14T23:05:16.622993+0000","last_fullsized":"2024-06-14T23:05:16.622993+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"243'1407","reported_seq":1422,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:27.345331+0000","last_change":"2024-06-14T23:05:03.621148+0000","last_active":"2024-06-14T23:05:27.345331+0000","last_peered":"2024-06-14T23:05:27.345331+0000","last_clean":"2024-06-14T23:05:27.345331+0000","last_became_active":"2024-06-14T23:05:03.620864+0000","last_became_peered":"2024-06-14T23:05:03.620864+0000","last_unstale":"2024-06-14T23:05:27.345331+0000","last_undegraded":"2024-06-14T23:05:27.345331+0000","last_fullsized":"2024-06-14T23:05:27.345331+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:30:42.892991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92209152,"num_objects":1407,"num_object_clones":0,"num_object_copies":4221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1407,"num_write_kb":90048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":194,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.230383+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:05:16.230383+0000","last_peered":"2024-06-14T23:05:16.230383+0000","last_clean":"2024-06-14T23:05:16.230383+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:05:16.230383+0000","last_undegraded":"2024-06-14T23:05:16.230383+0000","last_fullsized":"2024-06-14T23:05:16.230383+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":220,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889007+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:05:15.889007+0000","last_peered":"2024-06-14T23:05:15.889007+0000","last_clean":"2024-06-14T23:05:15.889007+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:05:15.889007+0000","last_undegraded":"2024-06-14T23:05:15.889007+0000","last_fullsized":"2024-06-14T23:05:15.889007+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"243'1703","reported_seq":1717,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:29.512711+0000","last_change":"2024-06-14T23:05:03.625005+0000","last_active":"2024-06-14T23:05:29.512711+0000","last_peered":"2024-06-14T23:05:29.512711+0000","last_clean":"2024-06-14T23:05:29.512711+0000","last_became_active":"2024-06-14T23:05:03.624698+0000","last_became_peered":"2024-06-14T23:05:03.624698+0000","last_unstale":"2024-06-14T23:05:29.512711+0000","last_undegraded":"2024-06-14T23:05:29.512711+0000","last_fullsized":"2024-06-14T23:05:29.512711+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1703,"log_dups_size":0,"ondisk_log_size":1703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:30:27.319001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111607808,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1703,"num_write_kb":108992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":121,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.623113+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T23:05:16.623113+0000","last_peered":"2024-06-14T23:05:16.623113+0000","last_clean":"2024-06-14T23:05:16.623113+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T23:05:16.623113+0000","last_undegraded":"2024-06-14T23:05:16.623113+0000","last_fullsized":"2024-06-14T23:05:16.623113+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.220431+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T23:05:16.220431+0000","last_peered":"2024-06-14T23:05:16.220431+0000","last_clean":"2024-06-14T23:05:16.220431+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T23:05:16.220431+0000","last_undegraded":"2024-06-14T23:05:16.220431+0000","last_fullsized":"2024-06-14T23:05:16.220431+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"243'1672","reported_seq":1686,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:29.150569+0000","last_change":"2024-06-14T23:05:03.624271+0000","last_active":"2024-06-14T23:05:29.150569+0000","last_peered":"2024-06-14T23:05:29.150569+0000","last_clean":"2024-06-14T23:05:29.150569+0000","last_became_active":"2024-06-14T23:05:03.623977+0000","last_became_peered":"2024-06-14T23:05:03.623977+0000","last_unstale":"2024-06-14T23:05:29.150569+0000","last_undegraded":"2024-06-14T23:05:29.150569+0000","last_fullsized":"2024-06-14T23:05:29.150569+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1672,"log_dups_size":0,"ondisk_log_size":1672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:11:53.025160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1672,"num_write_kb":107008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":204,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.623115+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T23:05:16.623115+0000","last_peered":"2024-06-14T23:05:16.623115+0000","last_clean":"2024-06-14T23:05:16.623115+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T23:05:16.623115+0000","last_undegraded":"2024-06-14T23:05:16.623115+0000","last_fullsized":"2024-06-14T23:05:16.623115+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":198,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971597+0000","last_change":"2024-06-14T22:49:25.487542+0000","last_active":"2024-06-14T23:05:15.971597+0000","last_peered":"2024-06-14T23:05:15.971597+0000","last_clean":"2024-06-14T23:05:15.971597+0000","last_became_active":"2024-06-14T22:49:25.487240+0000","last_became_peered":"2024-06-14T22:49:25.487240+0000","last_unstale":"2024-06-14T23:05:15.971597+0000","last_undegraded":"2024-06-14T23:05:15.971597+0000","last_fullsized":"2024-06-14T23:05:15.971597+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:26.990859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"243'1581","reported_seq":1595,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:29.132060+0000","last_change":"2024-06-14T23:05:03.626281+0000","last_active":"2024-06-14T23:05:29.132060+0000","last_peered":"2024-06-14T23:05:29.132060+0000","last_clean":"2024-06-14T23:05:29.132060+0000","last_became_active":"2024-06-14T23:05:03.625996+0000","last_became_peered":"2024-06-14T23:05:03.625996+0000","last_unstale":"2024-06-14T23:05:29.132060+0000","last_undegraded":"2024-06-14T23:05:29.132060+0000","last_fullsized":"2024-06-14T23:05:29.132060+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:52:21.957192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103612416,"num_objects":1581,"num_object_clones":0,"num_object_copies":4743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1581,"num_write_kb":101184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":495,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.104348+0000","last_change":"2024-06-14T23:04:40.403447+0000","last_active":"2024-06-14T23:05:16.104348+0000","last_peered":"2024-06-14T23:05:16.104348+0000","last_clean":"2024-06-14T23:05:16.104348+0000","last_became_active":"2024-06-14T23:04:37.904456+0000","last_became_peered":"2024-06-14T23:04:37.904456+0000","last_unstale":"2024-06-14T23:05:16.104348+0000","last_undegraded":"2024-06-14T23:05:16.104348+0000","last_fullsized":"2024-06-14T23:05:16.104348+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:04:40.403365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:04:40.403365+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:45.174359+0000","scrub_duration":0.0014330689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":114,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971441+0000","last_change":"2024-06-14T22:58:09.752985+0000","last_active":"2024-06-14T23:05:15.971441+0000","last_peered":"2024-06-14T23:05:15.971441+0000","last_clean":"2024-06-14T23:05:15.971441+0000","last_became_active":"2024-06-14T22:58:09.752630+0000","last_became_peered":"2024-06-14T22:58:09.752630+0000","last_unstale":"2024-06-14T23:05:15.971441+0000","last_undegraded":"2024-06-14T23:05:15.971441+0000","last_fullsized":"2024-06-14T23:05:15.971441+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:22.320986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889341+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T23:05:15.889341+0000","last_peered":"2024-06-14T23:05:15.889341+0000","last_clean":"2024-06-14T23:05:15.889341+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T23:05:15.889341+0000","last_undegraded":"2024-06-14T23:05:15.889341+0000","last_fullsized":"2024-06-14T23:05:15.889341+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"243'1698","reported_seq":1713,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:30.489209+0000","last_change":"2024-06-14T23:05:03.807700+0000","last_active":"2024-06-14T23:05:30.489209+0000","last_peered":"2024-06-14T23:05:30.489209+0000","last_clean":"2024-06-14T23:05:30.489209+0000","last_became_active":"2024-06-14T23:05:03.807400+0000","last_became_peered":"2024-06-14T23:05:03.807400+0000","last_unstale":"2024-06-14T23:05:30.489209+0000","last_undegraded":"2024-06-14T23:05:30.489209+0000","last_fullsized":"2024-06-14T23:05:30.489209+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1698,"log_dups_size":0,"ondisk_log_size":1698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:56:35.097613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111280128,"num_objects":1698,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1698,"num_write_kb":108672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":588,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.622806+0000","last_change":"2024-06-14T23:04:37.902544+0000","last_active":"2024-06-14T23:05:16.622806+0000","last_peered":"2024-06-14T23:05:16.622806+0000","last_clean":"2024-06-14T23:05:16.622806+0000","last_became_active":"2024-06-14T23:04:37.902191+0000","last_became_peered":"2024-06-14T23:04:37.902191+0000","last_unstale":"2024-06-14T23:05:16.622806+0000","last_undegraded":"2024-06-14T23:05:16.622806+0000","last_fullsized":"2024-06-14T23:05:16.622806+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:05:09.672062+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":84,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.888996+0000","last_change":"2024-06-14T23:05:09.660081+0000","last_active":"2024-06-14T23:05:15.888996+0000","last_peered":"2024-06-14T23:05:15.888996+0000","last_clean":"2024-06-14T23:05:15.888996+0000","last_became_active":"2024-06-14T23:05:09.659778+0000","last_became_peered":"2024-06-14T23:05:09.659778+0000","last_unstale":"2024-06-14T23:05:15.888996+0000","last_undegraded":"2024-06-14T23:05:15.888996+0000","last_fullsized":"2024-06-14T23:05:15.888996+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:41:09.113348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":233,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889511+0000","last_change":"2024-06-14T23:04:34.887791+0000","last_active":"2024-06-14T23:05:15.889511+0000","last_peered":"2024-06-14T23:05:15.889511+0000","last_clean":"2024-06-14T23:05:15.889511+0000","last_became_active":"2024-06-14T23:04:34.887099+0000","last_became_peered":"2024-06-14T23:04:34.887099+0000","last_unstale":"2024-06-14T23:05:15.889511+0000","last_undegraded":"2024-06-14T23:05:15.889511+0000","last_fullsized":"2024-06-14T23:05:15.889511+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:46:07.470301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"243'1721","reported_seq":1735,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:29.112277+0000","last_change":"2024-06-14T23:05:03.808354+0000","last_active":"2024-06-14T23:05:29.112277+0000","last_peered":"2024-06-14T23:05:29.112277+0000","last_clean":"2024-06-14T23:05:29.112277+0000","last_became_active":"2024-06-14T23:05:03.808017+0000","last_became_peered":"2024-06-14T23:05:03.808017+0000","last_unstale":"2024-06-14T23:05:29.112277+0000","last_undegraded":"2024-06-14T23:05:29.112277+0000","last_fullsized":"2024-06-14T23:05:29.112277+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1721,"log_dups_size":0,"ondisk_log_size":1721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:46:06.873436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112787456,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1721,"num_write_kb":110144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":224,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889055+0000","last_change":"2024-06-14T23:04:34.896257+0000","last_active":"2024-06-14T23:05:15.889055+0000","last_peered":"2024-06-14T23:05:15.889055+0000","last_clean":"2024-06-14T23:05:15.889055+0000","last_became_active":"2024-06-14T23:04:34.895926+0000","last_became_peered":"2024-06-14T23:04:34.895926+0000","last_unstale":"2024-06-14T23:05:15.889055+0000","last_undegraded":"2024-06-14T23:05:15.889055+0000","last_fullsized":"2024-06-14T23:05:15.889055+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:17.800902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889639+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T23:05:15.889639+0000","last_peered":"2024-06-14T23:05:15.889639+0000","last_clean":"2024-06-14T23:05:15.889639+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T23:05:15.889639+0000","last_undegraded":"2024-06-14T23:05:15.889639+0000","last_fullsized":"2024-06-14T23:05:15.889639+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"243'1728","reported_seq":1743,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:30.079832+0000","last_change":"2024-06-14T23:05:03.810157+0000","last_active":"2024-06-14T23:05:30.079832+0000","last_peered":"2024-06-14T23:05:30.079832+0000","last_clean":"2024-06-14T23:05:30.079832+0000","last_became_active":"2024-06-14T23:05:03.809591+0000","last_became_peered":"2024-06-14T23:05:03.809591+0000","last_unstale":"2024-06-14T23:05:30.079832+0000","last_undegraded":"2024-06-14T23:05:30.079832+0000","last_fullsized":"2024-06-14T23:05:30.079832+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1728,"log_dups_size":0,"ondisk_log_size":1728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:04:46.257184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":1728,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1728,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":580,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.622906+0000","last_change":"2024-06-14T23:04:37.902330+0000","last_active":"2024-06-14T23:05:16.622906+0000","last_peered":"2024-06-14T23:05:16.622906+0000","last_clean":"2024-06-14T23:05:16.622906+0000","last_became_active":"2024-06-14T23:04:37.901760+0000","last_became_peered":"2024-06-14T23:04:37.901760+0000","last_unstale":"2024-06-14T23:05:16.622906+0000","last_undegraded":"2024-06-14T23:05:16.622906+0000","last_fullsized":"2024-06-14T23:05:16.622906+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:57.295069+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":200,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891940+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:05:15.891940+0000","last_peered":"2024-06-14T23:05:15.891940+0000","last_clean":"2024-06-14T23:05:15.891940+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:05:15.891940+0000","last_undegraded":"2024-06-14T23:05:15.891940+0000","last_fullsized":"2024-06-14T23:05:15.891940+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":213,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891903+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:05:15.891903+0000","last_peered":"2024-06-14T23:05:15.891903+0000","last_clean":"2024-06-14T23:05:15.891903+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:05:15.891903+0000","last_undegraded":"2024-06-14T23:05:15.891903+0000","last_fullsized":"2024-06-14T23:05:15.891903+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"243'1771","reported_seq":1786,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:29.508271+0000","last_change":"2024-06-14T23:05:09.659680+0000","last_active":"2024-06-14T23:05:29.508271+0000","last_peered":"2024-06-14T23:05:29.508271+0000","last_clean":"2024-06-14T23:05:29.508271+0000","last_became_active":"2024-06-14T23:05:09.659391+0000","last_became_peered":"2024-06-14T23:05:09.659391+0000","last_unstale":"2024-06-14T23:05:29.508271+0000","last_undegraded":"2024-06-14T23:05:29.508271+0000","last_fullsized":"2024-06-14T23:05:29.508271+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1771,"log_dups_size":0,"ondisk_log_size":1771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:48:20.431065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116064256,"num_objects":1771,"num_object_clones":0,"num_object_copies":5313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1771,"num_write_kb":113344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":534,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971297+0000","last_change":"2024-06-14T23:00:30.610273+0000","last_active":"2024-06-14T23:05:15.971297+0000","last_peered":"2024-06-14T23:05:15.971297+0000","last_clean":"2024-06-14T23:05:15.971297+0000","last_became_active":"2024-06-14T23:00:09.766095+0000","last_became_peered":"2024-06-14T23:00:09.766095+0000","last_unstale":"2024-06-14T23:05:15.971297+0000","last_undegraded":"2024-06-14T23:05:15.971297+0000","last_fullsized":"2024-06-14T23:05:15.971297+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.610180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.610180+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:53.521891+0000","scrub_duration":0.00217396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":228,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.622959+0000","last_change":"2024-06-14T23:04:34.889456+0000","last_active":"2024-06-14T23:05:16.622959+0000","last_peered":"2024-06-14T23:05:16.622959+0000","last_clean":"2024-06-14T23:05:16.622959+0000","last_became_active":"2024-06-14T23:04:34.888908+0000","last_became_peered":"2024-06-14T23:04:34.888908+0000","last_unstale":"2024-06-14T23:05:16.622959+0000","last_undegraded":"2024-06-14T23:05:16.622959+0000","last_fullsized":"2024-06-14T23:05:16.622959+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:00:24.632734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":147,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.623182+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T23:05:16.623182+0000","last_peered":"2024-06-14T23:05:16.623182+0000","last_clean":"2024-06-14T23:05:16.623182+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T23:05:16.623182+0000","last_undegraded":"2024-06-14T23:05:16.623182+0000","last_fullsized":"2024-06-14T23:05:16.623182+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":186,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.220828+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T23:05:16.220828+0000","last_peered":"2024-06-14T23:05:16.220828+0000","last_clean":"2024-06-14T23:05:16.220828+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T23:05:16.220828+0000","last_undegraded":"2024-06-14T23:05:16.220828+0000","last_fullsized":"2024-06-14T23:05:16.220828+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"243'1545","reported_seq":1559,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:29.127483+0000","last_change":"2024-06-14T23:05:03.808225+0000","last_active":"2024-06-14T23:05:29.127483+0000","last_peered":"2024-06-14T23:05:29.127483+0000","last_clean":"2024-06-14T23:05:29.127483+0000","last_became_active":"2024-06-14T23:05:03.807879+0000","last_became_peered":"2024-06-14T23:05:03.807879+0000","last_unstale":"2024-06-14T23:05:29.127483+0000","last_undegraded":"2024-06-14T23:05:29.127483+0000","last_fullsized":"2024-06-14T23:05:29.127483+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1545,"log_dups_size":0,"ondisk_log_size":1545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:13:19.536572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101253120,"num_objects":1545,"num_object_clones":0,"num_object_copies":4635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1545,"num_write_kb":98880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":543,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891295+0000","last_change":"2024-06-14T23:00:30.719758+0000","last_active":"2024-06-14T23:05:15.891295+0000","last_peered":"2024-06-14T23:05:15.891295+0000","last_clean":"2024-06-14T23:05:15.891295+0000","last_became_active":"2024-06-14T23:00:17.403817+0000","last_became_peered":"2024-06-14T23:00:17.403817+0000","last_unstale":"2024-06-14T23:05:15.891295+0000","last_undegraded":"2024-06-14T23:05:15.891295+0000","last_fullsized":"2024-06-14T23:05:15.891295+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:03:47.728089+0000","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.890879+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:05:15.890879+0000","last_peered":"2024-06-14T23:05:15.890879+0000","last_clean":"2024-06-14T23:05:15.890879+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:05:15.890879+0000","last_undegraded":"2024-06-14T23:05:15.890879+0000","last_fullsized":"2024-06-14T23:05:15.890879+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":197,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971342+0000","last_change":"2024-06-14T22:54:06.277248+0000","last_active":"2024-06-14T23:05:15.971342+0000","last_peered":"2024-06-14T23:05:15.971342+0000","last_clean":"2024-06-14T23:05:15.971342+0000","last_became_active":"2024-06-14T22:54:06.276947+0000","last_became_peered":"2024-06-14T22:54:06.276947+0000","last_unstale":"2024-06-14T23:05:15.971342+0000","last_undegraded":"2024-06-14T23:05:15.971342+0000","last_fullsized":"2024-06-14T23:05:15.971342+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:53.903023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":229,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971673+0000","last_change":"2024-06-14T23:04:34.880222+0000","last_active":"2024-06-14T23:05:15.971673+0000","last_peered":"2024-06-14T23:05:15.971673+0000","last_clean":"2024-06-14T23:05:15.971673+0000","last_became_active":"2024-06-14T23:04:34.879923+0000","last_became_peered":"2024-06-14T23:04:34.879923+0000","last_unstale":"2024-06-14T23:05:15.971673+0000","last_undegraded":"2024-06-14T23:05:15.971673+0000","last_fullsized":"2024-06-14T23:05:15.971673+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:04:31.686965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"243'1641","reported_seq":1655,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:29.113420+0000","last_change":"2024-06-14T23:05:03.808305+0000","last_active":"2024-06-14T23:05:29.113420+0000","last_peered":"2024-06-14T23:05:29.113420+0000","last_clean":"2024-06-14T23:05:29.113420+0000","last_became_active":"2024-06-14T23:05:03.807960+0000","last_became_peered":"2024-06-14T23:05:03.807960+0000","last_unstale":"2024-06-14T23:05:29.113420+0000","last_undegraded":"2024-06-14T23:05:29.113420+0000","last_fullsized":"2024-06-14T23:05:29.113420+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1641,"log_dups_size":0,"ondisk_log_size":1641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:14.633117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107544576,"num_objects":1641,"num_object_clones":0,"num_object_copies":4923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1641,"num_write_kb":105024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":551,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971335+0000","last_change":"2024-06-14T23:00:31.643024+0000","last_active":"2024-06-14T23:05:15.971335+0000","last_peered":"2024-06-14T23:05:15.971335+0000","last_clean":"2024-06-14T23:05:15.971335+0000","last_became_active":"2024-06-14T23:00:17.400819+0000","last_became_peered":"2024-06-14T23:00:17.400819+0000","last_unstale":"2024-06-14T23:05:15.971335+0000","last_undegraded":"2024-06-14T23:05:15.971335+0000","last_fullsized":"2024-06-14T23:05:15.971335+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.642940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.642940+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:52:45.475940+0000","scrub_duration":0.001647253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.104533+0000","last_change":"2024-06-14T23:04:34.879251+0000","last_active":"2024-06-14T23:05:16.104533+0000","last_peered":"2024-06-14T23:05:16.104533+0000","last_clean":"2024-06-14T23:05:16.104533+0000","last_became_active":"2024-06-14T23:04:34.878470+0000","last_became_peered":"2024-06-14T23:04:34.878470+0000","last_unstale":"2024-06-14T23:05:16.104533+0000","last_undegraded":"2024-06-14T23:05:16.104533+0000","last_fullsized":"2024-06-14T23:05:16.104533+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:55.949794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":232,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:19.911883+0000","last_change":"2024-06-14T23:04:34.885379+0000","last_active":"2024-06-14T23:05:19.911883+0000","last_peered":"2024-06-14T23:05:19.911883+0000","last_clean":"2024-06-14T23:05:19.911883+0000","last_became_active":"2024-06-14T23:04:34.885076+0000","last_became_peered":"2024-06-14T23:04:34.885076+0000","last_unstale":"2024-06-14T23:05:19.911883+0000","last_undegraded":"2024-06-14T23:05:19.911883+0000","last_fullsized":"2024-06-14T23:05:19.911883+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:23:32.132929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":198,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891790+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:05:15.891790+0000","last_peered":"2024-06-14T23:05:15.891790+0000","last_clean":"2024-06-14T23:05:15.891790+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:05:15.891790+0000","last_undegraded":"2024-06-14T23:05:15.891790+0000","last_fullsized":"2024-06-14T23:05:15.891790+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"243'1749","reported_seq":1764,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:30.067097+0000","last_change":"2024-06-14T23:05:03.809160+0000","last_active":"2024-06-14T23:05:30.067097+0000","last_peered":"2024-06-14T23:05:30.067097+0000","last_clean":"2024-06-14T23:05:30.067097+0000","last_became_active":"2024-06-14T23:05:03.808799+0000","last_became_peered":"2024-06-14T23:05:03.808799+0000","last_unstale":"2024-06-14T23:05:30.067097+0000","last_undegraded":"2024-06-14T23:05:30.067097+0000","last_fullsized":"2024-06-14T23:05:30.067097+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1749,"log_dups_size":0,"ondisk_log_size":1749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:53.813154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114622464,"num_objects":1749,"num_object_clones":0,"num_object_copies":5247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1749,"num_write_kb":111936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":579,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.622850+0000","last_change":"2024-06-14T23:04:37.903143+0000","last_active":"2024-06-14T23:05:16.622850+0000","last_peered":"2024-06-14T23:05:16.622850+0000","last_clean":"2024-06-14T23:05:16.622850+0000","last_became_active":"2024-06-14T23:04:37.902356+0000","last_became_peered":"2024-06-14T23:04:37.902356+0000","last_unstale":"2024-06-14T23:05:16.622850+0000","last_undegraded":"2024-06-14T23:05:16.622850+0000","last_fullsized":"2024-06-14T23:05:16.622850+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.818076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.818076+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:01:11.632458+0000","scrub_duration":0.0013568390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":208,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.623307+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T23:05:16.623307+0000","last_peered":"2024-06-14T23:05:16.623307+0000","last_clean":"2024-06-14T23:05:16.623307+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T23:05:16.623307+0000","last_undegraded":"2024-06-14T23:05:16.623307+0000","last_fullsized":"2024-06-14T23:05:16.623307+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":204,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889413+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T23:05:15.889413+0000","last_peered":"2024-06-14T23:05:15.889413+0000","last_clean":"2024-06-14T23:05:15.889413+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T23:05:15.889413+0000","last_undegraded":"2024-06-14T23:05:15.889413+0000","last_fullsized":"2024-06-14T23:05:15.889413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":246,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891081+0000","last_change":"2024-06-14T23:04:34.880836+0000","last_active":"2024-06-14T23:05:15.891081+0000","last_peered":"2024-06-14T23:05:15.891081+0000","last_clean":"2024-06-14T23:05:15.891081+0000","last_became_active":"2024-06-14T23:04:34.880540+0000","last_became_peered":"2024-06-14T23:04:34.880540+0000","last_unstale":"2024-06-14T23:05:15.891081+0000","last_undegraded":"2024-06-14T23:05:15.891081+0000","last_fullsized":"2024-06-14T23:05:15.891081+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:06.981511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"243'1901","reported_seq":1916,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:30.484839+0000","last_change":"2024-06-14T23:05:03.625434+0000","last_active":"2024-06-14T23:05:30.484839+0000","last_peered":"2024-06-14T23:05:30.484839+0000","last_clean":"2024-06-14T23:05:30.484839+0000","last_became_active":"2024-06-14T23:05:03.625174+0000","last_became_peered":"2024-06-14T23:05:03.625174+0000","last_unstale":"2024-06-14T23:05:30.484839+0000","last_undegraded":"2024-06-14T23:05:30.484839+0000","last_fullsized":"2024-06-14T23:05:30.484839+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1901,"log_dups_size":0,"ondisk_log_size":1901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:00.079552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124583936,"num_objects":1901,"num_object_clones":0,"num_object_copies":5703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1901,"num_write_kb":121664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":493,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.220809+0000","last_change":"2024-06-14T23:00:42.661659+0000","last_active":"2024-06-14T23:05:16.220809+0000","last_peered":"2024-06-14T23:05:16.220809+0000","last_clean":"2024-06-14T23:05:16.220809+0000","last_became_active":"2024-06-14T23:00:17.398852+0000","last_became_peered":"2024-06-14T23:00:17.398852+0000","last_unstale":"2024-06-14T23:05:16.220809+0000","last_undegraded":"2024-06-14T23:05:16.220809+0000","last_fullsized":"2024-06-14T23:05:16.220809+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:47:14.444805+0000","scrub_duration":0.0053445919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":241,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.623295+0000","last_change":"2024-06-14T23:04:34.880181+0000","last_active":"2024-06-14T23:05:16.623295+0000","last_peered":"2024-06-14T23:05:16.623295+0000","last_clean":"2024-06-14T23:05:16.623295+0000","last_became_active":"2024-06-14T23:04:34.879842+0000","last_became_peered":"2024-06-14T23:04:34.879842+0000","last_unstale":"2024-06-14T23:05:16.623295+0000","last_undegraded":"2024-06-14T23:05:16.623295+0000","last_fullsized":"2024-06-14T23:05:16.623295+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:14:18.433225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891414+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:05:15.891414+0000","last_peered":"2024-06-14T23:05:15.891414+0000","last_clean":"2024-06-14T23:05:15.891414+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:05:15.891414+0000","last_undegraded":"2024-06-14T23:05:15.891414+0000","last_fullsized":"2024-06-14T23:05:15.891414+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":198,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891116+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:05:15.891116+0000","last_peered":"2024-06-14T23:05:15.891116+0000","last_clean":"2024-06-14T23:05:15.891116+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:05:15.891116+0000","last_undegraded":"2024-06-14T23:05:15.891116+0000","last_fullsized":"2024-06-14T23:05:15.891116+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"243'1748","reported_seq":1761,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:30.132019+0000","last_change":"2024-06-14T23:05:03.621987+0000","last_active":"2024-06-14T23:05:30.132019+0000","last_peered":"2024-06-14T23:05:30.132019+0000","last_clean":"2024-06-14T23:05:30.132019+0000","last_became_active":"2024-06-14T23:05:03.621724+0000","last_became_peered":"2024-06-14T23:05:03.621724+0000","last_unstale":"2024-06-14T23:05:30.132019+0000","last_undegraded":"2024-06-14T23:05:30.132019+0000","last_fullsized":"2024-06-14T23:05:30.132019+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1748,"log_dups_size":0,"ondisk_log_size":1748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:12.558177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114556928,"num_objects":1748,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1748,"num_write_kb":111872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":493,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.220704+0000","last_change":"2024-06-14T23:00:31.555325+0000","last_active":"2024-06-14T23:05:16.220704+0000","last_peered":"2024-06-14T23:05:16.220704+0000","last_clean":"2024-06-14T23:05:16.220704+0000","last_became_active":"2024-06-14T23:00:17.400871+0000","last_became_peered":"2024-06-14T23:00:17.400871+0000","last_unstale":"2024-06-14T23:05:16.220704+0000","last_undegraded":"2024-06-14T23:05:16.220704+0000","last_fullsized":"2024-06-14T23:05:16.220704+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:58.191436+0000","scrub_duration":0.001665094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":231,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891406+0000","last_change":"2024-06-14T23:04:34.880684+0000","last_active":"2024-06-14T23:05:15.891406+0000","last_peered":"2024-06-14T23:05:15.891406+0000","last_clean":"2024-06-14T23:05:15.891406+0000","last_became_active":"2024-06-14T23:04:34.880127+0000","last_became_peered":"2024-06-14T23:04:34.880127+0000","last_unstale":"2024-06-14T23:05:15.891406+0000","last_undegraded":"2024-06-14T23:05:15.891406+0000","last_fullsized":"2024-06-14T23:05:15.891406+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:53:52.966928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":208,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.230336+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:05:16.230336+0000","last_peered":"2024-06-14T23:05:16.230336+0000","last_clean":"2024-06-14T23:05:16.230336+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:05:16.230336+0000","last_undegraded":"2024-06-14T23:05:16.230336+0000","last_fullsized":"2024-06-14T23:05:16.230336+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":183,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971531+0000","last_change":"2024-06-14T22:44:51.841407+0000","last_active":"2024-06-14T23:05:15.971531+0000","last_peered":"2024-06-14T23:05:15.971531+0000","last_clean":"2024-06-14T23:05:15.971531+0000","last_became_active":"2024-06-14T22:44:51.840699+0000","last_became_peered":"2024-06-14T22:44:51.840699+0000","last_unstale":"2024-06-14T23:05:15.971531+0000","last_undegraded":"2024-06-14T23:05:15.971531+0000","last_fullsized":"2024-06-14T23:05:15.971531+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:32.171930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"243'1689","reported_seq":1704,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:30.128558+0000","last_change":"2024-06-14T23:05:03.621656+0000","last_active":"2024-06-14T23:05:30.128558+0000","last_peered":"2024-06-14T23:05:30.128558+0000","last_clean":"2024-06-14T23:05:30.128558+0000","last_became_active":"2024-06-14T23:05:03.620834+0000","last_became_peered":"2024-06-14T23:05:03.620834+0000","last_unstale":"2024-06-14T23:05:30.128558+0000","last_undegraded":"2024-06-14T23:05:30.128558+0000","last_fullsized":"2024-06-14T23:05:30.128558+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1689,"log_dups_size":0,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:59:26.118782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110690304,"num_objects":1689,"num_object_clones":0,"num_object_copies":5067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1689,"num_write_kb":108096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":580,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889373+0000","last_change":"2024-06-14T23:04:37.901715+0000","last_active":"2024-06-14T23:05:15.889373+0000","last_peered":"2024-06-14T23:05:15.889373+0000","last_clean":"2024-06-14T23:05:15.889373+0000","last_became_active":"2024-06-14T23:04:37.901351+0000","last_became_peered":"2024-06-14T23:04:37.901351+0000","last_unstale":"2024-06-14T23:05:15.889373+0000","last_undegraded":"2024-06-14T23:05:15.889373+0000","last_fullsized":"2024-06-14T23:05:15.889373+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.174132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.174132+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:55.141528+0000","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":105,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971623+0000","last_change":"2024-06-14T22:58:09.752729+0000","last_active":"2024-06-14T23:05:15.971623+0000","last_peered":"2024-06-14T23:05:15.971623+0000","last_clean":"2024-06-14T23:05:15.971623+0000","last_became_active":"2024-06-14T22:58:09.752393+0000","last_became_peered":"2024-06-14T22:58:09.752393+0000","last_unstale":"2024-06-14T23:05:15.971623+0000","last_undegraded":"2024-06-14T23:05:15.971623+0000","last_fullsized":"2024-06-14T23:05:15.971623+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:21:58.662622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":197,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891509+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T23:05:15.891509+0000","last_peered":"2024-06-14T23:05:15.891509+0000","last_clean":"2024-06-14T23:05:15.891509+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T23:05:15.891509+0000","last_undegraded":"2024-06-14T23:05:15.891509+0000","last_fullsized":"2024-06-14T23:05:15.891509+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":188,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.229702+0000","last_change":"2024-06-14T22:58:32.754079+0000","last_active":"2024-06-14T23:05:16.229702+0000","last_peered":"2024-06-14T23:05:16.229702+0000","last_clean":"2024-06-14T23:05:16.229702+0000","last_became_active":"2024-06-14T22:58:32.753814+0000","last_became_peered":"2024-06-14T22:58:32.753814+0000","last_unstale":"2024-06-14T23:05:16.229702+0000","last_undegraded":"2024-06-14T23:05:16.229702+0000","last_fullsized":"2024-06-14T23:05:16.229702+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:37:07.414370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.e","version":"243'1654","reported_seq":1668,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:29.149986+0000","last_change":"2024-06-14T23:05:03.623137+0000","last_active":"2024-06-14T23:05:29.149986+0000","last_peered":"2024-06-14T23:05:29.149986+0000","last_clean":"2024-06-14T23:05:29.149986+0000","last_became_active":"2024-06-14T23:05:03.622833+0000","last_became_peered":"2024-06-14T23:05:03.622833+0000","last_unstale":"2024-06-14T23:05:29.149986+0000","last_undegraded":"2024-06-14T23:05:29.149986+0000","last_fullsized":"2024-06-14T23:05:29.149986+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":1654,"log_dups_size":0,"ondisk_log_size":1654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:53:45.610358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108396544,"num_objects":1654,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1654,"num_write_kb":105856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":543,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891761+0000","last_change":"2024-06-14T23:00:32.716013+0000","last_active":"2024-06-14T23:05:15.891761+0000","last_peered":"2024-06-14T23:05:15.891761+0000","last_clean":"2024-06-14T23:05:15.891761+0000","last_became_active":"2024-06-14T23:00:17.398349+0000","last_became_peered":"2024-06-14T23:00:17.398349+0000","last_unstale":"2024-06-14T23:05:15.891761+0000","last_undegraded":"2024-06-14T23:05:15.891761+0000","last_fullsized":"2024-06-14T23:05:15.891761+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.715907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.715907+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:23:40.212866+0000","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":201,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971563+0000","last_change":"2024-06-14T22:49:25.497181+0000","last_active":"2024-06-14T23:05:15.971563+0000","last_peered":"2024-06-14T23:05:15.971563+0000","last_clean":"2024-06-14T23:05:15.971563+0000","last_became_active":"2024-06-14T22:49:25.496508+0000","last_became_peered":"2024-06-14T22:49:25.496508+0000","last_unstale":"2024-06-14T23:05:15.971563+0000","last_undegraded":"2024-06-14T23:05:15.971563+0000","last_fullsized":"2024-06-14T23:05:15.971563+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:29.327403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":228,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.220870+0000","last_change":"2024-06-14T23:04:34.890587+0000","last_active":"2024-06-14T23:05:16.220870+0000","last_peered":"2024-06-14T23:05:16.220870+0000","last_clean":"2024-06-14T23:05:16.220870+0000","last_became_active":"2024-06-14T23:04:34.890254+0000","last_became_peered":"2024-06-14T23:04:34.890254+0000","last_unstale":"2024-06-14T23:05:16.220870+0000","last_undegraded":"2024-06-14T23:05:16.220870+0000","last_fullsized":"2024-06-14T23:05:16.220870+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:28:06.189496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":186,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.888887+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T23:05:15.888887+0000","last_peered":"2024-06-14T23:05:15.888887+0000","last_clean":"2024-06-14T23:05:15.888887+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T23:05:15.888887+0000","last_undegraded":"2024-06-14T23:05:15.888887+0000","last_fullsized":"2024-06-14T23:05:15.888887+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889462+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T23:05:15.889462+0000","last_peered":"2024-06-14T23:05:15.889462+0000","last_clean":"2024-06-14T23:05:15.889462+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T23:05:15.889462+0000","last_undegraded":"2024-06-14T23:05:15.889462+0000","last_fullsized":"2024-06-14T23:05:15.889462+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":209,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889386+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T23:05:15.889386+0000","last_peered":"2024-06-14T23:05:15.889386+0000","last_clean":"2024-06-14T23:05:15.889386+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T23:05:15.889386+0000","last_undegraded":"2024-06-14T23:05:15.889386+0000","last_fullsized":"2024-06-14T23:05:15.889386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":230,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891064+0000","last_change":"2024-06-14T23:04:34.885556+0000","last_active":"2024-06-14T23:05:15.891064+0000","last_peered":"2024-06-14T23:05:15.891064+0000","last_clean":"2024-06-14T23:05:15.891064+0000","last_became_active":"2024-06-14T23:04:34.885222+0000","last_became_peered":"2024-06-14T23:04:34.885222+0000","last_unstale":"2024-06-14T23:05:15.891064+0000","last_undegraded":"2024-06-14T23:05:15.891064+0000","last_fullsized":"2024-06-14T23:05:15.891064+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:19:04.372165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":123,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891853+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:05:15.891853+0000","last_peered":"2024-06-14T23:05:15.891853+0000","last_clean":"2024-06-14T23:05:15.891853+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:05:15.891853+0000","last_undegraded":"2024-06-14T23:05:15.891853+0000","last_fullsized":"2024-06-14T23:05:15.891853+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":204,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.623215+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:05:16.623215+0000","last_peered":"2024-06-14T23:05:16.623215+0000","last_clean":"2024-06-14T23:05:16.623215+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:05:16.623215+0000","last_undegraded":"2024-06-14T23:05:16.623215+0000","last_fullsized":"2024-06-14T23:05:16.623215+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":147,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.220232+0000","last_change":"2024-06-14T22:58:09.753103+0000","last_active":"2024-06-14T23:05:16.220232+0000","last_peered":"2024-06-14T23:05:16.220232+0000","last_clean":"2024-06-14T23:05:16.220232+0000","last_became_active":"2024-06-14T22:58:09.752761+0000","last_became_peered":"2024-06-14T22:58:09.752761+0000","last_unstale":"2024-06-14T23:05:16.220232+0000","last_undegraded":"2024-06-14T23:05:16.220232+0000","last_fullsized":"2024-06-14T23:05:16.220232+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:20:20.619331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":196,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971525+0000","last_change":"2024-06-14T22:54:06.285013+0000","last_active":"2024-06-14T23:05:15.971525+0000","last_peered":"2024-06-14T23:05:15.971525+0000","last_clean":"2024-06-14T23:05:15.971525+0000","last_became_active":"2024-06-14T22:54:06.283594+0000","last_became_peered":"2024-06-14T22:54:06.283594+0000","last_unstale":"2024-06-14T23:05:15.971525+0000","last_undegraded":"2024-06-14T23:05:15.971525+0000","last_fullsized":"2024-06-14T23:05:15.971525+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:06.363884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":198,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891700+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T23:05:15.891700+0000","last_peered":"2024-06-14T23:05:15.891700+0000","last_clean":"2024-06-14T23:05:15.891700+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T23:05:15.891700+0000","last_undegraded":"2024-06-14T23:05:15.891700+0000","last_fullsized":"2024-06-14T23:05:15.891700+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.220800+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T23:05:16.220800+0000","last_peered":"2024-06-14T23:05:16.220800+0000","last_clean":"2024-06-14T23:05:16.220800+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T23:05:16.220800+0000","last_undegraded":"2024-06-14T23:05:16.220800+0000","last_fullsized":"2024-06-14T23:05:16.220800+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":193,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-06-14T23:05:20.269608+0000","last_change":"2024-06-14T22:58:09.785074+0000","last_active":"2024-06-14T23:05:20.269608+0000","last_peered":"2024-06-14T23:05:20.269608+0000","last_clean":"2024-06-14T23:05:20.269608+0000","last_became_active":"2024-06-14T22:58:09.784309+0000","last_became_peered":"2024-06-14T22:58:09.784309+0000","last_unstale":"2024-06-14T23:05:20.269608+0000","last_undegraded":"2024-06-14T23:05:20.269608+0000","last_fullsized":"2024-06-14T23:05:20.269608+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:23.581930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":199,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971635+0000","last_change":"2024-06-14T22:54:06.277938+0000","last_active":"2024-06-14T23:05:15.971635+0000","last_peered":"2024-06-14T23:05:15.971635+0000","last_clean":"2024-06-14T23:05:15.971635+0000","last_became_active":"2024-06-14T22:54:06.277632+0000","last_became_peered":"2024-06-14T22:54:06.277632+0000","last_unstale":"2024-06-14T23:05:15.971635+0000","last_undegraded":"2024-06-14T23:05:15.971635+0000","last_fullsized":"2024-06-14T23:05:15.971635+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:16.564712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":212,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.890988+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:05:15.890988+0000","last_peered":"2024-06-14T23:05:15.890988+0000","last_clean":"2024-06-14T23:05:15.890988+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:05:15.890988+0000","last_undegraded":"2024-06-14T23:05:15.890988+0000","last_fullsized":"2024-06-14T23:05:15.890988+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":35,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.891154+0000","last_change":"2024-06-14T23:04:12.880798+0000","last_active":"2024-06-14T23:05:15.891154+0000","last_peered":"2024-06-14T23:05:15.891154+0000","last_clean":"2024-06-14T23:05:15.891154+0000","last_became_active":"2024-06-14T23:04:12.880499+0000","last_became_peered":"2024-06-14T23:04:12.880499+0000","last_unstale":"2024-06-14T23:05:15.891154+0000","last_undegraded":"2024-06-14T23:05:15.891154+0000","last_fullsized":"2024-06-14T23:05:15.891154+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:29:35.037498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":242,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889218+0000","last_change":"2024-06-14T23:04:34.883863+0000","last_active":"2024-06-14T23:05:15.889218+0000","last_peered":"2024-06-14T23:05:15.889218+0000","last_clean":"2024-06-14T23:05:15.889218+0000","last_became_active":"2024-06-14T23:04:34.881668+0000","last_became_peered":"2024-06-14T23:04:34.881668+0000","last_unstale":"2024-06-14T23:05:15.889218+0000","last_undegraded":"2024-06-14T23:05:15.889218+0000","last_fullsized":"2024-06-14T23:05:15.889218+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:44:45.737416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.104503+0000","last_change":"2024-06-14T23:04:34.879498+0000","last_active":"2024-06-14T23:05:16.104503+0000","last_peered":"2024-06-14T23:05:16.104503+0000","last_clean":"2024-06-14T23:05:16.104503+0000","last_became_active":"2024-06-14T23:04:34.878932+0000","last_became_peered":"2024-06-14T23:04:34.878932+0000","last_unstale":"2024-06-14T23:05:16.104503+0000","last_undegraded":"2024-06-14T23:05:16.104503+0000","last_fullsized":"2024-06-14T23:05:16.104503+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:56:50.201189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":185,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971538+0000","last_change":"2024-06-14T22:44:51.841584+0000","last_active":"2024-06-14T23:05:15.971538+0000","last_peered":"2024-06-14T23:05:15.971538+0000","last_clean":"2024-06-14T23:05:15.971538+0000","last_became_active":"2024-06-14T22:44:51.841025+0000","last_became_peered":"2024-06-14T22:44:51.841025+0000","last_unstale":"2024-06-14T23:05:15.971538+0000","last_undegraded":"2024-06-14T23:05:15.971538+0000","last_fullsized":"2024-06-14T23:05:15.971538+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:59:59.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":220,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889310+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:05:15.889310+0000","last_peered":"2024-06-14T23:05:15.889310+0000","last_clean":"2024-06-14T23:05:15.889310+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:05:15.889310+0000","last_undegraded":"2024-06-14T23:05:15.889310+0000","last_fullsized":"2024-06-14T23:05:15.889310+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":206,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.623075+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:05:16.623075+0000","last_peered":"2024-06-14T23:05:16.623075+0000","last_clean":"2024-06-14T23:05:16.623075+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:05:16.623075+0000","last_undegraded":"2024-06-14T23:05:16.623075+0000","last_fullsized":"2024-06-14T23:05:16.623075+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":220,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.889009+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:05:15.889009+0000","last_peered":"2024-06-14T23:05:15.889009+0000","last_clean":"2024-06-14T23:05:15.889009+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:05:15.889009+0000","last_undegraded":"2024-06-14T23:05:15.889009+0000","last_fullsized":"2024-06-14T23:05:15.889009+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":189,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.623338+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T23:05:16.623338+0000","last_peered":"2024-06-14T23:05:16.623338+0000","last_clean":"2024-06-14T23:05:16.623338+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T23:05:16.623338+0000","last_undegraded":"2024-06-14T23:05:16.623338+0000","last_fullsized":"2024-06-14T23:05:16.623338+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.104339+0000","last_change":"2024-06-14T23:04:34.879700+0000","last_active":"2024-06-14T23:05:16.104339+0000","last_peered":"2024-06-14T23:05:16.104339+0000","last_clean":"2024-06-14T23:05:16.104339+0000","last_became_active":"2024-06-14T23:04:34.878924+0000","last_became_peered":"2024-06-14T23:04:34.878924+0000","last_unstale":"2024-06-14T23:05:16.104339+0000","last_undegraded":"2024-06-14T23:05:16.104339+0000","last_fullsized":"2024-06-14T23:05:16.104339+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:28.826486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":114,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:16.104624+0000","last_change":"2024-06-14T23:05:09.659896+0000","last_active":"2024-06-14T23:05:16.104624+0000","last_peered":"2024-06-14T23:05:16.104624+0000","last_clean":"2024-06-14T23:05:16.104624+0000","last_became_active":"2024-06-14T23:05:09.659539+0000","last_became_peered":"2024-06-14T23:05:09.659539+0000","last_unstale":"2024-06-14T23:05:16.104624+0000","last_undegraded":"2024-06-14T23:05:16.104624+0000","last_fullsized":"2024-06-14T23:05:16.104624+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:21.285896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.888789+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T23:05:15.888789+0000","last_peered":"2024-06-14T23:05:15.888789+0000","last_clean":"2024-06-14T23:05:15.888789+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T23:05:15.888789+0000","last_undegraded":"2024-06-14T23:05:15.888789+0000","last_fullsized":"2024-06-14T23:05:15.888789+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":231,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-06-14T23:05:15.971533+0000","last_change":"2024-06-14T23:04:34.888253+0000","last_active":"2024-06-14T23:05:15.971533+0000","last_peered":"2024-06-14T23:05:15.971533+0000","last_clean":"2024-06-14T23:05:15.971533+0000","last_became_active":"2024-06-14T23:04:34.881268+0000","last_became_peered":"2024-06-14T23:04:34.881268+0000","last_unstale":"2024-06-14T23:05:15.971533+0000","last_undegraded":"2024-06-14T23:05:15.971533+0000","last_fullsized":"2024-06-14T23:05:15.971533+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:11:14.951006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1762983936,"num_objects":26901,"num_object_clones":0,"num_object_copies":80703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26901,"num_write_kb":1721664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5247336448,"data_stored":5247336448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26901,"ondisk_log_size":26901,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1208,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1771,"ondisk_log_size":1771,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234377,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1477620,"kb_used_data":578668,"kb_used_omap":1,"kb_used_meta":898942,"kb_avail":92267532,"statfs":{"total":95995035648,"available":94481952768,"internally_reserved":0,"allocated":592556032,"data_stored":591342533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920516952},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:05:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.94699999999999995,"15min":1.022},"min":{"1min":0.28399999999999997,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.8999999999999999,"5min":3.3519999999999999,"15min":6.9539999999999997},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.234,"15min":1.161},"min":{"1min":0.26900000000000002,"5min":0.22600000000000001,"15min":0.20300000000000001},"max":{"1min":1.742,"5min":6.9550000000000001,"15min":6.9550000000000001},"last":1.3080000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":1.0149999999999999,"15min":1.6879999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.6970000000000001,"5min":6.3090000000000002,"15min":83.295000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.88700000000000001,"15min":1.7050000000000001},"min":{"1min":0.28299999999999997,"5min":0.22900000000000001,"15min":0.223},"max":{"1min":2.8149999999999999,"5min":6.2279999999999998,"15min":82.972999999999999},"last":0.55700000000000005}]},{"osd":2,"last update":"Fri Jun 14 23:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.97799999999999998,"15min":0.88900000000000001},"min":{"1min":0.36599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":3.456,"5min":7.468,"15min":7.468},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":1.022,"15min":0.96499999999999997},"min":{"1min":0.375,"5min":0.252,"15min":0.252},"max":{"1min":2.879,"5min":6.0469999999999997,"15min":8.1150000000000002},"last":0.69699999999999995}]},{"osd":3,"last update":"Fri Jun 14 23:05:17 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.2689999999999999,"15min":1.2310000000000001},"min":{"1min":0.32500000000000001,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":7,"5min":7.8609999999999998,"15min":55.683999999999997},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.2090000000000001,"15min":1.2450000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.29799999999999999},"max":{"1min":1.4790000000000001,"5min":7.7670000000000003,"15min":55.533999999999999},"last":1.337}]},{"osd":4,"last update":"Fri Jun 14 23:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.2190000000000001,"15min":0.92800000000000005},"min":{"1min":0.25,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.3810000000000002,"5min":11.342000000000001,"15min":11.342000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.127,"5min":1.284,"15min":0.99199999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.0960000000000001,"5min":12.651,"15min":16.879000000000001},"last":0.373}]},{"osd":5,"last update":"Fri Jun 14 23:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.1240000000000001,"15min":1.046},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.383,"5min":6.4320000000000004,"15min":6.9000000000000004},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1080000000000001,"15min":1.044},"min":{"1min":0.318,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":5.4240000000000004,"5min":6.2809999999999997,"15min":6.8529999999999998},"last":1.266}]},{"osd":6,"last update":"Fri Jun 14 23:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.363,"15min":1.04},"min":{"1min":0.35499999999999998,"5min":0.307,"15min":0.307},"max":{"1min":4.444,"5min":8.952,"15min":8.952},"last":1.3859999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.323,"15min":1.0169999999999999},"min":{"1min":0.308,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.0629999999999997,"5min":7.0629999999999997,"15min":12.951000000000001},"last":0.88300000000000001}]}]},{"osd":6,"up_from":46,"seq":197568496013,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1694180,"kb_used_data":801052,"kb_used_omap":1,"kb_used_meta":893118,"kb_avail":92050972,"statfs":{"total":95995035648,"available":94260195328,"internally_reserved":0,"allocated":820277248,"data_stored":819079664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914553176},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:05:19 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":2.117,"15min":1.4490000000000001},"min":{"1min":0.38,"5min":0.23400000000000001,"15min":0.224},"max":{"1min":3.847,"5min":45.579999999999998,"15min":45.579999999999998},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":1.756,"15min":1.222},"min":{"1min":0.19800000000000001,"5min":0.19,"15min":0.19},"max":{"1min":2.8980000000000001,"5min":44.918999999999997,"15min":44.918999999999997},"last":1.254}]},{"osd":1,"last update":"Fri Jun 14 23:04:49 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.96899999999999997,"15min":1.1419999999999999},"min":{"1min":0.22900000000000001,"5min":0.223,"15min":0.223},"max":{"1min":3.6539999999999999,"5min":4.6260000000000003,"15min":9.9559999999999995},"last":2.831},{"interface":"front","average":{"1min":1.258,"5min":1.0629999999999999,"15min":1.206},"min":{"1min":0.38600000000000001,"5min":0.27400000000000002,"15min":0.247},"max":{"1min":3.9849999999999999,"5min":6.0750000000000002,"15min":9.5510000000000002},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:04:49 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.2170000000000001,"15min":0.90000000000000002},"min":{"1min":0.48599999999999999,"5min":0.30299999999999999,"15min":0.27800000000000002},"max":{"1min":4.0069999999999997,"5min":9.9600000000000009,"15min":9.9600000000000009},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.391,"5min":1.4059999999999999,"15min":1.0069999999999999},"min":{"1min":0.46100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.9220000000000002,"5min":10.823,"15min":10.823},"last":0.42099999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:05:19 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.256,"15min":1.1240000000000001},"min":{"1min":0.42799999999999999,"5min":0.32000000000000001,"15min":0.27300000000000002},"max":{"1min":3.8839999999999999,"5min":5.0949999999999998,"15min":29.152000000000001},"last":1.143},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.2030000000000001,"15min":1.1319999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":2.9769999999999999,"5min":7.3799999999999999,"15min":25.864000000000001},"last":1.4059999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:04:36 2024","interfaces":[{"interface":"back","average":{"1min":2.0960000000000001,"5min":1.2430000000000001,"15min":1.0580000000000001},"min":{"1min":0.32700000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.7119999999999997,"5min":7.7709999999999999,"15min":47.698},"last":0.51200000000000001},{"interface":"front","average":{"1min":2.0310000000000001,"5min":1.143,"15min":1.0229999999999999},"min":{"1min":0.32800000000000001,"5min":0.28000000000000003,"15min":0.24299999999999999},"max":{"1min":3.9159999999999999,"5min":4.6790000000000003,"15min":48.197000000000003},"last":0.31900000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:04:36 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.194,"15min":0.97899999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.1420000000000003,"5min":7.3540000000000001,"15min":8.6500000000000004},"last":2.7989999999999999},{"interface":"front","average":{"1min":1.8799999999999999,"5min":1.0920000000000001,"15min":1},"min":{"1min":0.35299999999999998,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":4.548,"5min":6.1079999999999997,"15min":8.6349999999999998},"last":0.47699999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:05:10 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.01,"15min":0.96099999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.9020000000000001,"5min":4.4720000000000004,"15min":8.6950000000000003},"last":2.9060000000000001},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.0469999999999999,"15min":1.006},"min":{"1min":0.39400000000000002,"5min":0.34899999999999998,"15min":0.28899999999999998},"max":{"1min":2.5739999999999998,"5min":4.1470000000000002,"15min":8.673},"last":0.58899999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757652,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1310880,"kb_used_data":419736,"kb_used_omap":1,"kb_used_meta":891134,"kb_avail":92434272,"statfs":{"total":95995035648,"available":94652694528,"internally_reserved":0,"allocated":429809664,"data_stored":428612534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912521560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:04:40 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":2.0659999999999998,"15min":1.411},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":8.3710000000000004,"5min":14.94,"15min":14.94},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.857,"5min":1.9319999999999999,"15min":1.3280000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":8.3550000000000004,"5min":9.3819999999999997,"15min":9.3819999999999997},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:04:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.78400000000000003,"15min":1.0569999999999999},"min":{"1min":0.26100000000000001,"5min":0.23999999999999999,"15min":0.23300000000000001},"max":{"1min":7.4370000000000003,"5min":7.4370000000000003,"15min":7.694},"last":0.753},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.3140000000000001,"15min":1.2529999999999999},"min":{"1min":0.28199999999999997,"5min":0.27600000000000002,"15min":0.23400000000000001},"max":{"1min":3.6960000000000002,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":1.919}]},{"osd":2,"last update":"Fri Jun 14 23:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.569,"15min":1.0549999999999999},"min":{"1min":0.32500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.931,"5min":14.891,"15min":14.891},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.244,"5min":1.6919999999999999,"15min":1.081},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.2080000000000002,"5min":9.532,"15min":9.532},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.4370000000000001,"15min":1.3400000000000001},"min":{"1min":0.26500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":3.4089999999999998,"5min":9.2149999999999999,"15min":45.744},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.579,"15min":1.4450000000000001},"min":{"1min":0.36099999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.27,"5min":14.914,"15min":45.838000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Jun 14 23:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":1.0049999999999999,"15min":0.80200000000000005},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.23300000000000001},"max":{"1min":1.637,"5min":14.352,"15min":14.352},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.085,"5min":1.458,"15min":0.98899999999999999},"min":{"1min":0.29299999999999998,"5min":0.28399999999999997,"15min":0.22700000000000001},"max":{"1min":5.4080000000000004,"5min":9.5980000000000008,"15min":9.5980000000000008},"last":0.63500000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:04:45 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.52,"15min":1.0640000000000001},"min":{"1min":0.35199999999999998,"5min":0.28100000000000003,"15min":0.253},"max":{"1min":6.819,"5min":15.382999999999999,"15min":15.382999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.49,"15min":1.036},"min":{"1min":0.307,"5min":0.29599999999999999,"15min":0.25},"max":{"1min":6.6879999999999997,"5min":14.855,"15min":14.855},"last":2.1699999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:05:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.3939999999999999,"15min":1.0700000000000001},"min":{"1min":0.39000000000000001,"5min":0.30499999999999999,"15min":0.30199999999999999},"max":{"1min":6.29,"5min":14.818,"15min":14.818},"last":2.101},{"interface":"front","average":{"1min":0.81899999999999995,"5min":1.1020000000000001,"15min":0.96399999999999997},"min":{"1min":0.33400000000000002,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":4.7329999999999997,"5min":14.429,"15min":14.429},"last":0.78100000000000003}]}]},{"osd":4,"up_from":30,"seq":128849019288,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1314284,"kb_used_data":419428,"kb_used_omap":1,"kb_used_meta":894782,"kb_avail":92430868,"statfs":{"total":95995035648,"available":94649208832,"internally_reserved":0,"allocated":429494272,"data_stored":428288630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916257112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:04:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0619999999999998,"5min":1.6850000000000001,"15min":1.1819999999999999},"min":{"1min":0.23499999999999999,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":6.101,"5min":10.567,"15min":10.567},"last":1.901},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.669,"15min":1.1919999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.4989999999999997,"5min":8.3390000000000004,"15min":8.3390000000000004},"last":0.254}]},{"osd":1,"last update":"Fri Jun 14 23:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.849,"15min":1.4039999999999999},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.2679999999999998,"5min":56.917000000000002,"15min":56.917000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.1200000000000001,"5min":2.0059999999999998,"15min":1.482},"min":{"1min":0.26700000000000002,"5min":0.25700000000000001,"15min":0.249},"max":{"1min":5.6619999999999999,"5min":56.789999999999999,"15min":56.789999999999999},"last":0.45200000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3029999999999999,"15min":0.89100000000000001},"min":{"1min":0.29599999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":5.2779999999999996,"5min":8.4860000000000007,"15min":8.9250000000000007},"last":1.7709999999999999},{"interface":"front","average":{"1min":1.71,"5min":1.4399999999999999,"15min":0.96799999999999997},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.6319999999999997,"5min":10.478,"15min":10.478},"last":1.786}]},{"osd":3,"last update":"Fri Jun 14 23:04:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":1.258,"15min":0.97199999999999998},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.8759999999999999,"5min":6.327,"15min":6.617},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.587,"5min":1.5469999999999999,"15min":1.095},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":6.1440000000000001,"5min":10.510999999999999,"15min":10.510999999999999},"last":0.52900000000000003}]},{"osd":5,"last update":"Fri Jun 14 23:05:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":1.3,"15min":1.5269999999999999},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.24199999999999999},"max":{"1min":2.254,"5min":12.315,"15min":139.13200000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":1.2689999999999999,"15min":1.552},"min":{"1min":0.308,"5min":0.29699999999999999,"15min":0.26100000000000001},"max":{"1min":3.0110000000000001,"5min":10.959,"15min":138.86000000000001},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:04:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.24,"15min":0.879},"min":{"1min":0.38100000000000001,"5min":0.28000000000000003,"15min":0.25800000000000001},"max":{"1min":1.853,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.212,"5min":1.1599999999999999,"15min":0.83299999999999996},"min":{"1min":0.32100000000000001,"5min":0.27900000000000003,"15min":0.254},"max":{"1min":4.3680000000000003,"5min":8,"15min":8.8819999999999997},"last":0.51300000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:05:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.089,"15min":0.90400000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.0890000000000004,"5min":6.0890000000000004,"15min":8.9489999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.0700000000000001,"15min":1.0069999999999999},"min":{"1min":0.32500000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.5169999999999999,"5min":5.3479999999999999,"15min":8.4730000000000008},"last":0.38900000000000001}]}]},{"osd":3,"up_from":138,"seq":592705486987,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099488,"kb_used_data":716952,"kb_used_omap":4,"kb_used_meta":382523,"kb_avail":92645664,"statfs":{"total":95995035648,"available":94869159936,"internally_reserved":0,"allocated":734158848,"data_stored":732961491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":391704030},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":1.3859999999999999,"15min":0.98299999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.184},"max":{"1min":1.171,"5min":11.071999999999999,"15min":11.071999999999999},"last":0.222},{"interface":"front","average":{"1min":0.53800000000000003,"5min":1.429,"15min":0.92100000000000004},"min":{"1min":0.20799999999999999,"5min":0.19,"15min":0.17999999999999999},"max":{"1min":3.173,"5min":8.2330000000000005,"15min":8.2330000000000005},"last":0.19900000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":1.222,"15min":1.0169999999999999},"min":{"1min":0.247,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.27,"5min":9.6080000000000005,"15min":24.097999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.2470000000000001,"15min":1.0529999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":1.857,"5min":6.6189999999999998,"15min":24.132999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.425,"15min":0.96399999999999997},"min":{"1min":0.26900000000000002,"5min":0.245,"15min":0.23599999999999999},"max":{"1min":3.1299999999999999,"5min":9.4730000000000008,"15min":9.4730000000000008},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.3560000000000001,"15min":0.91000000000000003},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20899999999999999},"max":{"1min":3.1539999999999999,"5min":10.369,"15min":10.369},"last":0.27400000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.4990000000000001,"15min":1.0649999999999999},"min":{"1min":0.34499999999999997,"5min":0.33000000000000002,"15min":0.27900000000000003},"max":{"1min":3.3290000000000002,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":1.331,"15min":1.0640000000000001},"min":{"1min":0.35599999999999998,"5min":0.29499999999999998,"15min":0.27300000000000002},"max":{"1min":4.2889999999999997,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.53000000000000003}]},{"osd":5,"last update":"Fri Jun 14 23:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.171,"15min":1.0820000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78600000000000003,"5min":6.2560000000000002,"15min":6.2560000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":1.3180000000000001,"15min":1.073},"min":{"1min":0.36599999999999999,"5min":0.32400000000000001,"15min":0.30199999999999999},"max":{"1min":2.1819999999999999,"5min":8.4019999999999992,"15min":8.4019999999999992},"last":0.72999999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.4199999999999999,"15min":1.0680000000000001},"min":{"1min":0.38200000000000001,"5min":0.34499999999999997,"15min":0.32400000000000001},"max":{"1min":4.6870000000000003,"5min":4.984,"15min":4.984},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.661,"15min":1.1319999999999999},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.31},"max":{"1min":3.883,"5min":7.7320000000000002,"15min":7.7320000000000002},"last":0.44900000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":1.345,"15min":1.1439999999999999},"min":{"1min":0.371,"5min":0.35599999999999998,"15min":0.35299999999999998},"max":{"1min":3.3010000000000002,"5min":5.5300000000000002,"15min":7.7240000000000002},"last":1.8680000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":1.044,"15min":1.0589999999999999},"min":{"1min":0.377,"5min":0.36299999999999999,"15min":0.32400000000000001},"max":{"1min":3.7109999999999999,"5min":5.2160000000000002,"15min":5.8300000000000001},"last":0.79500000000000004}]}]},{"osd":2,"up_from":133,"seq":571230650514,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":974688,"kb_used_data":688024,"kb_used_omap":6,"kb_used_meta":286649,"kb_avail":92770464,"statfs":{"total":95995035648,"available":94996955136,"internally_reserved":0,"allocated":704536576,"data_stored":703339433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":293529077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":0.99299999999999999,"15min":0.70399999999999996},"min":{"1min":0.217,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":8.4510000000000005,"5min":8.4510000000000005,"15min":8.4510000000000005},"last":3.113},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.4590000000000001,"15min":0.91300000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":5.4219999999999997,"5min":11.369,"15min":11.369},"last":1.8240000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.86199999999999999,"15min":0.873},"min":{"1min":0.22500000000000001,"5min":0.221,"15min":0.20799999999999999},"max":{"1min":1.498,"5min":9.1349999999999998,"15min":21.920000000000002},"last":2.9060000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.82499999999999996,"15min":0.84199999999999997},"min":{"1min":0.25600000000000001,"5min":0.217,"15min":0.217},"max":{"1min":5.4820000000000002,"5min":5.7439999999999998,"15min":21.573},"last":0.69899999999999995}]},{"osd":3,"last update":"Fri Jun 14 23:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":1.357,"15min":1.0529999999999999},"min":{"1min":0.27900000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.76600000000000001,"5min":11.24,"15min":21.66},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":1.393,"15min":1.0700000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":2.7029999999999998,"5min":11.271000000000001,"15min":22.341000000000001},"last":2.7029999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":2.2120000000000002,"15min":1.286},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.318},"max":{"1min":6.21,"5min":31.928000000000001,"15min":31.928000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.8460000000000001,"15min":1.1499999999999999},"min":{"1min":0.38,"5min":0.308,"15min":0.28899999999999998},"max":{"1min":2.3330000000000002,"5min":31.954999999999998,"15min":31.954999999999998},"last":2.6749999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.409,"15min":0.999},"min":{"1min":0.33300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.7510000000000003,"5min":31.539999999999999,"15min":31.539999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.329,"5min":1.776,"15min":1.1180000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29399999999999998},"max":{"1min":5.4530000000000003,"5min":31.481999999999999,"15min":31.481999999999999},"last":0.71499999999999997}]},{"osd":6,"last update":"Fri Jun 14 23:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.7949999999999999,"15min":1.0780000000000001},"min":{"1min":0.38100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":2.8460000000000001,"5min":32.005000000000003,"15min":32.005000000000003},"last":2.5710000000000002},{"interface":"front","average":{"1min":1.413,"5min":1.706,"15min":1.079},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.30399999999999999},"max":{"1min":5.5330000000000004,"5min":32.079999999999998,"15min":32.079999999999998},"last":0.72899999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":1.2210000000000001,"15min":0.996},"min":{"1min":0.41999999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.998,"5min":10.614000000000001,"15min":10.614000000000001},"last":2.8069999999999999},{"interface":"front","average":{"1min":1.129,"5min":0.98999999999999999,"15min":0.83999999999999997},"min":{"1min":0.39600000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":7.6260000000000003},"last":0.65600000000000003}]}]},{"osd":1,"up_from":116,"seq":498216206531,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1348184,"kb_used_data":842896,"kb_used_omap":5,"kb_used_meta":505274,"kb_avail":92396968,"statfs":{"total":95995035648,"available":94614495232,"internally_reserved":0,"allocated":863125504,"data_stored":861935705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":517400856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.2410000000000001,"15min":1.4790000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.17499999999999999},"max":{"1min":6.3730000000000002,"5min":9.5350000000000001,"15min":11.257999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.401,"15min":1.2949999999999999},"min":{"1min":0.24099999999999999,"5min":0.19700000000000001,"15min":0.186},"max":{"1min":3.6760000000000002,"5min":13.672000000000001,"15min":13.672000000000001},"last":0.872}]},{"osd":2,"last update":"Fri Jun 14 23:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":1.4390000000000001,"15min":1.073},"min":{"1min":0.216,"5min":0.21299999999999999,"15min":0.19500000000000001},"max":{"1min":6.4100000000000001,"5min":9.5470000000000006,"15min":9.5470000000000006},"last":1.361},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.1970000000000001,"15min":0.82399999999999995},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":6.3259999999999996,"5min":8.0060000000000002,"15min":8.0060000000000002},"last":0.94899999999999995}]},{"osd":3,"last update":"Fri Jun 14 23:05:22 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":1.012,"15min":1.0669999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":0.497,"5min":6.2060000000000004,"15min":21.422000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":1.175,"15min":1.1539999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.214},"max":{"1min":1.0009999999999999,"5min":8.1219999999999999,"15min":21.016999999999999},"last":1.3879999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:04:56 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.1639999999999999,"15min":1.335},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.2939999999999996,"5min":6.5350000000000001,"15min":8.8599999999999994},"last":1.145},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.232,"15min":1.4970000000000001},"min":{"1min":0.33400000000000002,"5min":0.29899999999999999,"15min":0.29199999999999998},"max":{"1min":2.7749999999999999,"5min":6.2960000000000003,"15min":11.161},"last":1.2769999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":1.1459999999999999,"15min":1.4359999999999999},"min":{"1min":0.33200000000000002,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":2.9980000000000002,"5min":6.0049999999999999,"15min":10.103},"last":1.222},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0840000000000001,"15min":1.319},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.28599999999999998},"max":{"1min":3.9220000000000002,"5min":5.8250000000000002,"15min":8.125},"last":1.004}]},{"osd":6,"last update":"Fri Jun 14 23:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.885,"15min":1.6359999999999999},"min":{"1min":0.37,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":3.0739999999999998,"5min":63.795999999999999,"15min":63.795999999999999},"last":1.26},{"interface":"front","average":{"1min":0.79500000000000004,"5min":1.8180000000000001,"15min":1.5589999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.28499999999999998},"max":{"1min":3.0640000000000001,"5min":63.811,"15min":63.811},"last":0.71999999999999997}]},{"osd":7,"last update":"Fri Jun 14 23:04:56 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":1.105,"15min":1.4890000000000001},"min":{"1min":0.41399999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.7989999999999999,"5min":8.0440000000000005,"15min":9.4429999999999996},"last":1.3300000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.96099999999999997,"15min":1.3009999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.36499999999999999},"max":{"1min":0.88300000000000001,"5min":5.9580000000000002,"15min":9.5449999999999999},"last":1.0980000000000001}]}]},{"osd":0,"up_from":91,"seq":390842024185,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1166104,"kb_used_data":426576,"kb_used_omap":5,"kb_used_meta":739514,"kb_avail":92579048,"statfs":{"total":95995035648,"available":94800945152,"internally_reserved":0,"allocated":436813824,"data_stored":435622024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":757262426},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.98799999999999999,"15min":1.4279999999999999},"min":{"1min":0.17499999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":6.7080000000000002,"5min":9.4380000000000006,"15min":16.305},"last":1.284},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.1910000000000001,"15min":1.5609999999999999},"min":{"1min":0.222,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":2.4089999999999998,"5min":9.9990000000000006,"15min":16.184999999999999},"last":0.48499999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:05:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":1.4750000000000001,"15min":0.997},"min":{"1min":0.25600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.73499999999999999,"5min":10.053000000000001,"15min":16.757000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.755,"5min":1.3560000000000001,"15min":1.042},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.222},"max":{"1min":6.2809999999999997,"5min":9.2769999999999992,"15min":16.055},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Jun 14 23:04:30 2024","interfaces":[{"interface":"back","average":{"1min":1.8999999999999999,"5min":1.482,"15min":1.0700000000000001},"min":{"1min":0.39800000000000002,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":11.083,"5min":11.083,"15min":16.873999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.427,"15min":1.048},"min":{"1min":0.27500000000000002,"5min":0.224,"15min":0.224},"max":{"1min":11.375,"5min":11.375,"15min":16.085999999999999},"last":1.214}]},{"osd":4,"last update":"Fri Jun 14 23:04:30 2024","interfaces":[{"interface":"back","average":{"1min":2.419,"5min":1.7529999999999999,"15min":1.4339999999999999},"min":{"1min":0.33700000000000002,"5min":0.30199999999999999,"15min":0.27600000000000002},"max":{"1min":11.162000000000001,"5min":16.527000000000001,"15min":16.527000000000001},"last":1.153},{"interface":"front","average":{"1min":1.427,"5min":1.492,"15min":1.329},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.2300000000000004,"5min":14.475,"15min":15.976000000000001},"last":0.76600000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:04:30 2024","interfaces":[{"interface":"back","average":{"1min":2.149,"5min":1.591,"15min":1.518},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":10.462,"5min":10.462,"15min":16.108000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.867,"5min":1.754,"15min":1.5},"min":{"1min":0.434,"5min":0.32500000000000001,"15min":0.31900000000000001},"max":{"1min":6.0650000000000004,"5min":11.436,"15min":16.006},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Jun 14 23:04:30 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.571,"15min":1.421},"min":{"1min":0.40000000000000002,"5min":0.33600000000000002,"15min":0.30599999999999999},"max":{"1min":3.7850000000000001,"5min":14.247,"15min":16.152000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.831,"5min":1.986,"15min":1.4750000000000001},"min":{"1min":0.441,"5min":0.35299999999999998,"15min":0.29399999999999998},"max":{"1min":5.6269999999999998,"5min":14.523999999999999,"15min":16.227},"last":0.72699999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:04:30 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.6279999999999999,"15min":1.3959999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":6.0970000000000004,"5min":11.364000000000001,"15min":16.199999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.8380000000000001,"5min":1.587,"15min":1.458},"min":{"1min":0.45800000000000002,"5min":0.36699999999999999,"15min":0.314},"max":{"1min":11.195,"5min":11.195,"15min":16.259},"last":1.1919999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":447807488,"data_stored":447807488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":915800064,"data_stored":915800064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":711524352,"data_stored":711524352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":781516800,"data_stored":781516800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":435879936,"data_stored":435879936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":449183744,"data_stored":449183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":867041280,"data_stored":867041280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":638582784,"data_stored":638582784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:05:33.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set unique_pool_6 pgp_num 16 2024-06-14T23:05:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:33.990 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:33.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:34.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:34.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:34.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:34.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:34.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:33 smithi137 ceph-mon[90634]: from='client.55645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:05:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:33 smithi137 ceph-mon[90634]: pgmap v944: 99 pgs: 99 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-06-14T23:05:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2813073810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:05:34.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 37108 37092 165.553 135.125 0.00901493 0.00603435 2024-06-14T23:05:34.280 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:33 smithi086 ceph-mon[97913]: from='client.55645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:05:34.281 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:33 smithi086 ceph-mon[97913]: pgmap v944: 99 pgs: 99 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-06-14T23:05:34.281 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2813073810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:05:34.281 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:33 smithi086 ceph-mon[102754]: from='client.55645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:05:34.281 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:33 smithi086 ceph-mon[102754]: pgmap v944: 99 pgs: 99 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-06-14T23:05:34.281 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2813073810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:05:34.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:34.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:34.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:34.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:34.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:34.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:34.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:34.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:34.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:34.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:34.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:34.578 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:34.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:34.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:34.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:34.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:34.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:34.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:34.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:34.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:34.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:34.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:35.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:35.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:05:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:05:34] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.43.0" 2024-06-14T23:05:35.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:34 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:05:35.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:34 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:05:35.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2813073810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:05:35.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:34 smithi086 ceph-mon[97913]: osdmap e244: 8 total, 8 up, 8 in 2024-06-14T23:05:35.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:34 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:05:35.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:34 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:05:35.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2813073810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:05:35.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:34 smithi086 ceph-mon[102754]: osdmap e244: 8 total, 8 up, 8 in 2024-06-14T23:05:35.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:35.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:35.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:35.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:35.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:34 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:05:35.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:34 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:05:35.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2813073810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:05:35.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:34 smithi137 ceph-mon[90634]: osdmap e244: 8 total, 8 up, 8 in 2024-06-14T23:05:35.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:35.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:35.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 39170 39154 163.108 128.875 0.00484311 0.00612577 2024-06-14T23:05:35.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:35.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:35.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:35.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:35.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:35.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:35.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:35.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:35.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:35.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:35.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:35.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:35.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:35.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:35.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:35.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:36.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:36.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:36.148 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:35 smithi086 ceph-mon[97913]: pgmap v946: 99 pgs: 99 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 158 MiB/s wr, 2.52k op/s 2024-06-14T23:05:36.148 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:35 smithi086 ceph-mon[102754]: pgmap v946: 99 pgs: 99 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 158 MiB/s wr, 2.52k op/s 2024-06-14T23:05:36.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:36.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:36.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:36.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:35 smithi137 ceph-mon[90634]: pgmap v946: 99 pgs: 99 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 158 MiB/s wr, 2.52k op/s 2024-06-14T23:05:36.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 41242 41226 161.007 129.5 0.00657261 0.0062067 2024-06-14T23:05:36.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:36.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:36.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:36.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:36.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:36.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:36.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:36.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:36.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:36.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:36.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:36.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:36.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:36.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:36.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:36.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:37.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:37.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:37.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:37.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:37.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:37.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:05:37.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:37.269 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 16 43238 43222 158.873 124.75 0.00641453 0.00628977 2024-06-14T23:05:37.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:37.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:37.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:37.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:37.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:37.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:37.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:37.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:37.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:37.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:37.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:37.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:37.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:37.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:37.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:37.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:37 smithi086 ceph-mon[97913]: pgmap v947: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-06-14T23:05:37.950 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:37 smithi086 ceph-mon[102754]: pgmap v947: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24715 9.d 9:b7390049:::benchmark_data_smithi086_152062_object24714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.661292+0000", 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 585.19910529100002, 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.018053425, 2024-06-14T23:05:37.965 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24715 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:37.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661292+0000", 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661292+0000", 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661290+0000", 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:37.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661315+0000", 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5228999999999998e-05 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661325+0000", 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8689999999999999e-06 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661332+0000", 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8990000000000004e-06 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661364+0000", 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2121e-05 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.661375+0000", 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0645999999999999e-05 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.665805+0000", 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.004429807 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:37.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.666631+0000", 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082573 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.666788+0000", 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000157484 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673421+0000", 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066326279999999998 2024-06-14T23:05:37.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673465+0000", 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4218999999999998e-05 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679261+0000", 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057959120000000003 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679308+0000", 2024-06-14T23:05:37.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6998e-05 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679345+0000", 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7234999999999998e-05 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24723 9.d 9:b2f904ff:::benchmark_data_smithi086_152062_object24722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.664488+0000", 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 585.19590851400005, 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0150036360000001, 2024-06-14T23:05:37.972 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24723 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664488+0000", 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664488+0000", 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664487+0000", 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664523+0000", 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5485000000000003e-05 2024-06-14T23:05:37.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664538+0000", 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5412000000000001e-05 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664545+0000", 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.781e-06 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664598+0000", 2024-06-14T23:05:37.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2976000000000002e-05 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.664607+0000", 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6290000000000005e-06 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668290+0000", 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0036830910000001 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:37.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668426+0000", 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013543800000000001 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.668573+0000", 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147498 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673521+0000", 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049472129999999998 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.676833+0000", 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0033120250000000001 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679379+0000", 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025462699999999998 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679445+0000", 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6476999999999994e-05 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679492+0000", 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6640000000000001e-05 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24729 9.d 9:b6335b33:::benchmark_data_smithi086_152062_object24728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.667823+0000", 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 585.192573541, 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0118812850000001, 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:37.979 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24729 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667823+0000", 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667823+0000", 2024-06-14T23:05:37.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667822+0000", 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667847+0000", 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4666000000000001e-05 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:37.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667864+0000", 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7611000000000001e-05 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.667878+0000", 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3182e-05 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668333+0000", 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000454931 2024-06-14T23:05:37.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668342+0000", 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1630000000000002e-06 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.669852+0000", 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.00151035 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.670010+0000", 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000158257 2024-06-14T23:05:37.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.670183+0000", 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017219199999999999 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.677648+0000", 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007465546 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679516+0000", 2024-06-14T23:05:37.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001868429 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679566+0000", 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9395999999999999e-05 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679670+0000", 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010412900000000001 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:37.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679705+0000", 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4591999999999997e-05 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24730 9.d 9:be481d2c:::benchmark_data_smithi086_152062_object24729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.668425+0000", 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 585.19197166499998, 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0114436330000001, 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24730 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668425+0000", 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668447+0000", 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2699000000000001e-05 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668454+0000", 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6880000000000002e-06 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668459+0000", 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4940000000000003e-06 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668500+0000", 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0577000000000001e-05 2024-06-14T23:05:37.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:05:37.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.668508+0000", 2024-06-14T23:05:37.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5229999999999998e-06 2024-06-14T23:05:37.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.671828+0000", 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.003320011 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.671965+0000", 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000137081 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.672100+0000", 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013549800000000001 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.677685+0000", 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055849710000000002 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679521+0000", 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001835761 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679784+0000", 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000263465 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679825+0000", 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0980999999999999e-05 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679869+0000", 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3461000000000003e-05 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:24738 9.d 9:b3d2a961:::benchmark_data_smithi086_152062_object24737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-14T23:05:37.993 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:55:52.671698+0000", 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 585.18869854800005, 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.009193183, 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 24738 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671698+0000", 2024-06-14T23:05:37.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671698+0000", 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671697+0000", 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671720+0000", 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3345000000000001e-05 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671734+0000", 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3669000000000001e-05 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671742+0000", 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7470000000000005e-06 2024-06-14T23:05:37.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671822+0000", 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.051e-05 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:52.671834+0000", 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1847e-05 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673548+0000", 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.001713718 2024-06-14T23:05:37.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673671+0000", 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012297600000000001 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.673796+0000", 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012489900000000001 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679254+0000", 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054580119999999999 2024-06-14T23:05:37.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.679525+0000", 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00027125800000000002 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680809+0000", 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012839290000000001 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680856+0000", 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7094e-05 2024-06-14T23:05:37.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:55:53.680891+0000", 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5438999999999997e-05 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249750 9.2 9:4ea8c052:::benchmark_data_smithi086_152062_object117268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.641870+0000", 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 369.21852709699999, 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.071249919999999994, 2024-06-14T23:05:38.000 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249750 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641870+0000", 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641870+0000", 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641869+0000", 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641874+0000", 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.605e-06 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641889+0000", 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5305999999999999e-05 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.641896+0000", 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7360000000000004e-06 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.680196+0000", 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038300010000000002 2024-06-14T23:05:38.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.680308+0000", 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112149 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.680416+0000", 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108577 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693896+0000", 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.013479804999999999 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.693903+0000", 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4049999999999997e-06 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.713033+0000", 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019130945999999999 2024-06-14T23:05:38.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.713081+0000", 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7457999999999999e-05 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.713120+0000", 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8748999999999998e-05 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.006 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.35580.0:249775 9.2 9:4b4108a1:::benchmark_data_smithi086_152062_object117293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T22:59:28.674316+0000", 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 369.18608095799999, 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.079108740999999996, 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.35580", 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2967504444", 2024-06-14T23:05:38.007 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 249775 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674316+0000", 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674316+0000", 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674315+0000", 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674321+0000", 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2870000000000001e-06 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674334+0000", 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3183999999999999e-05 2024-06-14T23:05:38.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.674341+0000", 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7009999999999999e-06 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.690107+0000", 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015766227000000001 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.690222+0000", 2024-06-14T23:05:38.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115213 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.690337+0000", 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114992 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.697790+0000", 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074525579999999998 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.705669+0000", 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0078787340000000001 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753341+0000", 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047672842999999999 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753388+0000", 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7018999999999998e-05 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T22:59:28.753425+0000", 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6276000000000003e-05 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:117237 10.2 10:4e671e84:::benchmark_data_smithi086_189745_object117236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:53.026979+0000", 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 224.83341737699999, 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.096224995999999993, 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.013 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117237 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.026979+0000", 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.026979+0000", 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.026978+0000", 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027001+0000", 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2802999999999998e-05 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027016+0000", 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5668999999999999e-05 2024-06-14T23:05:38.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027023+0000", 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9890000000000003e-06 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027062+0000", 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8475e-05 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027171+0000", 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109152 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.027268+0000", 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6864e-05 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.035630+0000", 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083618860000000007 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.042402+0000", 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067718520000000001 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.123118+0000", 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.080716689999999994 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.123175+0000", 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6718999999999997e-05 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.123204+0000", 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9386999999999998e-05 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:117241 10.2 10:40bacbc2:::benchmark_data_smithi086_189745_object117240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:53.030165+0000", 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 224.830231416, 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.097320450000000003, 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:05:38.019 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117241 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030165+0000", 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030165+0000", 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030164+0000", 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030214+0000", 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0188999999999997e-05 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030233+0000", 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8879999999999999e-05 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.030240+0000", 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2690000000000002e-06 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.031168+0000", 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000928151 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.031297+0000", 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012866900000000001 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.031433+0000", 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013556899999999999 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.035638+0000", 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042048800000000002 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.042528+0000", 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0068906940000000002 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.127425+0000", 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.084896588999999995 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.127459+0000", 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3997e-05 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.127486+0000", 2024-06-14T23:05:38.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7046000000000002e-05 2024-06-14T23:05:38.025 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.025 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.025 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.025 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:117247 10.2 10:4a6dbd53:::benchmark_data_smithi086_189745_object117246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:05:38.025 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:01:53.038191+0000", 2024-06-14T23:05:38.026 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 224.82220617199999, 2024-06-14T23:05:38.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.096470944000000003, 2024-06-14T23:05:38.026 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.026 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.026 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 117247 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038191+0000", 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038191+0000", 2024-06-14T23:05:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038189+0000", 2024-06-14T23:05:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038215+0000", 2024-06-14T23:05:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5704e-05 2024-06-14T23:05:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038233+0000", 2024-06-14T23:05:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8318999999999999e-05 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038241+0000", 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.276e-06 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038326+0000", 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4302999999999996e-05 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038564+0000", 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000238741 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.038720+0000", 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000155718 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.042551+0000", 2024-06-14T23:05:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038304789999999999 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.043231+0000", 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068008399999999996 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.134573+0000", 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.091342603999999994 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.134622+0000", 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8588000000000003e-05 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:01:53.134662+0000", 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9749000000000002e-05 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:208238 10.2 10:45e8300f:::benchmark_data_smithi086_189745_object77270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:03:29.019259+0000", 2024-06-14T23:05:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 128.84113825099999, 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.075338019000000006, 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 208238 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.019259+0000", 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.019259+0000", 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.019257+0000", 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.019265+0000", 2024-06-14T23:05:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3729999999999993e-06 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.019284+0000", 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8561999999999999e-05 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.019294+0000", 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0417999999999999e-05 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.022979+0000", 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036846539999999999 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.023447+0000", 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00046795499999999999 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.023883+0000", 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000436883 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.029893+0000", 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060092990000000001 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.057680+0000", 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.02778721 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.094206+0000", 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036525850999999998 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.094494+0000", 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00028773399999999999 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.094597+0000", 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103062 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:208239 10.2 10:4b1c0263:::benchmark_data_smithi086_189745_object77271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-06-14T23:05:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:03:29.025774+0000", 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 128.834622564, 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.069078123000000005, 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 208239 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.025774+0000", 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.025774+0000", 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.025773+0000", 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.025780+0000", 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4410000000000004e-06 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.025799+0000", 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8278e-05 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.025807+0000", 2024-06-14T23:05:38.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6489999999999997e-06 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.029925+0000", 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041180519999999997 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.030053+0000", 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000128054 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:05:38.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.030175+0000", 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012210899999999999 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.040486+0000", 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010311114 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.057687+0000", 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017200264 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.094640+0000", 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036952798000000002 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.094813+0000", 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017347700000000001 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.094852+0000", 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9387000000000001e-05 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.045 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:208244 10.2 10:4b710359:::benchmark_data_smithi086_189745_object77276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:03:29.029576+0000", 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 128.83082110000001, 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.065396803000000003, 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:05:38.046 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 208244 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.029576+0000", 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.029576+0000", 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.029574+0000", 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.029582+0000", 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9470000000000003e-06 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.029601+0000", 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8544000000000001e-05 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.029677+0000", 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6403999999999996e-05 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.034690+0000", 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0050129679999999996 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.034841+0000", 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015139799999999999 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.035078+0000", 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000236569 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.040657+0000", 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055789730000000001 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.057691+0000", 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017034108999999999 2024-06-14T23:05:38.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.094896+0000", 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037204464 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.094941+0000", 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5129000000000001e-05 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:03:29.094973+0000", 2024-06-14T23:05:38.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1890999999999997e-05 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.36237.0:24239 11.7 e243/237 11:e2dabb81:::benchmark_data_smithi086_226760_object24238:head v 243'1536, mlcod=243'1532)", 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:05:28.216029+0000", 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.6443678560000006, 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.084066189999999999, 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.216029+0000", 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.216029+0000", 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.216028+0000", 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.216061+0000", 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2589000000000003e-05 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.216071+0000", 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0302000000000001e-05 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.216077+0000", 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6720000000000001e-06 2024-06-14T23:05:38.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.216114+0000", 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6606999999999997e-05 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.216133+0000", 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9167000000000001e-05 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.269076+0000", 2024-06-14T23:05:38.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.052942889 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.300095+0000", 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031019482000000001 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.36237.0:24241 11.7 e243/237 11:eceaebdc:::benchmark_data_smithi086_226760_object24240:head v 243'1537, mlcod=243'1532)", 2024-06-14T23:05:38.056 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:05:28.217239+0000", 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.6431574490000003, 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.082902580000000003, 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.217239+0000", 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.217239+0000", 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.217239+0000", 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.218061+0000", 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00082244600000000001 2024-06-14T23:05:38.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.260573+0000", 2024-06-14T23:05:38.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042511383 2024-06-14T23:05:38.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.260590+0000", 2024-06-14T23:05:38.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7319000000000001e-05 2024-06-14T23:05:38.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.262354+0000", 2024-06-14T23:05:38.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017638420000000001 2024-06-14T23:05:38.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.262370+0000", 2024-06-14T23:05:38.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6748999999999998e-05 2024-06-14T23:05:38.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.300102+0000", 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037731366000000002 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:28.300142+0000", 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0151999999999998e-05 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.062 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:33059 11.c 11:3f5ab2c1:::benchmark_data_smithi086_226760_object33058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:05:32.216284+0000", 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.6441127289999997, 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.088900249000000001, 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 33059 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.063 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.216284+0000", 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.216284+0000", 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.216282+0000", 2024-06-14T23:05:38.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.216307+0000", 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4542e-05 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.216322+0000", 2024-06-14T23:05:38.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4599e-05 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.216329+0000", 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1500000000000002e-06 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.216380+0000", 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1450999999999999e-05 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.216545+0000", 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016499 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.216694+0000", 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014826999999999999 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.219021+0000", 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023278639999999998 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.219488+0000", 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000466296 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.305057+0000", 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.085569354 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.305144+0000", 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6786000000000006e-05 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.305184+0000", 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0578999999999999e-05 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:33070 11.c 11:377270a4:::benchmark_data_smithi086_226760_object33069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:05:32.225368+0000", 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.6350289440000001, 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.079956940000000004, 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.069 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 33070 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.225368+0000", 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.225368+0000", 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.225367+0000", 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.225385+0000", 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7997999999999999e-05 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.225396+0000", 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1027e-05 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.225415+0000", 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8462999999999999e-05 2024-06-14T23:05:38.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.225486+0000", 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1107999999999997e-05 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.225583+0000", 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7207000000000006e-05 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-06-14T23:05:38.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.225672+0000", 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9247999999999997e-05 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.226975+0000", 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001302377 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.228436+0000", 2024-06-14T23:05:38.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001461279 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.305238+0000", 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076802290999999995 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.305287+0000", 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8606999999999997e-05 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.305325+0000", 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8164999999999998e-05 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:33075 11.c 11:3d60bcbc:::benchmark_data_smithi086_226760_object33074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:05:32.226942+0000", 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.6334552049999997, 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.078552413000000001, 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.076 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 33075 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.226942+0000", 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.226942+0000", 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.226941+0000", 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.226962+0000", 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0392000000000001e-05 2024-06-14T23:05:38.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.226968+0000", 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7440000000000001e-06 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.226973+0000", 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.831e-06 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227013+0000", 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9641999999999998e-05 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227211+0000", 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019805199999999999 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227348+0000", 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013675800000000001 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.231277+0000", 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003929157 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.232236+0000", 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00095947399999999998 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.305410+0000", 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.073174300999999997 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.305461+0000", 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0327000000000001e-05 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.305494+0000", 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3265000000000001e-05 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.082 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:33076 11.c 11:31436627:::benchmark_data_smithi086_226760_object33075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:05:32.227232+0000", 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.6331650389999997, 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.080104700000000001, 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 33076 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227232+0000", 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227232+0000", 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227231+0000", 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:05:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227250+0000", 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9340000000000001e-05 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227264+0000", 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3964999999999999e-05 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227269+0000", 2024-06-14T23:05:38.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5929999999999999e-06 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.228459+0000", 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001190214 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.228578+0000", 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011873 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.228709+0000", 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013064899999999999 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.231311+0000", 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026019200000000002 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.232241+0000", 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000930196 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.307263+0000", 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.075022440999999995 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.307309+0000", 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5998999999999998e-05 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.307337+0000", 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7373e-05 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:33077 11.c 11:3111b3ca:::benchmark_data_smithi086_226760_object33076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:05:32.227362+0000", 2024-06-14T23:05:38.089 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 5.6330346750000002, 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.080232861000000003, 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 33077 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227362+0000", 2024-06-14T23:05:38.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227362+0000", 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227362+0000", 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227378+0000", 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6503000000000001e-05 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227396+0000", 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8144000000000001e-05 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.227401+0000", 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7539999999999997e-06 2024-06-14T23:05:38.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.229955+0000", 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002554373 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.230061+0000", 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000105515 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-06-14T23:05:38.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.230175+0000", 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011461800000000001 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.232245+0000", 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020691120000000001 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.233260+0000", 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010157499999999999 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.307528+0000", 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.074267334000000004 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.307565+0000", 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7400000000000001e-05 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:05:32.307595+0000", 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9971000000000002e-05 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.095 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.096 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.096 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:05:38.096 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:05:38.096 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:05:38.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:38.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:38.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:38.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:38.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:37 smithi137 ceph-mon[90634]: pgmap v947: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-06-14T23:05:38.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:38.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:38.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 45008 44992 156.193 110.625 0.0120515 0.00639746 2024-06-14T23:05:38.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:38.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:38.422 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:38.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:38.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:38.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:38.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:38.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:38.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:38.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:38.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:38.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:38.881 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:05:38.881 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:05:38.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:38.917 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:05:38.918 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:05:38.945 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:05:38.945 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:05:38.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:38.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:38.981 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:05:38.981 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:05:39.014 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:05:39.014 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:05:39.046 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:05:39.047 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:05:39.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:39.083 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:05:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:05:39.089 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:39.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:39.116 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:05:39.117 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:05:39.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:39.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:39.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:39.243 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:38 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1814571659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:39.243 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:38 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1814571659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:39.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:38 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1814571659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:39.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 46790 46774 153.834 111.375 0.00393877 0.00649518 2024-06-14T23:05:39.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:39.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:39.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:39.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:39.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:39.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:39.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:39.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T23:05:39.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:39.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:39.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:39.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:39.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:39.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:39.813 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:39.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:39.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:39.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:40.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:40.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:40.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:40.176 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:40.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:40.176 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:39 smithi086 ceph-mon[102754]: pgmap v948: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-06-14T23:05:40.176 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:39 smithi086 ceph-mon[97913]: pgmap v948: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-06-14T23:05:40.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:39 smithi137 ceph-mon[90634]: pgmap v948: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-06-14T23:05:40.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:05:40.269752+0000 min lat: 0.00101502 max lat: 0.0907664 avg lat: 0.00658667 2024-06-14T23:05:40.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:05:40.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 48563 48547 151.682 110.812 0.00864821 0.00658667 2024-06-14T23:05:40.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:40.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:40.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:40.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:40.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:40.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:40.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:40.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:40.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:40.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:40.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:40.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:40.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:40.860 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:40.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:40.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:40.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:40.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:40.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:41.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:41.129 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:41.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:41.230 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:41.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:41.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:41.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 15 50392 50377 149.905 114.375 0.00567358 0.00666581 2024-06-14T23:05:41.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:41.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:41.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:41.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:41.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:41.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:41.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:41.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:41.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:41.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:41.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:41.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:41.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:41.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:41.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:41.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:42.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:42.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:42.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:42.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:42.229 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:42.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:42.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:42.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:41 smithi137 ceph-mon[90634]: pgmap v949: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-06-14T23:05:42.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 52191 52175 148.199 112.375 0.00458895 0.00674328 2024-06-14T23:05:42.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:42.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:41 smithi086 ceph-mon[97913]: pgmap v949: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-06-14T23:05:42.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:41 smithi086 ceph-mon[102754]: pgmap v949: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-06-14T23:05:42.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:42.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:42.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:42.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:42.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:42.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:05:42.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:42.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:42.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:42.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:42.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:42.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:42.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:42.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:42.894 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:42.910 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:42.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:43.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:43.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:43.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:43.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:43.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:43.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:43.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-14T23:05:43.234 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:43.234 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:43.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-14T23:05:43.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:43.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:43.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-14T23:05:43.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:43.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:43.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 53762 53746 146.024 98.1875 0.0181887 0.00684303 2024-06-14T23:05:43.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:43.331 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:43.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:43.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:43.473 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:43.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:43.575 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:43.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:43.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:43.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:43.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:43.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:43.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:43.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:43.866 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:43.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:43.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:44.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:44.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:44.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:44.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:44.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:44.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:44.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:44.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:43 smithi137 ceph-mon[90634]: pgmap v950: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-06-14T23:05:44.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-06-14T23:05:44.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:43 smithi137 ceph-mon[90634]: osdmap e245: 8 total, 8 up, 8 in 2024-06-14T23:05:44.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 55430 55414 144.284 104.25 0.0101168 0.00692616 2024-06-14T23:05:44.273 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:44.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:44.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:43 smithi086 ceph-mon[102754]: pgmap v950: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-06-14T23:05:44.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-06-14T23:05:44.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:43 smithi086 ceph-mon[102754]: osdmap e245: 8 total, 8 up, 8 in 2024-06-14T23:05:44.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:43 smithi086 ceph-mon[97913]: pgmap v950: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-06-14T23:05:44.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-06-14T23:05:44.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:43 smithi086 ceph-mon[97913]: osdmap e245: 8 total, 8 up, 8 in 2024-06-14T23:05:44.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:44.409 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:44.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:44.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:44.539 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:44.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:44.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:44.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:44.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:44.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:44.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:44.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:44.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:44.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:44.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:44.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:44.955 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:05:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:05:44] "GET /metrics HTTP/1.1" 200 36594 "" "Prometheus/2.43.0" 2024-06-14T23:05:45.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:45.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:45.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:45.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:45.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:45.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:45.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:44 smithi137 ceph-mon[90634]: osdmap e246: 8 total, 8 up, 8 in 2024-06-14T23:05:45.271 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 57053 57037 142.57 101.438 0.0066058 0.00700934 2024-06-14T23:05:45.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:45.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:44 smithi086 ceph-mon[97913]: osdmap e246: 8 total, 8 up, 8 in 2024-06-14T23:05:45.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:44 smithi086 ceph-mon[102754]: osdmap e246: 8 total, 8 up, 8 in 2024-06-14T23:05:45.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:45.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:45.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:45.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:45.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:45.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:45.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:45.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:45.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:45.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:45.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:45.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:45.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:45.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:45.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:45.980 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 11 pgp_num to 16 2024-06-14T23:05:46.010 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:46.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:46.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:46.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:46.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:46.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:45 smithi086 ceph-mon[97913]: pgmap v953: 99 pgs: 99 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-06-14T23:05:46.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3094872394' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:05:46.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:45 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:05:46.235 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:45 smithi086 ceph-mon[102754]: pgmap v953: 99 pgs: 99 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-06-14T23:05:46.235 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3094872394' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:05:46.235 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:45 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:05:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:45 smithi137 ceph-mon[90634]: pgmap v953: 99 pgs: 99 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-06-14T23:05:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3094872394' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:05:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:45 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:05:46.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:46.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 58620 58604 140.853 97.9375 0.00778512 0.00709394 2024-06-14T23:05:46.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:46.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:46.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:46.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:46.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:46.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:46.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:46.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:46.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:46.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:46.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:46.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:46.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:46.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:46.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:46.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:46.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:47.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:47.109 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:46 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:05:47.109 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:46 smithi086 ceph-mon[102754]: osdmap e247: 8 total, 8 up, 8 in 2024-06-14T23:05:47.109 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:46 smithi086 ceph-mon[102754]: pgmap v955: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-14T23:05:47.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:46 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:05:47.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:46 smithi086 ceph-mon[97913]: osdmap e247: 8 total, 8 up, 8 in 2024-06-14T23:05:47.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:46 smithi086 ceph-mon[97913]: pgmap v955: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-14T23:05:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:47.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:47.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:47.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:47.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:46 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:05:47.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:46 smithi137 ceph-mon[90634]: osdmap e247: 8 total, 8 up, 8 in 2024-06-14T23:05:47.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:46 smithi137 ceph-mon[90634]: pgmap v955: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-14T23:05:47.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:47.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:47.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 60146 60130 139.168 95.375 0.0107842 0.00718044 2024-06-14T23:05:47.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:47.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:47.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:47.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:47.537 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:47.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:47.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:47.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:47.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:47.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:47.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:47.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:47.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:47.921 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:47.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:47.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:47.945 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:05:47.945 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:05:48.002 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:05:48.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:48.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/515589027' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:05:48.056 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/515589027' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:05:48.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:48.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:48.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:48.220 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:48.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:48.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 61709 61693 137.687 97.6875 0.00869496 0.00725818 2024-06-14T23:05:48.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:48.357 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:48.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:48.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:48.492 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:48.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:48.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/515589027' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:05:48.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:48.624 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:48.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:48.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:48.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:48.763 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:48.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:48.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:48.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:48.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:48.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:49.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:49.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:49.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:49.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:49.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:49.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:49.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:49.270 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 63256 63240 136.273 96.6875 0.0107339 0.00733313 2024-06-14T23:05:49.280 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:49 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:05:49.281 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/515589027' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:05:49.281 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:49 smithi086 ceph-mon[97913]: osdmap e248: 8 total, 8 up, 8 in 2024-06-14T23:05:49.281 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:49 smithi086 ceph-mon[97913]: pgmap v957: 99 pgs: 99 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-14T23:05:49.281 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:49 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:05:49.281 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/515589027' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:05:49.281 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:49 smithi086 ceph-mon[102754]: osdmap e248: 8 total, 8 up, 8 in 2024-06-14T23:05:49.281 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:49 smithi086 ceph-mon[102754]: pgmap v957: 99 pgs: 99 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-14T23:05:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:49.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:49.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:49.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:49.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:49.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:49.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:49 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:05:49.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/515589027' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:05:49.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:49 smithi137 ceph-mon[90634]: osdmap e248: 8 total, 8 up, 8 in 2024-06-14T23:05:49.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:49 smithi137 ceph-mon[90634]: pgmap v957: 99 pgs: 99 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-14T23:05:49.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:49.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:49.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:49.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:49.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:49.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:49.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:49.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:49.840 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:49.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:49.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:49.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:49.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:49.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:50.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:50.128 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:50.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:50.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:50.261 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:50.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:50.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:50.271 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 64803 64787 134.953 96.6875 0.00401373 0.00740529 2024-06-14T23:05:50.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:50.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:50.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:50.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:50.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:50.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:50.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:50.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:50.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:50.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:50.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:50.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:50.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:50.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:51.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:51.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:51.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:51.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:51.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:51.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:51.271 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 66331 66315 133.68 95.5 0.00483899 0.00747596 2024-06-14T23:05:51.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:51.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:05:51.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:51.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:51.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:51.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:51.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:51.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:51.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:51.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:51.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:51.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:51.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:51.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:51.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:51.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:51.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:51.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:52.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:52.118 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:51 smithi086 ceph-mon[102754]: pgmap v958: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-06-14T23:05:52.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:51 smithi086 ceph-mon[97913]: pgmap v958: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-06-14T23:05:52.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:52.167 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:52.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:52.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:51 smithi137 ceph-mon[90634]: pgmap v958: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-06-14T23:05:52.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:52.271 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 67667 67651 132.112 83.5 0.00817343 0.00756426 2024-06-14T23:05:52.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:52.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:52.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:52.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:52.539 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:05:52.539 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:05:52.539 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T23:05:52.539 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.652226 2024-06-14T23:05:52.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 3 0.6522263171066482 2024-06-14T23:05:52.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:52.582 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:52.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:52.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:52.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:52.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:52.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:52.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:52.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:52.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:52.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:52.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:53.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:53.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:53.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:53.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:53.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:53.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:53.271 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 15 69009 68994 130.652 83.9375 0.00665937 0.00764906 2024-06-14T23:05:53.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:53.379 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:53.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:53.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:53.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:53.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:53.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:53.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:53.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:53.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:53.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:53.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:53.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:53.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:53.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:53.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:54.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:54.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:54.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:54.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:54.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:54.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:54.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:54.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:53 smithi137 ceph-mon[90634]: pgmap v959: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-14T23:05:54.271 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 70246 70230 129.081 77.25 0.00821046 0.00774231 2024-06-14T23:05:54.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:54.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:54.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:54.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:54.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:53 smithi086 ceph-mon[97913]: pgmap v959: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-14T23:05:54.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:53 smithi086 ceph-mon[102754]: pgmap v959: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-14T23:05:54.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:54.432 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:54.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:54.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:54.562 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:54.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:54.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:54.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:54.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:54.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:54.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:54.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:54.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:54.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:54.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:55.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:55.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:55.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:55.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:55.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:55.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:55.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:55.271 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 71209 71193 127.113 60.1875 0.0135104 0.00786206 2024-06-14T23:05:55.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:55.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:05:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:05:54] "GET /metrics HTTP/1.1" 200 36594 "" "Prometheus/2.43.0" 2024-06-14T23:05:55.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:55.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:55.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:55.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:55.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:55.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:55.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:55.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:55.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:55.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:55.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:55.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:05:55.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:55.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:05:55.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:56.033 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:56.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:56.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:56.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:55 smithi086 ceph-mon[97913]: pgmap v960: 99 pgs: 99 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T23:05:56.057 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:55 smithi086 ceph-mon[102754]: pgmap v960: 99 pgs: 99 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T23:05:56.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:56.170 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:56.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:56.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-14T23:05:56.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:55 smithi137 ceph-mon[90634]: pgmap v960: 99 pgs: 99 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-14T23:05:56.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:56.278 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 72534 72518 125.865 82.8125 0.0138913 0.00793916 2024-06-14T23:05:56.278 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:56.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:56.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:56.349 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:05:56.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:56.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:56.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:56.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:56.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:56.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:56.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:56.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:56.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:56.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:56.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:56.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:56.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:56.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:05:57.034 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:57.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:57.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:05:57.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:57.165 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:57.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:57.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:57.277 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 73819 73803 124.631 80.3125 0.0113748 0.00801903 2024-06-14T23:05:57.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:57.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:57.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:57.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:57.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:57.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:57.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:57.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:57.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:57.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:57.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:57.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:57.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:57.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:57.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:57.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:57.998 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:05:57.998 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:05:57.998 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:05:57.999 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:05:58.000 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:05:58.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:57 smithi086 ceph-mon[97913]: pgmap v961: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-14T23:05:58.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:58.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:58.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/748564302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:58.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:57 smithi086 ceph-mon[102754]: pgmap v961: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-14T23:05:58.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:58.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:58.000 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/748564302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:58.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:05:58.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:58.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:05:58.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:57 smithi137 ceph-mon[90634]: pgmap v961: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-14T23:05:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:05:58.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:05:58.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/748564302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:05:58.251 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:58.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:58.278 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 75070 75054 123.407 78.1875 0.0103479 0.00809702 2024-06-14T23:05:58.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:58.395 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:58.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:58.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:58.526 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:58.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:58.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:05:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:58.667 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:58.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:05:58.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:58.801 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:58.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:58.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:58.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:58.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:58.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:05:59.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:59.078 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:59.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:59.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:59.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:59.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:59.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:59.278 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 15 76183 76168 122.028 69.625 0.00640031 0.00818959 2024-06-14T23:05:59.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:59.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:59.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:59.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:59.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:05:59.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:59.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:05:59.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:05:59.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:59.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:59.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:05:59.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:05:59.807 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:05:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:59.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:05:59.882 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:05:59.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:05:59.949 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:05:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:05:59.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:00.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:00.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:00.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:00.141 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:59 smithi086 ceph-mon[97913]: pgmap v962: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T23:06:00.141 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:05:59 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1772132218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:00.141 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:59 smithi086 ceph-mon[102754]: pgmap v962: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T23:06:00.141 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:05:59 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1772132218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:00.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:00.217 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:00.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:00.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:59 smithi137 ceph-mon[90634]: pgmap v962: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T23:06:00.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:05:59 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1772132218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:00.278 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:06:00.277703+0000 min lat: 0.00101502 max lat: 0.15347 avg lat: 0.00826804 2024-06-14T23:06:00.278 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:06:00.278 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 77400 77384 120.878 76 0.00721456 0.00826804 2024-06-14T23:06:00.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:00.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:00.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:00.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:00.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:00.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:00.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:00.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:00.625 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:00.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:00.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:00.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:00.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:00.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:00.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:00.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:00.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:00.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:01.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:01.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:01.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:00 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:01.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1772132218' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:06:01.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:00 smithi086 ceph-mon[97913]: osdmap e249: 8 total, 8 up, 8 in 2024-06-14T23:06:01.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:00 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:01.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1772132218' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:06:01.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:00 smithi086 ceph-mon[102754]: osdmap e249: 8 total, 8 up, 8 in 2024-06-14T23:06:01.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:01.177 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:01.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:01.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:00 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:01.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1772132218' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:06:01.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:00 smithi137 ceph-mon[90634]: osdmap e249: 8 total, 8 up, 8 in 2024-06-14T23:06:01.278 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 78748 78732 119.984 84.25 0.0115287 0.00832997 2024-06-14T23:06:01.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:01.296 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:01.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:01.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:01.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:01.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:01.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:01.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:01.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:01.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:01.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:01.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:01.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:01.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:01.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:01.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:02.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:02.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:02.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:06:02.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:02.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:02.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:02.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:02.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:01 smithi137 ceph-mon[90634]: pgmap v964: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-14T23:06:02.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:01 smithi086 ceph-mon[97913]: pgmap v964: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-14T23:06:02.271 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:01 smithi086 ceph-mon[102754]: pgmap v964: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-14T23:06:02.279 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 16 80037 80021 119.044 80.5625 0.00803653 0.00839511 2024-06-14T23:06:02.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:02.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:02.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:02.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:02.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:02.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:02.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:02.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:02.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:02.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:02.694 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:02.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:02.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:02.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:02.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:02.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:02.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:02.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:03.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:03.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:03.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:03.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:03.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:06:03.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:03.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:03.279 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 81275 81259 118.074 77.375 0.016911 0.00846249 2024-06-14T23:06:03.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:03.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:03.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:03.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:03.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:03.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:03.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:03.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:03.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:03.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:03.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:03.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:03.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:03.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:04.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:04.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:04.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:04.083 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.3 primary-affinity to 0.652226 (8427442) 2024-06-14T23:06:04.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:04.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:04.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:04.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:03 smithi137 ceph-mon[90634]: pgmap v965: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-14T23:06:04.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/535079030' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6522263171066482}]: dispatch 2024-06-14T23:06:04.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:03 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6522263171066482}]: dispatch 2024-06-14T23:06:04.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:04.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:04.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:04.279 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 16 82163 82147 116.652 55.5 0.00304105 0.008507 2024-06-14T23:06:04.281 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:04.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:03 smithi086 ceph-mon[97913]: pgmap v965: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-14T23:06:04.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/535079030' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6522263171066482}]: dispatch 2024-06-14T23:06:04.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:03 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6522263171066482}]: dispatch 2024-06-14T23:06:04.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:03 smithi086 ceph-mon[102754]: pgmap v965: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-14T23:06:04.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/535079030' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6522263171066482}]: dispatch 2024-06-14T23:06:04.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:03 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6522263171066482}]: dispatch 2024-06-14T23:06:04.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:04.398 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:04.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:04.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:04.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:04.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:04.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:04.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:04.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:04.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:04.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:04.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:04.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:04.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:04.988 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:04 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6522263171066482}]': finished 2024-06-14T23:06:04.989 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:04 smithi086 ceph-mon[102754]: osdmap e250: 8 total, 8 up, 8 in 2024-06-14T23:06:04.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:04 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6522263171066482}]': finished 2024-06-14T23:06:04.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:04 smithi086 ceph-mon[97913]: osdmap e250: 8 total, 8 up, 8 in 2024-06-14T23:06:04.989 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:06:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:06:04] "GET /metrics HTTP/1.1" 200 36602 "" "Prometheus/2.43.0" 2024-06-14T23:06:04.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:05.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:05.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:05.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:05.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:05.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:05.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:04 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.6522263171066482}]': finished 2024-06-14T23:06:05.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:04 smithi137 ceph-mon[90634]: osdmap e250: 8 total, 8 up, 8 in 2024-06-14T23:06:05.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:05.282 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 82585 82569 114.646 26.375 0.0129876 0.00871787 2024-06-14T23:06:05.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:05.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:05.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:05.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:05.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:05.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:05.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:05.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:05.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:05.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:05.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:05.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:05.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:06.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:06.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:06.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:06.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:06.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:06.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:06.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:05 smithi137 ceph-mon[90634]: pgmap v967: 99 pgs: 4 peering, 95 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-14T23:06:06.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:05 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:06:06.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:05 smithi137 ceph-mon[90634]: osdmap e251: 8 total, 8 up, 8 in 2024-06-14T23:06:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:06.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:06.281 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 16 83769 83753 113.758 74 0.00842261 0.00878581 2024-06-14T23:06:06.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:05 smithi086 ceph-mon[97913]: pgmap v967: 99 pgs: 4 peering, 95 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-14T23:06:06.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:05 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:06:06.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:05 smithi086 ceph-mon[97913]: osdmap e251: 8 total, 8 up, 8 in 2024-06-14T23:06:06.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:05 smithi086 ceph-mon[102754]: pgmap v967: 99 pgs: 4 peering, 95 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-14T23:06:06.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:05 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:06:06.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:05 smithi086 ceph-mon[102754]: osdmap e251: 8 total, 8 up, 8 in 2024-06-14T23:06:06.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:06.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:06.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:06.452 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:06.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:06.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:06.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:06.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:06.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:06.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:06.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:06.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:06.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:06.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:06.964 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:06.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:07.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:07.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:07.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:07.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:07.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:07.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:07.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:07.281 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 85006 84990 112.983 77.3125 0.013537 0.00884626 2024-06-14T23:06:07.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:07.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:07.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:07.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:07.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:07.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:07.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:07.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:07.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:07.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:07.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:07.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:07.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:07.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:07.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:07.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:07.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:08.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:08.065 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:08.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:08.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:08.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:08.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:08.202 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:06:08.202 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:06:08.202 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:06:08.202 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:06:08.202 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:06:08.203 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:07 smithi086 ceph-mon[97913]: pgmap v969: 99 pgs: 4 peering, 95 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T23:06:08.203 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:07 smithi086 ceph-mon[102754]: pgmap v969: 99 pgs: 4 peering, 95 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T23:06:08.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:07 smithi137 ceph-mon[90634]: pgmap v969: 99 pgs: 4 peering, 95 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-14T23:06:08.281 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 86214 86198 112.202 75.5 0.0106463 0.00890793 2024-06-14T23:06:08.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:08.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:08.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:08.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:08.481 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:08.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:08.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:08.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:08.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:08.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:08.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:08.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:08.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:08.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:08.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:08.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-14T23:06:08.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:09.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:09.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:09.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:09.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:09.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:09.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:09.281 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 87399 87383 111.424 74.0625 0.015247 0.00896936 2024-06-14T23:06:09.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:09.315 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:09.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:09.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:09.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:09.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:09.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:09.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:09.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:09.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:09.711 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:06:09.711 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:06:09.711 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T23:06:09.711 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-06-14T23:06:09.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 0 0 2024-06-14T23:06:09.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:09.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:09.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:09.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:09.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:09.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:10.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:10.005 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:06:10.006 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:06:10.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:10.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:10.045 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:06:10.045 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:06:10.079 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:06:10.080 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:06:10.118 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:06:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:06:10.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:10.154 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:09 smithi086 ceph-mon[102754]: pgmap v970: 99 pgs: 4 peering, 95 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 47 MiB/s wr, 749 op/s 2024-06-14T23:06:10.155 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:06:10.155 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:06:10.155 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:09 smithi086 ceph-mon[97913]: pgmap v970: 99 pgs: 4 peering, 95 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 47 MiB/s wr, 749 op/s 2024-06-14T23:06:10.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:10.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:10.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:10.185 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:06:10.185 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:06:10.221 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:06:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:06:10.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:09 smithi137 ceph-mon[90634]: pgmap v970: 99 pgs: 4 peering, 95 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 47 MiB/s wr, 749 op/s 2024-06-14T23:06:10.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:06:10.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:06:10.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:10.281 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 88457 88441 110.518 66.125 0.0137703 0.0090433 2024-06-14T23:06:10.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:10.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:10.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:10.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:10.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:10.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:10.570 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:10.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:10.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:10.688 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:10.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:10.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:10.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:10.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:10.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:10.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:10.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:10.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:11.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:11.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2806648410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:11.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2806648410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:11.102 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:11.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:11.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:11.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:11.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:11.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2806648410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:11.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 89654 89638 109.815 74.8125 0.0101169 0.00910161 2024-06-14T23:06:11.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:11.372 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:11.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:11.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:11.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:11.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:11.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:11.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:11.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:11.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:11.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:11.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:11.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:11.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:11.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:11.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:11.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:12.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:12.054 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:12.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:12.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:12.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:12.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:12.228 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:11 smithi086 ceph-mon[102754]: pgmap v971: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-14T23:06:12.228 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:11 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T23:06:12.228 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:11 smithi086 ceph-mon[97913]: pgmap v971: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-14T23:06:12.228 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:11 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T23:06:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:11 smithi137 ceph-mon[90634]: pgmap v971: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-14T23:06:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:11 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T23:06:12.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 16 90840 90824 109.129 74.125 0.00757152 0.00915884 2024-06-14T23:06:12.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:12.328 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:12.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:12.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:12.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:12.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:12.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:12.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:12.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:12.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:12.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:12.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:12.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:12.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:12.963 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:06:12.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:13.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:13.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:13.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/331369030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:06:13.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:13.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/331369030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:06:13.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:13.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:06:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:13.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:13.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:13.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:13.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/331369030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:06:13.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:13.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 16 91948 91932 108.377 69.25 0.0325484 0.00922193 2024-06-14T23:06:13.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:13.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:13.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:13.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:13.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:13.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:13.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:13.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:13.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:13.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:13.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:13.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:13.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:13.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:13.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:14.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:14.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:14.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:14.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:13 smithi086 ceph-mon[97913]: pgmap v972: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T23:06:14.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:13 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:14.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/331369030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:06:14.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:13 smithi086 ceph-mon[97913]: osdmap e252: 8 total, 8 up, 8 in 2024-06-14T23:06:14.092 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:13 smithi086 ceph-mon[102754]: pgmap v972: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T23:06:14.092 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:13 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:14.092 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/331369030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:06:14.092 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:13 smithi086 ceph-mon[102754]: osdmap e252: 8 total, 8 up, 8 in 2024-06-14T23:06:14.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:14.173 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:14.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:14.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:13 smithi137 ceph-mon[90634]: pgmap v972: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T23:06:14.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:13 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:14.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/331369030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:06:14.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:13 smithi137 ceph-mon[90634]: osdmap e252: 8 total, 8 up, 8 in 2024-06-14T23:06:14.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:14.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 93065 93049 107.663 69.8125 0.00718878 0.00928339 2024-06-14T23:06:14.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:14.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:14.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:14.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:14.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:14.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:14.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:14.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:14.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:14.677 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:06:14.678 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:06:14.703 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:14.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:14.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:14.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:14.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:14.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:14.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:14.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:15.014 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:06:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:06:14] "GET /metrics HTTP/1.1" 200 36602 "" "Prometheus/2.43.0" 2024-06-14T23:06:15.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:15.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:15.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:15.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:15.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:15.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:15.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 94190 94174 106.984 70.3125 0.0356465 0.00934197 2024-06-14T23:06:15.340 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:15.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:15.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:15.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:15.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:15.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:15.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:15.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:15.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:15.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:15.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:15.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:15.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:06:15.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:15.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:16.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:16.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:15 smithi086 ceph-mon[102754]: pgmap v974: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:06:16.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:15 smithi086 ceph-mon[97913]: pgmap v974: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:06:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:16.116 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:16.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:16.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:16.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:16.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:15 smithi137 ceph-mon[90634]: pgmap v974: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:06:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:16.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:16.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 95265 95249 106.273 67.1875 0.0149071 0.00940464 2024-06-14T23:06:16.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:16.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:16.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:16.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:16.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:16.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:16.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:16.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:16.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:16.802 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:16.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:16.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:16.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:16.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:17.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:17.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:17.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:17.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:17.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:17.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:17.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 96392 96376 105.644 70.4375 0.0360031 0.00946019 2024-06-14T23:06:17.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:17.334 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:17.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:17.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:17.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:17.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:17.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:17.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-14T23:06:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:17.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:17.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:17.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:17.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:17.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:17.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:17.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:17.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:17.985 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.0 primary-affinity to 0 (802) 2024-06-14T23:06:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:18.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:18.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:18.146 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:18.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:18.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:17 smithi137 ceph-mon[90634]: pgmap v975: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:06:18.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1216583039' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-06-14T23:06:18.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:18.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:18.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:18.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:18.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 16 97311 97295 104.813 57.4375 0.00331912 0.00950643 2024-06-14T23:06:18.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:17 smithi086 ceph-mon[97913]: pgmap v975: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:06:18.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1216583039' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-06-14T23:06:18.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:18.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:17 smithi086 ceph-mon[102754]: pgmap v975: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:06:18.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1216583039' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-06-14T23:06:18.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:18.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:18.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:18.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:18.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:18.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:18.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:18.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:18.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:18.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:18.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:18.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:18.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:18.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:18.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:18.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:18.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:19.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:19.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:19.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:19.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:19.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:19.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1216583039' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-06-14T23:06:19.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:18 smithi137 ceph-mon[90634]: osdmap e253: 8 total, 8 up, 8 in 2024-06-14T23:06:19.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:19.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 97601 97585 103.344 18.125 0.0194956 0.00967153 2024-06-14T23:06:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:19.286 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:19.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:19.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1216583039' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-06-14T23:06:19.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:18 smithi086 ceph-mon[102754]: osdmap e253: 8 total, 8 up, 8 in 2024-06-14T23:06:19.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1216583039' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-06-14T23:06:19.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:18 smithi086 ceph-mon[97913]: osdmap e253: 8 total, 8 up, 8 in 2024-06-14T23:06:19.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:19.420 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:19.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:19.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:19.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:19.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:19.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:19.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:19.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:19.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:19.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:19.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:19.822 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:19.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:19.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:19.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:20.028 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:06:20.028 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:06:20.062 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:06:20.062 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:06:20.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:20.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:20.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:20.093 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:06:20.093 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:06:20.134 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:06:20.134 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:06:20.165 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:06:20.166 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:06:20.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:20.201 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:06:20.201 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:06:20.202 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:19 smithi086 ceph-mon[102754]: pgmap v977: 99 pgs: 99 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s 2024-06-14T23:06:20.202 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:19 smithi086 ceph-mon[102754]: osdmap e254: 8 total, 8 up, 8 in 2024-06-14T23:06:20.202 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:19 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3021108288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:20.202 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:19 smithi086 ceph-mon[97913]: pgmap v977: 99 pgs: 99 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s 2024-06-14T23:06:20.202 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:19 smithi086 ceph-mon[97913]: osdmap e254: 8 total, 8 up, 8 in 2024-06-14T23:06:20.202 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:19 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3021108288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:20.231 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:06:20.231 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:06:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:20.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:20.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:20.263 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:06:20.263 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:06:20.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:06:20.283232+0000 min lat: 0.00101502 max lat: 1.03707 avg lat: 0.00973125 2024-06-14T23:06:20.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:06:20.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 98640 98624 102.704 64.9375 0.0120438 0.00973125 2024-06-14T23:06:20.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:20.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:20.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:20.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:20.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:19 smithi137 ceph-mon[90634]: pgmap v977: 99 pgs: 99 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s 2024-06-14T23:06:20.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:19 smithi137 ceph-mon[90634]: osdmap e254: 8 total, 8 up, 8 in 2024-06-14T23:06:20.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:19 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3021108288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:20.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:20.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:20.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:20.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:20.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:20.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:20.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:20.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:20.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:20.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:21.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:21.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:21.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:21.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:21.185 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:21.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:21.283 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 99691 99675 102.097 65.6875 0.00401733 0.009785 2024-06-14T23:06:21.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:21.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:21.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:21.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:21 smithi086 ceph-mon[102754]: 1.2 scrub starts 2024-06-14T23:06:21.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:21 smithi086 ceph-mon[102754]: 1.2 scrub ok 2024-06-14T23:06:21.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:21 smithi086 ceph-mon[102754]: pgmap v979: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 21 KiB/s rd, 80 MiB/s wr, 1.30k op/s; 48 KiB/s, 0 objects/s recovering 2024-06-14T23:06:21.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:21 smithi086 ceph-mon[97913]: 1.2 scrub starts 2024-06-14T23:06:21.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:21 smithi086 ceph-mon[97913]: 1.2 scrub ok 2024-06-14T23:06:21.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:21 smithi086 ceph-mon[97913]: pgmap v979: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 21 KiB/s rd, 80 MiB/s wr, 1.30k op/s; 48 KiB/s, 0 objects/s recovering 2024-06-14T23:06:21.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:21.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:21.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:21.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:21 smithi137 ceph-mon[90634]: 1.2 scrub starts 2024-06-14T23:06:21.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:21 smithi137 ceph-mon[90634]: 1.2 scrub ok 2024-06-14T23:06:21.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:21 smithi137 ceph-mon[90634]: pgmap v979: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 21 KiB/s rd, 80 MiB/s wr, 1.30k op/s; 48 KiB/s, 0 objects/s recovering 2024-06-14T23:06:21.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:21.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:21.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:21.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:21.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:21.720 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:21.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:21.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:21.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:21.852 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:21.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:21.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:21.992 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:21.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:22.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:22.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:22.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:22.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:22.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:22.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 100817 100801 101.586 70.375 0.00866533 0.00983895 2024-06-14T23:06:22.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:22 smithi086 ceph-mon[102754]: 1.0 scrub starts 2024-06-14T23:06:22.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:22 smithi086 ceph-mon[102754]: 1.0 scrub ok 2024-06-14T23:06:22.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:22 smithi086 ceph-mon[97913]: 1.0 scrub starts 2024-06-14T23:06:22.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:22 smithi086 ceph-mon[97913]: 1.0 scrub ok 2024-06-14T23:06:22.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:22.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:22.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:22.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:22.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:22 smithi137 ceph-mon[90634]: 1.0 scrub starts 2024-06-14T23:06:22.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:22 smithi137 ceph-mon[90634]: 1.0 scrub ok 2024-06-14T23:06:22.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:22.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:22.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:22.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:22.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:22.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:22.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:22.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:22.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:22.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:22.965 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:22.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:23.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:23.104 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:23.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:23.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:23.241 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:23.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:23.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 101911 101895 101.058 68.375 0.0104254 0.00989073 2024-06-14T23:06:23.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:23 smithi086 ceph-mon[102754]: pgmap v980: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 21 KiB/s rd, 51 MiB/s wr, 829 op/s; 47 KiB/s, 0 objects/s recovering 2024-06-14T23:06:23.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:23 smithi086 ceph-mon[97913]: pgmap v980: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 21 KiB/s rd, 51 MiB/s wr, 829 op/s; 47 KiB/s, 0 objects/s recovering 2024-06-14T23:06:23.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:23.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:23.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:23.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:23.494 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:23 smithi137 ceph-mon[90634]: pgmap v980: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 21 KiB/s rd, 51 MiB/s wr, 829 op/s; 47 KiB/s, 0 objects/s recovering 2024-06-14T23:06:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:23.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:23.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:23.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:23.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:23.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:23.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:23.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:23.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:23.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:24.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:24.065 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:24.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:24.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:24.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:24.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:24.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 102971 102955 100.515 66.25 0.0139237 0.00994301 2024-06-14T23:06:24.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:24.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:24.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:24.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:24.447 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:06:24.448 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:06:24.448 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-14T23:06:24.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:06:24.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:24.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:24.510 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:06:24.510 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:06:24.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.510 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:103190 11.b 11:d90a04b3:::benchmark_data_smithi086_226760_object103189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-06-14T23:06:24.510 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:24.488806+0000", 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.016914511, 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016953128000000001, 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 103190 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:06:24.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.488806+0000", 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.488806+0000", 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.488804+0000", 2024-06-14T23:06:24.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.488831+0000", 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6891000000000001e-05 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.488850+0000", 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8439000000000001e-05 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:06:24.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.488858+0000", 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9419999999999995e-06 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.489212+0000", 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035485699999999998 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.489356+0000", 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014304399999999999 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.489495+0000", 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000139095 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.498340+0000", 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0088451810000000006 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.498547+0000", 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000207419 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:06:24.515 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.36237.0:103195 11.2 e254/237 11:4e2e2697:::benchmark_data_smithi086_226760_object103194:head v 254'6326, mlcod=254'6324)", 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:24.497561+0000", 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0081598009999999995, 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0082914300000000007, 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.497561+0000", 2024-06-14T23:06:24.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.497561+0000", 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.497560+0000", 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.497588+0000", 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8005e-05 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.497606+0000", 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8084000000000002e-05 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.497615+0000", 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2490000000000007e-06 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.498480+0000", 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00086535599999999996 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.498497+0000", 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7091e-05 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.505822+0000", 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073248089999999998 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.36237.0:103199 11.2 e254/237 11:4069d2d3:::benchmark_data_smithi086_226760_object103198:head v 254'6327, mlcod=254'6324)", 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:24.499103+0000", 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0066176339999999998, 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0068239700000000004, 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:06:24.520 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.499103+0000", 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.499103+0000", 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.499102+0000", 2024-06-14T23:06:24.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.499125+0000", 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3027000000000002e-05 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.499135+0000", 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0368e-05 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.499141+0000", 2024-06-14T23:06:24.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8279999999999999e-06 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.502366+0000", 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0032248730000000001 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.502383+0000", 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6779999999999999e-05 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:06:24.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.36237.0:103197 11.0 e254/237 11:0ba86a96:::benchmark_data_smithi086_226760_object103196:head v 254'6291, mlcod=254'6289)", 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:24.500563+0000", 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.005157671, 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005426508, 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.500563+0000", 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.500563+0000", 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.500562+0000", 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.500586+0000", 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4125999999999999e-05 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.500599+0000", 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2824e-05 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.500606+0000", 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2949999999999996e-06 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.501736+0000", 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011301550000000001 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:06:24.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:24.501755+0000", 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9269000000000001e-05 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 4 2024-06-14T23:06:24.527 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:06:24.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:24.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:24.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:24.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:24.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:24.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:24.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:06:24.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:24.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:24.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:25.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:25.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:25.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:25.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:25.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:25.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:25.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:25.285 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 103878 103862 99.8403 56.6875 0.0102619 0.0100115 2024-06-14T23:06:25.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:25.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:06:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:06:24] "GET /metrics HTTP/1.1" 200 36602 "" "Prometheus/2.43.0" 2024-06-14T23:06:25.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:25.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:25.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:25.508 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:25.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:25.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-14T23:06:25.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:25.621 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:25.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:25.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:25.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:25.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:25.844 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:25.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:25.902 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:06:25.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:25.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:25.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:26.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:25 smithi086 ceph-mon[97913]: pgmap v981: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 7.9 KiB/s, 0 objects/s recovering 2024-06-14T23:06:26.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4242174436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:26.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:25 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:26.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:25 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:06:26.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:25 smithi086 ceph-mon[102754]: pgmap v981: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 7.9 KiB/s, 0 objects/s recovering 2024-06-14T23:06:26.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4242174436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:26.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:25 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:26.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:25 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:06:26.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:26.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:26.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:26.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:26.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:25 smithi137 ceph-mon[90634]: pgmap v981: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 7.9 KiB/s, 0 objects/s recovering 2024-06-14T23:06:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4242174436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:25 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:25 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:06:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:26.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:26.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:26.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 104784 104768 99.1855 56.625 0.0229395 0.0100772 2024-06-14T23:06:26.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:26.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:26.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:26.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:26.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:26.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:26.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:26.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:26.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:26.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:26.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:26.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:26.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:26.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:26.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:26.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:27.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:27.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:27.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:27.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:27.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:27.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:26 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:06:27.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:26 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:06:27.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:26 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:06:27.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:26 smithi137 ceph-mon[90634]: osdmap e255: 8 total, 8 up, 8 in 2024-06-14T23:06:27.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 105841 105825 98.6912 66.0625 0.0121577 0.010128 2024-06-14T23:06:27.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:27.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:26 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:06:27.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:26 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:06:27.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:26 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:06:27.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:26 smithi086 ceph-mon[97913]: osdmap e255: 8 total, 8 up, 8 in 2024-06-14T23:06:27.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:26 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:06:27.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:26 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:06:27.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:26 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:06:27.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:26 smithi086 ceph-mon[102754]: osdmap e255: 8 total, 8 up, 8 in 2024-06-14T23:06:27.346 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:27.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:27.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:27.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:27.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:27.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:27.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:27.630 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:27.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:27.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:27.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:27.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:27.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:27.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:27.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:27.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:28.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:28.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:28.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:28.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:28.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:28.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:28.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:28.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:27 smithi137 ceph-mon[90634]: pgmap v983: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s; 7.9 KiB/s, 0 objects/s recovering 2024-06-14T23:06:28.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:28.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:28.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:28.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 106768 106752 98.0919 57.9375 0.00923395 0.0101896 2024-06-14T23:06:28.317 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:28.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:28.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:27 smithi086 ceph-mon[97913]: pgmap v983: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s; 7.9 KiB/s, 0 objects/s recovering 2024-06-14T23:06:28.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:28.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:28.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:27 smithi086 ceph-mon[102754]: pgmap v983: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s; 7.9 KiB/s, 0 objects/s recovering 2024-06-14T23:06:28.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:28.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:28.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:28.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:28.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:28.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:28.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:28.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:28.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:28.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:28.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:28.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:28.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:28.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:28.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:28.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:29.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:29.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:29.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:29.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:29.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:29.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:29.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 107689 107673 97.5045 57.5625 0.0159292 0.0102503 2024-06-14T23:06:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:29.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:29.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:29.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:29.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:29.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:29.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:29.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:29.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:29.697 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:29.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:29.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:29.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:29.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:29.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:29.966 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:29.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:30.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:30.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:30.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:30.189 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:29 smithi086 ceph-mon[102754]: pgmap v984: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 983 op/s; 6.4 KiB/s, 0 objects/s recovering 2024-06-14T23:06:30.197 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:29 smithi086 ceph-mon[97913]: pgmap v984: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 983 op/s; 6.4 KiB/s, 0 objects/s recovering 2024-06-14T23:06:30.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:29 smithi137 ceph-mon[90634]: pgmap v984: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 983 op/s; 6.4 KiB/s, 0 objects/s recovering 2024-06-14T23:06:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:30.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:30.284 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 108558 108542 96.8875 54.3125 0.0352549 0.0103161 2024-06-14T23:06:30.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:30.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:30.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:30.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:30.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:30.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:30.661 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:30.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:30.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:30.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:30.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:30.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:30.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:30.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:30.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:30.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:31.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:31.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:31.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:31.225 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:31.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:31.285 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 109496 109480 96.3486 58.625 0.0279696 0.0103744 2024-06-14T23:06:31.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:31.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:31.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:31.519 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:31.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:31.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:31.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:31.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:31.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:31.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:31.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:31.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:31.949 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:31.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:32.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:32.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:32.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:32.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:31 smithi086 ceph-mon[97913]: pgmap v985: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-14T23:06:32.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:31 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:32.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:31 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:32.119 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:31 smithi086 ceph-mon[102754]: pgmap v985: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-14T23:06:32.119 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:31 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:32.119 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:31 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:32.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:32.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:32.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:32.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:06:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:31 smithi137 ceph-mon[90634]: pgmap v985: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-14T23:06:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:31 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:32.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:31 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:32.285 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 110410 110394 95.8037 57.125 0.0131792 0.0104327 2024-06-14T23:06:32.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:32.349 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:32.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:32.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:32.420 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:06:32.424 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:06:32.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:32.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:32.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:32.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:32.608 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:32.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:32.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:32.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:32.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:32.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:32.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:32.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:32.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:33.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:33.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:33.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:33.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:33.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:33.285 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 111326 111310 95.2752 57.25 0.0194452 0.0104908 2024-06-14T23:06:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:33.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:33.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:33.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:33.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:33.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:33.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:33.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:33.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:33.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:33.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:33.854 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:33.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:33.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:33.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:33.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:34.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:34.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:34.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:34.222 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:33 smithi086 ceph-mon[102754]: from='client.36306 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:06:34.223 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:33 smithi086 ceph-mon[102754]: pgmap v986: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-14T23:06:34.223 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:33 smithi086 ceph-mon[97913]: from='client.36306 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:06:34.223 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:33 smithi086 ceph-mon[97913]: pgmap v986: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-14T23:06:34.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:33 smithi137 ceph-mon[90634]: from='client.36306 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:06:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:33 smithi137 ceph-mon[90634]: pgmap v986: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-06-14T23:06:34.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:34.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:34.285 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 15 112313 112298 94.822 61.75 0.0238871 0.0105413 2024-06-14T23:06:34.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:34.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:34.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:34.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:34.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:34.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:34.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:34.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:34.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:34.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:34.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:34.817 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:34.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:34.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:34.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:34.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:35.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:35.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:06:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:06:34] "GET /metrics HTTP/1.1" 200 36614 "" "Prometheus/2.43.0" 2024-06-14T23:06:35.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:35.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:35.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:35.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:35.268 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:35.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:35.285 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 113291 113275 94.3718 61.0625 0.0143145 0.0105908 2024-06-14T23:06:35.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:35.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:35.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:35.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:35.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:35.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:35.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:35.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:35.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:35.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:35.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:35.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:35.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:35.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:36.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:36.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:36.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:36.202 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:35 smithi086 ceph-mon[97913]: pgmap v987: 99 pgs: 99 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s 2024-06-14T23:06:36.203 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3284298328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:36.203 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:35 smithi086 ceph-mon[102754]: pgmap v987: 99 pgs: 99 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s 2024-06-14T23:06:36.203 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3284298328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:36.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:36.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:36.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:35 smithi137 ceph-mon[90634]: pgmap v987: 99 pgs: 99 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s 2024-06-14T23:06:36.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3284298328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:36.285 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 114274 114258 93.9385 61.4375 0.00620673 0.01064 2024-06-14T23:06:36.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:36.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:36.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:36.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:36.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:36.561 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":985,"stamp":"2024-06-14T23:06:30.814879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7084185542,"num_objects":108111,"num_object_clones":0,"num_object_copies":324333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108148,"num_write_kb":6917597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":590980,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109913,"ondisk_log_size":109913,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":26353812,"kb_used_data":20637012,"kb_used_omap":29,"kb_used_meta":5716706,"kb_avail":723607404,"statfs":{"total":767960285184,"available":740973981696,"internally_reserved":0,"allocated":21132300288,"data_stored":21122334110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":5853907621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,3,6,8,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":652893365,"num_objects":9963,"num_object_clones":0,"num_object_copies":29889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9963,"num_whiteouts":0,"num_read":-90,"num_read_kb":-167,"num_write":9921,"num_write_kb":637257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-322019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003285"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":187,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.991576+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:06:18.991576+0000","last_peered":"2024-06-14T23:06:18.991576+0000","last_clean":"2024-06-14T23:06:18.991576+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:06:18.991576+0000","last_undegraded":"2024-06-14T23:06:18.991576+0000","last_fullsized":"2024-06-14T23:06:18.991576+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":250,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.997205+0000","last_change":"2024-06-14T23:04:34.895560+0000","last_active":"2024-06-14T23:06:18.997205+0000","last_peered":"2024-06-14T23:06:18.997205+0000","last_clean":"2024-06-14T23:06:18.997205+0000","last_became_active":"2024-06-14T23:04:34.887055+0000","last_became_peered":"2024-06-14T23:04:34.887055+0000","last_unstale":"2024-06-14T23:06:18.997205+0000","last_undegraded":"2024-06-14T23:06:18.997205+0000","last_fullsized":"2024-06-14T23:06:18.997205+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:26:36.259241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":215,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996772+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:06:18.996772+0000","last_peered":"2024-06-14T23:06:18.996772+0000","last_clean":"2024-06-14T23:06:18.996772+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:06:18.996772+0000","last_undegraded":"2024-06-14T23:06:18.996772+0000","last_fullsized":"2024-06-14T23:06:18.996772+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":209,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996720+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:06:18.996720+0000","last_peered":"2024-06-14T23:06:18.996720+0000","last_clean":"2024-06-14T23:06:18.996720+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:06:18.996720+0000","last_undegraded":"2024-06-14T23:06:18.996720+0000","last_fullsized":"2024-06-14T23:06:18.996720+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":92,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.767350+0000","last_change":"2024-06-14T23:06:18.999778+0000","last_active":"2024-06-14T23:06:19.767350+0000","last_peered":"2024-06-14T23:06:19.767350+0000","last_clean":"2024-06-14T23:06:19.767350+0000","last_became_active":"2024-06-14T23:06:18.999306+0000","last_became_peered":"2024-06-14T23:06:18.999306+0000","last_unstale":"2024-06-14T23:06:19.767350+0000","last_undegraded":"2024-06-14T23:06:19.767350+0000","last_fullsized":"2024-06-14T23:06:19.767350+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:18:01.226932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":122,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993019+0000","last_change":"2024-06-14T22:55:53.664052+0000","last_active":"2024-06-14T23:06:18.993019+0000","last_peered":"2024-06-14T23:06:18.993019+0000","last_clean":"2024-06-14T23:06:18.993019+0000","last_became_active":"2024-06-14T22:55:53.663748+0000","last_became_peered":"2024-06-14T22:55:53.663748+0000","last_unstale":"2024-06-14T23:06:18.993019+0000","last_undegraded":"2024-06-14T23:06:18.993019+0000","last_fullsized":"2024-06-14T23:06:18.993019+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:57:22.745515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":211,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993046+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:06:18.993046+0000","last_peered":"2024-06-14T23:06:18.993046+0000","last_clean":"2024-06-14T23:06:18.993046+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:06:18.993046+0000","last_undegraded":"2024-06-14T23:06:18.993046+0000","last_fullsized":"2024-06-14T23:06:18.993046+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":224,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.999690+0000","last_change":"2024-06-14T22:44:51.836427+0000","last_active":"2024-06-14T23:06:18.999690+0000","last_peered":"2024-06-14T23:06:18.999690+0000","last_clean":"2024-06-14T23:06:18.999690+0000","last_became_active":"2024-06-14T22:44:51.835897+0000","last_became_peered":"2024-06-14T22:44:51.835897+0000","last_unstale":"2024-06-14T23:06:18.999690+0000","last_undegraded":"2024-06-14T23:06:18.999690+0000","last_fullsized":"2024-06-14T23:06:18.999690+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:13:10.466902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"255'6788","reported_seq":6814,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.491888+0000","last_change":"2024-06-14T23:05:03.621489+0000","last_active":"2024-06-14T23:06:29.491888+0000","last_peered":"2024-06-14T23:06:29.491888+0000","last_clean":"2024-06-14T23:06:29.491888+0000","last_became_active":"2024-06-14T23:05:03.621114+0000","last_became_peered":"2024-06-14T23:05:03.621114+0000","last_unstale":"2024-06-14T23:06:29.491888+0000","last_undegraded":"2024-06-14T23:06:29.491888+0000","last_fullsized":"2024-06-14T23:06:29.491888+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6788,"log_dups_size":0,"ondisk_log_size":6788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:33.811591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444858368,"num_objects":6788,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":434432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":205,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996155+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:06:18.996155+0000","last_peered":"2024-06-14T23:06:18.996155+0000","last_clean":"2024-06-14T23:06:18.996155+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:06:18.996155+0000","last_undegraded":"2024-06-14T23:06:18.996155+0000","last_fullsized":"2024-06-14T23:06:18.996155+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":198,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.995046+0000","last_change":"2024-06-14T22:34:46.494776+0000","last_active":"2024-06-14T23:06:18.995046+0000","last_peered":"2024-06-14T23:06:18.995046+0000","last_clean":"2024-06-14T23:06:18.995046+0000","last_became_active":"2024-06-14T22:34:46.494500+0000","last_became_peered":"2024-06-14T22:34:46.494500+0000","last_unstale":"2024-06-14T23:06:18.995046+0000","last_undegraded":"2024-06-14T23:06:18.995046+0000","last_fullsized":"2024-06-14T23:06:18.995046+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:11.014592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"255'6818","reported_seq":6857,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:30.532996+0000","last_change":"2024-06-14T23:06:19.064148+0000","last_active":"2024-06-14T23:06:30.532996+0000","last_peered":"2024-06-14T23:06:30.532996+0000","last_clean":"2024-06-14T23:06:30.532996+0000","last_became_active":"2024-06-14T23:06:19.007544+0000","last_became_peered":"2024-06-14T23:06:19.007544+0000","last_unstale":"2024-06-14T23:06:30.532996+0000","last_undegraded":"2024-06-14T23:06:30.532996+0000","last_fullsized":"2024-06-14T23:06:30.532996+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:47.143283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446824448,"num_objects":6818,"num_object_clones":0,"num_object_copies":20454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6818,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":206,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.997031+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:06:18.997031+0000","last_peered":"2024-06-14T23:06:18.997031+0000","last_clean":"2024-06-14T23:06:18.997031+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:06:18.997031+0000","last_undegraded":"2024-06-14T23:06:18.997031+0000","last_fullsized":"2024-06-14T23:06:18.997031+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":232,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.994098+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:06:18.994098+0000","last_peered":"2024-06-14T23:06:18.994098+0000","last_clean":"2024-06-14T23:06:18.994098+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:06:18.994098+0000","last_undegraded":"2024-06-14T23:06:18.994098+0000","last_fullsized":"2024-06-14T23:06:18.994098+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"255'6815","reported_seq":6841,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.448811+0000","last_change":"2024-06-14T23:05:03.625005+0000","last_active":"2024-06-14T23:06:29.448811+0000","last_peered":"2024-06-14T23:06:29.448811+0000","last_clean":"2024-06-14T23:06:29.448811+0000","last_became_active":"2024-06-14T23:05:03.624698+0000","last_became_peered":"2024-06-14T23:05:03.624698+0000","last_unstale":"2024-06-14T23:06:29.448811+0000","last_undegraded":"2024-06-14T23:06:29.448811+0000","last_fullsized":"2024-06-14T23:06:29.448811+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6815,"log_dups_size":0,"ondisk_log_size":6815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:30:27.319001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446627840,"num_objects":6815,"num_object_clones":0,"num_object_copies":20445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6815,"num_write_kb":436160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":133,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996779+0000","last_change":"2024-06-14T22:55:53.664957+0000","last_active":"2024-06-14T23:06:18.996779+0000","last_peered":"2024-06-14T23:06:18.996779+0000","last_clean":"2024-06-14T23:06:18.996779+0000","last_became_active":"2024-06-14T22:55:53.664613+0000","last_became_peered":"2024-06-14T22:55:53.664613+0000","last_unstale":"2024-06-14T23:06:18.996779+0000","last_undegraded":"2024-06-14T23:06:18.996779+0000","last_fullsized":"2024-06-14T23:06:18.996779+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:49.878019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":215,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.000086+0000","last_change":"2024-06-14T22:49:25.495484+0000","last_active":"2024-06-14T23:06:19.000086+0000","last_peered":"2024-06-14T23:06:19.000086+0000","last_clean":"2024-06-14T23:06:19.000086+0000","last_became_active":"2024-06-14T22:49:25.495029+0000","last_became_peered":"2024-06-14T22:49:25.495029+0000","last_unstale":"2024-06-14T23:06:19.000086+0000","last_undegraded":"2024-06-14T23:06:19.000086+0000","last_fullsized":"2024-06-14T23:06:19.000086+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:26:22.637176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"255'6835","reported_seq":6860,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.165615+0000","last_change":"2024-06-14T23:05:03.624271+0000","last_active":"2024-06-14T23:06:29.165615+0000","last_peered":"2024-06-14T23:06:29.165615+0000","last_clean":"2024-06-14T23:06:29.165615+0000","last_became_active":"2024-06-14T23:05:03.623977+0000","last_became_peered":"2024-06-14T23:05:03.623977+0000","last_unstale":"2024-06-14T23:06:29.165615+0000","last_undegraded":"2024-06-14T23:06:29.165615+0000","last_fullsized":"2024-06-14T23:06:29.165615+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6835,"log_dups_size":0,"ondisk_log_size":6835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:11:53.025160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447938560,"num_objects":6835,"num_object_clones":0,"num_object_copies":20505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6835,"num_write_kb":437440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":216,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996592+0000","last_change":"2024-06-14T22:54:06.280349+0000","last_active":"2024-06-14T23:06:18.996592+0000","last_peered":"2024-06-14T23:06:18.996592+0000","last_clean":"2024-06-14T23:06:18.996592+0000","last_became_active":"2024-06-14T22:54:06.280090+0000","last_became_peered":"2024-06-14T22:54:06.280090+0000","last_unstale":"2024-06-14T23:06:18.996592+0000","last_undegraded":"2024-06-14T23:06:18.996592+0000","last_fullsized":"2024-06-14T23:06:18.996592+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:31.286194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":69,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:20.207544+0000","last_change":"2024-06-14T23:06:19.002672+0000","last_active":"2024-06-14T23:06:20.207544+0000","last_peered":"2024-06-14T23:06:20.207544+0000","last_clean":"2024-06-14T23:06:20.207544+0000","last_became_active":"2024-06-14T23:06:19.002175+0000","last_became_peered":"2024-06-14T23:06:19.002175+0000","last_unstale":"2024-06-14T23:06:20.207544+0000","last_undegraded":"2024-06-14T23:06:20.207544+0000","last_fullsized":"2024-06-14T23:06:20.207544+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:50.841812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"255'6633","reported_seq":6659,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.124425+0000","last_change":"2024-06-14T23:05:03.626281+0000","last_active":"2024-06-14T23:06:29.124425+0000","last_peered":"2024-06-14T23:06:29.124425+0000","last_clean":"2024-06-14T23:06:29.124425+0000","last_became_active":"2024-06-14T23:05:03.625996+0000","last_became_peered":"2024-06-14T23:05:03.625996+0000","last_unstale":"2024-06-14T23:06:29.124425+0000","last_undegraded":"2024-06-14T23:06:29.124425+0000","last_fullsized":"2024-06-14T23:06:29.124425+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6633,"log_dups_size":0,"ondisk_log_size":6633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:52:21.957192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434700288,"num_objects":6633,"num_object_clones":0,"num_object_copies":19899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":507,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.990472+0000","last_change":"2024-06-14T23:04:40.403447+0000","last_active":"2024-06-14T23:06:18.990472+0000","last_peered":"2024-06-14T23:06:18.990472+0000","last_clean":"2024-06-14T23:06:18.990472+0000","last_became_active":"2024-06-14T23:04:37.904456+0000","last_became_peered":"2024-06-14T23:04:37.904456+0000","last_unstale":"2024-06-14T23:06:18.990472+0000","last_undegraded":"2024-06-14T23:06:18.990472+0000","last_fullsized":"2024-06-14T23:06:18.990472+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:04:40.403365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:04:40.403365+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:45.174359+0000","scrub_duration":0.0014330689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":25,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.003276+0000","last_change":"2024-06-14T23:06:19.002737+0000","last_active":"2024-06-14T23:06:19.003276+0000","last_peered":"2024-06-14T23:06:19.003276+0000","last_clean":"2024-06-14T23:06:19.003276+0000","last_became_active":"2024-06-14T23:06:19.002078+0000","last_became_peered":"2024-06-14T23:06:19.002078+0000","last_unstale":"2024-06-14T23:06:19.003276+0000","last_undegraded":"2024-06-14T23:06:19.003276+0000","last_fullsized":"2024-06-14T23:06:19.003276+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:00:16.317592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":215,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993899+0000","last_change":"2024-06-14T22:54:06.280497+0000","last_active":"2024-06-14T23:06:18.993899+0000","last_peered":"2024-06-14T23:06:18.993899+0000","last_clean":"2024-06-14T23:06:18.993899+0000","last_became_active":"2024-06-14T22:54:06.280022+0000","last_became_peered":"2024-06-14T22:54:06.280022+0000","last_unstale":"2024-06-14T23:06:18.993899+0000","last_undegraded":"2024-06-14T23:06:18.993899+0000","last_fullsized":"2024-06-14T23:06:18.993899+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:55:30.056306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"255'6653","reported_seq":6678,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:30.530246+0000","last_change":"2024-06-14T23:05:03.807700+0000","last_active":"2024-06-14T23:06:30.530246+0000","last_peered":"2024-06-14T23:06:30.530246+0000","last_clean":"2024-06-14T23:06:30.530246+0000","last_became_active":"2024-06-14T23:05:03.807400+0000","last_became_peered":"2024-06-14T23:05:03.807400+0000","last_unstale":"2024-06-14T23:06:30.530246+0000","last_undegraded":"2024-06-14T23:06:30.530246+0000","last_fullsized":"2024-06-14T23:06:30.530246+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6653,"log_dups_size":0,"ondisk_log_size":6653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:56:35.097613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436011008,"num_objects":6653,"num_object_clones":0,"num_object_copies":19959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6653,"num_write_kb":425792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":600,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.994848+0000","last_change":"2024-06-14T23:04:37.902544+0000","last_active":"2024-06-14T23:06:18.994848+0000","last_peered":"2024-06-14T23:06:18.994848+0000","last_clean":"2024-06-14T23:06:18.994848+0000","last_became_active":"2024-06-14T23:04:37.902191+0000","last_became_peered":"2024-06-14T23:04:37.902191+0000","last_unstale":"2024-06-14T23:06:18.994848+0000","last_undegraded":"2024-06-14T23:06:18.994848+0000","last_fullsized":"2024-06-14T23:06:18.994848+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:05:09.672062+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":96,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993252+0000","last_change":"2024-06-14T23:05:09.660081+0000","last_active":"2024-06-14T23:06:18.993252+0000","last_peered":"2024-06-14T23:06:18.993252+0000","last_clean":"2024-06-14T23:06:18.993252+0000","last_became_active":"2024-06-14T23:05:09.659778+0000","last_became_peered":"2024-06-14T23:05:09.659778+0000","last_unstale":"2024-06-14T23:06:18.993252+0000","last_undegraded":"2024-06-14T23:06:18.993252+0000","last_fullsized":"2024-06-14T23:06:18.993252+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:41:09.113348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":245,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993760+0000","last_change":"2024-06-14T23:04:34.887791+0000","last_active":"2024-06-14T23:06:18.993760+0000","last_peered":"2024-06-14T23:06:18.993760+0000","last_clean":"2024-06-14T23:06:18.993760+0000","last_became_active":"2024-06-14T23:04:34.887099+0000","last_became_peered":"2024-06-14T23:04:34.887099+0000","last_unstale":"2024-06-14T23:06:18.993760+0000","last_undegraded":"2024-06-14T23:06:18.993760+0000","last_fullsized":"2024-06-14T23:06:18.993760+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:46:07.470301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"255'6858","reported_seq":6884,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.123702+0000","last_change":"2024-06-14T23:05:03.808354+0000","last_active":"2024-06-14T23:06:29.123702+0000","last_peered":"2024-06-14T23:06:29.123702+0000","last_clean":"2024-06-14T23:06:29.123702+0000","last_became_active":"2024-06-14T23:05:03.808017+0000","last_became_peered":"2024-06-14T23:05:03.808017+0000","last_unstale":"2024-06-14T23:06:29.123702+0000","last_undegraded":"2024-06-14T23:06:29.123702+0000","last_fullsized":"2024-06-14T23:06:29.123702+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6858,"log_dups_size":0,"ondisk_log_size":6858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:46:06.873436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449445888,"num_objects":6858,"num_object_clones":0,"num_object_copies":20574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6858,"num_write_kb":438912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":236,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.994152+0000","last_change":"2024-06-14T23:04:34.896257+0000","last_active":"2024-06-14T23:06:18.994152+0000","last_peered":"2024-06-14T23:06:18.994152+0000","last_clean":"2024-06-14T23:06:18.994152+0000","last_became_active":"2024-06-14T23:04:34.895926+0000","last_became_peered":"2024-06-14T23:04:34.895926+0000","last_unstale":"2024-06-14T23:06:18.994152+0000","last_undegraded":"2024-06-14T23:06:18.994152+0000","last_fullsized":"2024-06-14T23:06:18.994152+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:17.800902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":215,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.994082+0000","last_change":"2024-06-14T22:49:25.478419+0000","last_active":"2024-06-14T23:06:18.994082+0000","last_peered":"2024-06-14T23:06:18.994082+0000","last_clean":"2024-06-14T23:06:18.994082+0000","last_became_active":"2024-06-14T22:49:25.477799+0000","last_became_peered":"2024-06-14T22:49:25.477799+0000","last_unstale":"2024-06-14T23:06:18.994082+0000","last_undegraded":"2024-06-14T23:06:18.994082+0000","last_fullsized":"2024-06-14T23:06:18.994082+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:00:26.142003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"255'6639","reported_seq":6666,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:30.196407+0000","last_change":"2024-06-14T23:05:03.810157+0000","last_active":"2024-06-14T23:06:30.196407+0000","last_peered":"2024-06-14T23:06:30.196407+0000","last_clean":"2024-06-14T23:06:30.196407+0000","last_became_active":"2024-06-14T23:05:03.809591+0000","last_became_peered":"2024-06-14T23:05:03.809591+0000","last_unstale":"2024-06-14T23:06:30.196407+0000","last_undegraded":"2024-06-14T23:06:30.196407+0000","last_fullsized":"2024-06-14T23:06:30.196407+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:04:46.257184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435093504,"num_objects":6639,"num_object_clones":0,"num_object_copies":19917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6639,"num_write_kb":424896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":592,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.995957+0000","last_change":"2024-06-14T23:04:37.902330+0000","last_active":"2024-06-14T23:06:18.995957+0000","last_peered":"2024-06-14T23:06:18.995957+0000","last_clean":"2024-06-14T23:06:18.995957+0000","last_became_active":"2024-06-14T23:04:37.901760+0000","last_became_peered":"2024-06-14T23:04:37.901760+0000","last_unstale":"2024-06-14T23:06:18.995957+0000","last_undegraded":"2024-06-14T23:06:18.995957+0000","last_fullsized":"2024-06-14T23:06:18.995957+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:57.295069+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":212,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.995503+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:06:18.995503+0000","last_peered":"2024-06-14T23:06:18.995503+0000","last_clean":"2024-06-14T23:06:18.995503+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:06:18.995503+0000","last_undegraded":"2024-06-14T23:06:18.995503+0000","last_fullsized":"2024-06-14T23:06:18.995503+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":225,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993155+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:06:18.993155+0000","last_peered":"2024-06-14T23:06:18.993155+0000","last_clean":"2024-06-14T23:06:18.993155+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:06:18.993155+0000","last_undegraded":"2024-06-14T23:06:18.993155+0000","last_fullsized":"2024-06-14T23:06:18.993155+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"255'6981","reported_seq":7008,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.504831+0000","last_change":"2024-06-14T23:05:09.659680+0000","last_active":"2024-06-14T23:06:29.504831+0000","last_peered":"2024-06-14T23:06:29.504831+0000","last_clean":"2024-06-14T23:06:29.504831+0000","last_became_active":"2024-06-14T23:05:09.659391+0000","last_became_peered":"2024-06-14T23:05:09.659391+0000","last_unstale":"2024-06-14T23:06:29.504831+0000","last_undegraded":"2024-06-14T23:06:29.504831+0000","last_fullsized":"2024-06-14T23:06:29.504831+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6981,"log_dups_size":0,"ondisk_log_size":6981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:48:20.431065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457506816,"num_objects":6981,"num_object_clones":0,"num_object_copies":20943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6981,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":508,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:21.536771+0000","last_change":"2024-06-14T23:06:21.536713+0000","last_active":"2024-06-14T23:06:21.536771+0000","last_peered":"2024-06-14T23:06:21.536771+0000","last_clean":"2024-06-14T23:06:21.536771+0000","last_became_active":"2024-06-14T23:06:19.001469+0000","last_became_peered":"2024-06-14T23:06:19.001469+0000","last_unstale":"2024-06-14T23:06:21.536771+0000","last_undegraded":"2024-06-14T23:06:21.536771+0000","last_fullsized":"2024-06-14T23:06:21.536771+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:06:21.536637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:06:21.536637+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:25:15.843643+0000","scrub_duration":0.0035987950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":240,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996326+0000","last_change":"2024-06-14T23:04:34.889456+0000","last_active":"2024-06-14T23:06:18.996326+0000","last_peered":"2024-06-14T23:06:18.996326+0000","last_clean":"2024-06-14T23:06:18.996326+0000","last_became_active":"2024-06-14T23:04:34.888908+0000","last_became_peered":"2024-06-14T23:04:34.888908+0000","last_unstale":"2024-06-14T23:06:18.996326+0000","last_undegraded":"2024-06-14T23:06:18.996326+0000","last_fullsized":"2024-06-14T23:06:18.996326+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:00:24.632734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":159,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996105+0000","last_change":"2024-06-14T22:58:09.752136+0000","last_active":"2024-06-14T23:06:18.996105+0000","last_peered":"2024-06-14T23:06:18.996105+0000","last_clean":"2024-06-14T23:06:18.996105+0000","last_became_active":"2024-06-14T22:58:09.751515+0000","last_became_peered":"2024-06-14T22:58:09.751515+0000","last_unstale":"2024-06-14T23:06:18.996105+0000","last_undegraded":"2024-06-14T23:06:18.996105+0000","last_fullsized":"2024-06-14T23:06:18.996105+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:00.154049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":198,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.000694+0000","last_change":"2024-06-14T22:34:47.047908+0000","last_active":"2024-06-14T23:06:19.000694+0000","last_peered":"2024-06-14T23:06:19.000694+0000","last_clean":"2024-06-14T23:06:19.000694+0000","last_became_active":"2024-06-14T22:34:47.047080+0000","last_became_peered":"2024-06-14T22:34:47.047080+0000","last_unstale":"2024-06-14T23:06:19.000694+0000","last_undegraded":"2024-06-14T23:06:19.000694+0000","last_fullsized":"2024-06-14T23:06:19.000694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:15.029100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"255'6717","reported_seq":6751,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.766199+0000","last_change":"2024-06-14T23:06:04.928899+0000","last_active":"2024-06-14T23:06:29.766199+0000","last_peered":"2024-06-14T23:06:29.766199+0000","last_clean":"2024-06-14T23:06:29.766199+0000","last_became_active":"2024-06-14T23:06:04.923998+0000","last_became_peered":"2024-06-14T23:06:04.923998+0000","last_unstale":"2024-06-14T23:06:29.766199+0000","last_undegraded":"2024-06-14T23:06:29.766199+0000","last_fullsized":"2024-06-14T23:06:29.766199+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6717,"log_dups_size":0,"ondisk_log_size":6717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:03:03.356810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440205312,"num_objects":6717,"num_object_clones":0,"num_object_copies":20151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6717,"num_write_kb":429888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":555,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.992069+0000","last_change":"2024-06-14T23:00:30.719758+0000","last_active":"2024-06-14T23:06:18.992069+0000","last_peered":"2024-06-14T23:06:18.992069+0000","last_clean":"2024-06-14T23:06:18.992069+0000","last_became_active":"2024-06-14T23:00:17.403817+0000","last_became_peered":"2024-06-14T23:00:17.403817+0000","last_unstale":"2024-06-14T23:06:18.992069+0000","last_undegraded":"2024-06-14T23:06:18.992069+0000","last_fullsized":"2024-06-14T23:06:18.992069+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:03:47.728089+0000","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":216,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.990897+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:06:18.990897+0000","last_peered":"2024-06-14T23:06:18.990897+0000","last_clean":"2024-06-14T23:06:18.990897+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:06:18.990897+0000","last_undegraded":"2024-06-14T23:06:18.990897+0000","last_fullsized":"2024-06-14T23:06:18.990897+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":93,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.002403+0000","last_change":"2024-06-14T23:06:19.001589+0000","last_active":"2024-06-14T23:06:19.002403+0000","last_peered":"2024-06-14T23:06:19.002403+0000","last_clean":"2024-06-14T23:06:19.002403+0000","last_became_active":"2024-06-14T23:06:19.001260+0000","last_became_peered":"2024-06-14T23:06:19.001260+0000","last_unstale":"2024-06-14T23:06:19.002403+0000","last_undegraded":"2024-06-14T23:06:19.002403+0000","last_fullsized":"2024-06-14T23:06:19.002403+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:54:46.316751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":221,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.999106+0000","last_change":"2024-06-14T23:06:18.998828+0000","last_active":"2024-06-14T23:06:18.999106+0000","last_peered":"2024-06-14T23:06:18.999106+0000","last_clean":"2024-06-14T23:06:18.999106+0000","last_became_active":"2024-06-14T23:06:18.998461+0000","last_became_peered":"2024-06-14T23:06:18.998461+0000","last_unstale":"2024-06-14T23:06:18.999106+0000","last_undegraded":"2024-06-14T23:06:18.999106+0000","last_fullsized":"2024-06-14T23:06:18.999106+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:18:21.826501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"255'6627","reported_seq":6652,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.169099+0000","last_change":"2024-06-14T23:05:03.808305+0000","last_active":"2024-06-14T23:06:29.169099+0000","last_peered":"2024-06-14T23:06:29.169099+0000","last_clean":"2024-06-14T23:06:29.169099+0000","last_became_active":"2024-06-14T23:05:03.807960+0000","last_became_peered":"2024-06-14T23:05:03.807960+0000","last_unstale":"2024-06-14T23:06:29.169099+0000","last_undegraded":"2024-06-14T23:06:29.169099+0000","last_fullsized":"2024-06-14T23:06:29.169099+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6627,"log_dups_size":0,"ondisk_log_size":6627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:14.633117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434307072,"num_objects":6627,"num_object_clones":0,"num_object_copies":19881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6627,"num_write_kb":424128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":485,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.887990+0000","last_change":"2024-06-14T23:06:19.887928+0000","last_active":"2024-06-14T23:06:19.887990+0000","last_peered":"2024-06-14T23:06:19.887990+0000","last_clean":"2024-06-14T23:06:19.887990+0000","last_became_active":"2024-06-14T23:06:18.997847+0000","last_became_peered":"2024-06-14T23:06:18.997847+0000","last_unstale":"2024-06-14T23:06:19.887990+0000","last_undegraded":"2024-06-14T23:06:19.887990+0000","last_fullsized":"2024-06-14T23:06:19.887990+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:06:19.887856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:06:19.887856+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:40.735391+0000","scrub_duration":0.012536587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.990619+0000","last_change":"2024-06-14T23:04:34.879251+0000","last_active":"2024-06-14T23:06:18.990619+0000","last_peered":"2024-06-14T23:06:18.990619+0000","last_clean":"2024-06-14T23:06:18.990619+0000","last_became_active":"2024-06-14T23:04:34.878470+0000","last_became_peered":"2024-06-14T23:04:34.878470+0000","last_unstale":"2024-06-14T23:06:18.990619+0000","last_undegraded":"2024-06-14T23:06:18.990619+0000","last_fullsized":"2024-06-14T23:06:18.990619+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:55.949794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":244,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.991794+0000","last_change":"2024-06-14T23:04:34.885379+0000","last_active":"2024-06-14T23:06:18.991794+0000","last_peered":"2024-06-14T23:06:18.991794+0000","last_clean":"2024-06-14T23:06:18.991794+0000","last_became_active":"2024-06-14T23:04:34.885076+0000","last_became_peered":"2024-06-14T23:04:34.885076+0000","last_unstale":"2024-06-14T23:06:18.991794+0000","last_undegraded":"2024-06-14T23:06:18.991794+0000","last_fullsized":"2024-06-14T23:06:18.991794+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:23:32.132929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":210,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993985+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:06:18.993985+0000","last_peered":"2024-06-14T23:06:18.993985+0000","last_clean":"2024-06-14T23:06:18.993985+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:06:18.993985+0000","last_undegraded":"2024-06-14T23:06:18.993985+0000","last_fullsized":"2024-06-14T23:06:18.993985+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"255'6820","reported_seq":6854,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.732571+0000","last_change":"2024-06-14T23:06:04.928960+0000","last_active":"2024-06-14T23:06:29.732571+0000","last_peered":"2024-06-14T23:06:29.732571+0000","last_clean":"2024-06-14T23:06:29.732571+0000","last_became_active":"2024-06-14T23:06:04.923775+0000","last_became_peered":"2024-06-14T23:06:04.923775+0000","last_unstale":"2024-06-14T23:06:29.732571+0000","last_undegraded":"2024-06-14T23:06:29.732571+0000","last_fullsized":"2024-06-14T23:06:29.732571+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6820,"log_dups_size":0,"ondisk_log_size":6820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:10:46.076558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446955520,"num_objects":6820,"num_object_clones":0,"num_object_copies":20460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6820,"num_write_kb":436480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":591,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.994969+0000","last_change":"2024-06-14T23:04:37.903143+0000","last_active":"2024-06-14T23:06:18.994969+0000","last_peered":"2024-06-14T23:06:18.994969+0000","last_clean":"2024-06-14T23:06:18.994969+0000","last_became_active":"2024-06-14T23:04:37.902356+0000","last_became_peered":"2024-06-14T23:04:37.902356+0000","last_unstale":"2024-06-14T23:06:18.994969+0000","last_undegraded":"2024-06-14T23:06:18.994969+0000","last_fullsized":"2024-06-14T23:06:18.994969+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.818076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.818076+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:01:11.632458+0000","scrub_duration":0.0013568390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":220,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996185+0000","last_change":"2024-06-14T22:49:25.488737+0000","last_active":"2024-06-14T23:06:18.996185+0000","last_peered":"2024-06-14T23:06:18.996185+0000","last_clean":"2024-06-14T23:06:18.996185+0000","last_became_active":"2024-06-14T22:49:25.488424+0000","last_became_peered":"2024-06-14T22:49:25.488424+0000","last_unstale":"2024-06-14T23:06:18.996185+0000","last_undegraded":"2024-06-14T23:06:18.996185+0000","last_fullsized":"2024-06-14T23:06:18.996185+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:57:02.740802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":216,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993964+0000","last_change":"2024-06-14T22:44:51.839399+0000","last_active":"2024-06-14T23:06:18.993964+0000","last_peered":"2024-06-14T23:06:18.993964+0000","last_clean":"2024-06-14T23:06:18.993964+0000","last_became_active":"2024-06-14T22:44:51.839080+0000","last_became_peered":"2024-06-14T22:44:51.839080+0000","last_unstale":"2024-06-14T23:06:18.993964+0000","last_undegraded":"2024-06-14T23:06:18.993964+0000","last_fullsized":"2024-06-14T23:06:18.993964+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:40.892882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":259,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.991033+0000","last_change":"2024-06-14T23:04:34.880836+0000","last_active":"2024-06-14T23:06:18.991033+0000","last_peered":"2024-06-14T23:06:18.991033+0000","last_clean":"2024-06-14T23:06:18.991033+0000","last_became_active":"2024-06-14T23:04:34.880540+0000","last_became_peered":"2024-06-14T23:04:34.880540+0000","last_unstale":"2024-06-14T23:06:18.991033+0000","last_undegraded":"2024-06-14T23:06:18.991033+0000","last_fullsized":"2024-06-14T23:06:18.991033+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:06.981511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"255'6777","reported_seq":6804,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:30.557975+0000","last_change":"2024-06-14T23:05:03.625434+0000","last_active":"2024-06-14T23:06:30.557975+0000","last_peered":"2024-06-14T23:06:30.557975+0000","last_clean":"2024-06-14T23:06:30.557975+0000","last_became_active":"2024-06-14T23:05:03.625174+0000","last_became_peered":"2024-06-14T23:05:03.625174+0000","last_unstale":"2024-06-14T23:06:30.557975+0000","last_undegraded":"2024-06-14T23:06:30.557975+0000","last_fullsized":"2024-06-14T23:06:30.557975+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:00.079552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444137472,"num_objects":6777,"num_object_clones":0,"num_object_copies":20331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6777,"num_write_kb":433728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":505,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.000057+0000","last_change":"2024-06-14T23:00:42.661659+0000","last_active":"2024-06-14T23:06:19.000057+0000","last_peered":"2024-06-14T23:06:19.000057+0000","last_clean":"2024-06-14T23:06:19.000057+0000","last_became_active":"2024-06-14T23:00:17.398852+0000","last_became_peered":"2024-06-14T23:00:17.398852+0000","last_unstale":"2024-06-14T23:06:19.000057+0000","last_undegraded":"2024-06-14T23:06:19.000057+0000","last_fullsized":"2024-06-14T23:06:19.000057+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:47:14.444805+0000","scrub_duration":0.0053445919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":253,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996854+0000","last_change":"2024-06-14T23:04:34.880181+0000","last_active":"2024-06-14T23:06:18.996854+0000","last_peered":"2024-06-14T23:06:18.996854+0000","last_clean":"2024-06-14T23:06:18.996854+0000","last_became_active":"2024-06-14T23:04:34.879842+0000","last_became_peered":"2024-06-14T23:04:34.879842+0000","last_unstale":"2024-06-14T23:06:18.996854+0000","last_undegraded":"2024-06-14T23:06:18.996854+0000","last_fullsized":"2024-06-14T23:06:18.996854+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:14:18.433225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":216,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.991270+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:06:18.991270+0000","last_peered":"2024-06-14T23:06:18.991270+0000","last_clean":"2024-06-14T23:06:18.991270+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:06:18.991270+0000","last_undegraded":"2024-06-14T23:06:18.991270+0000","last_fullsized":"2024-06-14T23:06:18.991270+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":211,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.991119+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:06:18.991119+0000","last_peered":"2024-06-14T23:06:18.991119+0000","last_clean":"2024-06-14T23:06:18.991119+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:06:18.991119+0000","last_undegraded":"2024-06-14T23:06:18.991119+0000","last_fullsized":"2024-06-14T23:06:18.991119+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"255'6864","reported_seq":6891,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:30.205631+0000","last_change":"2024-06-14T23:05:03.621987+0000","last_active":"2024-06-14T23:06:30.205631+0000","last_peered":"2024-06-14T23:06:30.205631+0000","last_clean":"2024-06-14T23:06:30.205631+0000","last_became_active":"2024-06-14T23:05:03.621724+0000","last_became_peered":"2024-06-14T23:05:03.621724+0000","last_unstale":"2024-06-14T23:06:30.205631+0000","last_undegraded":"2024-06-14T23:06:30.205631+0000","last_fullsized":"2024-06-14T23:06:30.205631+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6864,"log_dups_size":0,"ondisk_log_size":6864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:12.558177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449839104,"num_objects":6864,"num_object_clones":0,"num_object_copies":20592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6864,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":505,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.000196+0000","last_change":"2024-06-14T23:00:31.555325+0000","last_active":"2024-06-14T23:06:19.000196+0000","last_peered":"2024-06-14T23:06:19.000196+0000","last_clean":"2024-06-14T23:06:19.000196+0000","last_became_active":"2024-06-14T23:00:17.400871+0000","last_became_peered":"2024-06-14T23:00:17.400871+0000","last_unstale":"2024-06-14T23:06:19.000196+0000","last_undegraded":"2024-06-14T23:06:19.000196+0000","last_fullsized":"2024-06-14T23:06:19.000196+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:43:58.191436+0000","scrub_duration":0.001665094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":244,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.991193+0000","last_change":"2024-06-14T23:04:34.880684+0000","last_active":"2024-06-14T23:06:18.991193+0000","last_peered":"2024-06-14T23:06:18.991193+0000","last_clean":"2024-06-14T23:06:18.991193+0000","last_became_active":"2024-06-14T23:04:34.880127+0000","last_became_peered":"2024-06-14T23:04:34.880127+0000","last_unstale":"2024-06-14T23:06:18.991193+0000","last_undegraded":"2024-06-14T23:06:18.991193+0000","last_fullsized":"2024-06-14T23:06:18.991193+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:53:52.966928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":220,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996417+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:06:18.996417+0000","last_peered":"2024-06-14T23:06:18.996417+0000","last_clean":"2024-06-14T23:06:18.996417+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:06:18.996417+0000","last_undegraded":"2024-06-14T23:06:18.996417+0000","last_fullsized":"2024-06-14T23:06:18.996417+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.004135+0000","last_change":"2024-06-14T23:06:19.003881+0000","last_active":"2024-06-14T23:06:19.004135+0000","last_peered":"2024-06-14T23:06:19.004135+0000","last_clean":"2024-06-14T23:06:19.004135+0000","last_became_active":"2024-06-14T23:06:19.003589+0000","last_became_peered":"2024-06-14T23:06:19.003589+0000","last_unstale":"2024-06-14T23:06:19.004135+0000","last_undegraded":"2024-06-14T23:06:19.004135+0000","last_fullsized":"2024-06-14T23:06:19.004135+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:52.332299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"255'6707","reported_seq":6734,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:30.203570+0000","last_change":"2024-06-14T23:05:03.621656+0000","last_active":"2024-06-14T23:06:30.203570+0000","last_peered":"2024-06-14T23:06:30.203570+0000","last_clean":"2024-06-14T23:06:30.203570+0000","last_became_active":"2024-06-14T23:05:03.620834+0000","last_became_peered":"2024-06-14T23:05:03.620834+0000","last_unstale":"2024-06-14T23:06:30.203570+0000","last_undegraded":"2024-06-14T23:06:30.203570+0000","last_fullsized":"2024-06-14T23:06:30.203570+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:59:26.118782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439549952,"num_objects":6707,"num_object_clones":0,"num_object_copies":20121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6707,"num_write_kb":429248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":592,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993446+0000","last_change":"2024-06-14T23:04:37.901715+0000","last_active":"2024-06-14T23:06:18.993446+0000","last_peered":"2024-06-14T23:06:18.993446+0000","last_clean":"2024-06-14T23:06:18.993446+0000","last_became_active":"2024-06-14T23:04:37.901351+0000","last_became_peered":"2024-06-14T23:04:37.901351+0000","last_unstale":"2024-06-14T23:06:18.993446+0000","last_undegraded":"2024-06-14T23:06:18.993446+0000","last_fullsized":"2024-06-14T23:06:18.993446+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.174132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.174132+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:55.141528+0000","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":146,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.767374+0000","last_change":"2024-06-14T23:06:19.000734+0000","last_active":"2024-06-14T23:06:19.767374+0000","last_peered":"2024-06-14T23:06:19.767374+0000","last_clean":"2024-06-14T23:06:19.767374+0000","last_became_active":"2024-06-14T23:06:19.000432+0000","last_became_peered":"2024-06-14T23:06:19.000432+0000","last_unstale":"2024-06-14T23:06:19.767374+0000","last_undegraded":"2024-06-14T23:06:19.767374+0000","last_fullsized":"2024-06-14T23:06:19.767374+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:48.268127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":209,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.994323+0000","last_change":"2024-06-14T22:49:25.489975+0000","last_active":"2024-06-14T23:06:18.994323+0000","last_peered":"2024-06-14T23:06:18.994323+0000","last_clean":"2024-06-14T23:06:18.994323+0000","last_became_active":"2024-06-14T22:49:25.486457+0000","last_became_peered":"2024-06-14T22:49:25.486457+0000","last_unstale":"2024-06-14T23:06:18.994323+0000","last_undegraded":"2024-06-14T23:06:18.994323+0000","last_fullsized":"2024-06-14T23:06:18.994323+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:23.396144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":147,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.995258+0000","last_change":"2024-06-14T23:06:04.932485+0000","last_active":"2024-06-14T23:06:18.995258+0000","last_peered":"2024-06-14T23:06:18.995258+0000","last_clean":"2024-06-14T23:06:18.995258+0000","last_became_active":"2024-06-14T23:06:04.932166+0000","last_became_peered":"2024-06-14T23:06:04.932166+0000","last_unstale":"2024-06-14T23:06:18.995258+0000","last_undegraded":"2024-06-14T23:06:18.995258+0000","last_fullsized":"2024-06-14T23:06:18.995258+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:46.149456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"255'6555","reported_seq":6581,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-06-14T23:06:29.176994+0000","last_change":"2024-06-14T23:05:03.623137+0000","last_active":"2024-06-14T23:06:29.176994+0000","last_peered":"2024-06-14T23:06:29.176994+0000","last_clean":"2024-06-14T23:06:29.176994+0000","last_became_active":"2024-06-14T23:05:03.622833+0000","last_became_peered":"2024-06-14T23:05:03.622833+0000","last_unstale":"2024-06-14T23:06:29.176994+0000","last_undegraded":"2024-06-14T23:06:29.176994+0000","last_fullsized":"2024-06-14T23:06:29.176994+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6555,"log_dups_size":0,"ondisk_log_size":6555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:53:45.610358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429588480,"num_objects":6555,"num_object_clones":0,"num_object_copies":19665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6555,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":555,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.992210+0000","last_change":"2024-06-14T23:00:32.716013+0000","last_active":"2024-06-14T23:06:18.992210+0000","last_peered":"2024-06-14T23:06:18.992210+0000","last_clean":"2024-06-14T23:06:18.992210+0000","last_became_active":"2024-06-14T23:00:17.398349+0000","last_became_peered":"2024-06-14T23:00:17.398349+0000","last_unstale":"2024-06-14T23:06:18.992210+0000","last_undegraded":"2024-06-14T23:06:18.992210+0000","last_fullsized":"2024-06-14T23:06:18.992210+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.715907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.715907+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:23:40.212866+0000","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":38,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:20.207600+0000","last_change":"2024-06-14T23:06:19.000517+0000","last_active":"2024-06-14T23:06:20.207600+0000","last_peered":"2024-06-14T23:06:20.207600+0000","last_clean":"2024-06-14T23:06:20.207600+0000","last_became_active":"2024-06-14T23:06:19.000141+0000","last_became_peered":"2024-06-14T23:06:19.000141+0000","last_unstale":"2024-06-14T23:06:20.207600+0000","last_undegraded":"2024-06-14T23:06:20.207600+0000","last_fullsized":"2024-06-14T23:06:20.207600+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:57:42.599655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":240,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.000231+0000","last_change":"2024-06-14T23:04:34.890587+0000","last_active":"2024-06-14T23:06:19.000231+0000","last_peered":"2024-06-14T23:06:19.000231+0000","last_clean":"2024-06-14T23:06:19.000231+0000","last_became_active":"2024-06-14T23:04:34.890254+0000","last_became_peered":"2024-06-14T23:04:34.890254+0000","last_unstale":"2024-06-14T23:06:19.000231+0000","last_undegraded":"2024-06-14T23:06:19.000231+0000","last_fullsized":"2024-06-14T23:06:19.000231+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:28:06.189496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":198,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993116+0000","last_change":"2024-06-14T22:34:46.494329+0000","last_active":"2024-06-14T23:06:18.993116+0000","last_peered":"2024-06-14T23:06:18.993116+0000","last_clean":"2024-06-14T23:06:18.993116+0000","last_became_active":"2024-06-14T22:34:46.494063+0000","last_became_peered":"2024-06-14T22:34:46.494063+0000","last_unstale":"2024-06-14T23:06:18.993116+0000","last_undegraded":"2024-06-14T23:06:18.993116+0000","last_fullsized":"2024-06-14T23:06:18.993116+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:08.087138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":215,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993870+0000","last_change":"2024-06-14T22:49:25.476803+0000","last_active":"2024-06-14T23:06:18.993870+0000","last_peered":"2024-06-14T23:06:18.993870+0000","last_clean":"2024-06-14T23:06:18.993870+0000","last_became_active":"2024-06-14T22:49:25.476331+0000","last_became_peered":"2024-06-14T22:49:25.476331+0000","last_unstale":"2024-06-14T23:06:18.993870+0000","last_undegraded":"2024-06-14T23:06:18.993870+0000","last_fullsized":"2024-06-14T23:06:18.993870+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:08.700533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":221,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.994285+0000","last_change":"2024-06-14T22:34:45.812438+0000","last_active":"2024-06-14T23:06:18.994285+0000","last_peered":"2024-06-14T23:06:18.994285+0000","last_clean":"2024-06-14T23:06:18.994285+0000","last_became_active":"2024-06-14T22:34:45.810932+0000","last_became_peered":"2024-06-14T22:34:45.810932+0000","last_unstale":"2024-06-14T23:06:18.994285+0000","last_undegraded":"2024-06-14T23:06:18.994285+0000","last_fullsized":"2024-06-14T23:06:18.994285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:06.211001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":243,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.990992+0000","last_change":"2024-06-14T23:04:34.885556+0000","last_active":"2024-06-14T23:06:18.990992+0000","last_peered":"2024-06-14T23:06:18.990992+0000","last_clean":"2024-06-14T23:06:18.990992+0000","last_became_active":"2024-06-14T23:04:34.885222+0000","last_became_peered":"2024-06-14T23:04:34.885222+0000","last_unstale":"2024-06-14T23:06:18.990992+0000","last_undegraded":"2024-06-14T23:06:18.990992+0000","last_fullsized":"2024-06-14T23:06:18.990992+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:19:04.372165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":136,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993374+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:06:18.993374+0000","last_peered":"2024-06-14T23:06:18.993374+0000","last_clean":"2024-06-14T23:06:18.993374+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:06:18.993374+0000","last_undegraded":"2024-06-14T23:06:18.993374+0000","last_fullsized":"2024-06-14T23:06:18.993374+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":216,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996689+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:06:18.996689+0000","last_peered":"2024-06-14T23:06:18.996689+0000","last_clean":"2024-06-14T23:06:18.996689+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:06:18.996689+0000","last_undegraded":"2024-06-14T23:06:18.996689+0000","last_fullsized":"2024-06-14T23:06:18.996689+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":103,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.994404+0000","last_change":"2024-06-14T23:06:04.937475+0000","last_active":"2024-06-14T23:06:18.994404+0000","last_peered":"2024-06-14T23:06:18.994404+0000","last_clean":"2024-06-14T23:06:18.994404+0000","last_became_active":"2024-06-14T23:06:04.934624+0000","last_became_peered":"2024-06-14T23:06:04.934624+0000","last_unstale":"2024-06-14T23:06:18.994404+0000","last_undegraded":"2024-06-14T23:06:18.994404+0000","last_fullsized":"2024-06-14T23:06:18.994404+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:39:20.514729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":92,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.001293+0000","last_change":"2024-06-14T23:06:19.000252+0000","last_active":"2024-06-14T23:06:19.001293+0000","last_peered":"2024-06-14T23:06:19.001293+0000","last_clean":"2024-06-14T23:06:19.001293+0000","last_became_active":"2024-06-14T23:06:18.999939+0000","last_became_peered":"2024-06-14T23:06:18.999939+0000","last_unstale":"2024-06-14T23:06:19.001293+0000","last_undegraded":"2024-06-14T23:06:19.001293+0000","last_fullsized":"2024-06-14T23:06:19.001293+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:50.509140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":211,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.995065+0000","last_change":"2024-06-14T22:54:06.281615+0000","last_active":"2024-06-14T23:06:18.995065+0000","last_peered":"2024-06-14T23:06:18.995065+0000","last_clean":"2024-06-14T23:06:18.995065+0000","last_became_active":"2024-06-14T22:54:06.279782+0000","last_became_peered":"2024-06-14T22:54:06.279782+0000","last_unstale":"2024-06-14T23:06:18.995065+0000","last_undegraded":"2024-06-14T23:06:18.995065+0000","last_fullsized":"2024-06-14T23:06:18.995065+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.996864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":215,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.000143+0000","last_change":"2024-06-14T22:49:25.487221+0000","last_active":"2024-06-14T23:06:19.000143+0000","last_peered":"2024-06-14T23:06:19.000143+0000","last_clean":"2024-06-14T23:06:19.000143+0000","last_became_active":"2024-06-14T22:49:25.486839+0000","last_became_peered":"2024-06-14T22:49:25.486839+0000","last_unstale":"2024-06-14T23:06:19.000143+0000","last_undegraded":"2024-06-14T23:06:19.000143+0000","last_fullsized":"2024-06-14T23:06:19.000143+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:24:12.416491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":114,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:20.207464+0000","last_change":"2024-06-14T23:06:18.998767+0000","last_active":"2024-06-14T23:06:20.207464+0000","last_peered":"2024-06-14T23:06:20.207464+0000","last_clean":"2024-06-14T23:06:20.207464+0000","last_became_active":"2024-06-14T23:06:18.998481+0000","last_became_peered":"2024-06-14T23:06:18.998481+0000","last_unstale":"2024-06-14T23:06:20.207464+0000","last_undegraded":"2024-06-14T23:06:20.207464+0000","last_fullsized":"2024-06-14T23:06:20.207464+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:58:13.282478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":93,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.767333+0000","last_change":"2024-06-14T23:06:18.999600+0000","last_active":"2024-06-14T23:06:19.767333+0000","last_peered":"2024-06-14T23:06:19.767333+0000","last_clean":"2024-06-14T23:06:19.767333+0000","last_became_active":"2024-06-14T23:06:18.999293+0000","last_became_peered":"2024-06-14T23:06:18.999293+0000","last_unstale":"2024-06-14T23:06:19.767333+0000","last_undegraded":"2024-06-14T23:06:19.767333+0000","last_fullsized":"2024-06-14T23:06:19.767333+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:28:12.373005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":225,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:20.207500+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:06:20.207500+0000","last_peered":"2024-06-14T23:06:20.207500+0000","last_clean":"2024-06-14T23:06:20.207500+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:06:20.207500+0000","last_undegraded":"2024-06-14T23:06:20.207500+0000","last_fullsized":"2024-06-14T23:06:20.207500+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.990794+0000","last_change":"2024-06-14T23:05:43.964532+0000","last_active":"2024-06-14T23:06:18.990794+0000","last_peered":"2024-06-14T23:06:18.990794+0000","last_clean":"2024-06-14T23:06:18.990794+0000","last_became_active":"2024-06-14T23:05:43.964255+0000","last_became_peered":"2024-06-14T23:05:43.964255+0000","last_unstale":"2024-06-14T23:06:18.990794+0000","last_undegraded":"2024-06-14T23:06:18.990794+0000","last_fullsized":"2024-06-14T23:06:18.990794+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:06.098124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":254,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993466+0000","last_change":"2024-06-14T23:04:34.883863+0000","last_active":"2024-06-14T23:06:18.993466+0000","last_peered":"2024-06-14T23:06:18.993466+0000","last_clean":"2024-06-14T23:06:18.993466+0000","last_became_active":"2024-06-14T23:04:34.881668+0000","last_became_peered":"2024-06-14T23:04:34.881668+0000","last_unstale":"2024-06-14T23:06:18.993466+0000","last_undegraded":"2024-06-14T23:06:18.993466+0000","last_fullsized":"2024-06-14T23:06:18.993466+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:44:45.737416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":36,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.992446+0000","last_change":"2024-06-14T23:04:34.879498+0000","last_active":"2024-06-14T23:06:18.992446+0000","last_peered":"2024-06-14T23:06:18.992446+0000","last_clean":"2024-06-14T23:06:18.992446+0000","last_became_active":"2024-06-14T23:04:34.878932+0000","last_became_peered":"2024-06-14T23:04:34.878932+0000","last_unstale":"2024-06-14T23:06:18.992446+0000","last_undegraded":"2024-06-14T23:06:18.992446+0000","last_fullsized":"2024-06-14T23:06:18.992446+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:56:50.201189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":28,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.003404+0000","last_change":"2024-06-14T23:06:19.003121+0000","last_active":"2024-06-14T23:06:19.003404+0000","last_peered":"2024-06-14T23:06:19.003404+0000","last_clean":"2024-06-14T23:06:19.003404+0000","last_became_active":"2024-06-14T23:06:19.002731+0000","last_became_peered":"2024-06-14T23:06:19.002731+0000","last_unstale":"2024-06-14T23:06:19.003404+0000","last_undegraded":"2024-06-14T23:06:19.003404+0000","last_fullsized":"2024-06-14T23:06:19.003404+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:31:04.851974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":232,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993790+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:06:18.993790+0000","last_peered":"2024-06-14T23:06:18.993790+0000","last_clean":"2024-06-14T23:06:18.993790+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:06:18.993790+0000","last_undegraded":"2024-06-14T23:06:18.993790+0000","last_fullsized":"2024-06-14T23:06:18.993790+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":218,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996472+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:06:18.996472+0000","last_peered":"2024-06-14T23:06:18.996472+0000","last_clean":"2024-06-14T23:06:18.996472+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:06:18.996472+0000","last_undegraded":"2024-06-14T23:06:18.996472+0000","last_fullsized":"2024-06-14T23:06:18.996472+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":232,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.993364+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:06:18.993364+0000","last_peered":"2024-06-14T23:06:18.993364+0000","last_clean":"2024-06-14T23:06:18.993364+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:06:18.993364+0000","last_undegraded":"2024-06-14T23:06:18.993364+0000","last_fullsized":"2024-06-14T23:06:18.993364+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":201,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.996833+0000","last_change":"2024-06-14T22:34:45.808714+0000","last_active":"2024-06-14T23:06:18.996833+0000","last_peered":"2024-06-14T23:06:18.996833+0000","last_clean":"2024-06-14T23:06:18.996833+0000","last_became_active":"2024-06-14T22:34:45.808375+0000","last_became_peered":"2024-06-14T22:34:45.808375+0000","last_unstale":"2024-06-14T23:06:18.996833+0000","last_undegraded":"2024-06-14T23:06:18.996833+0000","last_fullsized":"2024-06-14T23:06:18.996833+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:33:11.357666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":36,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.990550+0000","last_change":"2024-06-14T23:04:34.879700+0000","last_active":"2024-06-14T23:06:18.990550+0000","last_peered":"2024-06-14T23:06:18.990550+0000","last_clean":"2024-06-14T23:06:18.990550+0000","last_became_active":"2024-06-14T23:04:34.878924+0000","last_became_peered":"2024-06-14T23:04:34.878924+0000","last_unstale":"2024-06-14T23:06:18.990550+0000","last_undegraded":"2024-06-14T23:06:18.990550+0000","last_fullsized":"2024-06-14T23:06:18.990550+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:28.826486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":126,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.992549+0000","last_change":"2024-06-14T23:05:09.659896+0000","last_active":"2024-06-14T23:06:18.992549+0000","last_peered":"2024-06-14T23:06:18.992549+0000","last_clean":"2024-06-14T23:06:18.992549+0000","last_became_active":"2024-06-14T23:05:09.659539+0000","last_became_peered":"2024-06-14T23:05:09.659539+0000","last_unstale":"2024-06-14T23:06:18.992549+0000","last_undegraded":"2024-06-14T23:06:18.992549+0000","last_fullsized":"2024-06-14T23:06:18.992549+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:21.285896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":215,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:18.992910+0000","last_change":"2024-06-14T22:49:25.474023+0000","last_active":"2024-06-14T23:06:18.992910+0000","last_peered":"2024-06-14T23:06:18.992910+0000","last_clean":"2024-06-14T23:06:18.992910+0000","last_became_active":"2024-06-14T22:49:25.473686+0000","last_became_peered":"2024-06-14T22:49:25.473686+0000","last_unstale":"2024-06-14T23:06:18.992910+0000","last_undegraded":"2024-06-14T23:06:18.992910+0000","last_fullsized":"2024-06-14T23:06:18.992910+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:22.212415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":39,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-06-14T23:06:19.002949+0000","last_change":"2024-06-14T23:06:19.002447+0000","last_active":"2024-06-14T23:06:19.002949+0000","last_peered":"2024-06-14T23:06:19.002949+0000","last_clean":"2024-06-14T23:06:19.002949+0000","last_became_active":"2024-06-14T23:06:19.002029+0000","last_became_peered":"2024-06-14T23:06:19.002029+0000","last_unstale":"2024-06-14T23:06:19.002949+0000","last_undegraded":"2024-06-14T23:06:19.002949+0000","last_fullsized":"2024-06-14T23:06:19.002949+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:27.132531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7083589632,"num_objects":108087,"num_object_clones":0,"num_object_copies":324261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108087,"num_write_kb":6917568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21195587584,"data_stored":21195587584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108087,"ondisk_log_size":108087,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1771,"ondisk_log_size":1771,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234389,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3455300,"kb_used_data":2556348,"kb_used_omap":1,"kb_used_meta":898942,"kb_avail":90289852,"statfs":{"total":95995035648,"available":92456808448,"internally_reserved":0,"allocated":2617700352,"data_stored":2616439825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920516952},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.95299999999999996,"15min":0.98199999999999998},"min":{"1min":0.214,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.7509999999999999,"5min":3.1699999999999999,"15min":6.9539999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.3089999999999999,"15min":1.103},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.20300000000000001},"max":{"1min":4.3550000000000004,"5min":6.9550000000000001,"15min":6.9550000000000001},"last":3.2839999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:05:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.079,"15min":1.6459999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.1259999999999999,"5min":6.3090000000000002,"15min":83.295000000000002},"last":3.3490000000000002},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.94099999999999995,"15min":1.6639999999999999},"min":{"1min":0.34599999999999997,"5min":0.22900000000000001,"15min":0.223},"max":{"1min":5.2889999999999997,"5min":6.2279999999999998,"15min":82.972999999999999},"last":0.38500000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:05:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.0309999999999999,"15min":0.91000000000000003},"min":{"1min":0.25700000000000001,"5min":0.253,"15min":0.253},"max":{"1min":3.2770000000000001,"5min":7.468,"15min":7.468},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.874,"5min":1.0700000000000001,"15min":0.98499999999999999},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.252},"max":{"1min":2.718,"5min":6.0469999999999997,"15min":8.1150000000000002},"last":0.44900000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:06:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.3580000000000001,"15min":1.2669999999999999},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":2.5659999999999998,"5min":7.8609999999999998,"15min":55.683999999999997},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.458,"5min":1.3899999999999999,"15min":1.302},"min":{"1min":0.41799999999999998,"5min":0.30099999999999999,"15min":0.29799999999999999},"max":{"1min":4.3010000000000002,"5min":7.7670000000000003,"15min":55.533999999999999},"last":3.3679999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.254,"15min":0.92900000000000005},"min":{"1min":0.26600000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.2719999999999998,"5min":11.342000000000001,"15min":11.342000000000001},"last":3.3279999999999998},{"interface":"front","average":{"1min":0.502,"5min":1.278,"15min":0.98599999999999999},"min":{"1min":0.29699999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.913,"5min":12.651,"15min":16.879000000000001},"last":0.27500000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.131,"15min":1.0329999999999999},"min":{"1min":0.27800000000000002,"5min":0.252,"15min":0.23100000000000001},"max":{"1min":2.2170000000000001,"5min":6.4320000000000004,"15min":6.9000000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":1.163,"15min":1.038},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.26000000000000001},"max":{"1min":2.3170000000000002,"5min":6.2809999999999997,"15min":6.8529999999999998},"last":3.2509999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":1.367,"15min":1.0449999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6320000000000001,"5min":8.952,"15min":8.952},"last":3.3069999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.2989999999999999,"15min":1.008},"min":{"1min":0.34000000000000002,"5min":0.29599999999999999,"15min":0.28299999999999997},"max":{"1min":1.4850000000000001,"5min":7.0629999999999997,"15min":12.951000000000001},"last":2.121}]}]},{"osd":6,"up_from":46,"seq":197568496025,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4346612,"kb_used_data":3449388,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":89398540,"statfs":{"total":95995035648,"available":91544104960,"internally_reserved":0,"allocated":3532173312,"data_stored":3530928700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":2.0670000000000002,"15min":1.4199999999999999},"min":{"1min":0.253,"5min":0.23400000000000001,"15min":0.23200000000000001},"max":{"1min":3.8919999999999999,"5min":45.579999999999998,"15min":45.579999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.25,"5min":1.819,"15min":1.2110000000000001},"min":{"1min":0.27400000000000002,"5min":0.19,"15min":0.19},"max":{"1min":5.3460000000000001,"5min":44.918999999999997,"15min":44.918999999999997},"last":0.33600000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:05:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.075,"15min":1.1299999999999999},"min":{"1min":0.34100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":3.3820000000000001,"5min":4.6260000000000003,"15min":6.6680000000000001},"last":5.7220000000000004},{"interface":"front","average":{"1min":1.077,"5min":1.091,"15min":1.1930000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.247},"max":{"1min":7.0570000000000004,"5min":7.0570000000000004,"15min":9.5510000000000002},"last":3.907}]},{"osd":2,"last update":"Fri Jun 14 23:05:50 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.2010000000000001,"15min":0.91400000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.27800000000000002},"max":{"1min":2.0270000000000001,"5min":9.9600000000000009,"15min":9.9600000000000009},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":1.359,"15min":1.0089999999999999},"min":{"1min":0.35799999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.0569999999999999,"5min":10.823,"15min":10.823},"last":7.0570000000000004}]},{"osd":3,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3120000000000001,"15min":1.179},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":4.3440000000000003,"5min":5.0949999999999998,"15min":29.152000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.23,"15min":1.175},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":2.8519999999999999,"5min":7.3799999999999999,"15min":25.864000000000001},"last":0.442}]},{"osd":4,"last update":"Fri Jun 14 23:05:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":1.1040000000000001,"15min":0.89200000000000002},"min":{"1min":0.28899999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9950000000000001,"5min":7.7119999999999997,"15min":7.7709999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":1.149,"15min":0.84499999999999997},"min":{"1min":0.31900000000000001,"5min":0.28000000000000003,"15min":0.24299999999999999},"max":{"1min":0.87,"5min":4.6790000000000003,"15min":9.9610000000000003},"last":0.38800000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:05:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":1.1579999999999999,"15min":0.96299999999999997},"min":{"1min":0.314,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.7989999999999999,"5min":7.3540000000000001,"15min":8.6500000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.0249999999999999,"15min":0.98699999999999999},"min":{"1min":0.32500000000000001,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":1.95,"5min":4.548,"15min":8.6349999999999998},"last":0.374}]},{"osd":7,"last update":"Fri Jun 14 23:06:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.998,"15min":0.96699999999999997},"min":{"1min":0.37,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.9060000000000001,"5min":4.4720000000000004,"15min":8.6950000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":1.0469999999999999,"15min":1.012},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.28899999999999998},"max":{"1min":1.9079999999999999,"5min":4.1470000000000002,"15min":8.673},"last":0.41799999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757664,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2623600,"kb_used_data":1736552,"kb_used_omap":1,"kb_used_meta":887038,"kb_avail":91121552,"statfs":{"total":95995035648,"available":93308469248,"internally_reserved":0,"allocated":1778229248,"data_stored":1776985090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908327256},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":2.028,"15min":1.361},"min":{"1min":0.30399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":1.4179999999999999,"5min":14.94,"15min":14.94},"last":0.223},{"interface":"front","average":{"1min":0.54900000000000004,"5min":1.829,"15min":1.28},"min":{"1min":0.22800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":1.4650000000000001,"5min":9.3819999999999997,"15min":9.3819999999999997},"last":0.83399999999999996}]},{"osd":1,"last update":"Fri Jun 14 23:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.84599999999999997,"15min":1.0229999999999999},"min":{"1min":0.39000000000000001,"5min":0.23999999999999999,"15min":0.23300000000000001},"max":{"1min":2.0310000000000001,"5min":7.4370000000000003,"15min":7.4370000000000003},"last":3.9390000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.3680000000000001,"15min":1.2350000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.247},"max":{"1min":2.4390000000000001,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.37}]},{"osd":2,"last update":"Fri Jun 14 23:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.6200000000000001,"15min":1.0800000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.569,"5min":14.891,"15min":14.891},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":1.728,"15min":1.0980000000000001},"min":{"1min":0.35399999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.762,"5min":9.532,"15min":9.532},"last":0.441}]},{"osd":3,"last update":"Fri Jun 14 23:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.4370000000000001,"15min":1.3400000000000001},"min":{"1min":0.26500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":3.4089999999999998,"5min":9.2149999999999999,"15min":45.744},"last":3.9169999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.579,"15min":1.4450000000000001},"min":{"1min":0.36099999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.27,"5min":14.914,"15min":45.838000000000001},"last":3.883}]},{"osd":4,"last update":"Fri Jun 14 23:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.97699999999999998,"15min":0.81000000000000005},"min":{"1min":0.27200000000000002,"5min":0.26600000000000001,"15min":0.24399999999999999},"max":{"1min":1.887,"5min":14.352,"15min":14.352},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.466,"15min":1.0009999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":4.6660000000000004,"5min":9.5980000000000008,"15min":9.5980000000000008},"last":0.88600000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":1.5289999999999999,"15min":1.0780000000000001},"min":{"1min":0.379,"5min":0.28100000000000003,"15min":0.253},"max":{"1min":4.7800000000000002,"5min":15.382999999999999,"15min":15.382999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.516,"15min":1.0489999999999999},"min":{"1min":0.40400000000000003,"5min":0.29599999999999999,"15min":0.25},"max":{"1min":2.1699999999999999,"5min":14.855,"15min":14.855},"last":0.42299999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.4179999999999999,"15min":1.073},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30199999999999999},"max":{"1min":2.101,"5min":14.818,"15min":14.818},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.141,"15min":0.97299999999999998},"min":{"1min":0.376,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":2.1539999999999999,"5min":14.429,"15min":14.429},"last":4.0910000000000002}]}]},{"osd":4,"up_from":30,"seq":128849019300,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2606652,"kb_used_data":1715956,"kb_used_omap":1,"kb_used_meta":890686,"kb_avail":91138500,"statfs":{"total":95995035648,"available":93325824000,"internally_reserved":0,"allocated":1757138944,"data_stored":1755886274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912062808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.595,"15min":1.1559999999999999},"min":{"1min":0.27600000000000002,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":1.901,"5min":10.567,"15min":10.567},"last":2.0409999999999999},{"interface":"front","average":{"1min":0.505,"5min":1.4610000000000001,"15min":1.179},"min":{"1min":0.254,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.67300000000000004,"5min":6.7519999999999998,"15min":8.3390000000000004},"last":0.27400000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:05:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.778,"15min":1.365},"min":{"1min":0.27600000000000002,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":2.004,"5min":56.917000000000002,"15min":56.917000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.749,"5min":1.946,"15min":1.4390000000000001},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.249},"max":{"1min":2.2000000000000002,"5min":56.789999999999999,"15min":56.789999999999999},"last":3.1150000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:05:50 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.3420000000000001,"15min":0.90700000000000003},"min":{"1min":0.315,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.5550000000000002,"5min":8.4860000000000007,"15min":8.9250000000000007},"last":1.752},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.458,"15min":0.98199999999999998},"min":{"1min":0.29099999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.5390000000000001,"5min":10.478,"15min":10.478},"last":1.7649999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":1.157,"15min":0.97199999999999998},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.85299999999999998,"5min":6.327,"15min":6.617},"last":2.9169999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":1.452,"15min":1.0920000000000001},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27500000000000002},"max":{"1min":0.80300000000000005,"5min":10.510999999999999,"15min":10.510999999999999},"last":0.29099999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.3,"15min":1.5309999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":2.0339999999999998,"5min":12.315,"15min":139.13200000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.252,"15min":1.548},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26100000000000001},"max":{"1min":1.659,"5min":10.959,"15min":138.86000000000001},"last":0.80200000000000005}]},{"osd":6,"last update":"Fri Jun 14 23:05:38 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.1859999999999999,"15min":0.89400000000000002},"min":{"1min":0.36299999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.5640000000000001,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.621,"5min":1.1859999999999999,"15min":0.84299999999999997},"min":{"1min":0.34999999999999998,"5min":0.27900000000000003,"15min":0.254},"max":{"1min":1.4590000000000001,"5min":8,"15min":8.8819999999999997},"last":0.59999999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.0940000000000001,"15min":0.91500000000000004},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.851,"5min":6.0890000000000004,"15min":8.9489999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.0900000000000001,"15min":1.024},"min":{"1min":0.37,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.7629999999999999,"5min":5.3479999999999999,"15min":8.4730000000000008},"last":0.36499999999999999}]}]},{"osd":3,"up_from":138,"seq":592705486999,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3437168,"kb_used_data":3010344,"kb_used_omap":4,"kb_used_meta":426811,"kb_avail":90307984,"statfs":{"total":95995035648,"available":92475375616,"internally_reserved":0,"allocated":3082592256,"data_stored":3081347871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":437054942},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.3740000000000001,"15min":1.0209999999999999},"min":{"1min":0.222,"5min":0.187,"15min":0.184},"max":{"1min":3.8620000000000001,"5min":11.071999999999999,"15min":11.071999999999999},"last":0.214},{"interface":"front","average":{"1min":1.103,"5min":1.4710000000000001,"15min":0.97099999999999997},"min":{"1min":0.19800000000000001,"5min":0.19,"15min":0.17999999999999999},"max":{"1min":6.6109999999999998,"5min":8.2330000000000005,"15min":8.2330000000000005},"last":0.56299999999999994}]},{"osd":1,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.327,"15min":1.085},"min":{"1min":0.25700000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.5499999999999998,"5min":9.6080000000000005,"15min":24.097999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.587,"5min":1.3999999999999999,"15min":1.133},"min":{"1min":0.25800000000000001,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":6.6539999999999999,"5min":6.6539999999999999,"15min":24.132999999999999},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.5960000000000001,"15min":1.0469999999999999},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.23599999999999999},"max":{"1min":6.7430000000000003,"5min":9.4730000000000008,"15min":9.4730000000000008},"last":0.624},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4419999999999999,"15min":0.97399999999999998},"min":{"1min":0.27400000000000002,"5min":0.23799999999999999,"15min":0.20899999999999999},"max":{"1min":4.6900000000000004,"5min":10.369,"15min":10.369},"last":0.36299999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.4850000000000001,"15min":1.105},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.27900000000000003},"max":{"1min":4.7249999999999996,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":1.208},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.262,"15min":1.087},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.4620000000000002,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":5.9800000000000004,"5min":2.1949999999999998,"15min":1.4379999999999999},"min":{"1min":0.33200000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":81.438999999999993,"5min":81.438999999999993,"15min":81.438999999999993},"last":3.1179999999999999},{"interface":"front","average":{"1min":6.5289999999999999,"5min":2.4140000000000001,"15min":1.4670000000000001},"min":{"1min":0.38,"5min":0.32400000000000001,"15min":0.30199999999999999},"max":{"1min":81.626999999999995,"5min":81.626999999999995,"15min":81.626999999999995},"last":3.496}]},{"osd":6,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":6.7469999999999999,"5min":2.5579999999999998,"15min":1.4750000000000001},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.32400000000000001},"max":{"1min":82.248999999999995,"5min":82.248999999999995,"15min":82.248999999999995},"last":0.59599999999999997},{"interface":"front","average":{"1min":6.5620000000000003,"5min":2.71,"15min":1.528},"min":{"1min":0.44900000000000001,"5min":0.31,"15min":0.31},"max":{"1min":81.585999999999999,"5min":81.585999999999999,"15min":81.585999999999999},"last":1.022}]},{"osd":7,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.373,"15min":1.2010000000000001},"min":{"1min":0.36499999999999999,"5min":0.35599999999999998,"15min":0.35299999999999998},"max":{"1min":4.7809999999999997,"5min":5.5300000000000002,"15min":7.7240000000000002},"last":1.7150000000000001},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.085,"15min":1.0860000000000001},"min":{"1min":0.42299999999999999,"5min":0.36299999999999999,"15min":0.32400000000000001},"max":{"1min":3.5169999999999999,"5min":5.2160000000000002,"15min":5.8300000000000001},"last":0.42299999999999999}]}]},{"osd":2,"up_from":133,"seq":571230650526,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2932464,"kb_used_data":2605608,"kb_used_omap":6,"kb_used_meta":326841,"kb_avail":90812688,"statfs":{"total":95995035648,"available":92992192512,"internally_reserved":0,"allocated":2668142592,"data_stored":2666898421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":334685685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:05:55 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":1.581,"15min":0.90900000000000003},"min":{"1min":0.183,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":55.851999999999997,"5min":55.851999999999997,"15min":55.851999999999997},"last":0.36299999999999999},{"interface":"front","average":{"1min":3.1890000000000001,"5min":1.998,"15min":1.1020000000000001},"min":{"1min":0.23699999999999999,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":56.171999999999997,"5min":56.171999999999997,"15min":56.171999999999997},"last":0.53500000000000003}]},{"osd":1,"last update":"Fri Jun 14 23:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.94699999999999995,"15min":0.90700000000000003},"min":{"1min":0.24399999999999999,"5min":0.221,"15min":0.20799999999999999},"max":{"1min":3.0859999999999999,"5min":9.1349999999999998,"15min":21.920000000000002},"last":1.4099999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.89300000000000002,"15min":0.872},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.9790000000000001,"5min":5.7439999999999998,"15min":21.573},"last":1.194}]},{"osd":3,"last update":"Fri Jun 14 23:05:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":1.357,"15min":1.0529999999999999},"min":{"1min":0.27900000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.76600000000000001,"5min":11.24,"15min":21.66},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":1.393,"15min":1.0700000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":2.7029999999999998,"5min":11.271000000000001,"15min":22.341000000000001},"last":1.1319999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":2.2029999999999998,"15min":1.2889999999999999},"min":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.318},"max":{"1min":1.867,"5min":31.928000000000001,"15min":31.928000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.056,"5min":1.9219999999999999,"15min":1.181},"min":{"1min":0.32400000000000001,"5min":0.308,"15min":0.28899999999999998},"max":{"1min":5.7229999999999999,"5min":31.954999999999998,"15min":31.954999999999998},"last":0.59999999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:05:55 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.4199999999999999,"15min":1.0289999999999999},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.1139999999999999,"5min":31.539999999999999,"15min":31.539999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.749,"5min":1.7829999999999999,"15min":1.1299999999999999},"min":{"1min":0.33900000000000002,"5min":0.32500000000000001,"15min":0.29399999999999998},"max":{"1min":2.9279999999999999,"5min":31.481999999999999,"15min":31.481999999999999},"last":1.163}]},{"osd":6,"last update":"Fri Jun 14 23:05:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.9039999999999999,"15min":1.1180000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.7489999999999997,"5min":32.005000000000003,"15min":32.005000000000003},"last":1.387},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.7529999999999999,"15min":1.1020000000000001},"min":{"1min":0.34999999999999998,"5min":0.32800000000000001,"15min":0.30399999999999999},"max":{"1min":3.1469999999999998,"5min":32.079999999999998,"15min":32.079999999999998},"last":0.41199999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:05:55 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.2569999999999999,"15min":1.0329999999999999},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":5.6840000000000002,"5min":10.614000000000001,"15min":10.614000000000001},"last":0.624},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.0069999999999999,"15min":0.86399999999999999},"min":{"1min":0.378,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.8079999999999998,"5min":5.5119999999999996,"15min":7.6260000000000003},"last":0.433}]}]},{"osd":1,"up_from":116,"seq":498216206543,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4036776,"kb_used_data":3441248,"kb_used_omap":5,"kb_used_meta":595514,"kb_avail":89708376,"statfs":{"total":95995035648,"available":91861377024,"internally_reserved":0,"allocated":3523837952,"data_stored":3522601125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":609806616},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:05:58 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.3620000000000001,"15min":1.375},"min":{"1min":0.20000000000000001,"5min":0.182,"15min":0.17499999999999999},"max":{"1min":4.1109999999999998,"5min":9.5350000000000001,"15min":11.257999999999999},"last":9.077},{"interface":"front","average":{"1min":1.377,"5min":1.577,"15min":1.2589999999999999},"min":{"1min":0.20000000000000001,"5min":0.19700000000000001,"15min":0.186},"max":{"1min":5.3390000000000004,"5min":13.672000000000001,"15min":13.672000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:05:58 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.54,"15min":1.1160000000000001},"min":{"1min":0.253,"5min":0.21299999999999999,"15min":0.19500000000000001},"max":{"1min":4.1369999999999996,"5min":9.5470000000000006,"15min":9.5470000000000006},"last":4.6299999999999999},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.3240000000000001,"15min":0.88100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":5.4610000000000003,"5min":8.0060000000000002,"15min":8.0060000000000002},"last":4.0410000000000004}]},{"osd":3,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.873,"5min":1.2669999999999999,"15min":1.1699999999999999},"min":{"1min":0.23599999999999999,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":7.5679999999999996,"5min":7.5679999999999996,"15min":21.422000000000001},"last":4.6790000000000003},{"interface":"front","average":{"1min":1.796,"5min":1.369,"15min":1.2490000000000001},"min":{"1min":0.36299999999999999,"5min":0.24099999999999999,"15min":0.214},"max":{"1min":4.4729999999999999,"5min":8.1219999999999999,"15min":21.016999999999999},"last":5.8109999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:05:58 2024","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.28,"15min":1.264},"min":{"1min":0.375,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.5579999999999998,"5min":6.5350000000000001,"15min":8.8599999999999994},"last":1.7230000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.2829999999999999,"15min":1.4199999999999999},"min":{"1min":0.40899999999999997,"5min":0.29899999999999999,"15min":0.29199999999999998},"max":{"1min":3.867,"5min":6.2960000000000003,"15min":11.161},"last":2.4470000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:05:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0960000000000001,"5min":1.4390000000000001,"15min":1.4470000000000001},"min":{"1min":0.35799999999999998,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":19.132999999999999,"5min":19.132999999999999,"15min":19.132999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":2.1739999999999999,"5min":1.393,"15min":1.3069999999999999},"min":{"1min":0.40100000000000002,"5min":0.29399999999999998,"15min":0.28599999999999998},"max":{"1min":19.123000000000001,"5min":19.123000000000001,"15min":19.123000000000001},"last":1.7589999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:05:58 2024","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":2.2349999999999999,"15min":1.669},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":24.856000000000002,"5min":63.795999999999999,"15min":63.795999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":2.548,"5min":2.1760000000000002,"15min":1.571},"min":{"1min":0.39900000000000002,"5min":0.32900000000000001,"15min":0.28499999999999998},"max":{"1min":24.904,"5min":63.811,"15min":63.811},"last":1.7969999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:05:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.1799999999999999,"15min":1.401},"min":{"1min":0.47099999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.8940000000000001,"5min":8.0440000000000005,"15min":9.4429999999999996},"last":2.5},{"interface":"front","average":{"1min":1.238,"5min":1.0740000000000001,"15min":1.266},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":5.3970000000000002,"5min":5.9580000000000002,"15min":9.5449999999999999},"last":0.71799999999999997}]}]},{"osd":0,"up_from":91,"seq":390842024197,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2915240,"kb_used_data":2121568,"kb_used_omap":5,"kb_used_meta":793658,"kb_avail":90829912,"statfs":{"total":95995035648,"available":93009829888,"internally_reserved":0,"allocated":2172485632,"data_stored":2171246804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":812705882},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.054,"15min":1.373},"min":{"1min":0.192,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.4460000000000002,"5min":9.4380000000000006,"15min":16.305},"last":0.186},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.266,"15min":1.4510000000000001},"min":{"1min":0.20699999999999999,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":5.1180000000000003,"5min":9.9990000000000006,"15min":16.184999999999999},"last":1.002}]},{"osd":2,"last update":"Fri Jun 14 23:06:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.629,"15min":1.0580000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":6.8499999999999996,"5min":10.053000000000001,"15min":16.757000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.488,"15min":1.095},"min":{"1min":0.223,"5min":0.223,"15min":0.222},"max":{"1min":6.8129999999999997,"5min":9.2769999999999992,"15min":16.055},"last":0.94499999999999995}]},{"osd":3,"last update":"Fri Jun 14 23:05:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":1.409,"15min":1.077},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.1429999999999998,"5min":11.083,"15min":16.873999999999999},"last":1.026},{"interface":"front","average":{"1min":0.41099999999999998,"5min":1.357,"15min":1.0509999999999999},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.224},"max":{"1min":1.214,"5min":11.375,"15min":16.085999999999999},"last":0.96899999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:05:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":1.6859999999999999,"15min":1.405},"min":{"1min":0.33200000000000002,"5min":0.30199999999999999,"15min":0.27600000000000002},"max":{"1min":1.153,"5min":16.527000000000001,"15min":16.527000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":1.504,"15min":1.3069999999999999},"min":{"1min":0.32500000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.8260000000000001,"5min":14.475,"15min":15.976000000000001},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:05:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.607,"15min":1.508},"min":{"1min":0.375,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.1619999999999999,"5min":10.462,"15min":16.108000000000001},"last":1.073},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.6060000000000001,"15min":1.4630000000000001},"min":{"1min":0.33700000000000002,"5min":0.32500000000000001,"15min":0.31900000000000001},"max":{"1min":2.5169999999999999,"5min":9.2279999999999998,"15min":16.006},"last":0.69599999999999995}]},{"osd":6,"last update":"Fri Jun 14 23:05:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.641,"15min":1.4259999999999999},"min":{"1min":0.39900000000000002,"5min":0.38100000000000001,"15min":0.30599999999999999},"max":{"1min":6.1669999999999998,"5min":14.247,"15min":16.152000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.861,"15min":1.4490000000000001},"min":{"1min":0.38800000000000001,"5min":0.36499999999999999,"15min":0.29399999999999998},"max":{"1min":2.8980000000000001,"5min":14.523999999999999,"15min":16.227},"last":0.752}]},{"osd":7,"last update":"Fri Jun 14 23:05:33 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.4810000000000001,"15min":1.373},"min":{"1min":0.35899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.3639999999999999,"5min":9.2579999999999991,"15min":16.199999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.629,"5min":1.482,"15min":1.4279999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.314},"max":{"1min":1.1919999999999999,"5min":11.195,"15min":16.259},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2170028032,"data_stored":2170028032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3538026496,"data_stored":3538026496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2671640576,"data_stored":2671640576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3101949952,"data_stored":3101949952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1759182848,"data_stored":1759182848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1784545280,"data_stored":1784545280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3545104384,"data_stored":3545104384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2625110016,"data_stored":2625110016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:06:36.566 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-14T23:06:36.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-06-14T23:06:36.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:36.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:36.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:36.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:36.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:36.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:36.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:36.901 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:36.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:36.930 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:06:36.930 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:06:36.930 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:06:36.931 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:06:36.932 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:06:36.932 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:06:37.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:37.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:37.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:37.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:37.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:37.166 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:37.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:37.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:37.286 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 16 115231 115215 93.4947 59.8125 0.0267426 0.0106909 2024-06-14T23:06:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:37.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:37.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:37.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:37.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:37.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:37.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:37.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:37.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:37.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:37.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:37.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:37.844 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:37.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:37.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:37.997 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:37.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:38.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:37 smithi086 ceph-mon[97913]: pgmap v988: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s 2024-06-14T23:06:38.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:37 smithi086 ceph-mon[102754]: pgmap v988: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s 2024-06-14T23:06:38.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:38.133 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:38.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:38.147 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:38.194 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:06:38.194 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:06:38.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:38.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:37 smithi137 ceph-mon[90634]: pgmap v988: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s 2024-06-14T23:06:38.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:38.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:38.286 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 116145 116129 93.0282 57.125 0.0116208 0.0107448 2024-06-14T23:06:38.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:38.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:38.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:38.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:38.467 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:38.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:38.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:38.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:38.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:38.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:38.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:38.827 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:38.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:38.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:38.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:39.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:39.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:39.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:39.224 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:39.254 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:39.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:39.286 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 117103 117087 92.6086 59.875 0.026743 0.0107929 2024-06-14T23:06:39.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:39.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:39.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:39.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:39.525 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:39.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:39.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:39.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:39.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:39.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:39.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:39.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:06:39.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:39.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:39.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:39.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:39.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:40.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:40.076 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:40.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:40.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:40.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:40.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:40.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:40.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:39 smithi137 ceph-mon[90634]: pgmap v989: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s 2024-06-14T23:06:40.258 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:39 smithi086 ceph-mon[97913]: pgmap v989: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s 2024-06-14T23:06:40.258 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:39 smithi086 ceph-mon[102754]: pgmap v989: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s 2024-06-14T23:06:40.286 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:06:40.286331+0000 min lat: 0.00101502 max lat: 1.03707 avg lat: 0.0108406 2024-06-14T23:06:40.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:06:40.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 118061 118045 92.1994 59.875 0.0182455 0.0108406 2024-06-14T23:06:40.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-14T23:06:40.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:40.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:40.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:40.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:40.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:40.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:40.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:40.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:40.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:40.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:40.783 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:40.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:40.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:40.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:40.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:41.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:41.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:41.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:41.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:41.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:41.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:41.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 118961 118945 91.7555 56.25 0.0201189 0.0108939 2024-06-14T23:06:41.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:41.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:41.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:41.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:41.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:41.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:41.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:41.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:41.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:41.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:41.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:41.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:41.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:41.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:41.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:41.911 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:41.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:41 smithi086 ceph-mon[97913]: pgmap v990: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-14T23:06:41.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:42.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:42.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:42.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:42.160 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:42.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:42.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:41 smithi137 ceph-mon[90634]: pgmap v990: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-14T23:06:42.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:42.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 119863 119847 91.324 56.375 0.019941 0.0109431 2024-06-14T23:06:42.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:42.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:42.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:41 smithi086 ceph-mon[102754]: pgmap v990: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-14T23:06:42.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:42.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:42.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:42.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:42.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:42.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:42.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:42.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:42.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:42.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:42.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:42.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:43.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:43.013 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:43.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:43.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:43.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:43.159 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:43.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:43.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:43.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:43.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 16 120795 120779 90.9255 58.25 0.0145332 0.0109926 2024-06-14T23:06:43.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:43.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:43.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:43.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:43.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:43.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:43.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:43.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:43.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:43.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:43.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:43.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:43.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:43.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:43.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:43.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:43.977 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:06:43.984 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:43.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:44.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:44.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:44.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:44.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:43 smithi137 ceph-mon[90634]: pgmap v991: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-06-14T23:06:44.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/947605592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:06:44.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:43 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:06:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:44.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:44.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:44.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 121674 121658 90.4971 54.9375 0.013246 0.0110452 2024-06-14T23:06:44.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:43 smithi086 ceph-mon[102754]: pgmap v991: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-06-14T23:06:44.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/947605592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:06:44.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:43 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:06:44.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:43 smithi086 ceph-mon[97913]: pgmap v991: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-06-14T23:06:44.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/947605592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:06:44.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:43 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:06:44.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:44.407 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:44.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:44.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:44.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:44.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:44.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:44.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:44.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:44.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:44.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:44.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:44.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:44.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:44.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:45.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:45.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:45.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:45.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:45.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:45.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:45.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:44 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:45.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:44 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:06:45.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:44 smithi137 ceph-mon[90634]: osdmap e256: 8 total, 8 up, 8 in 2024-06-14T23:06:45.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 122676 122660 90.1692 62.625 0.0158377 0.0110854 2024-06-14T23:06:45.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:45.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:44 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:45.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:44 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:06:45.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:44 smithi086 ceph-mon[97913]: osdmap e256: 8 total, 8 up, 8 in 2024-06-14T23:06:45.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:44 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:45.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:44 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:06:45.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:44 smithi086 ceph-mon[102754]: osdmap e256: 8 total, 8 up, 8 in 2024-06-14T23:06:45.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:06:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:06:44] "GET /metrics HTTP/1.1" 200 36603 "" "Prometheus/2.43.0" 2024-06-14T23:06:45.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:45.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:45.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:45.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:45.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:45.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:45.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:45.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:45.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:45.763 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:45.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:45.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:45.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:45.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:45.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:46.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:46.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:46.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:46.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:46.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:46.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:45 smithi137 ceph-mon[90634]: pgmap v993: 99 pgs: 99 active+clean; 7.4 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 59 MiB/s wr, 940 op/s 2024-06-14T23:06:46.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:46.266 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:45 smithi086 ceph-mon[97913]: pgmap v993: 99 pgs: 99 active+clean; 7.4 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 59 MiB/s wr, 940 op/s 2024-06-14T23:06:46.266 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:45 smithi086 ceph-mon[102754]: pgmap v993: 99 pgs: 99 active+clean; 7.4 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 59 MiB/s wr, 940 op/s 2024-06-14T23:06:46.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 123641 123625 89.8221 60.3125 0.0155562 0.0111277 2024-06-14T23:06:46.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:46.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:46.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:46.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:46.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:46.498 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:46.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:46.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:46.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:46.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:46.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:46.709 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:46.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:46.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:46.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:46.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:46.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:46.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:46.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:47.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:47.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:47.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:47.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:47.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:47.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:47.275 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:06:47.275 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:06:47.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 124568 124552 89.4556 57.9375 0.0127431 0.0111737 2024-06-14T23:06:47.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:47.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:47.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:47.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:47.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:47.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:47.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:47.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:47.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:47.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:47.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:47.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:47.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:47.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:48.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:48.079 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:48.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:48.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:47 smithi086 ceph-mon[97913]: pgmap v994: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s 2024-06-14T23:06:48.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:47 smithi086 ceph-mon[102754]: pgmap v994: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s 2024-06-14T23:06:48.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:48.210 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:48.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:48.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:47 smithi137 ceph-mon[90634]: pgmap v994: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s 2024-06-14T23:06:48.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 125550 125534 89.1365 61.375 0.00709498 0.0112133 2024-06-14T23:06:48.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:48.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:48.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:48.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:48.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:48.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:48.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:48.633 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:48.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:48.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-14T23:06:48.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:48.766 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:48.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:48.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:48.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:48.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:49.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:49.035 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:06:49.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:49.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:49.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:49.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:49.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:49.287 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 126435 126419 88.7564 55.3125 0.0216102 0.0112617 2024-06-14T23:06:49.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:49.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:49.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:49.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:49.472 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:49.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:49.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:49.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:49.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:49.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:49.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:49.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:49.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:06:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:49.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:49.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:50.004 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 1 pg_num to 21 2024-06-14T23:06:50.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:50.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:50.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:50.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:50.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:50.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:49 smithi137 ceph-mon[90634]: pgmap v995: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s 2024-06-14T23:06:50.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/454800057' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-14T23:06:50.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:50.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:50.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:50.288 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0135 2024-06-14T23:06:50.288 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 127318 2024-06-14T23:06:50.288 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T23:06:50.288 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T23:06:50.289 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 88.4021 2024-06-14T23:06:50.289 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 41.5631 2024-06-14T23:06:50.289 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 243.438 2024-06-14T23:06:50.289 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 18.125 2024-06-14T23:06:50.289 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1414 2024-06-14T23:06:50.290 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 665.009 2024-06-14T23:06:50.290 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 3895 2024-06-14T23:06:50.290 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 290 2024-06-14T23:06:50.290 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0113079 2024-06-14T23:06:50.290 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.0171402 2024-06-14T23:06:50.290 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 1.03707 2024-06-14T23:06:50.291 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00101502 2024-06-14T23:06:50.293 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T23:06:50.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:49 smithi086 ceph-mon[102754]: pgmap v995: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s 2024-06-14T23:06:50.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/454800057' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-14T23:06:50.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:49 smithi086 ceph-mon[97913]: pgmap v995: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s 2024-06-14T23:06:50.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/454800057' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-14T23:06:50.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:50.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:50.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:50.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:50.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:50.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:50.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:50.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:50.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:50.788 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:50.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:50.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:50.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:50.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:50.955 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:50.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:51.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:51.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:51.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:51.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:51.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:51.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:51.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/454800057' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-06-14T23:06:51.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:51 smithi137 ceph-mon[90634]: osdmap e257: 8 total, 8 up, 8 in 2024-06-14T23:06:51.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-14T23:06:51.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/454800057' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-06-14T23:06:51.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:51 smithi086 ceph-mon[97913]: osdmap e257: 8 total, 8 up, 8 in 2024-06-14T23:06:51.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-14T23:06:51.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/454800057' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-06-14T23:06:51.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:51 smithi086 ceph-mon[102754]: osdmap e257: 8 total, 8 up, 8 in 2024-06-14T23:06:51.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-14T23:06:51.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:51.365 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:51.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:51.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:51.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:51.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:51.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:51.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:51.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:51.660 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:06:51.660 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:06:51.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:51.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:51.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:51.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:51.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:51.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:51.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:52.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:52.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:52.160 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:52.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:52.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:52.300 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:52.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:52.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:52 smithi086 ceph-mon[97913]: pgmap v997: 99 pgs: 99 active+clean; 7.7 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 127 B/s rd, 72 MiB/s wr, 1.17k op/s 2024-06-14T23:06:52.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:52 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-06-14T23:06:52.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:52 smithi086 ceph-mon[97913]: osdmap e258: 8 total, 8 up, 8 in 2024-06-14T23:06:52.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:52 smithi086 ceph-mon[102754]: pgmap v997: 99 pgs: 99 active+clean; 7.7 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 127 B/s rd, 72 MiB/s wr, 1.17k op/s 2024-06-14T23:06:52.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:52 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-06-14T23:06:52.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:52 smithi086 ceph-mon[102754]: osdmap e258: 8 total, 8 up, 8 in 2024-06-14T23:06:52.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:52.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:52.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:52.474 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:52 smithi137 ceph-mon[90634]: pgmap v997: 99 pgs: 99 active+clean; 7.7 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 127 B/s rd, 72 MiB/s wr, 1.17k op/s 2024-06-14T23:06:52.474 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:52 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-06-14T23:06:52.475 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:52 smithi137 ceph-mon[90634]: osdmap e258: 8 total, 8 up, 8 in 2024-06-14T23:06:52.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-14T23:06:52.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:52.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:52.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:52.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:52.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:52.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:52.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:52.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:52.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:52.908 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:06:52.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:53.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:53.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:53.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:53.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:53.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:53.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:53.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:53.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:53 smithi086 ceph-mon[97913]: osdmap e259: 8 total, 8 up, 8 in 2024-06-14T23:06:53.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:53 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:53.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:53 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:53.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:53 smithi086 ceph-mon[97913]: pgmap v1000: 109 pgs: 10 unknown, 99 active+clean; 7.7 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 170 B/s rd, 49 MiB/s wr, 793 op/s 2024-06-14T23:06:53.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:53 smithi086 ceph-mon[102754]: osdmap e259: 8 total, 8 up, 8 in 2024-06-14T23:06:53.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:53 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:53.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:53 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:53.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:53 smithi086 ceph-mon[102754]: pgmap v1000: 109 pgs: 10 unknown, 99 active+clean; 7.7 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 170 B/s rd, 49 MiB/s wr, 793 op/s 2024-06-14T23:06:53.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:53.393 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:53 smithi137 ceph-mon[90634]: osdmap e259: 8 total, 8 up, 8 in 2024-06-14T23:06:53.393 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:53 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:53.393 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:53 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:53.394 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:53 smithi137 ceph-mon[90634]: pgmap v1000: 109 pgs: 10 unknown, 99 active+clean; 7.7 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 170 B/s rd, 49 MiB/s wr, 793 op/s 2024-06-14T23:06:53.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:53.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:53.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:53.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:53.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:53.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:53.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:53.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:53.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:53.809 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:53.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:53.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:53.916 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:53.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:53.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:54.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:54.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:54.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:54.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:54.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:54.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:54.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:54.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:54.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:54.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:54.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:54.522 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:54.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:54.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:54 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:06:54.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:54 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:06:54.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:54 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:54.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:54 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:06:54.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:54 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:06:54.608 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:54 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:54.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:54.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:54.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:54.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:54.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:54.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:54.809 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:06:54.809 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:06:54.828 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:54 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:06:54.828 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:54 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:06:54.828 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:54 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:54.846 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:06:54.847 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:06:54.866 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:06:54.866 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:06:54.889 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:06:54.889 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:06:54.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:54.925 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:06:54.926 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:06:54.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:54.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:54.960 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:06:54.960 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:06:55.001 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:06:55.002 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:06:55.038 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:06:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:06:55.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:55.069 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:06:55.070 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:06:55.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:55.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:55.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:55.213 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:55.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:55.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:55.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:06:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:06:54] "GET /metrics HTTP/1.1" 200 36603 "" "Prometheus/2.43.0" 2024-06-14T23:06:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:55.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:55.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:55.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:55.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:55.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:55.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:55.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:55.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-14T23:06:55.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:55.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:55.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:55.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:55.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:55.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:55 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2584170928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:55.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:55 smithi086 ceph-mon[97913]: pgmap v1001: 109 pgs: 4 unknown, 105 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 170 B/s rd, 53 MiB/s wr, 1.20k op/s 2024-06-14T23:06:55.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:55 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2584170928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:55.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:55 smithi086 ceph-mon[102754]: pgmap v1001: 109 pgs: 4 unknown, 105 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 170 B/s rd, 53 MiB/s wr, 1.20k op/s 2024-06-14T23:06:55.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:55.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:55.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2584170928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:06:55.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:55 smithi137 ceph-mon[90634]: pgmap v1001: 109 pgs: 4 unknown, 105 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 170 B/s rd, 53 MiB/s wr, 1.20k op/s 2024-06-14T23:06:56.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:56.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:56.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:56.064 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:06:56.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:56.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:56.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:56.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:56.331 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:56.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:56.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:56.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:56.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:56.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:56.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:56.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:56.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:56.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:56.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:56.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:56.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:56.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:57.014 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:57.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:57.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:57.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:57.236 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-14T23:06:57.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-14T23:06:57.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-14T23:06:57.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:57.291 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:57.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:57.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:57.443 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:57.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:57.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:57.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:57.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:57.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:57.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:57.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:57.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:57.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:57.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:57.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:57.981 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:57.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:58.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:06:58.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:58.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:06:58.207 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:06:58.207 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:06:58.208 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-06-14T23:06:58.208 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-06-14T23:06:58.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.6 2024-06-14T23:06:58.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:06:58.231 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:57 smithi137 ceph-mon[90634]: pgmap v1002: 109 pgs: 109 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 149 B/s rd, 20 MiB/s wr, 975 op/s 2024-06-14T23:06:58.232 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-06-14T23:06:58.232 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:57 smithi137 ceph-mon[90634]: osdmap e260: 8 total, 8 up, 8 in 2024-06-14T23:06:58.232 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:58.232 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:58.241 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:58.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:06:58.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[102754]: pgmap v1002: 109 pgs: 109 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 149 B/s rd, 20 MiB/s wr, 975 op/s 2024-06-14T23:06:58.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-06-14T23:06:58.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[102754]: osdmap e260: 8 total, 8 up, 8 in 2024-06-14T23:06:58.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:58.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[97913]: pgmap v1002: 109 pgs: 109 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 149 B/s rd, 20 MiB/s wr, 975 op/s 2024-06-14T23:06:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-06-14T23:06:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[97913]: osdmap e260: 8 total, 8 up, 8 in 2024-06-14T23:06:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:06:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:06:58.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:06:58.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:58.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:06:58.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:58.495 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:06:58 smithi137 systemd[1]: Stopping Ceph osd.6 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:06:58.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:58.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:58.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:58.646 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:58.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:06:58.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:58.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:58.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:58.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:58.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:58.910 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:06:58.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:06:58.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:58.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:58.976 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:06:58.976 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:06:58.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:117233 10.e 10:733fe19a:::benchmark_data_smithi086_189745_object117232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.024480+0000", 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 305.94773743899998, 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.108642816, 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117233 2024-06-14T23:06:58.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024480+0000", 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024480+0000", 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:58.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024478+0000", 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024504+0000", 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5913000000000001e-05 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024517+0000", 2024-06-14T23:06:58.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3409e-05 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.024523+0000", 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.947e-06 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026253+0000", 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017299800000000001 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026395+0000", 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141648 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.026528+0000", 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013276999999999999 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.125950+0000", 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.099422650000000001 2024-06-14T23:06:58.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.126111+0000", 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160555 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.133029+0000", 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0069178759999999999 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:58.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.133083+0000", 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4388999999999997e-05 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.133123+0000", 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9734999999999998e-05 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:117239 10.e 10:73df0b96:::benchmark_data_smithi086_189745_object117238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-14T23:06:58.983 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:01:53.029598+0000", 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 305.94261948500002, 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.104880501, 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117239 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:58.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029598+0000", 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029598+0000", 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029596+0000", 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029646+0000", 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0021000000000001e-05 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029668+0000", 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2027000000000001e-05 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:58.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.029675+0000", 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.426e-06 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122576+0000", 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.092900782000000001 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122706+0000", 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013087900000000001 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.122869+0000", 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016291799999999999 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.129414+0000", 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065442110000000003 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.133165+0000", 2024-06-14T23:06:58.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037517330000000002 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.134397+0000", 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012313109999999999 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.134444+0000", 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6993999999999998e-05 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:01:53.134479+0000", 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5061000000000002e-05 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252334 10.e 10:703c40e1:::benchmark_data_smithi086_189745_object121366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.132222+0000", 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 157.83999579600001, 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15311908299999999, 2024-06-14T23:06:58.990 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:58.991 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:58.991 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:58.991 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:58.991 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252334 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132222+0000", 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132222+0000", 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:58.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132220+0000", 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132225+0000", 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5709999999999999e-06 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:58.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132237+0000", 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2354000000000001e-05 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132243+0000", 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4859999999999997e-06 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132349+0000", 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105629 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132454+0000", 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105302 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132573+0000", 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119226 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.142887+0000", 2024-06-14T23:06:58.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010313708 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.142945+0000", 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8133000000000001e-05 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.142955+0000", 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0077e-05 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285197+0000", 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.142242058 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285240+0000", 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3432e-05 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285341+0000", 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010046499999999999 2024-06-14T23:06:58.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252339 10.e 10:70dcdd8f:::benchmark_data_smithi086_189745_object121371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.137784+0000", 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 157.83443363999999, 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14782557299999999, 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:58.998 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252339 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137784+0000", 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137784+0000", 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:58.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137782+0000", 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137789+0000", 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8140000000000003e-06 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137802+0000", 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2449999999999999e-05 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137807+0000", 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6620000000000002e-06 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139832+0000", 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020249109999999999 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139964+0000", 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000132223 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.140134+0000", 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000169362 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143030+0000", 2024-06-14T23:06:59.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028967730000000001 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143035+0000", 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5199999999999999e-06 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.148876+0000", 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005840914 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285457+0000", 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13658071899999999 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285498+0000", 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1788e-05 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285610+0000", 2024-06-14T23:06:59.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011121299999999999 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252341 10.e 10:75c51104:::benchmark_data_smithi086_189745_object121373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.137972+0000", 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 157.83424591299999, 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14802895199999999, 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252341 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137972+0000", 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137972+0000", 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137971+0000", 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137975+0000", 2024-06-14T23:06:59.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3629999999999998e-06 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137982+0000", 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7340000000000008e-06 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137986+0000", 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1819999999999999e-06 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143052+0000", 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050652889999999997 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143170+0000", 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118755 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143294+0000", 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012369700000000001 2024-06-14T23:06:59.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.148940+0000", 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056457319999999997 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.148970+0000", 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0195000000000001e-05 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149013+0000", 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3353999999999999e-05 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285649+0000", 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.136635389 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285769+0000", 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120219 2024-06-14T23:06:59.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.286001+0000", 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000231642 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252342 10.e 10:7c0afdb9:::benchmark_data_smithi086_189745_object121374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.139687+0000", 2024-06-14T23:06:59.012 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 157.83253019, 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14649146099999999, 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252342 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139687+0000", 2024-06-14T23:06:59.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139687+0000", 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139686+0000", 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139692+0000", 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6049999999999997e-06 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139708+0000", 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6384000000000001e-05 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139714+0000", 2024-06-14T23:06:59.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6910000000000002e-06 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.146284+0000", 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065695550000000004 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.146430+0000", 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014615099999999999 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.146550+0000", 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120195 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149006+0000", 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002455975 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149044+0000", 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8155000000000003e-05 2024-06-14T23:06:59.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.151425+0000", 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023813290000000002 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.286039+0000", 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.134613646 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.286081+0000", 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1538000000000001e-05 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.286179+0000", 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8363999999999995e-05 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252346 10.e 10:7351a8c3:::benchmark_data_smithi086_189745_object121378:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.145483+0000", 2024-06-14T23:06:59.019 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 157.82673443799999, 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.142599638, 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252346 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145483+0000", 2024-06-14T23:06:59.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145483+0000", 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145482+0000", 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145487+0000", 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6800000000000001e-06 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145498+0000", 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1216000000000001e-05 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145504+0000", 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5180000000000001e-06 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149077+0000", 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035729210000000002 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149176+0000", 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9083e-05 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149278+0000", 2024-06-14T23:06:59.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101383 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.151657+0000", 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023791870000000001 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.151664+0000", 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1459999999999999e-06 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.152401+0000", 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073650199999999999 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.287931+0000", 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13553063700000001 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.287982+0000", 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0846000000000001e-05 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.288083+0000", 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100682 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252602 10.e 10:73840a31:::benchmark_data_smithi086_189745_object121634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.127817+0000", 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.84440018699999, 2024-06-14T23:06:59.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89931418799999996, 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252602 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127817+0000", 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127817+0000", 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127816+0000", 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127822+0000", 2024-06-14T23:06:59.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9739999999999999e-06 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127834+0000", 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1814000000000001e-05 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127839+0000", 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4689999999999996e-06 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127961+0000", 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121651 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127968+0000", 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6889999999999998e-06 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.992502+0000", 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86453456799999995 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.992591+0000", 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9153000000000005e-05 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.992668+0000", 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6419000000000001e-05 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024151+0000", 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031483186000000003 2024-06-14T23:06:59.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024178+0000", 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7098e-05 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027048+0000", 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028697240000000001 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027091+0000", 2024-06-14T23:06:59.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3090000000000002e-05 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027132+0000", 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0772999999999997e-05 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252608 10.e 10:774df1e1:::benchmark_data_smithi086_189745_object121640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.133959+0000", 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.83825846400001, 2024-06-14T23:06:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89326967700000004, 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252608 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133959+0000", 2024-06-14T23:06:59.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133959+0000", 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133958+0000", 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133964+0000", 2024-06-14T23:06:59.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5610000000000004e-06 2024-06-14T23:06:59.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133988+0000", 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3663999999999999e-05 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133995+0000", 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8249999999999999e-06 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.134043+0000", 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8720000000000001e-05 2024-06-14T23:06:59.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.134051+0000", 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8499999999999994e-06 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.995475+0000", 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86142369100000005 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.995601+0000", 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125754 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.995731+0000", 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000130681 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024203+0000", 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028471988 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024228+0000", 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.48e-05 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027140+0000", 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002911832 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027193+0000", 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3399000000000002e-05 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027229+0000", 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5485999999999999e-05 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252623 10.e 10:716ab276:::benchmark_data_smithi086_189745_object121655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.148259+0000", 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.82395821599999, 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87907613699999998, 2024-06-14T23:06:59.041 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252623 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148259+0000", 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148259+0000", 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148258+0000", 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148263+0000", 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6250000000000004e-06 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148273+0000", 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5950000000000005e-06 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148278+0000", 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1440000000000002e-06 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148323+0000", 2024-06-14T23:06:59.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4616000000000001e-05 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148336+0000", 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3771000000000001e-05 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.998870+0000", 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85053394100000002 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.999017+0000", 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014683900000000001 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.999155+0000", 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013765199999999999 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024253+0000", 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025097715999999999 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024278+0000", 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5605e-05 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027270+0000", 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029918240000000001 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027306+0000", 2024-06-14T23:06:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5534999999999998e-05 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027336+0000", 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9986999999999999e-05 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252629 10.e 10:71d7498e:::benchmark_data_smithi086_189745_object121661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.154557+0000", 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.817660856, 2024-06-14T23:06:59.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87284944099999995, 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252629 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.154557+0000", 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.154557+0000", 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.154540+0000", 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999833 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158057+0000", 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035170869999999999 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158076+0000", 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8581999999999999e-05 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158083+0000", 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0350000000000004e-06 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158731+0000", 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064791399999999998 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158738+0000", 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1169999999999998e-06 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.002862+0000", 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84412396499999998 2024-06-14T23:06:59.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.003212+0000", 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035084500000000002 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.003371+0000", 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015807500000000001 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024303+0000", 2024-06-14T23:06:59.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020932504000000001 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024327+0000", 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4170999999999999e-05 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027347+0000", 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030197610000000001 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027377+0000", 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9986999999999999e-05 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027406+0000", 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9277000000000001e-05 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252647 10.e 10:72729fba:::benchmark_data_smithi086_189745_object121679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:06:59.055 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.172478+0000", 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.79973973200001, 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.854998225, 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252647 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172478+0000", 2024-06-14T23:06:59.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172478+0000", 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172476+0000", 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172482+0000", 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2740000000000004e-06 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172498+0000", 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6018000000000001e-05 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172504+0000", 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0970000000000002e-06 2024-06-14T23:06:59.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172546+0000", 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2064000000000002e-05 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172554+0000", 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4070000000000004e-06 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.009106+0000", 2024-06-14T23:06:59.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83655226299999996 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.009214+0000", 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107619 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.009335+0000", 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121106 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024353+0000", 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015018337999999999 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024378+0000", 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4732e-05 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027411+0000", 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030333130000000002 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027443+0000", 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1362000000000002e-05 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027476+0000", 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3546e-05 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.062 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252651 10.e 10:711dfc34:::benchmark_data_smithi086_189745_object121683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.178987+0000", 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.79323038699999, 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84855638899999997, 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252651 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.178987+0000", 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.178987+0000", 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.178986+0000", 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.178994+0000", 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2700000000000004e-06 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.179007+0000", 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3383e-05 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.179013+0000", 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7339999999999996e-06 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.179048+0000", 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5388000000000001e-05 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.179054+0000", 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6319999999999996e-06 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.012456+0000", 2024-06-14T23:06:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83340202200000002 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.012590+0000", 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133652 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.012733+0000", 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000142991 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024409+0000", 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011676116 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027040+0000", 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026314519999999998 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027482+0000", 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044158400000000002 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027513+0000", 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1535999999999998e-05 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027544+0000", 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0182999999999999e-05 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252662 10.e 10:75f70bd0:::benchmark_data_smithi086_189745_object121694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.201859+0000", 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.770358491, 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82584559400000002, 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252662 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201859+0000", 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201859+0000", 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201858+0000", 2024-06-14T23:06:59.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201863+0000", 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5469999999999998e-06 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201873+0000", 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0339e-05 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201877+0000", 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0430000000000001e-06 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201927+0000", 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0423999999999997e-05 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201936+0000", 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3990000000000004e-06 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.015444+0000", 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.81350812299999997 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.015563+0000", 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0001189 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.015691+0000", 2024-06-14T23:06:59.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012809400000000001 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027265+0000", 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011573669999999999 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027549+0000", 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000284787 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027646+0000", 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6413e-05 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027677+0000", 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0862999999999999e-05 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027705+0000", 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8107e-05 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.076 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252665 10.e 10:7d9cb9fd:::benchmark_data_smithi086_189745_object121697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.214428+0000", 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.757789282, 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.81702328300000004, 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:06:59.077 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252665 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214428+0000", 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214428+0000", 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:06:59.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214427+0000", 2024-06-14T23:06:59.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:06:59.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214432+0000", 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7999999999999998e-06 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214442+0000", 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0379e-05 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214446+0000", 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0810000000000004e-06 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214494+0000", 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7839000000000001e-05 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214500+0000", 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9889999999999997e-06 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.019682+0000", 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80518220100000004 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.019778+0000", 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5921000000000005e-05 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.019885+0000", 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010651500000000001 2024-06-14T23:06:59.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027554+0000", 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076687140000000001 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027731+0000", 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000176863 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.031365+0000", 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036342330000000002 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.031413+0000", 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8579999999999999e-05 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.031452+0000", 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8331000000000003e-05 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36237.0:135097 11.d e261/237 11:b57b3051:::benchmark_data_smithi086_226760_object7776:head v 261'8402, mlcod=261'8400)", 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.307313+0000", 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.66490443600000004, 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14073733699999999, 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307313+0000", 2024-06-14T23:06:59.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307313+0000", 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307312+0000", 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307320+0000", 2024-06-14T23:06:59.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9000000000000006e-06 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307333+0000", 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3521e-05 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307339+0000", 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4439999999999999e-06 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307392+0000", 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3584000000000001e-05 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307411+0000", 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8717999999999999e-05 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447999+0000", 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14058846 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448051+0000", 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1109000000000001e-05 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:135101 11.e 11:7b4cbd7b:::benchmark_data_smithi086_226760_object7780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.308293+0000", 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.66392477599999999, 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15223837100000001, 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.090 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135101 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308293+0000", 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308293+0000", 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308291+0000", 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308299+0000", 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6960000000000005e-06 2024-06-14T23:06:59.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308312+0000", 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3261999999999999e-05 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308318+0000", 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1879999999999999e-06 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308365+0000", 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6332000000000003e-05 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308470+0000", 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105128 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448019+0000", 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13954949 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.453774+0000", 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005755151 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.453816+0000", 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1721999999999998e-05 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460440+0000", 2024-06-14T23:06:59.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066234040000000003 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460497+0000", 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7669e-05 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460531+0000", 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4076999999999999e-05 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36237.0:135100 11.c e261/237 11:33d4c343:::benchmark_data_smithi086_226760_object7779:head v 261'8533, mlcod=261'8531)", 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.308431+0000", 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.66378634599999997, 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.139588079, 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308431+0000", 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308431+0000", 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308431+0000", 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308439+0000", 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8979999999999996e-06 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308448+0000", 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4350000000000003e-06 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308452+0000", 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9600000000000002e-06 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308488+0000", 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5855000000000001e-05 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308505+0000", 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.749e-05 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447983+0000", 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.139477667 2024-06-14T23:06:59.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448019+0000", 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6445000000000001e-05 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36237.0:135104 11.d e261/237 11:bcf46504:::benchmark_data_smithi086_226760_object7783:head v 261'8403, mlcod=261'8400)", 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.316932+0000", 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.65528609299999996, 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14248889100000001, 2024-06-14T23:06:59.101 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.316932+0000", 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.316932+0000", 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.316931+0000", 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.316937+0000", 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6900000000000003e-06 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447970+0000", 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13103320500000001 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447984+0000", 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3740000000000001e-05 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448082+0000", 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7369999999999998e-05 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448099+0000", 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7337999999999999e-05 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.459388+0000", 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011289344999999999 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.459420+0000", 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2211e-05 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.105 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:135114 11.e 11:7774b920:::benchmark_data_smithi086_226760_object7793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.324411+0000", 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.64780677399999997, 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.136254772, 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:06:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135114 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.324411+0000", 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.324411+0000", 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:06:59.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.324409+0000", 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.324418+0000", 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7600000000000008e-06 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447972+0000", 2024-06-14T23:06:59.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12355432099999999 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447994+0000", 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2172e-05 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.451216+0000", 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032218099999999999 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.451315+0000", 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8837000000000003e-05 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.451394+0000", 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240999999999996e-05 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460361+0000", 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0089670789999999993 2024-06-14T23:06:59.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460563+0000", 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020152599999999999 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460568+0000", 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1529999999999996e-06 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460635+0000", 2024-06-14T23:06:59.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6655000000000005e-05 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460666+0000", 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1031000000000002e-05 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:06:59.112 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:06:59.123 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:59.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:06:59.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[102754]: osdmap e261: 8 total, 8 up, 8 in 2024-06-14T23:06:59.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2075064076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:59.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:59.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2075064076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:06:59.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[102754]: osdmap e262: 8 total, 8 up, 8 in 2024-06-14T23:06:59.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[97913]: osdmap e261: 8 total, 8 up, 8 in 2024-06-14T23:06:59.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2075064076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:59.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:59.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2075064076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:06:59.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:58 smithi086 ceph-mon[97913]: osdmap e262: 8 total, 8 up, 8 in 2024-06-14T23:06:59.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:59.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:58 smithi137 ceph-mon[90634]: osdmap e261: 8 total, 8 up, 8 in 2024-06-14T23:06:59.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2075064076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:06:59.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:58 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:06:59.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2075064076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:06:59.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:58 smithi137 ceph-mon[90634]: osdmap e262: 8 total, 8 up, 8 in 2024-06-14T23:06:59.254 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:59.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:59.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:06:59.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:59.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:06:59.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:59.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:59.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:59.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:06:59.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:59.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:06:59.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:06:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[50586]: 2024-06-14T23:06:59.465+0000 7fcb4bd20640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:06:59.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:06:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[50586]: 2024-06-14T23:06:59.465+0000 7fcb4bd20640 -1 osd.6 262 *** Got signal Terminated *** 2024-06-14T23:06:59.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:06:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[50586]: 2024-06-14T23:06:59.465+0000 7fcb4bd20640 -1 osd.6 262 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:06:59.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:06:59.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:06:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:59.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:06:59.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:06:59.918 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:06:59.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:06:59.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:06:59.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:00.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:00.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:00.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:00.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:59 smithi086 ceph-mon[97913]: pgmap v1005: 109 pgs: 109 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 4.4 MiB/s wr, 721 op/s 2024-06-14T23:07:00.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:59 smithi086 ceph-mon[97913]: osd.6 marked itself down and dead 2024-06-14T23:07:00.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:06:59 smithi086 ceph-mon[97913]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T23:07:00.062 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:59 smithi086 ceph-mon[102754]: pgmap v1005: 109 pgs: 109 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 4.4 MiB/s wr, 721 op/s 2024-06-14T23:07:00.062 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:59 smithi086 ceph-mon[102754]: osd.6 marked itself down and dead 2024-06-14T23:07:00.062 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:06:59 smithi086 ceph-mon[102754]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T23:07:00.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:00.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:00.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:00.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:00.199 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:59 smithi137 ceph-mon[90634]: pgmap v1005: 109 pgs: 109 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 4.4 MiB/s wr, 721 op/s 2024-06-14T23:07:00.199 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:59 smithi137 ceph-mon[90634]: osd.6 marked itself down and dead 2024-06-14T23:07:00.199 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:06:59 smithi137 ceph-mon[90634]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T23:07:00.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T23:07:00.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:00.303 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:00.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:00.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:00.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:00.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:00.457 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:07:00.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:00.551 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:00.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:00.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:00.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:00.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:00.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:00.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:00.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:00.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:00.895 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:00 smithi137 podman[157720]: 2024-06-14 23:07:00.628545379 +0000 UTC m=+1.619376251 container died 7210ba0a183df7f0f7d832c85018b85073345231aedffc43a2635e91422ddcf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-14T23:07:00.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:00.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:00.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:01.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:01.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:01.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:00 smithi086 ceph-mon[97913]: osdmap e263: 8 total, 7 up, 8 in 2024-06-14T23:07:01.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:00 smithi086 ceph-mon[97913]: osdmap e264: 8 total, 7 up, 8 in 2024-06-14T23:07:01.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:00 smithi086 ceph-mon[102754]: osdmap e263: 8 total, 7 up, 8 in 2024-06-14T23:07:01.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:00 smithi086 ceph-mon[102754]: osdmap e264: 8 total, 7 up, 8 in 2024-06-14T23:07:01.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:01.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:01.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:00 smithi137 ceph-mon[90634]: osdmap e263: 8 total, 7 up, 8 in 2024-06-14T23:07:01.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:00 smithi137 ceph-mon[90634]: osdmap e264: 8 total, 7 up, 8 in 2024-06-14T23:07:01.245 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:01 smithi137 podman[157720]: 2024-06-14 23:07:01.00449263 +0000 UTC m=+1.995323503 container cleanup 7210ba0a183df7f0f7d832c85018b85073345231aedffc43a2635e91422ddcf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-14T23:07:01.245 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:01 smithi137 bash[157720]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6 2024-06-14T23:07:01.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:01.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:01.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:01.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:01.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:01.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:01.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:01.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:01.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:01.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:01.643 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:01 smithi137 podman[157951]: 2024-06-14 23:07:01.273045722 +0000 UTC m=+0.701287749 container remove 7210ba0a183df7f0f7d832c85018b85073345231aedffc43a2635e91422ddcf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-14T23:07:01.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:01.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:01.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:01.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:07:01.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:01.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:01.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:01.946 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:07:01.947 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:07:01.952 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:01 smithi137 podman[158007]: 2024-06-14 23:07:01.750961672 +0000 UTC m=+0.718684037 container died 9c915e14757ebd4eb6f7e2f039696845446dbc8bd06fe6c0b4355b0663a3bab5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=hungry_galois, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-14T23:07:01.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:01.993 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:07:01.993 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:07:02.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:02.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:02.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:01 smithi086 ceph-mon[102754]: pgmap v1008: 109 pgs: 11 peering, 1 stale+active+undersized+remapped, 3 active+undersized+remapped, 9 stale+active+clean, 85 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 80 KiB/s, 0 objects/s recovering 2024-06-14T23:07:02.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:01 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:07:02.036 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1336548975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:02.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:01 smithi086 ceph-mon[102754]: osdmap e265: 8 total, 7 up, 8 in 2024-06-14T23:07:02.037 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:07:02.037 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:07:02.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:01 smithi086 ceph-mon[97913]: pgmap v1008: 109 pgs: 11 peering, 1 stale+active+undersized+remapped, 3 active+undersized+remapped, 9 stale+active+clean, 85 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 80 KiB/s, 0 objects/s recovering 2024-06-14T23:07:02.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:01 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:07:02.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1336548975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:02.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:01 smithi086 ceph-mon[97913]: osdmap e265: 8 total, 7 up, 8 in 2024-06-14T23:07:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:07:02.080 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:07:02.113 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:07:02.114 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:07:02.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:02.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:02.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:02.148 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:07:02.149 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:07:02.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-14T23:07:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:07:02.187 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:07:02.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:02.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:01 smithi137 ceph-mon[90634]: pgmap v1008: 109 pgs: 11 peering, 1 stale+active+undersized+remapped, 3 active+undersized+remapped, 9 stale+active+clean, 85 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 80 KiB/s, 0 objects/s recovering 2024-06-14T23:07:02.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:01 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-14T23:07:02.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1336548975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:02.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:01 smithi137 ceph-mon[90634]: osdmap e265: 8 total, 7 up, 8 in 2024-06-14T23:07:02.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:02.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:02.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:02.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:02.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:02.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:02.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:02.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:02.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:02.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:02.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:02.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:02.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:02.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:02.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:02.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:03.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:03.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:03.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:03.118 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:07:03.118 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:07:03.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:03.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:03.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:03.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:03.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:03.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:03.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:03.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:03.630 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:03.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:03.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:03.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:03.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:03.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:03.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:03.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:03.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:04.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:04.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:04.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:03 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:04.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:03 smithi086 ceph-mon[97913]: pgmap v1011: 109 pgs: 1 active+undersized, 11 peering, 1 stale+active+undersized+remapped, 3 active+undersized+remapped, 8 stale+active+clean, 85 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 10 KiB/s rd, 27 KiB/s wr, 1.11k op/s; 100 KiB/s, 0 objects/s recovering 2024-06-14T23:07:04.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:03 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:04.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:03 smithi086 ceph-mon[102754]: pgmap v1011: 109 pgs: 1 active+undersized, 11 peering, 1 stale+active+undersized+remapped, 3 active+undersized+remapped, 8 stale+active+clean, 85 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 10 KiB/s rd, 27 KiB/s wr, 1.11k op/s; 100 KiB/s, 0 objects/s recovering 2024-06-14T23:07:04.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:04.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:04.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:04.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:03 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:04.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:03 smithi137 ceph-mon[90634]: pgmap v1011: 109 pgs: 1 active+undersized, 11 peering, 1 stale+active+undersized+remapped, 3 active+undersized+remapped, 8 stale+active+clean, 85 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 10 KiB/s rd, 27 KiB/s wr, 1.11k op/s; 100 KiB/s, 0 objects/s recovering 2024-06-14T23:07:04.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:04.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:04.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:04.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:04.439 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:04.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:04.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:04.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:04.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:04.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-14T23:07:04.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:07:04.706 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:07:04.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:04.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:04.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:04.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:04.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:04.911 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:04 smithi137 podman[158345]: 2024-06-14 23:07:04.765855777 +0000 UTC m=+0.029796713 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:07:04.912 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:04 smithi137 podman[158345]: 2024-06-14 23:07:04.909562134 +0000 UTC m=+0.173503069 container create d9a16a606eee0c677511dc597f37fd35c1719a56cd17354a00d4ff8eb05b57fe (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:07:04.935 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:07:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:07:04] "GET /metrics HTTP/1.1" 200 36602 "" "Prometheus/2.43.0" 2024-06-14T23:07:04.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:04.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:04.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:05.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:05.111 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:05.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:05.188 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:04 smithi137 ceph-mon[90634]: Health check failed: Degraded data redundancy: 36003/349824 objects degraded (10.292%), 9 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:05.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:05.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:05.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:05.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:05.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:04 smithi086 ceph-mon[102754]: Health check failed: Degraded data redundancy: 36003/349824 objects degraded (10.292%), 9 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:05.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:04 smithi086 ceph-mon[97913]: Health check failed: Degraded data redundancy: 36003/349824 objects degraded (10.292%), 9 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:05.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:05.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:05.400 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:07:05.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:05.495 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:05 smithi137 podman[158345]: 2024-06-14 23:07:05.18705705 +0000 UTC m=+0.450997985 container init d9a16a606eee0c677511dc597f37fd35c1719a56cd17354a00d4ff8eb05b57fe (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-14T23:07:05.495 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:05 smithi137 podman[158345]: 2024-06-14 23:07:05.194486265 +0000 UTC m=+0.458427200 container start d9a16a606eee0c677511dc597f37fd35c1719a56cd17354a00d4ff8eb05b57fe (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-14T23:07:05.495 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:05 smithi137 podman[158345]: 2024-06-14 23:07:05.345603618 +0000 UTC m=+0.609544552 container attach d9a16a606eee0c677511dc597f37fd35c1719a56cd17354a00d4ff8eb05b57fe (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-14T23:07:05.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:05.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:05.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:05.628 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:05.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:05.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:05.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:05.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:05.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:05.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:05.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:05.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:06.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:06.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:06.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:06.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:05 smithi086 ceph-mon[102754]: pgmap v1012: 109 pgs: 3 active+clean+remapped, 18 active+undersized, 11 peering, 1 active+undersized+remapped, 2 stale+active+clean, 9 active+undersized+degraded, 65 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 14 KiB/s rd, 37 KiB/s wr, 1.06k op/s; 36003/349824 objects degraded (10.292%); 81 KiB/s, 0 objects/s recovering 2024-06-14T23:07:06.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:05 smithi086 ceph-mon[97913]: pgmap v1012: 109 pgs: 3 active+clean+remapped, 18 active+undersized, 11 peering, 1 active+undersized+remapped, 2 stale+active+clean, 9 active+undersized+degraded, 65 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 14 KiB/s rd, 37 KiB/s wr, 1.06k op/s; 36003/349824 objects degraded (10.292%); 81 KiB/s, 0 objects/s recovering 2024-06-14T23:07:06.091 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:05 smithi137 ceph-mon[90634]: pgmap v1012: 109 pgs: 3 active+clean+remapped, 18 active+undersized, 11 peering, 1 active+undersized+remapped, 2 stale+active+clean, 9 active+undersized+degraded, 65 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 14 KiB/s rd, 37 KiB/s wr, 1.06k op/s; 36003/349824 objects degraded (10.292%); 81 KiB/s, 0 objects/s recovering 2024-06-14T23:07:06.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:06.175 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:06.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:06.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:06.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:06.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:06.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:06.453 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:06 smithi137 podman[158345]: 2024-06-14 23:07:06.089612215 +0000 UTC m=+1.353553146 container died d9a16a606eee0c677511dc597f37fd35c1719a56cd17354a00d4ff8eb05b57fe (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612) 2024-06-14T23:07:06.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:06.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:06.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:06.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:06.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:06.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:06.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:07:06.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:06.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:06.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:06.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:06.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:06.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:06.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:06.968 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:07:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:07.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:07.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2142227167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:07:07.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:06 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:07:07.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2142227167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:07:07.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:06 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:07:07.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:07.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:07.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:07.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:07.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2142227167' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:07:07.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:06 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:07:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:07.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:07.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:07.403 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:07.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:07.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:07.531 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:07.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:07.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:07.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:07.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:07.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:07.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:07.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:07.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:07.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:07.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:07.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:07.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:07.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.6.service' 2024-06-14T23:07:08.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:08.043 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:07:08.044 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-06-14T23:07:08.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:08.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:08.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:08.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:08.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:08.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:07 smithi137 ceph-mon[90634]: pgmap v1013: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 18 KiB/s rd, 48 KiB/s wr, 852 op/s; 57302/342390 objects degraded (16.736%); 1/342390 objects misplaced (0.000%); 34 KiB/s, 0 objects/s recovering 2024-06-14T23:07:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:07 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T23:07:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:07 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:07 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:07:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:07 smithi137 ceph-mon[90634]: osdmap e266: 8 total, 7 up, 8 in 2024-06-14T23:07:08.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:08.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[97913]: pgmap v1013: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 18 KiB/s rd, 48 KiB/s wr, 852 op/s; 57302/342390 objects degraded (16.736%); 1/342390 objects misplaced (0.000%); 34 KiB/s, 0 objects/s recovering 2024-06-14T23:07:08.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T23:07:08.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:08.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:07:08.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[97913]: osdmap e266: 8 total, 7 up, 8 in 2024-06-14T23:07:08.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[102754]: pgmap v1013: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 18 KiB/s rd, 48 KiB/s wr, 852 op/s; 57302/342390 objects degraded (16.736%); 1/342390 objects misplaced (0.000%); 34 KiB/s, 0 objects/s recovering 2024-06-14T23:07:08.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-14T23:07:08.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:08.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:07:08.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:07 smithi086 ceph-mon[102754]: osdmap e266: 8 total, 7 up, 8 in 2024-06-14T23:07:08.343 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:08.343 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:08.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:08.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:08.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:08.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:08.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:08.628 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:08.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:08.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:08.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:08.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:08.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:08.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:08.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:08.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:08.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:09.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:09.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:09.108 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:07:09.108 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:07:09.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:09.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:09.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:09.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:09.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:09.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:09.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:09.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:09.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:09.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:09.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:09.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:09.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:09.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:09.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:09.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:09.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:09.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:09.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-14T23:07:09.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:09.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:09.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:10.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:10.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:09 smithi086 ceph-mon[97913]: pgmap v1015: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 7.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 16 KiB/s rd, 42 KiB/s wr, 557 op/s; 57302/342390 objects degraded (16.736%); 1/342390 objects misplaced (0.000%); 30 KiB/s, 0 objects/s recovering 2024-06-14T23:07:10.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:09 smithi086 ceph-mon[97913]: 1.14 scrub starts 2024-06-14T23:07:10.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:09 smithi086 ceph-mon[97913]: 1.14 scrub ok 2024-06-14T23:07:10.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:09 smithi086 ceph-mon[102754]: pgmap v1015: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 7.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 16 KiB/s rd, 42 KiB/s wr, 557 op/s; 57302/342390 objects degraded (16.736%); 1/342390 objects misplaced (0.000%); 30 KiB/s, 0 objects/s recovering 2024-06-14T23:07:10.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:09 smithi086 ceph-mon[102754]: 1.14 scrub starts 2024-06-14T23:07:10.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:09 smithi086 ceph-mon[102754]: 1.14 scrub ok 2024-06-14T23:07:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:10.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:10.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:10.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:07:10.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:10.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:09 smithi137 ceph-mon[90634]: pgmap v1015: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 7.0 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 16 KiB/s rd, 42 KiB/s wr, 557 op/s; 57302/342390 objects degraded (16.736%); 1/342390 objects misplaced (0.000%); 30 KiB/s, 0 objects/s recovering 2024-06-14T23:07:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:09 smithi137 ceph-mon[90634]: 1.14 scrub starts 2024-06-14T23:07:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:09 smithi137 ceph-mon[90634]: 1.14 scrub ok 2024-06-14T23:07:10.251 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:07:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:10.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:10.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:10.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:10.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:10.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:10.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:10.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:10.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:10.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:10.673 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:10.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:10.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:10.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:10.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:10.844 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:07:10.844 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:07:10.884 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:10.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:07:10.888 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:07:10.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:10.922 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:07:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:07:10.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:10.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:10.967 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:07:10.968 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:07:10.996 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:07:10.996 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:07:11.027 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:07:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:07:11.030 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-14T23:07:11.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:11.062 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:07:11.062 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:07:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:11.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:11.169 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[97913]: 1.13 scrub starts 2024-06-14T23:07:11.169 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[97913]: 1.13 scrub ok 2024-06-14T23:07:11.169 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[97913]: 1.4 scrub starts 2024-06-14T23:07:11.169 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[97913]: 1.4 scrub ok 2024-06-14T23:07:11.169 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2144523864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:11.170 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[102754]: 1.13 scrub starts 2024-06-14T23:07:11.170 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[102754]: 1.13 scrub ok 2024-06-14T23:07:11.170 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[102754]: 1.4 scrub starts 2024-06-14T23:07:11.170 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[102754]: 1.4 scrub ok 2024-06-14T23:07:11.170 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2144523864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:11.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:11.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:10 smithi137 ceph-mon[90634]: 1.13 scrub starts 2024-06-14T23:07:11.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:10 smithi137 ceph-mon[90634]: 1.13 scrub ok 2024-06-14T23:07:11.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:10 smithi137 ceph-mon[90634]: 1.4 scrub starts 2024-06-14T23:07:11.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:10 smithi137 ceph-mon[90634]: 1.4 scrub ok 2024-06-14T23:07:11.209 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2144523864' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:11.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:11.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:11.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:11.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:11.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:11.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:11.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:11.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:11.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:11.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:11.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:11.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:11.750 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:11.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:11.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:11.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:11.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:11.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:12.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:12.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:12.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:12.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:12.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:12.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:11 smithi137 ceph-mon[90634]: pgmap v1016: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 14 KiB/s rd, 37 KiB/s wr, 1.06k op/s; 54724/327258 objects degraded (16.722%); 1/327258 objects misplaced (0.000%); 26 KiB/s, 0 objects/s recovering 2024-06-14T23:07:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:11 smithi137 ceph-mon[90634]: 1.2 scrub starts 2024-06-14T23:07:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:11 smithi137 ceph-mon[90634]: 1.2 scrub ok 2024-06-14T23:07:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:11 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 54724/327258 objects degraded (16.722%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:11 smithi137 ceph-mon[90634]: 1.1 scrub starts 2024-06-14T23:07:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:11 smithi137 ceph-mon[90634]: 1.1 scrub ok 2024-06-14T23:07:12.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:12.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:12.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:12.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[97913]: pgmap v1016: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 14 KiB/s rd, 37 KiB/s wr, 1.06k op/s; 54724/327258 objects degraded (16.722%); 1/327258 objects misplaced (0.000%); 26 KiB/s, 0 objects/s recovering 2024-06-14T23:07:12.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[97913]: 1.2 scrub starts 2024-06-14T23:07:12.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[97913]: 1.2 scrub ok 2024-06-14T23:07:12.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 54724/327258 objects degraded (16.722%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:12.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[97913]: 1.1 scrub starts 2024-06-14T23:07:12.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[97913]: 1.1 scrub ok 2024-06-14T23:07:12.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[102754]: pgmap v1016: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 14 KiB/s rd, 37 KiB/s wr, 1.06k op/s; 54724/327258 objects degraded (16.722%); 1/327258 objects misplaced (0.000%); 26 KiB/s, 0 objects/s recovering 2024-06-14T23:07:12.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[102754]: 1.2 scrub starts 2024-06-14T23:07:12.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[102754]: 1.2 scrub ok 2024-06-14T23:07:12.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 54724/327258 objects degraded (16.722%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:12.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[102754]: 1.1 scrub starts 2024-06-14T23:07:12.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:11 smithi086 ceph-mon[102754]: 1.1 scrub ok 2024-06-14T23:07:12.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:12.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:12.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:12.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:12.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:12.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:12.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:12.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:12.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:12.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:12.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:12.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:12.975 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:12.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:13.045 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-06-14T23:07:13.045 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:07:13.045 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_6 2024-06-14T23:07:13.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:07:13.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:13.167 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:13.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:13.264 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:07:13.264 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:07:13.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:07:13.264 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:07:13.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.264 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252314 10.9 10:960f94a9:::benchmark_data_smithi086_189745_object121346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:07:13.264 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.011224+0000", 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 172.247213402, 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0900273949999999, 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252314 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011224+0000", 2024-06-14T23:07:13.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011224+0000", 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011223+0000", 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011230+0000", 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6009999999999997e-06 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011246+0000", 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6090999999999999e-05 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011252+0000", 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9660000000000001e-06 2024-06-14T23:07:13.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011316+0000", 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4170000000000004e-05 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011325+0000", 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1210000000000007e-06 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.041481+0000", 2024-06-14T23:07:13.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0301560869999999 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.041565+0000", 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4305e-05 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.041641+0000", 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6340000000000004e-05 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.059215+0000", 2024-06-14T23:07:13.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017573808 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101090+0000", 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041874407000000002 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101182+0000", 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2931999999999997e-05 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101222+0000", 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9650000000000002e-05 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101252+0000", 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9663999999999999e-05 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252317 10.9 10:91aeda83:::benchmark_data_smithi086_189745_object121349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.022551+0000", 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 172.235886397, 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.078809973, 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252317 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022551+0000", 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022551+0000", 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022550+0000", 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022556+0000", 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4269999999999999e-06 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022566+0000", 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0010000000000001e-05 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022570+0000", 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2189999999999998e-06 2024-06-14T23:07:13.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022606+0000", 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5809000000000002e-05 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022610+0000", 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8049999999999997e-06 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.045161+0000", 2024-06-14T23:07:13.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0225505130000001 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.045235+0000", 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4148999999999998e-05 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.045310+0000", 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4604000000000006e-05 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.059224+0000", 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013914517 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101136+0000", 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041911325999999999 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101303+0000", 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016734900000000001 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101339+0000", 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6380000000000001e-05 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101361+0000", 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2092000000000001e-05 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252374 10.9 10:999d37ba:::benchmark_data_smithi086_189745_object121406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.289210+0000", 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 171.96922791399999, 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.81232411699999996, 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252374 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.289210+0000", 2024-06-14T23:07:13.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.289210+0000", 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.289209+0000", 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.289215+0000", 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0329999999999997e-06 2024-06-14T23:07:13.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.289226+0000", 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1807000000000001e-05 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.289233+0000", 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1099999999999999e-06 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.289285+0000", 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2148000000000003e-05 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.289292+0000", 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.007e-06 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.048956+0000", 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.75966464499999997 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.049037+0000", 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0276000000000002e-05 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.049134+0000", 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7612e-05 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.066625+0000", 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017490634000000001 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101394+0000", 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034769126999999997 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101483+0000", 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8710999999999998e-05 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101514+0000", 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1687999999999999e-05 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101534+0000", 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9573999999999999e-05 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252411 10.9 10:980874fe:::benchmark_data_smithi086_189745_object121443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.328586+0000", 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 171.92985183900001, 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.77310465100000003, 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.285 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252411 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.328586+0000", 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.328586+0000", 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.328585+0000", 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.328589+0000", 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3349999999999998e-06 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.328596+0000", 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2200000000000003e-06 2024-06-14T23:07:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.328600+0000", 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0180000000000003e-06 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.328648+0000", 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8396000000000002e-05 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.328656+0000", 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0630000000000006e-06 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.052259+0000", 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72360204100000003 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.052326+0000", 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7513000000000002e-05 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.052395+0000", 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9441000000000001e-05 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.066632+0000", 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014236217000000001 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101423+0000", 2024-06-14T23:07:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034791242 2024-06-14T23:07:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101626+0000", 2024-06-14T23:07:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020318199999999999 2024-06-14T23:07:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101658+0000", 2024-06-14T23:07:13.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2274000000000003e-05 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101691+0000", 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2199999999999997e-05 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252678 10.3 10:ceff32ef:::benchmark_data_smithi086_189745_object121710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.023839+0000", 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.234599261, 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98525532999999998, 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252678 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023839+0000", 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023839+0000", 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023837+0000", 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023844+0000", 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.455e-06 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023858+0000", 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4619e-05 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023864+0000", 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3279999999999996e-06 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023971+0000", 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010751600000000001 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023979+0000", 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4390000000000003e-06 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.997909+0000", 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97393003199999995 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.998001+0000", 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2132000000000005e-05 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.998077+0000", 2024-06-14T23:07:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6552999999999997e-05 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002272+0000", 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041949149999999996 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.008946+0000", 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066735199999999996 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.008996+0000", 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0719e-05 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009049+0000", 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2899999999999998e-05 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009094+0000", 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4501e-05 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252708 10.3 10:c07e5468:::benchmark_data_smithi086_189745_object121740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.055163+0000", 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.203274937, 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95415895699999997, 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252708 2024-06-14T23:07:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055163+0000", 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055163+0000", 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055162+0000", 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055167+0000", 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2140000000000004e-06 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055177+0000", 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.013e-05 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055181+0000", 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8460000000000001e-06 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055222+0000", 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0878999999999999e-05 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055230+0000", 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.276e-06 2024-06-14T23:07:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002317+0000", 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94708661800000005 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002441+0000", 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124435 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002570+0000", 2024-06-14T23:07:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012846799999999999 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009132+0000", 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065624999999999998 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009196+0000", 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3718000000000002e-05 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009236+0000", 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0268000000000001e-05 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009286+0000", 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0078999999999999e-05 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009322+0000", 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5590999999999999e-05 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252709 10.3 10:c41e1665:::benchmark_data_smithi086_189745_object121741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.057035+0000", 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.201402691, 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95242822400000005, 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252709 2024-06-14T23:07:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057035+0000", 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057035+0000", 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057034+0000", 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057039+0000", 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1690000000000002e-06 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057049+0000", 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0148e-05 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057054+0000", 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7590000000000003e-06 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057093+0000", 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8566000000000003e-05 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057100+0000", 2024-06-14T23:07:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9859999999999997e-06 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.005750+0000", 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94864968599999999 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.005829+0000", 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9124999999999999e-05 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.005919+0000", 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0216000000000005e-05 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009353+0000", 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034340680000000002 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009357+0000", 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8970000000000001e-06 2024-06-14T23:07:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009391+0000", 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4400000000000003e-05 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009429+0000", 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7137e-05 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009463+0000", 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4786000000000002e-05 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252721 10.3 10:cae392c3:::benchmark_data_smithi086_189745_object121753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.067872+0000", 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.190565819, 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95780526600000004, 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252721 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067872+0000", 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067872+0000", 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067870+0000", 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067876+0000", 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6359999999999999e-06 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067888+0000", 2024-06-14T23:07:13.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1966e-05 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067894+0000", 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1739999999999997e-06 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067944+0000", 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0259000000000001e-05 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067951+0000", 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8650000000000003e-06 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009492+0000", 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94154064699999995 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009567+0000", 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5333999999999994e-05 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009654+0000", 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6735000000000002e-05 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017286+0000", 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076317240000000003 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017292+0000", 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1759999999999998e-06 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025571+0000", 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082794340000000004 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025636+0000", 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.444e-05 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025677+0000", 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1607999999999999e-05 2024-06-14T23:07:13.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252748 10.3 10:c31065d0:::benchmark_data_smithi086_189745_object121780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.118605+0000", 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.139832407, 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90724995399999997, 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.322 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252748 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118605+0000", 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118605+0000", 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118604+0000", 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118611+0000", 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7259999999999996e-06 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118622+0000", 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1523e-05 2024-06-14T23:07:13.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118628+0000", 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8119999999999996e-06 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118675+0000", 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6828e-05 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118685+0000", 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7990000000000005e-06 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.012981+0000", 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89429594199999995 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.013114+0000", 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013327200000000001 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.013233+0000", 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119317 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017334+0000", 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041004850000000001 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025532+0000", 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081980309999999997 2024-06-14T23:07:13.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025763+0000", 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023137200000000001 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025816+0000", 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2525000000000003e-05 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025855+0000", 2024-06-14T23:07:13.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9762999999999999e-05 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252755 10.3 10:c538d66a:::benchmark_data_smithi086_189745_object121787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.127097+0000", 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.13134052800001, 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89934203300000004, 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.329 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252755 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127097+0000", 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127097+0000", 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127096+0000", 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127102+0000", 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5610000000000004e-06 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127117+0000", 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4346e-05 2024-06-14T23:07:13.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127122+0000", 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6799999999999998e-06 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127178+0000", 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5414000000000002e-05 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127189+0000", 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1756000000000001e-05 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017353+0000", 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89016351900000001 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017457+0000", 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010399 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017571+0000", 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113579 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025689+0000", 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081183660000000001 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025731+0000", 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1641000000000003e-05 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026328+0000", 2024-06-14T23:07:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000597077 2024-06-14T23:07:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026402+0000", 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4680999999999999e-05 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026439+0000", 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7027000000000003e-05 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.337 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252760 10.3 10:cf81e509:::benchmark_data_smithi086_189745_object121792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.127159369, 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89530984300000005, 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252760 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131281+0000", 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6860000000000002e-06 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131287+0000", 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1029999999999998e-06 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131291+0000", 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8959999999999996e-06 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131331+0000", 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0021999999999997e-05 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131336+0000", 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3900000000000001e-06 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.021730+0000", 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89039394699999996 2024-06-14T23:07:13.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.021825+0000", 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5179000000000005e-05 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.021931+0000", 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010621899999999999 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025902+0000", 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039701329999999998 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025913+0000", 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1089e-05 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026503+0000", 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00059051700000000004 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026545+0000", 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1745000000000001e-05 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026588+0000", 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3331000000000002e-05 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252780 10.3 10:c1a81b18:::benchmark_data_smithi086_189745_object121812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.344 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.158993+0000", 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.099445105, 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.88218134500000001, 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252780 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.158993+0000", 2024-06-14T23:07:13.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.158993+0000", 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.158991+0000", 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.158998+0000", 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7039999999999997e-06 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.159010+0000", 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2255999999999999e-05 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.159016+0000", 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5350000000000004e-06 2024-06-14T23:07:13.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.159069+0000", 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2539e-05 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.159078+0000", 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5060000000000001e-06 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026962+0000", 2024-06-14T23:07:13.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86788375399999995 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.027071+0000", 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010958100000000001 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.027186+0000", 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011420499999999999 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.040994+0000", 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013808572 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041051+0000", 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7226999999999999e-05 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041089+0000", 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7701999999999999e-05 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041135+0000", 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6351999999999999e-05 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041174+0000", 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8625999999999999e-05 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.351 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252800 10.3 10:c6176e12:::benchmark_data_smithi086_189745_object121832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.189322+0000", 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.06911582399999, 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85191730099999996, 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252800 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189322+0000", 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189322+0000", 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189321+0000", 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189326+0000", 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7350000000000001e-06 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189338+0000", 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1694e-05 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189343+0000", 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5850000000000001e-06 2024-06-14T23:07:13.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189375+0000", 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2588e-05 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189380+0000", 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9710000000000003e-06 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.031447+0000", 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84206642600000003 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.031551+0000", 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104193 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.031661+0000", 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109812 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041023+0000", 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093619159999999996 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041082+0000", 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9357000000000001e-05 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041182+0000", 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100129 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041213+0000", 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0380000000000001e-05 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041239+0000", 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.675e-05 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.358 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252812 10.3 10:c0951fb7:::benchmark_data_smithi086_189745_object121844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.215115+0000", 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 170.04332261499999, 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84339402699999999, 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252812 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215115+0000", 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215115+0000", 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215114+0000", 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:07:13.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215119+0000", 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7150000000000003e-06 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215131+0000", 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1573000000000001e-05 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215135+0000", 2024-06-14T23:07:13.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2520000000000001e-06 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215169+0000", 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3906999999999999e-05 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215174+0000", 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8210000000000001e-06 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.036052+0000", 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82087817500000004 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.036175+0000", 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012256600000000001 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.036304+0000", 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012888999999999999 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.044326+0000", 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0080227460000000007 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.049185+0000", 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048587730000000003 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.058419+0000", 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0092341750000000007 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.058471+0000", 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1332000000000003e-05 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.058509+0000", 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8512e-05 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.365 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136541 11.5 11:ac4784f5:::benchmark_data_smithi086_226760_object9220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.932519+0000", 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.325918906, 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0219499169999999, 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136541 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932519+0000", 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932519+0000", 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932518+0000", 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932523+0000", 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.356e-06 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932535+0000", 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2167999999999999e-05 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932541+0000", 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7400000000000001e-06 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932575+0000", 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3946000000000003e-05 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932684+0000", 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010943299999999999 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932803+0000", 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118439 2024-06-14T23:07:13.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.946273+0000", 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013470650000000001 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.946281+0000", 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6260000000000003e-06 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.954404+0000", 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.008122773 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.954469+0000", 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4986000000000004e-05 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136539 11.8 11:1b96f922:::benchmark_data_smithi086_226760_object9218:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:07:13.371 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.308921785999999, 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.00353812, 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136539 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949518+0000", 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.83e-06 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949525+0000", 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4980000000000003e-06 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949529+0000", 2024-06-14T23:07:13.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8340000000000001e-06 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949580+0000", 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1199000000000002e-05 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949590+0000", 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9499999999999996e-06 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.952967+0000", 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.003376877 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953054+0000", 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7299000000000006e-05 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136542 11.8 11:118834dc:::benchmark_data_smithi086_226760_object9221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.949549+0000", 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.308889215000001, 2024-06-14T23:07:13.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.013177585, 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136542 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949549+0000", 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949549+0000", 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949548+0000", 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949551+0000", 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9459999999999998e-06 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949557+0000", 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6400000000000002e-06 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949560+0000", 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6849999999999999e-06 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949608+0000", 2024-06-14T23:07:13.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7905000000000003e-05 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949613+0000", 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5349999999999998e-06 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953079+0000", 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0034659319999999 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953172+0000", 2024-06-14T23:07:13.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3556999999999999e-05 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962668+0000", 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094954789999999994 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962726+0000", 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8313999999999998e-05 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136550 11.a 11:5c3c9776:::benchmark_data_smithi086_226760_object9229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.308862404999999, 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0136263190000001, 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136550 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949577+0000", 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7889999999999999e-06 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949583+0000", 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3689999999999998e-06 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949586+0000", 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3380000000000001e-06 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.950035+0000", 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044843299999999998 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.950042+0000", 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.922e-06 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.952960+0000", 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.002918226 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953096+0000", 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000136355 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953224+0000", 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000128019 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962228+0000", 2024-06-14T23:07:13.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090035859999999992 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963063+0000", 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000835453 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963118+0000", 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5297000000000003e-05 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963202+0000", 2024-06-14T23:07:13.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3250000000000004e-05 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136554 11.8 11:1bd3cba3:::benchmark_data_smithi086_226760_object9233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.957712+0000", 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.300725651, 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.004780947, 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:07:13.388 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136554 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957712+0000", 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957712+0000", 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957710+0000", 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957717+0000", 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0480000000000001e-06 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957731+0000", 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3701e-05 2024-06-14T23:07:13.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957738+0000", 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4860000000000002e-06 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957791+0000", 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3304999999999999e-05 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957799+0000", 2024-06-14T23:07:13.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9039999999999992e-06 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953252+0000", 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99545359899999997 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953371+0000", 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118045 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953468+0000", 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7633000000000005e-05 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957490+0000", 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040219909999999999 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962312+0000", 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048222350000000002 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962399+0000", 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6630000000000002e-05 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962493+0000", 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4086999999999996e-05 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136561 11.a 11:5eddd569:::benchmark_data_smithi086_226760_object9240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.966732+0000", 2024-06-14T23:07:13.394 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.291705657, 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0042532740000001, 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136561 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966732+0000", 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966732+0000", 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966731+0000", 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966736+0000", 2024-06-14T23:07:13.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0629999999999996e-06 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966747+0000", 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0335e-05 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966751+0000", 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0840000000000002e-06 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966802+0000", 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0970999999999997e-05 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966812+0000", 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9860000000000006e-06 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957486+0000", 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99067426400000003 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957592+0000", 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105921 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957699+0000", 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106839 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962260+0000", 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045612719999999999 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.970899+0000", 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086394460000000003 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.970938+0000", 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9079000000000003e-05 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.970985+0000", 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6937999999999997e-05 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.400 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:13.401 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:13.401 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:07:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:13.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:13.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:13 smithi137 ceph-mon[90634]: 1.11 scrub starts 2024-06-14T23:07:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:13 smithi137 ceph-mon[90634]: 1.11 scrub ok 2024-06-14T23:07:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:13 smithi137 ceph-mon[90634]: pgmap v1017: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 8.3 KiB/s rd, 22 KiB/s wr, 941 op/s; 54724/327258 objects degraded (16.722%); 1/327258 objects misplaced (0.000%); 16 KiB/s, 0 objects/s recovering 2024-06-14T23:07:13.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:13.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:13.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:13.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:13.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:13.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:13.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:13.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[97913]: 1.11 scrub starts 2024-06-14T23:07:13.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[97913]: 1.11 scrub ok 2024-06-14T23:07:13.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:13.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:13.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:13.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[97913]: pgmap v1017: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 8.3 KiB/s rd, 22 KiB/s wr, 941 op/s; 54724/327258 objects degraded (16.722%); 1/327258 objects misplaced (0.000%); 16 KiB/s, 0 objects/s recovering 2024-06-14T23:07:13.749 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[102754]: 1.11 scrub starts 2024-06-14T23:07:13.749 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[102754]: 1.11 scrub ok 2024-06-14T23:07:13.749 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:13.749 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:13.749 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:13.749 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:13 smithi086 ceph-mon[102754]: pgmap v1017: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 8.3 KiB/s rd, 22 KiB/s wr, 941 op/s; 54724/327258 objects degraded (16.722%); 1/327258 objects misplaced (0.000%); 16 KiB/s, 0 objects/s recovering 2024-06-14T23:07:13.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:07:13.788 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:07:13.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:13.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:13.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:13.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:13.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:14.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:14.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:14.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:14.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:14.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:14.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:14.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:14.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:14.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:14.394 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:14.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:14.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:14.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:14.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-14T23:07:14.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:14.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:14.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:14.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:14.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:14.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:14.803 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:07:14.861 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:07:14.871 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[102754]: 1.b scrub starts 2024-06-14T23:07:14.872 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[102754]: 1.b scrub ok 2024-06-14T23:07:14.872 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[102754]: 1.10 scrub starts 2024-06-14T23:07:14.872 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[102754]: 1.10 scrub ok 2024-06-14T23:07:14.872 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4036230999' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:07:14.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[97913]: 1.b scrub starts 2024-06-14T23:07:14.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[97913]: 1.b scrub ok 2024-06-14T23:07:14.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[97913]: 1.10 scrub starts 2024-06-14T23:07:14.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[97913]: 1.10 scrub ok 2024-06-14T23:07:14.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4036230999' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:07:14.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:14.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:14.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:15.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:15.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:15.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:15.141 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:14 smithi137 ceph-mon[90634]: 1.b scrub starts 2024-06-14T23:07:15.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:14 smithi137 ceph-mon[90634]: 1.b scrub ok 2024-06-14T23:07:15.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:14 smithi137 ceph-mon[90634]: 1.10 scrub starts 2024-06-14T23:07:15.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:14 smithi137 ceph-mon[90634]: 1.10 scrub ok 2024-06-14T23:07:15.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4036230999' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:07:15.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:15.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:15.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:15.265 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:07:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:07:14] "GET /metrics HTTP/1.1" 200 36627 "" "Prometheus/2.43.0" 2024-06-14T23:07:15.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:15.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:15.330 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:15.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:15.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:15.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:15.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:15.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:15.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:15.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:15.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:15.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:15.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:07:15.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:15.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:15.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:15.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:15 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:07:15.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4036230999' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:07:15.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:15 smithi086 ceph-mon[97913]: osdmap e267: 8 total, 7 up, 8 in 2024-06-14T23:07:15.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:15 smithi086 ceph-mon[97913]: pgmap v1019: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 53143/320532 objects degraded (16.580%); 1/320532 objects misplaced (0.000%) 2024-06-14T23:07:15.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:15 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:07:15.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4036230999' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:07:15.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:15 smithi086 ceph-mon[102754]: osdmap e267: 8 total, 7 up, 8 in 2024-06-14T23:07:15.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:15 smithi086 ceph-mon[102754]: pgmap v1019: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 53143/320532 objects degraded (16.580%); 1/320532 objects misplaced (0.000%) 2024-06-14T23:07:15.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:15.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:15.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:16.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:16.097 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:15 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:07:16.097 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4036230999' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:07:16.098 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:15 smithi137 ceph-mon[90634]: osdmap e267: 8 total, 7 up, 8 in 2024-06-14T23:07:16.098 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:15 smithi137 ceph-mon[90634]: pgmap v1019: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 53143/320532 objects degraded (16.580%); 1/320532 objects misplaced (0.000%) 2024-06-14T23:07:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:16.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:16.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:16.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:16.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:16.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:16.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:16.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:16.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:16.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:16.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:16.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:16.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:16.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:16.699 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:07:16.699 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:07:16.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.699 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:152233 11.3 11:c2b70c0f:::benchmark_data_smithi086_226760_object24912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-06-14T23:07:16.699 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:16.680527+0000", 2024-06-14T23:07:16.699 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.016428406999999999, 2024-06-14T23:07:16.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016449882999999998, 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 152233 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.680527+0000", 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.680527+0000", 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.680525+0000", 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.680533+0000", 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6389999999999992e-06 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.680547+0000", 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4535000000000001e-05 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.680553+0000", 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9880000000000001e-06 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.680608+0000", 2024-06-14T23:07:16.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4320000000000002e-05 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.680742+0000", 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013468700000000001 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1", 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.680869+0000", 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126539 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.685740+0000", 2024-06-14T23:07:16.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00487105 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:152235 11.3 11:c75204ba:::benchmark_data_smithi086_226760_object24914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:16.683333+0000", 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.013622336000000001, 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013693515, 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:07:16.704 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 152235 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.683333+0000", 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.683333+0000", 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.683332+0000", 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.683337+0000", 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2619999999999998e-06 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.683348+0000", 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0713999999999999e-05 2024-06-14T23:07:16.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.683352+0000", 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2030000000000002e-06 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.683563+0000", 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021077699999999999 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.683652+0000", 2024-06-14T23:07:16.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8774000000000001e-05 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1", 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.683729+0000", 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7764999999999998e-05 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.685871+0000", 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002141282 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:16.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36237.0:152241 11.f e267/237 11:f8a54349:::benchmark_data_smithi086_226760_object24920:head v 267'9383, mlcod=267'9381)", 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:16.695343+0000", 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0016123000000000001, 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001735784, 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.695343+0000", 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.695343+0000", 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.695342+0000", 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.695348+0000", 2024-06-14T23:07:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6609999999999997e-06 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.695358+0000", 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0249e-05 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.695363+0000", 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6149999999999999e-06 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.695402+0000", 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9354999999999998e-05 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.695417+0000", 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4533e-05 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:152223 11.3 11:ca61cf28:::benchmark_data_smithi086_226760_object24902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:16.670696+0000", 2024-06-14T23:07:16.712 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.02625955, 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026417676000000001, 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 152223 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.670696+0000", 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.670696+0000", 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.670695+0000", 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.670699+0000", 2024-06-14T23:07:16.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0149999999999997e-06 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.670709+0000", 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0144e-05 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.670715+0000", 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5790000000000003e-06 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.670746+0000", 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1272000000000001e-05 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.670833+0000", 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7169000000000005e-05 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1", 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.670908+0000", 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5337000000000001e-05 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:07:16.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:16.673719+0000", 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028103500000000001 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 4 2024-06-14T23:07:16.717 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:07:16.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:16.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:16.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:16.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:16.899 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:16.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:16.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:16 smithi137 ceph-mon[90634]: 1.12 deep-scrub starts 2024-06-14T23:07:16.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:16 smithi137 ceph-mon[90634]: 1.12 deep-scrub ok 2024-06-14T23:07:17.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:17.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:17.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:16 smithi086 ceph-mon[97913]: 1.12 deep-scrub starts 2024-06-14T23:07:17.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:16 smithi086 ceph-mon[97913]: 1.12 deep-scrub ok 2024-06-14T23:07:17.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:16 smithi086 ceph-mon[102754]: 1.12 deep-scrub starts 2024-06-14T23:07:17.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:16 smithi086 ceph-mon[102754]: 1.12 deep-scrub ok 2024-06-14T23:07:17.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:17.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:17.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:17.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:17.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:17.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:17.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:17.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:17.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-14T23:07:17.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:17.581 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:17.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:17.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:17.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:17.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:17.805 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:07:17.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:17.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:17.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:17.853 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:07:17.855 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:07:17.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:17.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:17.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:18.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:18.076 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:17 smithi137 ceph-mon[90634]: 1.c deep-scrub starts 2024-06-14T23:07:18.076 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:17 smithi137 ceph-mon[90634]: 1.c deep-scrub ok 2024-06-14T23:07:18.076 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:17 smithi137 ceph-mon[90634]: pgmap v1020: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 52237/312465 objects degraded (16.718%); 1/312465 objects misplaced (0.000%) 2024-06-14T23:07:18.076 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:17 smithi137 ceph-mon[90634]: 1.f scrub starts 2024-06-14T23:07:18.076 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:17 smithi137 ceph-mon[90634]: 1.f scrub ok 2024-06-14T23:07:18.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[102754]: 1.c deep-scrub starts 2024-06-14T23:07:18.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[102754]: 1.c deep-scrub ok 2024-06-14T23:07:18.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[102754]: pgmap v1020: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 52237/312465 objects degraded (16.718%); 1/312465 objects misplaced (0.000%) 2024-06-14T23:07:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[102754]: 1.f scrub starts 2024-06-14T23:07:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[102754]: 1.f scrub ok 2024-06-14T23:07:18.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[97913]: 1.c deep-scrub starts 2024-06-14T23:07:18.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[97913]: 1.c deep-scrub ok 2024-06-14T23:07:18.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[97913]: pgmap v1020: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 52237/312465 objects degraded (16.718%); 1/312465 objects misplaced (0.000%) 2024-06-14T23:07:18.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[97913]: 1.f scrub starts 2024-06-14T23:07:18.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:17 smithi086 ceph-mon[97913]: 1.f scrub ok 2024-06-14T23:07:18.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:18.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:18.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:18.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:18.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:18.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:18.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:18.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:18.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:18.517 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:18.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:18.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:18.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:18.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:18.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:18.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:18.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:18.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:07:18.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:18.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:18.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:18.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:19.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:19.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:19.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:19.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 52237/312465 objects degraded (16.718%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:19.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:18 smithi086 ceph-mon[97913]: from='client.36348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:07:19.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 52237/312465 objects degraded (16.718%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:19.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:18 smithi086 ceph-mon[102754]: from='client.36348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:07:19.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:19.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:19.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:19.202 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1020,"stamp":"2024-06-14T23:07:16.852119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6824721666,"num_objects":104155,"num_object_clones":0,"num_object_copies":312465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52237,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":104155,"num_whiteouts":0,"num_read":67,"num_read_kb":125,"num_write":150601,"num_write_kb":8148712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1160173,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123633,"ondisk_log_size":123633,"up":283,"acting":287,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":333,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":26536172,"kb_used_data":20626532,"kb_used_omap":29,"kb_used_meta":5909474,"kb_avail":723425044,"statfs":{"total":767960285184,"available":740787245056,"internally_reserved":0,"allocated":21121568768,"data_stored":21111135920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30043,"internal_metadata":6051302053},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,10,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-653940053,"num_objects":-9975,"num_object_clones":0,"num_object_copies":-29925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9975,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.893821"},"pg_stats":[{"pgid":"1.11","version":"77'161","reported_seq":641,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:12.090163+0000","last_change":"2024-06-14T23:07:12.090120+0000","last_active":"2024-06-14T23:07:12.090163+0000","last_peered":"2024-06-14T23:07:12.090163+0000","last_clean":"2024-06-14T23:07:12.090163+0000","last_became_active":"2024-06-14T23:07:00.939305+0000","last_became_peered":"2024-06-14T23:07:00.939305+0000","last_unstale":"2024-06-14T23:07:12.090163+0000","last_undegraded":"2024-06-14T23:07:12.090163+0000","last_fullsized":"2024-06-14T23:07:12.090163+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:15:19.613254+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":199,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.972120+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:07:06.972120+0000","last_peered":"2024-06-14T23:07:06.972120+0000","last_clean":"2024-06-14T23:07:06.972120+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:07:06.972120+0000","last_undegraded":"2024-06-14T23:07:06.972120+0000","last_fullsized":"2024-06-14T23:07:06.972120+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":261,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.931492+0000","last_change":"2024-06-14T23:04:34.895560+0000","last_active":"2024-06-14T23:07:01.931492+0000","last_peered":"2024-06-14T23:07:01.931492+0000","last_clean":"2024-06-14T23:07:01.931492+0000","last_became_active":"2024-06-14T23:04:34.887055+0000","last_became_peered":"2024-06-14T23:04:34.887055+0000","last_unstale":"2024-06-14T23:07:01.931492+0000","last_undegraded":"2024-06-14T23:07:01.931492+0000","last_fullsized":"2024-06-14T23:07:01.931492+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:26:36.259241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":573,"reported_epoch":266,"state":"active+clean+remapped","last_fresh":"2024-06-14T23:07:14.072620+0000","last_change":"2024-06-14T23:07:14.072561+0000","last_active":"2024-06-14T23:07:14.072620+0000","last_peered":"2024-06-14T23:07:14.072620+0000","last_clean":"2024-06-14T23:07:14.072620+0000","last_became_active":"2024-06-14T23:07:01.937723+0000","last_became_peered":"2024-06-14T23:07:01.937723+0000","last_unstale":"2024-06-14T23:07:14.072620+0000","last_undegraded":"2024-06-14T23:07:14.072620+0000","last_fullsized":"2024-06-14T23:07:14.072620+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:59.508626+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,0],"avail_no_missing":["5","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":226,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.933130+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:07:01.933130+0000","last_peered":"2024-06-14T23:07:01.933130+0000","last_clean":"2024-06-14T23:07:01.933130+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:07:01.933130+0000","last_undegraded":"2024-06-14T23:07:01.933130+0000","last_fullsized":"2024-06-14T23:07:01.933130+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":220,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.939087+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:07:01.939087+0000","last_peered":"2024-06-14T23:07:01.939087+0000","last_clean":"2024-06-14T23:07:01.939087+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:07:01.939087+0000","last_undegraded":"2024-06-14T23:07:01.939087+0000","last_fullsized":"2024-06-14T23:07:01.939087+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":606,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:09.705052+0000","last_change":"2024-06-14T23:07:09.705052+0000","last_active":"2024-06-14T23:07:09.705052+0000","last_peered":"2024-06-14T23:07:09.705052+0000","last_clean":"2024-06-14T23:07:09.705052+0000","last_became_active":"2024-06-14T23:07:00.938267+0000","last_became_peered":"2024-06-14T23:07:00.938267+0000","last_unstale":"2024-06-14T23:07:09.705052+0000","last_undegraded":"2024-06-14T23:07:09.705052+0000","last_fullsized":"2024-06-14T23:07:09.705052+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.704172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.704172+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:58:42.258413+0000","scrub_duration":0.0024350880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":104,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.967809+0000","last_change":"2024-06-14T23:06:18.999778+0000","last_active":"2024-06-14T23:07:06.967809+0000","last_peered":"2024-06-14T23:07:06.967809+0000","last_clean":"2024-06-14T23:07:06.967809+0000","last_became_active":"2024-06-14T23:06:18.999306+0000","last_became_peered":"2024-06-14T23:06:18.999306+0000","last_unstale":"2024-06-14T23:07:06.967809+0000","last_undegraded":"2024-06-14T23:07:06.967809+0000","last_fullsized":"2024-06-14T23:07:06.967809+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:18:01.226932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":140,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.933854+0000","last_change":"2024-06-14T23:07:00.940001+0000","last_active":"2024-06-14T23:07:01.933854+0000","last_peered":"2024-06-14T23:07:01.933854+0000","last_clean":"2024-06-14T23:06:58.913784+0000","last_became_active":"2024-06-14T23:07:00.940001+0000","last_became_peered":"2024-06-14T23:07:00.940001+0000","last_unstale":"2024-06-14T23:07:01.933854+0000","last_undegraded":"2024-06-14T23:07:01.933854+0000","last_fullsized":"2024-06-14T23:07:00.933148+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:30:48.706210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":579,"reported_epoch":265,"state":"active+clean+remapped","last_fresh":"2024-06-14T23:07:01.963056+0000","last_change":"2024-06-14T23:07:01.962710+0000","last_active":"2024-06-14T23:07:01.963056+0000","last_peered":"2024-06-14T23:07:01.963056+0000","last_clean":"2024-06-14T23:07:01.963056+0000","last_became_active":"2024-06-14T23:07:01.962124+0000","last_became_peered":"2024-06-14T23:07:01.962124+0000","last_unstale":"2024-06-14T23:07:01.963056+0000","last_undegraded":"2024-06-14T23:07:01.963056+0000","last_fullsized":"2024-06-14T23:07:01.963056+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:06:19.887856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:06:19.887856+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001647253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,0],"avail_no_missing":["5","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":223,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.967411+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:07:06.967411+0000","last_peered":"2024-06-14T23:07:06.967411+0000","last_clean":"2024-06-14T23:07:06.967411+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:07:06.967411+0000","last_undegraded":"2024-06-14T23:07:06.967411+0000","last_fullsized":"2024-06-14T23:07:06.967411+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":23,"reported_epoch":265,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:01.932814+0000","last_change":"2024-06-14T23:07:00.933275+0000","last_active":"2024-06-14T23:07:01.932814+0000","last_peered":"2024-06-14T23:07:01.932814+0000","last_clean":"2024-06-14T22:34:48.214958+0000","last_became_active":"2024-06-14T23:07:00.933275+0000","last_became_peered":"2024-06-14T23:07:00.933275+0000","last_unstale":"2024-06-14T23:07:01.932814+0000","last_undegraded":"2024-06-14T23:07:00.930110+0000","last_fullsized":"2024-06-14T23:07:00.930006+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:56:11.005533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":632,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934500+0000","last_change":"2024-06-14T23:07:00.948142+0000","last_active":"2024-06-14T23:07:01.934500+0000","last_peered":"2024-06-14T23:07:01.934500+0000","last_clean":"2024-06-14T23:07:01.934500+0000","last_became_active":"2024-06-14T23:07:00.947390+0000","last_became_peered":"2024-06-14T23:07:00.947390+0000","last_unstale":"2024-06-14T23:07:01.934500+0000","last_undegraded":"2024-06-14T23:07:01.934500+0000","last_fullsized":"2024-06-14T23:07:01.934500+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.174132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.174132+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"266'9442","reported_seq":9477,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:14.540857+0000","last_change":"2024-06-14T23:05:03.621489+0000","last_active":"2024-06-14T23:07:14.540857+0000","last_peered":"2024-06-14T23:07:14.540857+0000","last_clean":"2024-06-14T23:07:14.540857+0000","last_became_active":"2024-06-14T23:05:03.621114+0000","last_became_peered":"2024-06-14T23:05:03.621114+0000","last_unstale":"2024-06-14T23:07:14.540857+0000","last_undegraded":"2024-06-14T23:07:14.540857+0000","last_fullsized":"2024-06-14T23:07:14.540857+0000","mapping_epoch":237,"log_start":"243'1020","ondisk_log_start":"243'1020","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":8422,"log_dups_size":0,"ondisk_log_size":8422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:33.811591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428605440,"num_objects":6540,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9442,"num_write_kb":511424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":216,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.931901+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:07:01.931901+0000","last_peered":"2024-06-14T23:07:01.931901+0000","last_clean":"2024-06-14T23:07:01.931901+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:07:01.931901+0000","last_undegraded":"2024-06-14T23:07:01.931901+0000","last_fullsized":"2024-06-14T23:07:01.931901+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":216,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.932977+0000","last_change":"2024-06-14T23:07:00.957583+0000","last_active":"2024-06-14T23:07:01.932977+0000","last_peered":"2024-06-14T23:07:01.932977+0000","last_clean":"2024-06-14T23:06:58.923677+0000","last_became_active":"2024-06-14T23:07:00.957583+0000","last_became_peered":"2024-06-14T23:07:00.957583+0000","last_unstale":"2024-06-14T23:07:01.932977+0000","last_undegraded":"2024-06-14T23:07:01.932977+0000","last_fullsized":"2024-06-14T23:07:00.930331+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:57:26.296990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"77'161","reported_seq":601,"reported_epoch":266,"state":"active+clean+remapped","last_fresh":"2024-06-14T23:07:06.968319+0000","last_change":"2024-06-14T23:07:01.966264+0000","last_active":"2024-06-14T23:07:06.968319+0000","last_peered":"2024-06-14T23:07:06.968319+0000","last_clean":"2024-06-14T23:07:06.968319+0000","last_became_active":"2024-06-14T23:07:01.965788+0000","last_became_peered":"2024-06-14T23:07:01.965788+0000","last_unstale":"2024-06-14T23:07:06.968319+0000","last_undegraded":"2024-06-14T23:07:06.968319+0000","last_fullsized":"2024-06-14T23:07:06.968319+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.715907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.715907+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,5],"avail_no_missing":["3","0","5"],"object_location_counts":[{"shards":"0,3,5","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"267'9521","reported_seq":9573,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-06-14T23:07:15.602051+0000","last_change":"2024-06-14T23:06:19.064148+0000","last_active":"2024-06-14T23:07:15.602051+0000","last_peered":"2024-06-14T23:07:15.602051+0000","last_clean":"2024-06-14T23:07:15.602051+0000","last_became_active":"2024-06-14T23:06:19.007544+0000","last_became_peered":"2024-06-14T23:06:19.007544+0000","last_unstale":"2024-06-14T23:07:15.602051+0000","last_undegraded":"2024-06-14T23:07:15.602051+0000","last_fullsized":"2024-06-14T23:07:15.602051+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":9521,"log_dups_size":0,"ondisk_log_size":9521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:47.143283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424869888,"num_objects":6483,"num_object_clones":0,"num_object_copies":19449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9521,"num_write_kb":512128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":217,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.931838+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:07:01.931838+0000","last_peered":"2024-06-14T23:07:01.931838+0000","last_clean":"2024-06-14T23:07:01.931838+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:07:01.931838+0000","last_undegraded":"2024-06-14T23:07:01.931838+0000","last_fullsized":"2024-06-14T23:07:01.931838+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":243,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934076+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:07:01.934076+0000","last_peered":"2024-06-14T23:07:01.934076+0000","last_clean":"2024-06-14T23:07:01.934076+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:07:01.934076+0000","last_undegraded":"2024-06-14T23:07:01.934076+0000","last_fullsized":"2024-06-14T23:07:01.934076+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":542,"reported_epoch":265,"state":"active+clean+remapped","last_fresh":"2024-06-14T23:07:01.966387+0000","last_change":"2024-06-14T23:07:01.965579+0000","last_active":"2024-06-14T23:07:01.966387+0000","last_peered":"2024-06-14T23:07:01.966387+0000","last_clean":"2024-06-14T23:07:01.966387+0000","last_became_active":"2024-06-14T23:07:01.965031+0000","last_became_peered":"2024-06-14T23:07:01.965031+0000","last_unstale":"2024-06-14T23:07:01.966387+0000","last_undegraded":"2024-06-14T23:07:01.966387+0000","last_fullsized":"2024-06-14T23:07:01.966387+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053445919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,3],"avail_no_missing":["5","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"266'9457","reported_seq":9500,"reported_epoch":266,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:14.524945+0000","last_change":"2024-06-14T23:07:00.941000+0000","last_active":"2024-06-14T23:07:14.524945+0000","last_peered":"2024-06-14T23:07:14.524945+0000","last_clean":"2024-06-14T23:06:59.932215+0000","last_became_active":"2024-06-14T23:07:00.941000+0000","last_became_peered":"2024-06-14T23:07:00.941000+0000","last_unstale":"2024-06-14T23:07:14.524945+0000","last_undegraded":"2024-06-14T23:07:00.929008+0000","last_fullsized":"2024-06-14T23:07:00.928892+0000","mapping_epoch":263,"log_start":"243'1031","ondisk_log_start":"243'1031","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":8426,"log_dups_size":0,"ondisk_log_size":8426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:20:49.588484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431030272,"num_objects":6577,"num_object_clones":0,"num_object_copies":19731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9457,"num_write_kb":513088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":6577}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":151,"reported_epoch":265,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:01.932800+0000","last_change":"2024-06-14T23:07:00.954691+0000","last_active":"2024-06-14T23:07:01.932800+0000","last_peered":"2024-06-14T23:07:01.932800+0000","last_clean":"2024-06-14T23:06:58.919824+0000","last_became_active":"2024-06-14T23:07:00.954691+0000","last_became_peered":"2024-06-14T23:07:00.954691+0000","last_unstale":"2024-06-14T23:07:01.932800+0000","last_undegraded":"2024-06-14T23:07:00.929244+0000","last_fullsized":"2024-06-14T23:07:00.929131+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:21:34.661440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":80,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:06.970580+0000","last_change":"2024-06-14T23:07:00.936668+0000","last_active":"2024-06-14T23:07:06.970580+0000","last_peered":"2024-06-14T23:07:06.970580+0000","last_clean":"2024-06-14T22:48:59.842731+0000","last_became_active":"2024-06-14T23:07:00.936668+0000","last_became_peered":"2024-06-14T23:07:00.936668+0000","last_unstale":"2024-06-14T23:07:06.970580+0000","last_undegraded":"2024-06-14T23:07:06.970580+0000","last_fullsized":"2024-06-14T23:07:00.929716+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:04:45.947150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":474,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934128+0000","last_change":"2024-06-14T23:07:00.943849+0000","last_active":"2024-06-14T23:07:01.934128+0000","last_peered":"2024-06-14T23:07:01.934128+0000","last_clean":"2024-06-14T23:07:01.934128+0000","last_became_active":"2024-06-14T23:07:00.943273+0000","last_became_peered":"2024-06-14T23:07:00.943273+0000","last_unstale":"2024-06-14T23:07:01.934128+0000","last_undegraded":"2024-06-14T23:07:01.934128+0000","last_fullsized":"2024-06-14T23:07:01.934128+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":42,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"266'9503","reported_seq":9550,"reported_epoch":266,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:14.177278+0000","last_change":"2024-06-14T23:07:00.981362+0000","last_active":"2024-06-14T23:07:14.177278+0000","last_peered":"2024-06-14T23:07:14.177278+0000","last_clean":"2024-06-14T23:06:59.927883+0000","last_became_active":"2024-06-14T23:07:00.953063+0000","last_became_peered":"2024-06-14T23:07:00.953063+0000","last_unstale":"2024-06-14T23:07:14.177278+0000","last_undegraded":"2024-06-14T23:07:00.939999+0000","last_fullsized":"2024-06-14T23:07:00.931660+0000","mapping_epoch":263,"log_start":"244'2624","ondisk_log_start":"244'2624","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6879,"log_dups_size":0,"ondisk_log_size":6879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:06.399859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435093504,"num_objects":6639,"num_object_clones":0,"num_object_copies":19917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9503,"num_write_kb":516544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":6639}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":234,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.932347+0000","last_change":"2024-06-14T23:07:00.935368+0000","last_active":"2024-06-14T23:07:01.932347+0000","last_peered":"2024-06-14T23:07:01.932347+0000","last_clean":"2024-06-14T23:06:58.919985+0000","last_became_active":"2024-06-14T23:07:00.935368+0000","last_became_peered":"2024-06-14T23:07:00.935368+0000","last_unstale":"2024-06-14T23:07:01.932347+0000","last_undegraded":"2024-06-14T23:07:01.932347+0000","last_fullsized":"2024-06-14T23:07:00.929476+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:09:24.494264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":81,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.974107+0000","last_change":"2024-06-14T23:06:19.002672+0000","last_active":"2024-06-14T23:07:06.974107+0000","last_peered":"2024-06-14T23:07:06.974107+0000","last_clean":"2024-06-14T23:07:06.974107+0000","last_became_active":"2024-06-14T23:06:19.002175+0000","last_became_peered":"2024-06-14T23:06:19.002175+0000","last_unstale":"2024-06-14T23:07:06.974107+0000","last_undegraded":"2024-06-14T23:07:06.974107+0000","last_fullsized":"2024-06-14T23:07:06.974107+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:50.841812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"266'9226","reported_seq":9268,"reported_epoch":266,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:14.182867+0000","last_change":"2024-06-14T23:07:00.935046+0000","last_active":"2024-06-14T23:07:14.182867+0000","last_peered":"2024-06-14T23:07:14.182867+0000","last_clean":"2024-06-14T23:06:59.881114+0000","last_became_active":"2024-06-14T23:07:00.935046+0000","last_became_peered":"2024-06-14T23:07:00.935046+0000","last_unstale":"2024-06-14T23:07:14.182867+0000","last_undegraded":"2024-06-14T23:07:00.930260+0000","last_fullsized":"2024-06-14T23:07:00.930112+0000","mapping_epoch":263,"log_start":"244'2314","ondisk_log_start":"244'2314","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:22:09.319209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426639360,"num_objects":6510,"num_object_clones":0,"num_object_copies":19530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6510,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":503552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":6510}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":520,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.974808+0000","last_change":"2024-06-14T23:04:40.403447+0000","last_active":"2024-06-14T23:07:06.974808+0000","last_peered":"2024-06-14T23:07:06.974808+0000","last_clean":"2024-06-14T23:07:06.974808+0000","last_became_active":"2024-06-14T23:04:37.904456+0000","last_became_peered":"2024-06-14T23:04:37.904456+0000","last_unstale":"2024-06-14T23:07:06.974808+0000","last_undegraded":"2024-06-14T23:07:06.974808+0000","last_fullsized":"2024-06-14T23:07:06.974808+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:04:40.403365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:04:40.403365+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:45.174359+0000","scrub_duration":0.0014330689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":36,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934078+0000","last_change":"2024-06-14T23:06:19.002737+0000","last_active":"2024-06-14T23:07:01.934078+0000","last_peered":"2024-06-14T23:07:01.934078+0000","last_clean":"2024-06-14T23:07:01.934078+0000","last_became_active":"2024-06-14T23:06:19.002078+0000","last_became_peered":"2024-06-14T23:06:19.002078+0000","last_unstale":"2024-06-14T23:07:01.934078+0000","last_undegraded":"2024-06-14T23:07:01.934078+0000","last_fullsized":"2024-06-14T23:07:01.934078+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:00:16.317592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":233,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.934419+0000","last_change":"2024-06-14T23:07:00.939535+0000","last_active":"2024-06-14T23:07:01.934419+0000","last_peered":"2024-06-14T23:07:01.934419+0000","last_clean":"2024-06-14T23:06:58.915024+0000","last_became_active":"2024-06-14T23:07:00.939535+0000","last_became_peered":"2024-06-14T23:07:00.939535+0000","last_unstale":"2024-06-14T23:07:01.934419+0000","last_undegraded":"2024-06-14T23:07:01.934419+0000","last_fullsized":"2024-06-14T23:07:00.937252+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:32:07.563309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"267'9222","reported_seq":9262,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-06-14T23:07:15.612443+0000","last_change":"2024-06-14T23:05:03.807700+0000","last_active":"2024-06-14T23:07:15.612443+0000","last_peered":"2024-06-14T23:07:15.612443+0000","last_clean":"2024-06-14T23:07:15.612443+0000","last_became_active":"2024-06-14T23:05:03.807400+0000","last_became_peered":"2024-06-14T23:05:03.807400+0000","last_unstale":"2024-06-14T23:07:15.612443+0000","last_undegraded":"2024-06-14T23:07:15.612443+0000","last_fullsized":"2024-06-14T23:07:15.612443+0000","mapping_epoch":237,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:56:35.097613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423755776,"num_objects":6466,"num_object_clones":0,"num_object_copies":19398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":502016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":611,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.932710+0000","last_change":"2024-06-14T23:04:37.902544+0000","last_active":"2024-06-14T23:07:01.932710+0000","last_peered":"2024-06-14T23:07:01.932710+0000","last_clean":"2024-06-14T23:07:01.932710+0000","last_became_active":"2024-06-14T23:04:37.902191+0000","last_became_peered":"2024-06-14T23:04:37.902191+0000","last_unstale":"2024-06-14T23:07:01.932710+0000","last_undegraded":"2024-06-14T23:07:01.932710+0000","last_fullsized":"2024-06-14T23:07:01.932710+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:05:09.672062+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":114,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.934061+0000","last_change":"2024-06-14T23:07:00.937838+0000","last_active":"2024-06-14T23:07:01.934061+0000","last_peered":"2024-06-14T23:07:01.934061+0000","last_clean":"2024-06-14T23:06:58.914183+0000","last_became_active":"2024-06-14T23:07:00.937838+0000","last_became_peered":"2024-06-14T23:07:00.937838+0000","last_unstale":"2024-06-14T23:07:01.934061+0000","last_undegraded":"2024-06-14T23:07:01.934061+0000","last_fullsized":"2024-06-14T23:07:00.933228+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:03:09.821123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":256,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934702+0000","last_change":"2024-06-14T23:04:34.887791+0000","last_active":"2024-06-14T23:07:01.934702+0000","last_peered":"2024-06-14T23:07:01.934702+0000","last_clean":"2024-06-14T23:07:01.934702+0000","last_became_active":"2024-06-14T23:04:34.887099+0000","last_became_peered":"2024-06-14T23:04:34.887099+0000","last_unstale":"2024-06-14T23:07:01.934702+0000","last_undegraded":"2024-06-14T23:07:01.934702+0000","last_fullsized":"2024-06-14T23:07:01.934702+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:46:07.470301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":606,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:13.730354+0000","last_change":"2024-06-14T23:07:13.730354+0000","last_active":"2024-06-14T23:07:13.730354+0000","last_peered":"2024-06-14T23:07:13.730354+0000","last_clean":"2024-06-14T23:07:13.730354+0000","last_became_active":"2024-06-14T23:07:00.933661+0000","last_became_peered":"2024-06-14T23:07:00.933661+0000","last_unstale":"2024-06-14T23:07:13.730354+0000","last_undegraded":"2024-06-14T23:07:13.730354+0000","last_fullsized":"2024-06-14T23:07:13.730354+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:13.729891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:13.729891+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:57:14.387074+0000","scrub_duration":0.015960974999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"267'9657","reported_seq":9699,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:14.844421+0000","last_change":"2024-06-14T23:07:00.943403+0000","last_active":"2024-06-14T23:07:14.844421+0000","last_peered":"2024-06-14T23:07:14.844421+0000","last_clean":"2024-06-14T23:06:59.893668+0000","last_became_active":"2024-06-14T23:07:00.943403+0000","last_became_peered":"2024-06-14T23:07:00.943403+0000","last_unstale":"2024-06-14T23:07:14.844421+0000","last_undegraded":"2024-06-14T23:07:00.936661+0000","last_fullsized":"2024-06-14T23:07:00.936560+0000","mapping_epoch":263,"log_start":"244'3346","ondisk_log_start":"244'3346","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:03:10.130662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433127424,"num_objects":6609,"num_object_clones":0,"num_object_copies":19827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":520512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":6609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":247,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934095+0000","last_change":"2024-06-14T23:04:34.896257+0000","last_active":"2024-06-14T23:07:01.934095+0000","last_peered":"2024-06-14T23:07:01.934095+0000","last_clean":"2024-06-14T23:07:01.934095+0000","last_became_active":"2024-06-14T23:04:34.895926+0000","last_became_peered":"2024-06-14T23:04:34.895926+0000","last_unstale":"2024-06-14T23:07:01.934095+0000","last_undegraded":"2024-06-14T23:07:01.934095+0000","last_fullsized":"2024-06-14T23:07:01.934095+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:17.800902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":233,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.934406+0000","last_change":"2024-06-14T23:07:00.947413+0000","last_active":"2024-06-14T23:07:01.934406+0000","last_peered":"2024-06-14T23:07:01.934406+0000","last_clean":"2024-06-14T23:06:58.915858+0000","last_became_active":"2024-06-14T23:07:00.947413+0000","last_became_peered":"2024-06-14T23:07:00.947413+0000","last_unstale":"2024-06-14T23:07:01.934406+0000","last_undegraded":"2024-06-14T23:07:01.934406+0000","last_fullsized":"2024-06-14T23:07:00.936905+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:59.530190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"267'9244","reported_seq":9284,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-06-14T23:07:15.305522+0000","last_change":"2024-06-14T23:05:03.810157+0000","last_active":"2024-06-14T23:07:15.305522+0000","last_peered":"2024-06-14T23:07:15.305522+0000","last_clean":"2024-06-14T23:07:15.305522+0000","last_became_active":"2024-06-14T23:05:03.809591+0000","last_became_peered":"2024-06-14T23:05:03.809591+0000","last_unstale":"2024-06-14T23:07:15.305522+0000","last_undegraded":"2024-06-14T23:07:15.305522+0000","last_fullsized":"2024-06-14T23:07:15.305522+0000","mapping_epoch":237,"log_start":"244'2503","ondisk_log_start":"244'2503","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6741,"log_dups_size":0,"ondisk_log_size":6741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:04:46.257184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416677888,"num_objects":6358,"num_object_clones":0,"num_object_copies":19074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9244,"num_write_kb":499264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":610,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.932766+0000","last_change":"2024-06-14T23:07:00.939087+0000","last_active":"2024-06-14T23:07:01.932766+0000","last_peered":"2024-06-14T23:07:01.932766+0000","last_clean":"2024-06-14T23:06:58.920251+0000","last_became_active":"2024-06-14T23:07:00.939087+0000","last_became_peered":"2024-06-14T23:07:00.939087+0000","last_unstale":"2024-06-14T23:07:01.932766+0000","last_undegraded":"2024-06-14T23:07:01.932766+0000","last_fullsized":"2024-06-14T23:07:00.928570+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:43:29.127962+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":225,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.969062+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:07:06.969062+0000","last_peered":"2024-06-14T23:07:06.969062+0000","last_clean":"2024-06-14T23:07:06.969062+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:07:06.969062+0000","last_undegraded":"2024-06-14T23:07:06.969062+0000","last_fullsized":"2024-06-14T23:07:06.969062+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":237,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.972791+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:07:06.972791+0000","last_peered":"2024-06-14T23:07:06.972791+0000","last_clean":"2024-06-14T23:07:06.972791+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:07:06.972791+0000","last_undegraded":"2024-06-14T23:07:06.972791+0000","last_fullsized":"2024-06-14T23:07:06.972791+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"266'9718","reported_seq":9763,"reported_epoch":266,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:14.506340+0000","last_change":"2024-06-14T23:07:00.938983+0000","last_active":"2024-06-14T23:07:14.506340+0000","last_peered":"2024-06-14T23:07:14.506340+0000","last_clean":"2024-06-14T23:06:59.923360+0000","last_became_active":"2024-06-14T23:07:00.938983+0000","last_became_peered":"2024-06-14T23:07:00.938983+0000","last_unstale":"2024-06-14T23:07:14.506340+0000","last_undegraded":"2024-06-14T23:07:00.931768+0000","last_fullsized":"2024-06-14T23:07:00.931149+0000","mapping_epoch":263,"log_start":"243'1302","ondisk_log_start":"243'1302","created":237,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":8416,"log_dups_size":0,"ondisk_log_size":8416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:32:49.066667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438304768,"num_objects":6688,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9718,"num_write_kb":524992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":6688}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":536,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.931305+0000","last_change":"2024-06-14T23:07:00.939257+0000","last_active":"2024-06-14T23:07:01.931305+0000","last_peered":"2024-06-14T23:07:01.931305+0000","last_clean":"2024-06-14T23:06:58.916706+0000","last_became_active":"2024-06-14T23:07:00.939257+0000","last_became_peered":"2024-06-14T23:07:00.939257+0000","last_unstale":"2024-06-14T23:07:01.931305+0000","last_undegraded":"2024-06-14T23:07:01.931305+0000","last_fullsized":"2024-06-14T23:07:00.934551+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:06:21.536637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:06:21.536637+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035987950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":251,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.932181+0000","last_change":"2024-06-14T23:04:34.889456+0000","last_active":"2024-06-14T23:07:01.932181+0000","last_peered":"2024-06-14T23:07:01.932181+0000","last_clean":"2024-06-14T23:07:01.932181+0000","last_became_active":"2024-06-14T23:04:34.888908+0000","last_became_peered":"2024-06-14T23:04:34.888908+0000","last_unstale":"2024-06-14T23:07:01.932181+0000","last_undegraded":"2024-06-14T23:07:01.932181+0000","last_fullsized":"2024-06-14T23:07:01.932181+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:00:24.632734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":177,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.933075+0000","last_change":"2024-06-14T23:07:00.937968+0000","last_active":"2024-06-14T23:07:01.933075+0000","last_peered":"2024-06-14T23:07:01.933075+0000","last_clean":"2024-06-14T23:06:58.920459+0000","last_became_active":"2024-06-14T23:07:00.937968+0000","last_became_peered":"2024-06-14T23:07:00.937968+0000","last_unstale":"2024-06-14T23:07:01.933075+0000","last_undegraded":"2024-06-14T23:07:01.933075+0000","last_fullsized":"2024-06-14T23:07:00.930945+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:39:21.038750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.934449+0000","last_change":"2024-06-14T23:07:00.943133+0000","last_active":"2024-06-14T23:07:01.934449+0000","last_peered":"2024-06-14T23:07:01.934449+0000","last_clean":"2024-06-14T22:34:45.793132+0000","last_became_active":"2024-06-14T23:07:00.943133+0000","last_became_peered":"2024-06-14T23:07:00.943133+0000","last_unstale":"2024-06-14T23:07:01.934449+0000","last_undegraded":"2024-06-14T23:07:01.934449+0000","last_fullsized":"2024-06-14T23:07:00.936552+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:28:30.293511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.9","version":"267'9242","reported_seq":9293,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:14.870756+0000","last_change":"2024-06-14T23:07:00.938966+0000","last_active":"2024-06-14T23:07:14.870756+0000","last_peered":"2024-06-14T23:07:14.870756+0000","last_clean":"2024-06-14T23:06:59.934530+0000","last_became_active":"2024-06-14T23:07:00.938966+0000","last_became_peered":"2024-06-14T23:07:00.938966+0000","last_unstale":"2024-06-14T23:07:14.870756+0000","last_undegraded":"2024-06-14T23:07:00.936013+0000","last_fullsized":"2024-06-14T23:07:00.935895+0000","mapping_epoch":263,"log_start":"244'2970","ondisk_log_start":"244'2970","created":237,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6272,"log_dups_size":0,"ondisk_log_size":6272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:27:26.254569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427163648,"num_objects":6518,"num_object_clones":0,"num_object_copies":19554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9242,"num_write_kb":504320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":6518}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":583,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:06.974725+0000","last_change":"2024-06-14T23:07:00.938224+0000","last_active":"2024-06-14T23:07:06.974725+0000","last_peered":"2024-06-14T23:07:06.974725+0000","last_clean":"2024-06-14T23:06:58.905584+0000","last_became_active":"2024-06-14T23:07:00.938224+0000","last_became_peered":"2024-06-14T23:07:00.938224+0000","last_unstale":"2024-06-14T23:07:06.974725+0000","last_undegraded":"2024-06-14T23:07:06.974725+0000","last_fullsized":"2024-06-14T23:07:00.931024+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":228,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.971251+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:07:06.971251+0000","last_peered":"2024-06-14T23:07:06.971251+0000","last_clean":"2024-06-14T23:07:06.971251+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:07:06.971251+0000","last_undegraded":"2024-06-14T23:07:06.971251+0000","last_fullsized":"2024-06-14T23:07:06.971251+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":104,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.933828+0000","last_change":"2024-06-14T23:06:19.001589+0000","last_active":"2024-06-14T23:07:01.933828+0000","last_peered":"2024-06-14T23:07:01.933828+0000","last_clean":"2024-06-14T23:07:01.933828+0000","last_became_active":"2024-06-14T23:06:19.001260+0000","last_became_peered":"2024-06-14T23:06:19.001260+0000","last_unstale":"2024-06-14T23:07:01.933828+0000","last_undegraded":"2024-06-14T23:07:01.933828+0000","last_fullsized":"2024-06-14T23:07:01.933828+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:54:46.316751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":232,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.932236+0000","last_change":"2024-06-14T23:06:18.998828+0000","last_active":"2024-06-14T23:07:01.932236+0000","last_peered":"2024-06-14T23:07:01.932236+0000","last_clean":"2024-06-14T23:07:01.932236+0000","last_became_active":"2024-06-14T23:06:18.998461+0000","last_became_peered":"2024-06-14T23:06:18.998461+0000","last_unstale":"2024-06-14T23:07:01.932236+0000","last_undegraded":"2024-06-14T23:07:01.932236+0000","last_fullsized":"2024-06-14T23:07:01.932236+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:18:21.826501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"266'9304","reported_seq":9351,"reported_epoch":266,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:14.545176+0000","last_change":"2024-06-14T23:07:00.963005+0000","last_active":"2024-06-14T23:07:14.545176+0000","last_peered":"2024-06-14T23:07:14.545176+0000","last_clean":"2024-06-14T23:06:59.877480+0000","last_became_active":"2024-06-14T23:07:00.937777+0000","last_became_peered":"2024-06-14T23:07:00.937777+0000","last_unstale":"2024-06-14T23:07:14.545176+0000","last_undegraded":"2024-06-14T23:07:00.930596+0000","last_fullsized":"2024-06-14T23:07:00.928418+0000","mapping_epoch":263,"log_start":"243'2097","ondisk_log_start":"243'2097","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7207,"log_dups_size":0,"ondisk_log_size":7207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:52:19.274633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416940032,"num_objects":6362,"num_object_clones":0,"num_object_copies":19086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9304,"num_write_kb":501312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":6362}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":516,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:10.877048+0000","last_change":"2024-06-14T23:07:10.877003+0000","last_active":"2024-06-14T23:07:10.877048+0000","last_peered":"2024-06-14T23:07:10.877048+0000","last_clean":"2024-06-14T23:07:10.877048+0000","last_became_active":"2024-06-14T23:06:52.047842+0000","last_became_peered":"2024-06-14T23:06:52.047842+0000","last_unstale":"2024-06-14T23:07:10.877048+0000","last_undegraded":"2024-06-14T23:07:10.877048+0000","last_fullsized":"2024-06-14T23:07:10.877048+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:09:38.064292+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.975239+0000","last_change":"2024-06-14T23:04:34.879251+0000","last_active":"2024-06-14T23:07:06.975239+0000","last_peered":"2024-06-14T23:07:06.975239+0000","last_clean":"2024-06-14T23:07:06.975239+0000","last_became_active":"2024-06-14T23:04:34.878470+0000","last_became_peered":"2024-06-14T23:04:34.878470+0000","last_unstale":"2024-06-14T23:07:06.975239+0000","last_undegraded":"2024-06-14T23:07:06.975239+0000","last_fullsized":"2024-06-14T23:07:06.975239+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:55.949794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":262,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:06.971517+0000","last_change":"2024-06-14T23:07:00.939360+0000","last_active":"2024-06-14T23:07:06.971517+0000","last_peered":"2024-06-14T23:07:06.971517+0000","last_clean":"2024-06-14T23:06:58.904480+0000","last_became_active":"2024-06-14T23:07:00.939360+0000","last_became_peered":"2024-06-14T23:07:00.939360+0000","last_unstale":"2024-06-14T23:07:06.971517+0000","last_undegraded":"2024-06-14T23:07:06.971517+0000","last_fullsized":"2024-06-14T23:07:00.931293+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:31:48.673690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":223,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.972644+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:07:06.972644+0000","last_peered":"2024-06-14T23:07:06.972644+0000","last_clean":"2024-06-14T23:07:06.972644+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:07:06.972644+0000","last_undegraded":"2024-06-14T23:07:06.972644+0000","last_fullsized":"2024-06-14T23:07:06.972644+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"267'9455","reported_seq":9500,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-06-14T23:07:14.871050+0000","last_change":"2024-06-14T23:06:04.928960+0000","last_active":"2024-06-14T23:07:14.871050+0000","last_peered":"2024-06-14T23:07:14.871050+0000","last_clean":"2024-06-14T23:07:14.871050+0000","last_became_active":"2024-06-14T23:06:04.923775+0000","last_became_peered":"2024-06-14T23:06:04.923775+0000","last_unstale":"2024-06-14T23:07:14.871050+0000","last_undegraded":"2024-06-14T23:07:14.871050+0000","last_fullsized":"2024-06-14T23:07:14.871050+0000","mapping_epoch":250,"log_start":"244'3152","ondisk_log_start":"244'3152","created":237,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:10:46.076558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432209920,"num_objects":6595,"num_object_clones":0,"num_object_copies":19785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9455,"num_write_kb":513600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":622,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:11.050787+0000","last_change":"2024-06-14T23:07:11.050751+0000","last_active":"2024-06-14T23:07:11.050787+0000","last_peered":"2024-06-14T23:07:11.050787+0000","last_clean":"2024-06-14T23:07:11.050787+0000","last_became_active":"2024-06-14T23:06:52.051996+0000","last_became_peered":"2024-06-14T23:06:52.051996+0000","last_unstale":"2024-06-14T23:07:11.050787+0000","last_undegraded":"2024-06-14T23:07:11.050787+0000","last_fullsized":"2024-06-14T23:07:11.050787+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:52:38.070062+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":238,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.933173+0000","last_change":"2024-06-14T23:07:00.937465+0000","last_active":"2024-06-14T23:07:01.933173+0000","last_peered":"2024-06-14T23:07:01.933173+0000","last_clean":"2024-06-14T23:06:58.920522+0000","last_became_active":"2024-06-14T23:07:00.937465+0000","last_became_peered":"2024-06-14T23:07:00.937465+0000","last_unstale":"2024-06-14T23:07:01.933173+0000","last_undegraded":"2024-06-14T23:07:01.933173+0000","last_fullsized":"2024-06-14T23:07:00.931225+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:33:25.682405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":234,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.934723+0000","last_change":"2024-06-14T23:07:00.937556+0000","last_active":"2024-06-14T23:07:01.934723+0000","last_peered":"2024-06-14T23:07:01.934723+0000","last_clean":"2024-06-14T23:06:58.915309+0000","last_became_active":"2024-06-14T23:07:00.937556+0000","last_became_peered":"2024-06-14T23:07:00.937556+0000","last_unstale":"2024-06-14T23:07:01.934723+0000","last_undegraded":"2024-06-14T23:07:01.934723+0000","last_fullsized":"2024-06-14T23:07:00.933656+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:02:13.956900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":271,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.971895+0000","last_change":"2024-06-14T23:04:34.880836+0000","last_active":"2024-06-14T23:07:06.971895+0000","last_peered":"2024-06-14T23:07:06.971895+0000","last_clean":"2024-06-14T23:07:06.971895+0000","last_became_active":"2024-06-14T23:04:34.880540+0000","last_became_peered":"2024-06-14T23:04:34.880540+0000","last_unstale":"2024-06-14T23:07:06.971895+0000","last_undegraded":"2024-06-14T23:07:06.971895+0000","last_fullsized":"2024-06-14T23:07:06.971895+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:06.981511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"267'9436","reported_seq":9482,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:15.630649+0000","last_change":"2024-06-14T23:07:00.940414+0000","last_active":"2024-06-14T23:07:15.630649+0000","last_peered":"2024-06-14T23:07:15.630649+0000","last_clean":"2024-06-14T23:06:59.906458+0000","last_became_active":"2024-06-14T23:07:00.940414+0000","last_became_peered":"2024-06-14T23:07:00.940414+0000","last_unstale":"2024-06-14T23:07:15.630649+0000","last_undegraded":"2024-06-14T23:07:00.934060+0000","last_fullsized":"2024-06-14T23:07:00.933959+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:23:02.529916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414384152,"num_objects":6324,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9436,"num_write_kb":504257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":6324}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":418,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:06.967275+0000","last_change":"2024-06-14T23:07:00.941375+0000","last_active":"2024-06-14T23:07:06.967275+0000","last_peered":"2024-06-14T23:07:06.967275+0000","last_clean":"2024-06-14T22:39:44.528202+0000","last_became_active":"2024-06-14T23:07:00.941375+0000","last_became_peered":"2024-06-14T23:07:00.941375+0000","last_unstale":"2024-06-14T23:07:06.967275+0000","last_undegraded":"2024-06-14T23:07:06.967275+0000","last_fullsized":"2024-06-14T23:07:00.935304+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":264,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.933168+0000","last_change":"2024-06-14T23:04:34.880181+0000","last_active":"2024-06-14T23:07:01.933168+0000","last_peered":"2024-06-14T23:07:01.933168+0000","last_clean":"2024-06-14T23:07:01.933168+0000","last_became_active":"2024-06-14T23:04:34.879842+0000","last_became_peered":"2024-06-14T23:04:34.879842+0000","last_unstale":"2024-06-14T23:07:01.933168+0000","last_undegraded":"2024-06-14T23:07:01.933168+0000","last_fullsized":"2024-06-14T23:07:01.933168+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:14:18.433225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":228,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.972070+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:07:06.972070+0000","last_peered":"2024-06-14T23:07:06.972070+0000","last_clean":"2024-06-14T23:07:06.972070+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:07:06.972070+0000","last_undegraded":"2024-06-14T23:07:06.972070+0000","last_fullsized":"2024-06-14T23:07:06.972070+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":223,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.970974+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:07:06.970974+0000","last_peered":"2024-06-14T23:07:06.970974+0000","last_clean":"2024-06-14T23:07:06.970974+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:07:06.970974+0000","last_undegraded":"2024-06-14T23:07:06.970974+0000","last_fullsized":"2024-06-14T23:07:06.970974+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"267'9518","reported_seq":9556,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-06-14T23:07:15.311223+0000","last_change":"2024-06-14T23:05:03.621987+0000","last_active":"2024-06-14T23:07:15.311223+0000","last_peered":"2024-06-14T23:07:15.311223+0000","last_clean":"2024-06-14T23:07:15.311223+0000","last_became_active":"2024-06-14T23:05:03.621724+0000","last_became_peered":"2024-06-14T23:05:03.621724+0000","last_unstale":"2024-06-14T23:07:15.311223+0000","last_undegraded":"2024-06-14T23:07:15.311223+0000","last_fullsized":"2024-06-14T23:07:15.311223+0000","mapping_epoch":237,"log_start":"244'2796","ondisk_log_start":"244'2796","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6722,"log_dups_size":0,"ondisk_log_size":6722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:12.558177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430571520,"num_objects":6570,"num_object_clones":0,"num_object_copies":19710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9518,"num_write_kb":514816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":467,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:02.442098+0000","last_change":"2024-06-14T23:07:00.935806+0000","last_active":"2024-06-14T23:07:02.442098+0000","last_peered":"2024-06-14T23:07:02.442098+0000","last_clean":"2024-06-14T23:00:11.711333+0000","last_became_active":"2024-06-14T23:07:00.935806+0000","last_became_peered":"2024-06-14T23:07:00.935806+0000","last_unstale":"2024-06-14T23:07:02.442098+0000","last_undegraded":"2024-06-14T23:07:02.442098+0000","last_fullsized":"2024-06-14T23:07:00.930488+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":256,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.972046+0000","last_change":"2024-06-14T23:04:34.880684+0000","last_active":"2024-06-14T23:07:06.972046+0000","last_peered":"2024-06-14T23:07:06.972046+0000","last_clean":"2024-06-14T23:07:06.972046+0000","last_became_active":"2024-06-14T23:04:34.880127+0000","last_became_peered":"2024-06-14T23:04:34.880127+0000","last_unstale":"2024-06-14T23:07:06.972046+0000","last_undegraded":"2024-06-14T23:07:06.972046+0000","last_fullsized":"2024-06-14T23:07:06.972046+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:53:52.966928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":231,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.931944+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:07:01.931944+0000","last_peered":"2024-06-14T23:07:01.931944+0000","last_clean":"2024-06-14T23:07:01.931944+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:07:01.931944+0000","last_undegraded":"2024-06-14T23:07:01.931944+0000","last_fullsized":"2024-06-14T23:07:01.931944+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":40,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.932106+0000","last_change":"2024-06-14T23:06:19.003881+0000","last_active":"2024-06-14T23:07:01.932106+0000","last_peered":"2024-06-14T23:07:01.932106+0000","last_clean":"2024-06-14T23:07:01.932106+0000","last_became_active":"2024-06-14T23:06:19.003589+0000","last_became_peered":"2024-06-14T23:06:19.003589+0000","last_unstale":"2024-06-14T23:07:01.932106+0000","last_undegraded":"2024-06-14T23:07:01.932106+0000","last_fullsized":"2024-06-14T23:07:01.932106+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:52.332299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"267'9314","reported_seq":9353,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-06-14T23:07:15.282317+0000","last_change":"2024-06-14T23:05:03.621656+0000","last_active":"2024-06-14T23:07:15.282317+0000","last_peered":"2024-06-14T23:07:15.282317+0000","last_clean":"2024-06-14T23:07:15.282317+0000","last_became_active":"2024-06-14T23:05:03.620834+0000","last_became_peered":"2024-06-14T23:05:03.620834+0000","last_unstale":"2024-06-14T23:07:15.282317+0000","last_undegraded":"2024-06-14T23:07:15.282317+0000","last_fullsized":"2024-06-14T23:07:15.282317+0000","mapping_epoch":237,"log_start":"244'2625","ondisk_log_start":"244'2625","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:59:26.118782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425328640,"num_objects":6490,"num_object_clones":0,"num_object_copies":19470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9314,"num_write_kb":505728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":620,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.934601+0000","last_change":"2024-06-14T23:07:00.939551+0000","last_active":"2024-06-14T23:07:01.934601+0000","last_peered":"2024-06-14T23:07:01.934601+0000","last_clean":"2024-06-14T23:06:58.915758+0000","last_became_active":"2024-06-14T23:07:00.939551+0000","last_became_peered":"2024-06-14T23:07:00.939551+0000","last_unstale":"2024-06-14T23:07:01.934601+0000","last_undegraded":"2024-06-14T23:07:01.934601+0000","last_fullsized":"2024-06-14T23:07:00.936996+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:32.174132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:32.174132+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":158,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.967645+0000","last_change":"2024-06-14T23:06:19.000734+0000","last_active":"2024-06-14T23:07:06.967645+0000","last_peered":"2024-06-14T23:07:06.967645+0000","last_clean":"2024-06-14T23:07:06.967645+0000","last_became_active":"2024-06-14T23:06:19.000432+0000","last_became_peered":"2024-06-14T23:06:19.000432+0000","last_unstale":"2024-06-14T23:07:06.967645+0000","last_undegraded":"2024-06-14T23:07:06.967645+0000","last_fullsized":"2024-06-14T23:07:06.967645+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:48.268127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":228,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:06.982022+0000","last_change":"2024-06-14T23:07:00.944857+0000","last_active":"2024-06-14T23:07:06.982022+0000","last_peered":"2024-06-14T23:07:06.982022+0000","last_clean":"2024-06-14T23:06:58.908698+0000","last_became_active":"2024-06-14T23:07:00.944857+0000","last_became_peered":"2024-06-14T23:07:00.944857+0000","last_unstale":"2024-06-14T23:07:06.982022+0000","last_undegraded":"2024-06-14T23:07:06.982022+0000","last_fullsized":"2024-06-14T23:07:00.930574+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:12:27.794388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":159,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.968855+0000","last_change":"2024-06-14T23:06:04.932485+0000","last_active":"2024-06-14T23:07:06.968855+0000","last_peered":"2024-06-14T23:07:06.968855+0000","last_clean":"2024-06-14T23:07:06.968855+0000","last_became_active":"2024-06-14T23:06:04.932166+0000","last_became_peered":"2024-06-14T23:06:04.932166+0000","last_unstale":"2024-06-14T23:07:06.968855+0000","last_undegraded":"2024-06-14T23:07:06.968855+0000","last_fullsized":"2024-06-14T23:07:06.968855+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:46.149456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"266'9248","reported_seq":9285,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:14.182374+0000","last_change":"2024-06-14T23:05:03.623137+0000","last_active":"2024-06-14T23:07:14.182374+0000","last_peered":"2024-06-14T23:07:14.182374+0000","last_clean":"2024-06-14T23:07:14.182374+0000","last_became_active":"2024-06-14T23:05:03.622833+0000","last_became_peered":"2024-06-14T23:05:03.622833+0000","last_unstale":"2024-06-14T23:07:14.182374+0000","last_undegraded":"2024-06-14T23:07:14.182374+0000","last_fullsized":"2024-06-14T23:07:14.182374+0000","mapping_epoch":237,"log_start":"244'2530","ondisk_log_start":"244'2530","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:53:45.610358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419561472,"num_objects":6402,"num_object_clones":0,"num_object_copies":19206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9248,"num_write_kb":500800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":589,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:10.229317+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:07:10.229317+0000","last_peered":"2024-06-14T23:07:10.229317+0000","last_clean":"2024-06-14T23:07:10.229317+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:07:10.229317+0000","last_undegraded":"2024-06-14T23:07:10.229317+0000","last_fullsized":"2024-06-14T23:07:10.229317+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":50,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.979521+0000","last_change":"2024-06-14T23:06:19.000517+0000","last_active":"2024-06-14T23:07:06.979521+0000","last_peered":"2024-06-14T23:07:06.979521+0000","last_clean":"2024-06-14T23:07:06.979521+0000","last_became_active":"2024-06-14T23:06:19.000141+0000","last_became_peered":"2024-06-14T23:06:19.000141+0000","last_unstale":"2024-06-14T23:07:06.979521+0000","last_undegraded":"2024-06-14T23:07:06.979521+0000","last_fullsized":"2024-06-14T23:07:06.979521+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:57:42.599655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":218,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:06.975061+0000","last_change":"2024-06-14T23:07:00.938126+0000","last_active":"2024-06-14T23:07:06.975061+0000","last_peered":"2024-06-14T23:07:06.975061+0000","last_clean":"2024-06-14T23:04:27.842012+0000","last_became_active":"2024-06-14T23:07:00.938126+0000","last_became_peered":"2024-06-14T23:07:00.938126+0000","last_unstale":"2024-06-14T23:07:06.975061+0000","last_undegraded":"2024-06-14T23:07:06.975061+0000","last_fullsized":"2024-06-14T23:07:00.934039+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:35.150723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":216,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.933928+0000","last_change":"2024-06-14T23:07:00.935728+0000","last_active":"2024-06-14T23:07:01.933928+0000","last_peered":"2024-06-14T23:07:01.933928+0000","last_clean":"2024-06-14T23:06:58.913844+0000","last_became_active":"2024-06-14T23:07:00.935728+0000","last_became_peered":"2024-06-14T23:07:00.935728+0000","last_unstale":"2024-06-14T23:07:01.933928+0000","last_undegraded":"2024-06-14T23:07:01.933928+0000","last_fullsized":"2024-06-14T23:07:00.932830+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:34:30.704684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":233,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.934584+0000","last_change":"2024-06-14T23:07:00.937909+0000","last_active":"2024-06-14T23:07:01.934584+0000","last_peered":"2024-06-14T23:07:01.934584+0000","last_clean":"2024-06-14T23:06:58.915235+0000","last_became_active":"2024-06-14T23:07:00.937909+0000","last_became_peered":"2024-06-14T23:07:00.937909+0000","last_unstale":"2024-06-14T23:07:01.934584+0000","last_undegraded":"2024-06-14T23:07:01.934584+0000","last_fullsized":"2024-06-14T23:07:00.934608+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:03:05.662732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":239,"reported_epoch":265,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:01.934583+0000","last_change":"2024-06-14T23:07:00.944839+0000","last_active":"2024-06-14T23:07:01.934583+0000","last_peered":"2024-06-14T23:07:01.934583+0000","last_clean":"2024-06-14T23:06:58.915125+0000","last_became_active":"2024-06-14T23:07:00.944839+0000","last_became_peered":"2024-06-14T23:07:00.944839+0000","last_unstale":"2024-06-14T23:07:01.934583+0000","last_undegraded":"2024-06-14T23:07:00.937659+0000","last_fullsized":"2024-06-14T23:07:00.937552+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:33:00.141339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":261,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:06.970744+0000","last_change":"2024-06-14T23:07:00.939712+0000","last_active":"2024-06-14T23:07:06.970744+0000","last_peered":"2024-06-14T23:07:06.970744+0000","last_clean":"2024-06-14T23:06:58.903882+0000","last_became_active":"2024-06-14T23:07:00.939712+0000","last_became_peered":"2024-06-14T23:07:00.939712+0000","last_unstale":"2024-06-14T23:07:06.970744+0000","last_undegraded":"2024-06-14T23:07:06.970744+0000","last_fullsized":"2024-06-14T23:07:00.930889+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:04:23.848916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":148,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.965941+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:07:06.965941+0000","last_peered":"2024-06-14T23:07:06.965941+0000","last_clean":"2024-06-14T23:07:06.965941+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:07:06.965941+0000","last_undegraded":"2024-06-14T23:07:06.965941+0000","last_fullsized":"2024-06-14T23:07:06.965941+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":598,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:09.356684+0000","last_change":"2024-06-14T23:07:09.356684+0000","last_active":"2024-06-14T23:07:09.356684+0000","last_peered":"2024-06-14T23:07:09.356684+0000","last_clean":"2024-06-14T23:07:09.356684+0000","last_became_active":"2024-06-14T23:07:00.942504+0000","last_became_peered":"2024-06-14T23:07:00.942504+0000","last_unstale":"2024-06-14T23:07:09.356684+0000","last_undegraded":"2024-06-14T23:07:09.356684+0000","last_fullsized":"2024-06-14T23:07:09.356684+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:18:54.289960+0000","scrub_duration":0.013653241999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":227,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.932316+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:07:01.932316+0000","last_peered":"2024-06-14T23:07:01.932316+0000","last_clean":"2024-06-14T23:07:01.932316+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:07:01.932316+0000","last_undegraded":"2024-06-14T23:07:01.932316+0000","last_fullsized":"2024-06-14T23:07:01.932316+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":121,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:06.966652+0000","last_change":"2024-06-14T23:07:00.958108+0000","last_active":"2024-06-14T23:07:06.966652+0000","last_peered":"2024-06-14T23:07:06.966652+0000","last_clean":"2024-06-14T23:06:58.905082+0000","last_became_active":"2024-06-14T23:07:00.958108+0000","last_became_peered":"2024-06-14T23:07:00.958108+0000","last_unstale":"2024-06-14T23:07:06.966652+0000","last_undegraded":"2024-06-14T23:07:06.966652+0000","last_fullsized":"2024-06-14T23:07:00.937690+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:02:10.008170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":103,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.933072+0000","last_change":"2024-06-14T23:06:19.000252+0000","last_active":"2024-06-14T23:07:01.933072+0000","last_peered":"2024-06-14T23:07:01.933072+0000","last_clean":"2024-06-14T23:07:01.933072+0000","last_became_active":"2024-06-14T23:06:18.999939+0000","last_became_peered":"2024-06-14T23:06:18.999939+0000","last_unstale":"2024-06-14T23:07:01.933072+0000","last_undegraded":"2024-06-14T23:07:01.933072+0000","last_fullsized":"2024-06-14T23:07:01.933072+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:50.509140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":229,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:06.968774+0000","last_change":"2024-06-14T23:07:00.938629+0000","last_active":"2024-06-14T23:07:06.968774+0000","last_peered":"2024-06-14T23:07:06.968774+0000","last_clean":"2024-06-14T23:06:58.907628+0000","last_became_active":"2024-06-14T23:07:00.938629+0000","last_became_peered":"2024-06-14T23:07:00.938629+0000","last_unstale":"2024-06-14T23:07:06.968774+0000","last_undegraded":"2024-06-14T23:07:06.968774+0000","last_fullsized":"2024-06-14T23:07:00.933685+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:01:03.709194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":80,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.933255+0000","last_change":"2024-06-14T23:07:00.936462+0000","last_active":"2024-06-14T23:07:01.933255+0000","last_peered":"2024-06-14T23:07:01.933255+0000","last_clean":"2024-06-14T22:48:59.843431+0000","last_became_active":"2024-06-14T23:07:00.936462+0000","last_became_peered":"2024-06-14T23:07:00.936462+0000","last_unstale":"2024-06-14T23:07:01.933255+0000","last_undegraded":"2024-06-14T23:07:01.933255+0000","last_fullsized":"2024-06-14T23:07:00.930441+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:34:05.907903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":126,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.971316+0000","last_change":"2024-06-14T23:06:18.998767+0000","last_active":"2024-06-14T23:07:06.971316+0000","last_peered":"2024-06-14T23:07:06.971316+0000","last_clean":"2024-06-14T23:07:06.971316+0000","last_became_active":"2024-06-14T23:06:18.998481+0000","last_became_peered":"2024-06-14T23:06:18.998481+0000","last_unstale":"2024-06-14T23:07:06.971316+0000","last_undegraded":"2024-06-14T23:07:06.971316+0000","last_fullsized":"2024-06-14T23:07:06.971316+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:58:13.282478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":105,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.966222+0000","last_change":"2024-06-14T23:06:18.999600+0000","last_active":"2024-06-14T23:07:06.966222+0000","last_peered":"2024-06-14T23:07:06.966222+0000","last_clean":"2024-06-14T23:07:06.966222+0000","last_became_active":"2024-06-14T23:06:18.999293+0000","last_became_peered":"2024-06-14T23:06:18.999293+0000","last_unstale":"2024-06-14T23:07:06.966222+0000","last_undegraded":"2024-06-14T23:07:06.966222+0000","last_fullsized":"2024-06-14T23:07:06.966222+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:28:12.373005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":237,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.971133+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:07:06.971133+0000","last_peered":"2024-06-14T23:07:06.971133+0000","last_clean":"2024-06-14T23:07:06.971133+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:07:06.971133+0000","last_undegraded":"2024-06-14T23:07:06.971133+0000","last_fullsized":"2024-06-14T23:07:06.971133+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.994111+0000","last_change":"2024-06-14T23:05:43.964532+0000","last_active":"2024-06-14T23:07:06.994111+0000","last_peered":"2024-06-14T23:07:06.994111+0000","last_clean":"2024-06-14T23:07:06.994111+0000","last_became_active":"2024-06-14T23:05:43.964255+0000","last_became_peered":"2024-06-14T23:05:43.964255+0000","last_unstale":"2024-06-14T23:07:06.994111+0000","last_undegraded":"2024-06-14T23:07:06.994111+0000","last_fullsized":"2024-06-14T23:07:06.994111+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:06.098124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":265,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934798+0000","last_change":"2024-06-14T23:04:34.883863+0000","last_active":"2024-06-14T23:07:01.934798+0000","last_peered":"2024-06-14T23:07:01.934798+0000","last_clean":"2024-06-14T23:07:01.934798+0000","last_became_active":"2024-06-14T23:04:34.881668+0000","last_became_peered":"2024-06-14T23:04:34.881668+0000","last_unstale":"2024-06-14T23:07:01.934798+0000","last_undegraded":"2024-06-14T23:07:01.934798+0000","last_fullsized":"2024-06-14T23:07:01.934798+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:44:45.737416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":49,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.975605+0000","last_change":"2024-06-14T23:04:34.879498+0000","last_active":"2024-06-14T23:07:06.975605+0000","last_peered":"2024-06-14T23:07:06.975605+0000","last_clean":"2024-06-14T23:07:06.975605+0000","last_became_active":"2024-06-14T23:04:34.878932+0000","last_became_peered":"2024-06-14T23:04:34.878932+0000","last_unstale":"2024-06-14T23:07:06.975605+0000","last_undegraded":"2024-06-14T23:07:06.975605+0000","last_fullsized":"2024-06-14T23:07:06.975605+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:56:50.201189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":39,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934351+0000","last_change":"2024-06-14T23:06:19.003121+0000","last_active":"2024-06-14T23:07:01.934351+0000","last_peered":"2024-06-14T23:07:01.934351+0000","last_clean":"2024-06-14T23:07:01.934351+0000","last_became_active":"2024-06-14T23:06:19.002731+0000","last_became_peered":"2024-06-14T23:06:19.002731+0000","last_unstale":"2024-06-14T23:07:01.934351+0000","last_undegraded":"2024-06-14T23:07:01.934351+0000","last_fullsized":"2024-06-14T23:07:01.934351+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:31:04.851974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":243,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934492+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:07:01.934492+0000","last_peered":"2024-06-14T23:07:01.934492+0000","last_clean":"2024-06-14T23:07:01.934492+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:07:01.934492+0000","last_undegraded":"2024-06-14T23:07:01.934492+0000","last_fullsized":"2024-06-14T23:07:01.934492+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":229,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.932359+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:07:01.932359+0000","last_peered":"2024-06-14T23:07:01.932359+0000","last_clean":"2024-06-14T23:07:01.932359+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:07:01.932359+0000","last_undegraded":"2024-06-14T23:07:01.932359+0000","last_fullsized":"2024-06-14T23:07:01.932359+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":243,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.934183+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:07:01.934183+0000","last_peered":"2024-06-14T23:07:01.934183+0000","last_clean":"2024-06-14T23:07:01.934183+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:07:01.934183+0000","last_undegraded":"2024-06-14T23:07:01.934183+0000","last_fullsized":"2024-06-14T23:07:01.934183+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":219,"reported_epoch":265,"state":"active+undersized+degraded","last_fresh":"2024-06-14T23:07:01.933036+0000","last_change":"2024-06-14T23:07:00.940596+0000","last_active":"2024-06-14T23:07:01.933036+0000","last_peered":"2024-06-14T23:07:01.933036+0000","last_clean":"2024-06-14T23:06:58.921230+0000","last_became_active":"2024-06-14T23:07:00.940596+0000","last_became_peered":"2024-06-14T23:07:00.940596+0000","last_unstale":"2024-06-14T23:07:01.933036+0000","last_undegraded":"2024-06-14T23:07:00.929622+0000","last_fullsized":"2024-06-14T23:07:00.929522+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:54:43.704472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":49,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-14T23:07:06.974879+0000","last_change":"2024-06-14T23:04:34.879700+0000","last_active":"2024-06-14T23:07:06.974879+0000","last_peered":"2024-06-14T23:07:06.974879+0000","last_clean":"2024-06-14T23:07:06.974879+0000","last_became_active":"2024-06-14T23:04:34.878924+0000","last_became_peered":"2024-06-14T23:04:34.878924+0000","last_unstale":"2024-06-14T23:07:06.974879+0000","last_undegraded":"2024-06-14T23:07:06.974879+0000","last_fullsized":"2024-06-14T23:07:06.974879+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:28.826486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":145,"reported_epoch":266,"state":"active+undersized","last_fresh":"2024-06-14T23:07:10.625477+0000","last_change":"2024-06-14T23:07:00.941426+0000","last_active":"2024-06-14T23:07:10.625477+0000","last_peered":"2024-06-14T23:07:10.625477+0000","last_clean":"2024-06-14T23:06:58.920541+0000","last_became_active":"2024-06-14T23:07:00.941426+0000","last_became_peered":"2024-06-14T23:07:00.941426+0000","last_unstale":"2024-06-14T23:07:10.625477+0000","last_undegraded":"2024-06-14T23:07:10.625477+0000","last_fullsized":"2024-06-14T23:07:00.934612+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:00:22.305275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":233,"reported_epoch":265,"state":"active+undersized","last_fresh":"2024-06-14T23:07:01.933812+0000","last_change":"2024-06-14T23:07:00.947186+0000","last_active":"2024-06-14T23:07:01.933812+0000","last_peered":"2024-06-14T23:07:01.933812+0000","last_clean":"2024-06-14T23:06:58.913623+0000","last_became_active":"2024-06-14T23:07:00.947186+0000","last_became_peered":"2024-06-14T23:07:00.947186+0000","last_unstale":"2024-06-14T23:07:01.933812+0000","last_undegraded":"2024-06-14T23:07:01.933812+0000","last_fullsized":"2024-06-14T23:07:00.932514+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:03:54.831817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":50,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-06-14T23:07:01.932965+0000","last_change":"2024-06-14T23:06:19.002447+0000","last_active":"2024-06-14T23:07:01.932965+0000","last_peered":"2024-06-14T23:07:01.932965+0000","last_clean":"2024-06-14T23:07:01.932965+0000","last_became_active":"2024-06-14T23:06:19.002029+0000","last_became_peered":"2024-06-14T23:06:19.002029+0000","last_unstale":"2024-06-14T23:07:01.932965+0000","last_undegraded":"2024-06-14T23:07:01.932965+0000","last_fullsized":"2024-06-14T23:07:01.932965+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:27.132531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6824263704,"num_objects":104131,"num_object_clones":0,"num_object_copies":312393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104131,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150507,"num_write_kb":8148353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21008166912,"data_stored":21008154696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120197,"ondisk_log_size":120197,"up":40,"acting":40,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13600,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":85,"acting":85,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":21,"acting":21,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":452420,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":124,"num_write":33,"num_write_kb":330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1094569,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3381,"ondisk_log_size":3381,"up":53,"acting":57,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234398,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3424860,"kb_used_data":2513620,"kb_used_omap":1,"kb_used_meta":911230,"kb_avail":90320292,"statfs":{"total":95995035648,"available":92487979008,"internally_reserved":0,"allocated":2573946880,"data_stored":2572626279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":933099864},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.95299999999999996,"15min":0.98199999999999998},"min":{"1min":0.214,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.7509999999999999,"5min":3.1699999999999999,"15min":6.9539999999999997},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.3089999999999999,"15min":1.103},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.20300000000000001},"max":{"1min":4.3550000000000004,"5min":6.9550000000000001,"15min":6.9550000000000001},"last":0.29799999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:07:02 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.1439999999999999,"15min":1.629},"min":{"1min":0.27300000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.8490000000000002,"5min":6.3090000000000002,"15min":83.295000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":2.0830000000000002,"5min":1.161,"15min":1.6539999999999999},"min":{"1min":0.32900000000000001,"5min":0.22900000000000001,"15min":0.223},"max":{"1min":8.4640000000000004,"5min":8.4640000000000004,"15min":82.972999999999999},"last":3.5640000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:07:02 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.0860000000000001,"15min":0.96299999999999997},"min":{"1min":0.32200000000000001,"5min":0.253,"15min":0.253},"max":{"1min":5.4009999999999998,"5min":7.468,"15min":7.468},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.141,"15min":1.04},"min":{"1min":0.35499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":4.0369999999999999,"5min":6.0469999999999997,"15min":8.1150000000000002},"last":3.2210000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:06:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.3580000000000001,"15min":1.2669999999999999},"min":{"1min":0.34499999999999997,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":2.5659999999999998,"5min":7.8609999999999998,"15min":55.683999999999997},"last":2.8250000000000002},{"interface":"front","average":{"1min":1.458,"5min":1.3899999999999999,"15min":1.302},"min":{"1min":0.41799999999999998,"5min":0.30099999999999999,"15min":0.29799999999999999},"max":{"1min":4.3010000000000002,"5min":7.7670000000000003,"15min":55.533999999999999},"last":0.70899999999999996}]},{"osd":4,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.254,"15min":0.92900000000000005},"min":{"1min":0.26600000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.2719999999999998,"5min":11.342000000000001,"15min":11.342000000000001},"last":0.248},{"interface":"front","average":{"1min":0.502,"5min":1.278,"15min":0.98599999999999999},"min":{"1min":0.29699999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.913,"5min":12.651,"15min":16.879000000000001},"last":0.28199999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.131,"15min":1.0329999999999999},"min":{"1min":0.27800000000000002,"5min":0.252,"15min":0.23100000000000001},"max":{"1min":2.2170000000000001,"5min":6.4320000000000004,"15min":6.9000000000000004},"last":0.316},{"interface":"front","average":{"1min":0.78300000000000003,"5min":1.163,"15min":1.038},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.26000000000000001},"max":{"1min":2.3170000000000002,"5min":6.2809999999999997,"15min":6.8529999999999998},"last":0.33800000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":1.367,"15min":1.0449999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6320000000000001,"5min":8.952,"15min":8.952},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.2989999999999999,"15min":1.008},"min":{"1min":0.34000000000000002,"5min":0.29599999999999999,"15min":0.28299999999999997},"max":{"1min":1.4850000000000001,"5min":7.0629999999999997,"15min":12.951000000000001},"last":0.47099999999999997}]}]},{"osd":6,"up_from":46,"seq":197568496031,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4730444,"kb_used_data":3833212,"kb_used_omap":1,"kb_used_meta":897214,"kb_avail":89014708,"statfs":{"total":95995035648,"available":91151060992,"internally_reserved":0,"allocated":3925209088,"data_stored":3923934386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918747480},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":2.0670000000000002,"15min":1.4199999999999999},"min":{"1min":0.253,"5min":0.23400000000000001,"15min":0.23200000000000001},"max":{"1min":3.8919999999999999,"5min":45.579999999999998,"15min":45.579999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.25,"5min":1.819,"15min":1.2110000000000001},"min":{"1min":0.27400000000000002,"5min":0.19,"15min":0.19},"max":{"1min":5.3460000000000001,"5min":44.918999999999997,"15min":44.918999999999997},"last":1.0429999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:06:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.125,"15min":1.0860000000000001},"min":{"1min":0.40500000000000003,"5min":0.223,"15min":0.223},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":6.6680000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.105,"15min":1.153},"min":{"1min":0.35199999999999998,"5min":0.26600000000000001,"15min":0.247},"max":{"1min":3.907,"5min":7.0570000000000004,"15min":9.5510000000000002},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Jun 14 23:06:52 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.1859999999999999,"15min":0.96899999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":4.2229999999999999,"5min":4.2229999999999999,"15min":9.9600000000000009},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.387,"15min":1.081},"min":{"1min":0.33300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":7.0570000000000004,"5min":10.823,"15min":10.823},"last":0.46100000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3120000000000001,"15min":1.179},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27300000000000002},"max":{"1min":4.3440000000000003,"5min":5.0949999999999998,"15min":29.152000000000001},"last":1.788},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.23,"15min":1.175},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":2.8519999999999999,"5min":7.3799999999999999,"15min":25.864000000000001},"last":1.2609999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.998,"15min":0.86699999999999999},"min":{"1min":0.23999999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.571,"5min":7.7119999999999997,"15min":7.7709999999999999},"last":0.248},{"interface":"front","average":{"1min":0.45700000000000002,"5min":1.0069999999999999,"15min":0.81999999999999995},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.24299999999999999},"max":{"1min":0.91000000000000003,"5min":3.9159999999999999,"15min":9.9610000000000003},"last":0.75}]},{"osd":5,"last update":"Fri Jun 14 23:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.0509999999999999,"15min":0.93600000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.5070000000000001,"5min":6.1420000000000003,"15min":8.6500000000000004},"last":0.998},{"interface":"front","average":{"1min":0.86599999999999999,"5min":1.008,"15min":0.97899999999999998},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":3.3660000000000001,"5min":4.548,"15min":8.6349999999999998},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Jun 14 23:06:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.998,"15min":0.96699999999999997},"min":{"1min":0.37,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.9060000000000001,"5min":4.4720000000000004,"15min":8.6950000000000003},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":1.0469999999999999,"15min":1.012},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.28899999999999998},"max":{"1min":1.9079999999999999,"5min":4.1470000000000002,"15min":8.673},"last":0.42999999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757673,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2581256,"kb_used_data":1686016,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":91163896,"statfs":{"total":95995035648,"available":93351829504,"internally_reserved":0,"allocated":1726480384,"data_stored":1725176152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.9039999999999999,"15min":1.3560000000000001},"min":{"1min":0.223,"5min":0.22,"15min":0.22},"max":{"1min":2.2440000000000002,"5min":14.94,"15min":14.94},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.8169999999999999,"15min":1.3080000000000001},"min":{"1min":0.28899999999999998,"5min":0.219,"15min":0.219},"max":{"1min":4.71,"5min":9.3819999999999997,"15min":9.3819999999999997},"last":3.6309999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:06:57 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.0329999999999999,"15min":1.0129999999999999},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.23999999999999999},"max":{"1min":4.2889999999999997,"5min":7.4370000000000003,"15min":7.4370000000000003},"last":6.9480000000000004},{"interface":"front","average":{"1min":1.294,"5min":1.403,"15min":1.244},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.5880000000000001,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:06:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.6180000000000001,"15min":1.1399999999999999},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.9740000000000002,"5min":14.891,"15min":14.891},"last":3.1320000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.6379999999999999,"15min":1.1459999999999999},"min":{"1min":0.29799999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.3570000000000002,"5min":9.532,"15min":9.532},"last":4.2130000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:06:29 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.591,"15min":1.4039999999999999},"min":{"1min":0.36599999999999999,"5min":0.249,"15min":0.249},"max":{"1min":4.7389999999999999,"5min":9.2149999999999999,"15min":45.744},"last":5.2359999999999998},{"interface":"front","average":{"1min":1.222,"5min":1.667,"15min":1.484},"min":{"1min":0.29299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.883,"5min":14.914,"15min":45.838000000000001},"last":5.202}]},{"osd":4,"last update":"Fri Jun 14 23:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.97699999999999998,"15min":0.81000000000000005},"min":{"1min":0.27200000000000002,"5min":0.26600000000000001,"15min":0.24399999999999999},"max":{"1min":1.887,"5min":14.352,"15min":14.352},"last":6.9119999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.466,"15min":1.0009999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":4.6660000000000004,"5min":9.5980000000000008,"15min":9.5980000000000008},"last":0.95599999999999996}]},{"osd":6,"last update":"Fri Jun 14 23:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.458,"15min":1.0960000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":4.6719999999999997,"5min":15.382999999999999,"15min":15.382999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.417,"15min":1.071},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.6020000000000003,"5min":14.855,"15min":14.855},"last":0.439}]},{"osd":7,"last update":"Fri Jun 14 23:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.4179999999999999,"15min":1.073},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30199999999999999},"max":{"1min":2.101,"5min":14.818,"15min":14.818},"last":3.1640000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.141,"15min":0.97299999999999998},"min":{"1min":0.376,"5min":0.27000000000000002,"15min":0.25600000000000001},"max":{"1min":2.1539999999999999,"5min":14.429,"15min":14.429},"last":6.9779999999999998}]}]},{"osd":4,"up_from":30,"seq":128849019309,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2572568,"kb_used_data":1673680,"kb_used_omap":1,"kb_used_meta":898878,"kb_avail":91172584,"statfs":{"total":95995035648,"available":93360726016,"internally_reserved":0,"allocated":1713848320,"data_stored":1712531504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920451416},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:06:40 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.5369999999999999,"15min":1.1080000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.17100000000000001},"max":{"1min":2.0409999999999999,"5min":10.567,"15min":10.567},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.5,"15min":1.149},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":7.4900000000000002,"5min":7.4900000000000002,"15min":8.3390000000000004},"last":0.58699999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:06:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.216,"15min":1.331},"min":{"1min":0.29699999999999999,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":5.0430000000000001,"5min":6.6929999999999996,"15min":56.917000000000002},"last":1.538},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.3600000000000001,"15min":1.3999999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.3799999999999999,"5min":10.539999999999999,"15min":56.789999999999999},"last":0.81399999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:06:51 2024","interfaces":[{"interface":"back","average":{"1min":4.5170000000000003,"5min":2.0539999999999998,"15min":1.171},"min":{"1min":0.34799999999999998,"5min":0.26500000000000001,"15min":0.23000000000000001},"max":{"1min":46.866,"5min":46.866,"15min":46.866},"last":2.5790000000000002},{"interface":"front","average":{"1min":4.1289999999999996,"5min":2.1070000000000002,"15min":1.2190000000000001},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":46.851999999999997,"5min":46.851999999999997,"15min":46.851999999999997},"last":2.5649999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:06:40 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.2030000000000001,"15min":1.0209999999999999},"min":{"1min":0.38800000000000001,"5min":0.28599999999999998,"15min":0.27600000000000002},"max":{"1min":5.2149999999999999,"5min":6.327,"15min":6.617},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.508,"15min":1.1299999999999999},"min":{"1min":0.29099999999999998,"5min":0.28499999999999998,"15min":0.27500000000000002},"max":{"1min":3.5590000000000002,"5min":10.510999999999999,"15min":10.510999999999999},"last":1.4079999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.3,"15min":1.5309999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":2.0339999999999998,"5min":12.315,"15min":139.13200000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.252,"15min":1.548},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.26100000000000001},"max":{"1min":1.659,"5min":10.959,"15min":138.86000000000001},"last":0.27000000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:06:40 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.075,"15min":0.88300000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.7360000000000002,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.0509999999999999,"15min":0.83699999999999997},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.254},"max":{"1min":2.9009999999999998,"5min":5.8680000000000003,"15min":8.8819999999999997},"last":1.1220000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.02,"15min":0.90500000000000003},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.04,"5min":6.0890000000000004,"15min":8.9489999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.753,"5min":1.0369999999999999,"15min":1.018},"min":{"1min":0.30199999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.988,"5min":5.3479999999999999,"15min":8.4730000000000008},"last":0.28999999999999998}]}]},{"osd":3,"up_from":138,"seq":592705487008,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3403720,"kb_used_data":2940736,"kb_used_omap":4,"kb_used_meta":462907,"kb_avail":90341432,"statfs":{"total":95995035648,"available":92509626368,"internally_reserved":0,"allocated":3011313664,"data_stored":3010009205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":474017246},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.3740000000000001,"15min":1.0209999999999999},"min":{"1min":0.222,"5min":0.187,"15min":0.184},"max":{"1min":3.8620000000000001,"5min":11.071999999999999,"15min":11.071999999999999},"last":0.317},{"interface":"front","average":{"1min":1.103,"5min":1.4710000000000001,"15min":0.97099999999999997},"min":{"1min":0.19800000000000001,"5min":0.19,"15min":0.17999999999999999},"max":{"1min":6.6109999999999998,"5min":8.2330000000000005,"15min":8.2330000000000005},"last":0.17799999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.327,"15min":1.085},"min":{"1min":0.25700000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.5499999999999998,"5min":9.6080000000000005,"15min":24.097999999999999},"last":1.329},{"interface":"front","average":{"1min":1.587,"5min":1.3999999999999999,"15min":1.133},"min":{"1min":0.25800000000000001,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":6.6539999999999999,"5min":6.6539999999999999,"15min":24.132999999999999},"last":4.2320000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":1.5960000000000001,"15min":1.0469999999999999},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.23599999999999999},"max":{"1min":6.7430000000000003,"5min":9.4730000000000008,"15min":9.4730000000000008},"last":4.2720000000000002},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4419999999999999,"15min":0.97399999999999998},"min":{"1min":0.27400000000000002,"5min":0.23799999999999999,"15min":0.20899999999999999},"max":{"1min":4.6900000000000004,"5min":10.369,"15min":10.369},"last":0.46100000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.4850000000000001,"15min":1.105},"min":{"1min":0.36899999999999999,"5min":0.33000000000000002,"15min":0.27900000000000003},"max":{"1min":4.7249999999999996,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.262,"15min":1.087},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.4620000000000002,"5min":7.5350000000000001,"15min":7.5350000000000001},"last":0.44600000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":5.9800000000000004,"5min":2.1949999999999998,"15min":1.4379999999999999},"min":{"1min":0.33200000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":81.438999999999993,"5min":81.438999999999993,"15min":81.438999999999993},"last":0.42199999999999999},{"interface":"front","average":{"1min":6.5289999999999999,"5min":2.4140000000000001,"15min":1.4670000000000001},"min":{"1min":0.38,"5min":0.32400000000000001,"15min":0.30199999999999999},"max":{"1min":81.626999999999995,"5min":81.626999999999995,"15min":81.626999999999995},"last":0.29299999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":6.7469999999999999,"5min":2.5579999999999998,"15min":1.4750000000000001},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.32400000000000001},"max":{"1min":82.248999999999995,"5min":82.248999999999995,"15min":82.248999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":6.5620000000000003,"5min":2.71,"15min":1.528},"min":{"1min":0.44900000000000001,"5min":0.31,"15min":0.31},"max":{"1min":81.585999999999999,"5min":81.585999999999999,"15min":81.585999999999999},"last":0.72299999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.373,"15min":1.2010000000000001},"min":{"1min":0.36499999999999999,"5min":0.35599999999999998,"15min":0.35299999999999998},"max":{"1min":4.7809999999999997,"5min":5.5300000000000002,"15min":7.7240000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.085,"15min":1.0860000000000001},"min":{"1min":0.42299999999999999,"5min":0.36299999999999999,"15min":0.32400000000000001},"max":{"1min":3.5169999999999999,"5min":5.2160000000000002,"15min":5.8300000000000001},"last":0.499}]}]},{"osd":2,"up_from":133,"seq":571230650535,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2849692,"kb_used_data":2494932,"kb_used_omap":6,"kb_used_meta":354745,"kb_avail":90895460,"statfs":{"total":95995035648,"available":93076951040,"internally_reserved":0,"allocated":2554810368,"data_stored":2553494133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":363259381},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:06:56 2024","interfaces":[{"interface":"back","average":{"1min":2.2919999999999998,"5min":1.853,"15min":1.038},"min":{"1min":0.22700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":8.0579999999999998,"5min":55.851999999999997,"15min":55.851999999999997},"last":2.25},{"interface":"front","average":{"1min":1.829,"5min":2.0510000000000002,"15min":1.2},"min":{"1min":0.25800000000000001,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":9.798,"5min":56.171999999999997,"15min":56.171999999999997},"last":1.244}]},{"osd":1,"last update":"Fri Jun 14 23:06:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.2130000000000001,"15min":1.0249999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":9.8140000000000001,"5min":9.8140000000000001,"15min":21.920000000000002},"last":0.33900000000000002},{"interface":"front","average":{"1min":2.8500000000000001,"5min":1.2709999999999999,"15min":1.038},"min":{"1min":0.26800000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":14.416,"5min":14.416,"15min":21.573},"last":2.2839999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:06:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.556,"15min":1.141},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.516,"5min":11.24,"15min":21.66},"last":0.879},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.492,"15min":1.131},"min":{"1min":0.40899999999999997,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":6.5679999999999996,"5min":11.271000000000001,"15min":22.341000000000001},"last":2.6779999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:06:56 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.9590000000000001,"15min":1.343},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.3609999999999998,"5min":31.928000000000001,"15min":31.928000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.7829999999999999,"15min":1.2410000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.1520000000000001,"5min":31.954999999999998,"15min":31.954999999999998},"last":0.40300000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:06:56 2024","interfaces":[{"interface":"back","average":{"1min":2.2749999999999999,"5min":1.399,"15min":1.1439999999999999},"min":{"1min":0.317,"5min":0.29599999999999999,"15min":0.28299999999999997},"max":{"1min":7.7999999999999998,"5min":7.7999999999999998,"15min":31.539999999999999},"last":0.437},{"interface":"front","average":{"1min":1.256,"5min":1.4790000000000001,"15min":1.175},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.29399999999999998},"max":{"1min":4.2960000000000003,"5min":5.6340000000000003,"15min":31.481999999999999},"last":0.95999999999999996}]},{"osd":6,"last update":"Fri Jun 14 23:06:56 2024","interfaces":[{"interface":"back","average":{"1min":2.25,"5min":1.907,"15min":1.2310000000000001},"min":{"1min":0.35699999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.1440000000000001,"5min":32.005000000000003,"15min":32.005000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":2.3460000000000001,"5min":1.905,"15min":1.22},"min":{"1min":0.41199999999999998,"5min":0.34999999999999998,"15min":0.30399999999999999},"max":{"1min":7.7409999999999997,"5min":32.079999999999998,"15min":32.079999999999998},"last":0.68000000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:06:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1739999999999999,"5min":1.419,"15min":1.1379999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":9.7539999999999996,"5min":9.7539999999999996,"15min":10.614000000000001},"last":0.377},{"interface":"front","average":{"1min":2.4540000000000002,"5min":1.294,"15min":0.98799999999999999},"min":{"1min":0.433,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":7.6639999999999997,"5min":7.6639999999999997,"15min":7.6639999999999997},"last":0.46200000000000002}]}]},{"osd":1,"up_from":116,"seq":498216206552,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3961536,"kb_used_data":3338104,"kb_used_omap":5,"kb_used_meta":623418,"kb_avail":89783616,"statfs":{"total":95995035648,"available":91938422784,"internally_reserved":0,"allocated":3418218496,"data_stored":3416921595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":638380312},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:07:01 2024","interfaces":[{"interface":"back","average":{"1min":1.8999999999999999,"5min":1.456,"15min":1.3500000000000001},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":9.077,"5min":9.5350000000000001,"15min":11.257999999999999},"last":1.9259999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.4990000000000001,"15min":1.196},"min":{"1min":0.32100000000000001,"5min":0.19700000000000001,"15min":0.186},"max":{"1min":3.9590000000000001,"5min":13.672000000000001,"15min":13.672000000000001},"last":1.381}]},{"osd":2,"last update":"Fri Jun 14 23:07:01 2024","interfaces":[{"interface":"back","average":{"1min":2.2000000000000002,"5min":1.724,"15min":1.24},"min":{"1min":0.33200000000000002,"5min":0.21299999999999999,"15min":0.19500000000000001},"max":{"1min":6.0460000000000003,"5min":9.5470000000000006,"15min":9.5470000000000006},"last":1.621},{"interface":"front","average":{"1min":1.611,"5min":1.407,"15min":0.96599999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.0410000000000004,"5min":8.0060000000000002,"15min":8.0060000000000002},"last":0.35699999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.873,"5min":1.2669999999999999,"15min":1.1699999999999999},"min":{"1min":0.23599999999999999,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":7.5679999999999996,"5min":7.5679999999999996,"15min":21.422000000000001},"last":2.032},{"interface":"front","average":{"1min":1.796,"5min":1.369,"15min":1.2490000000000001},"min":{"1min":0.36299999999999999,"5min":0.24099999999999999,"15min":0.214},"max":{"1min":4.4729999999999999,"5min":8.1219999999999999,"15min":21.016999999999999},"last":1.732}]},{"osd":4,"last update":"Fri Jun 14 23:07:01 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.3220000000000001,"15min":1.2410000000000001},"min":{"1min":0.29599999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.7050000000000001,"5min":6.5350000000000001,"15min":8.8599999999999994},"last":2},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.3160000000000001,"15min":1.3640000000000001},"min":{"1min":0.33000000000000002,"5min":0.313,"15min":0.29199999999999998},"max":{"1min":5.8570000000000002,"5min":6.2960000000000003,"15min":10.148},"last":1.6870000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:07:01 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.4650000000000001,"15min":1.367},"min":{"1min":0.32900000000000001,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":3.8500000000000001,"5min":19.132999999999999,"15min":19.132999999999999},"last":4.1310000000000002},{"interface":"front","average":{"1min":1.274,"5min":1.4530000000000001,"15min":1.2649999999999999},"min":{"1min":0.379,"5min":0.29399999999999998,"15min":0.28599999999999998},"max":{"1min":4.7699999999999996,"5min":19.123000000000001,"15min":19.123000000000001},"last":2.0630000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:05:58 2024","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":2.2349999999999999,"15min":1.669},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":24.856000000000002,"5min":63.795999999999999,"15min":63.795999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":2.548,"5min":2.1760000000000002,"15min":1.571},"min":{"1min":0.39900000000000002,"5min":0.32900000000000001,"15min":0.28499999999999998},"max":{"1min":24.904,"5min":63.811,"15min":63.811},"last":0.92700000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:07:01 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.274,"15min":1.353},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.8230000000000004,"5min":8.0440000000000005,"15min":9.4429999999999996},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.1459999999999999,"15min":1.2150000000000001},"min":{"1min":0.38800000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.9300000000000002,"5min":5.9580000000000002,"15min":6.3529999999999998},"last":0.39000000000000001}]}]},{"osd":0,"up_from":91,"seq":390842024206,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3012096,"kb_used_data":2146232,"kb_used_omap":5,"kb_used_meta":865850,"kb_avail":90733056,"statfs":{"total":95995035648,"available":92910649344,"internally_reserved":0,"allocated":2197741568,"data_stored":2196442666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":886630490},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:06:46 2024","interfaces":[{"interface":"back","average":{"1min":1.851,"5min":1.2529999999999999,"15min":1.323},"min":{"1min":0.186,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":7.952,"5min":9.4380000000000006,"15min":16.305},"last":2.839},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.4019999999999999,"15min":1.4450000000000001},"min":{"1min":0.39500000000000002,"5min":0.20000000000000001,"15min":0.17899999999999999},"max":{"1min":3.4849999999999999,"5min":9.9990000000000006,"15min":16.184999999999999},"last":1.0800000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:07:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.5209999999999999,"15min":1.1220000000000001},"min":{"1min":0.27900000000000003,"5min":0.188,"15min":0.188},"max":{"1min":6.1299999999999999,"5min":9.9789999999999992,"15min":16.757000000000001},"last":0.505},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.375,"15min":1.139},"min":{"1min":0.32800000000000001,"5min":0.223,"15min":0.222},"max":{"1min":3.165,"5min":7.9800000000000004,"15min":16.055},"last":0.63200000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:06:36 2024","interfaces":[{"interface":"back","average":{"1min":2.2730000000000001,"5min":1.7,"15min":1.204},"min":{"1min":0.42699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":5.8819999999999997,"5min":11.083,"15min":16.873999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":2.2639999999999998,"5min":1.6519999999999999,"15min":1.1779999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.224},"max":{"1min":6.2990000000000004,"5min":11.375,"15min":16.085999999999999},"last":8.0259999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:06:36 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.5880000000000001,"15min":1.375},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.8420000000000001,"5min":12.436,"15min":16.527000000000001},"last":2.8140000000000001},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.448,"15min":1.3460000000000001},"min":{"1min":0.29199999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":3.339,"5min":5.0110000000000001,"15min":15.976000000000001},"last":0.29799999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:06:36 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.653,"15min":1.5449999999999999},"min":{"1min":0.34499999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.302,"5min":10.462,"15min":16.108000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.6439999999999999,"15min":1.4410000000000001},"min":{"1min":0.36299999999999999,"5min":0.32500000000000001,"15min":0.31900000000000001},"max":{"1min":3.2240000000000002,"5min":9.2279999999999998,"15min":16.006},"last":0.38}]},{"osd":6,"last update":"Fri Jun 14 23:06:36 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.5109999999999999,"15min":1.4419999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30599999999999999},"max":{"1min":3.367,"5min":6.1669999999999998,"15min":16.152000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.7130000000000001,"15min":1.466},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29399999999999998},"max":{"1min":4.274,"5min":9.3130000000000006,"15min":16.227},"last":0.63500000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:06:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.589,"15min":1.3939999999999999},"min":{"1min":0.40699999999999997,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.899,"5min":9.2579999999999991,"15min":16.199999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.528,"15min":1.4219999999999999},"min":{"1min":0.378,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.8050000000000002,"5min":11.195,"15min":16.259},"last":2.7719999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2192310272,"data_stored":2192310272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3391684608,"data_stored":3391684608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2545291264,"data_stored":2545287192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2990931968,"data_stored":2990931968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1706364928,"data_stored":1706360856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1714814976,"data_stored":1714814976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3903852544,"data_stored":3903848472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2562916352,"data_stored":2562916352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:07:19.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set unique_pool_6 pgp_num 16 2024-06-14T23:07:19.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 52237/312465 objects degraded (16.718%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:19.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:18 smithi137 ceph-mon[90634]: from='client.36348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:07:19.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:19.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:19.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:19.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:19.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:19.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:19.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:19.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:19.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:19.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:19.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:19.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:19.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:19.871 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:19 smithi137 ceph-mon[90634]: pgmap v1021: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 52237/312465 objects degraded (16.718%); 1/312465 objects misplaced (0.000%) 2024-06-14T23:07:19.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:19.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:19.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:20.003 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:07:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:07:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:07:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:07:20.003 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:07:20.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:20.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:20.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:19 smithi086 ceph-mon[102754]: pgmap v1021: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 52237/312465 objects degraded (16.718%); 1/312465 objects misplaced (0.000%) 2024-06-14T23:07:20.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:19 smithi086 ceph-mon[97913]: pgmap v1021: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 52237/312465 objects degraded (16.718%); 1/312465 objects misplaced (0.000%) 2024-06-14T23:07:20.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:20.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:20.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:20.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:20.276 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:20.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:20.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:20.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:20.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:20.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:20.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:20.581 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:20.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:20.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:20.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:20.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:20.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-14T23:07:20.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:20.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:20.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:20.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:20.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:20.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:21.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:21.107 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:07:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:21.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:21.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:21.142 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:07:21.142 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:07:21.179 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:07:21.180 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:07:21.217 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:07:21.218 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:07:21.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:21.251 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:07:21.349 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:07:21.350 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:07:21.350 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:07:21.350 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:07:21.351 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:07:21.351 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-14T23:07:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:21.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:21.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:21.352 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:07:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:07:21.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:21.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:21.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:21.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:21.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:21.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:21.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:21.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:21.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:21.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:21.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:21.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:21.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:22.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:22.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:21 smithi086 ceph-mon[97913]: pgmap v1022: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 49860/298269 objects degraded (16.716%); 1/298269 objects misplaced (0.000%) 2024-06-14T23:07:22.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/345838505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:22.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:21 smithi086 ceph-mon[102754]: pgmap v1022: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 49860/298269 objects degraded (16.716%); 1/298269 objects misplaced (0.000%) 2024-06-14T23:07:22.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/345838505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:22.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:22.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:22.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:22.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:21 smithi137 ceph-mon[90634]: pgmap v1022: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 49860/298269 objects degraded (16.716%); 1/298269 objects misplaced (0.000%) 2024-06-14T23:07:22.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/345838505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:22.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:22.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:22.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:22.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:22.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:22.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:22.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:22.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:22.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:22.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:22.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:22.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:22.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:22.871 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:22.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:22.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:23.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:23.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:23.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:23.119 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:22 smithi137 ceph-mon[90634]: 1.d scrub starts 2024-06-14T23:07:23.119 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:22 smithi137 ceph-mon[90634]: 1.d scrub ok 2024-06-14T23:07:23.148 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:23.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:23.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:23.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:23.273 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:23.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:23.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:22 smithi086 ceph-mon[97913]: 1.d scrub starts 2024-06-14T23:07:23.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:22 smithi086 ceph-mon[97913]: 1.d scrub ok 2024-06-14T23:07:23.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:22 smithi086 ceph-mon[102754]: 1.d scrub starts 2024-06-14T23:07:23.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:22 smithi086 ceph-mon[102754]: 1.d scrub ok 2024-06-14T23:07:23.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:23.401 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:07:23.401 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:07:23.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:23.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:23.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:23.537 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:23.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:23.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:23.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:23.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:23.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:23.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:23.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:23.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:23.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:23.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:23.988 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:23 smithi086 ceph-mon[97913]: pgmap v1023: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 49860/298269 objects degraded (16.716%); 1/298269 objects misplaced (0.000%) 2024-06-14T23:07:23.988 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:23 smithi086 ceph-mon[97913]: 1.e scrub starts 2024-06-14T23:07:23.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:23 smithi086 ceph-mon[97913]: 1.e scrub ok 2024-06-14T23:07:23.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 49860/298269 objects degraded (16.716%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:23.989 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:23 smithi086 ceph-mon[102754]: pgmap v1023: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 49860/298269 objects degraded (16.716%); 1/298269 objects misplaced (0.000%) 2024-06-14T23:07:23.989 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:23 smithi086 ceph-mon[102754]: 1.e scrub starts 2024-06-14T23:07:23.989 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:23 smithi086 ceph-mon[102754]: 1.e scrub ok 2024-06-14T23:07:23.990 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 49860/298269 objects degraded (16.716%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:24.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:24.078 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:24.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:24.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:24.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:24.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:24.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:23 smithi137 ceph-mon[90634]: pgmap v1023: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 49860/298269 objects degraded (16.716%); 1/298269 objects misplaced (0.000%) 2024-06-14T23:07:24.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:23 smithi137 ceph-mon[90634]: 1.e scrub starts 2024-06-14T23:07:24.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:23 smithi137 ceph-mon[90634]: 1.e scrub ok 2024-06-14T23:07:24.208 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 49860/298269 objects degraded (16.716%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:24.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-14T23:07:24.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:24.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:24.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:24.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:24.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:24.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:24.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:24.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:24.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:24.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:24.736 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:24.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:24.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:24.880 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:24.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:24.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:25.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:25.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:25.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:25.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:25.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:25.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:25.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:25.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:07:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:07:24] "GET /metrics HTTP/1.1" 200 36627 "" "Prometheus/2.43.0" 2024-06-14T23:07:25.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:25.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:25.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:25.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:25.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:25.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:25.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:25.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:25.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:25.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:25.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:25.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:25.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:25.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:25.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:26.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:26.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:26.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:26.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:26.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:26.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:25 smithi137 ceph-mon[90634]: pgmap v1024: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 957 op/s; 48287/291606 objects degraded (16.559%); 1/291606 objects misplaced (0.000%) 2024-06-14T23:07:26.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:26.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:25 smithi086 ceph-mon[97913]: pgmap v1024: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 957 op/s; 48287/291606 objects degraded (16.559%); 1/291606 objects misplaced (0.000%) 2024-06-14T23:07:26.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:25 smithi086 ceph-mon[102754]: pgmap v1024: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 957 op/s; 48287/291606 objects degraded (16.559%); 1/291606 objects misplaced (0.000%) 2024-06-14T23:07:26.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:26.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:26.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:26.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:26.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:26.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:26.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:26.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:26.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:26.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:26.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:26.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:26.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:26.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:26.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:27.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:27.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:27.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:27.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:27.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:07:27.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:07:27.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:27.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:27.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:27.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:27.443 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:27.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:27.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:27.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:27.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:27.662 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:27.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:27.745 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:27.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:27.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:27.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:27.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:27.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:28.008 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:28.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:28.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[102754]: pgmap v1025: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 47349/283380 objects degraded (16.709%); 1/283380 objects misplaced (0.000%) 2024-06-14T23:07:28.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:28.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:28.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3988210844' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:07:28.049 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:07:28.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[97913]: pgmap v1025: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 47349/283380 objects degraded (16.709%); 1/283380 objects misplaced (0.000%) 2024-06-14T23:07:28.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:28.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:28.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3988210844' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:07:28.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:27 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:07:28.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:28.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:28.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:28.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:28.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:27 smithi137 ceph-mon[90634]: pgmap v1025: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 47349/283380 objects degraded (16.709%); 1/283380 objects misplaced (0.000%) 2024-06-14T23:07:28.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:28.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:28.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3988210844' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:07:28.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:27 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:07:28.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:28.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:28.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 11 pgp_num to 16 2024-06-14T23:07:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:07:28.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:28.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:28.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:28.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:28.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:28.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:28.714 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:28.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:28.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:28.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:28.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:29.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:29.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:29.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:29.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:29.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:29.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:07:29.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:29.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/417141068' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:07:29.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 47349/283380 objects degraded (16.709%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:29.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:28 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:29.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:28 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:07:29.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/417141068' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:07:29.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:28 smithi137 ceph-mon[90634]: osdmap e268: 8 total, 7 up, 8 in 2024-06-14T23:07:29.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/417141068' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:07:29.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 47349/283380 objects degraded (16.709%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:29.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:29.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:07:29.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/417141068' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:07:29.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[97913]: osdmap e268: 8 total, 7 up, 8 in 2024-06-14T23:07:29.253 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/417141068' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:07:29.253 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 47349/283380 objects degraded (16.709%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:29.253 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:29.254 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:07:29.254 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/417141068' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:07:29.254 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:28 smithi086 ceph-mon[102754]: osdmap e268: 8 total, 7 up, 8 in 2024-06-14T23:07:29.261 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:29.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:29.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:29.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:29.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:29.536 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:29.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:29.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:29.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:29.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:29.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:29.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:29.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:29.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:29.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:29.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:30.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:30.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:30.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:30.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:30.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:30.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:29 smithi137 ceph-mon[90634]: pgmap v1027: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 968 op/s; 47349/283380 objects degraded (16.709%); 1/283380 objects misplaced (0.000%) 2024-06-14T23:07:30.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:30.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:29 smithi086 ceph-mon[102754]: pgmap v1027: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 968 op/s; 47349/283380 objects degraded (16.709%); 1/283380 objects misplaced (0.000%) 2024-06-14T23:07:30.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:29 smithi086 ceph-mon[97913]: pgmap v1027: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 968 op/s; 47349/283380 objects degraded (16.709%); 1/283380 objects misplaced (0.000%) 2024-06-14T23:07:30.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:30.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:30.451 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:30.473 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:30.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:30.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:30.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:30.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:30.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:30.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:30.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:30.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:30.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:30.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:30.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:30.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:31.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:31.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:31.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:31.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:31.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:31.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:31.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:31.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:31.434 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:31.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:31.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:31.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:31.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:31.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:31.702 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:31.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:31.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:31.827 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:31.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:31.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:31.906 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:07:31.906 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:07:31.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:31.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:32.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:32.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:32.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:32.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:31 smithi137 ceph-mon[90634]: pgmap v1028: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 45072/269937 objects degraded (16.697%); 1/269937 objects misplaced (0.000%) 2024-06-14T23:07:32.245 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:31 smithi086 ceph-mon[102754]: pgmap v1028: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 45072/269937 objects degraded (16.697%); 1/269937 objects misplaced (0.000%) 2024-06-14T23:07:32.245 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:31 smithi086 ceph-mon[97913]: pgmap v1028: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 45072/269937 objects degraded (16.697%); 1/269937 objects misplaced (0.000%) 2024-06-14T23:07:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:32.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:32.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:32.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:32.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:32.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:32.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:32.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:32.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:32.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:32.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:32.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:32.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:32.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-14T23:07:32.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:32.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:32.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:33.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:33.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:07:33.041 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:33.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:33.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:33.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:33.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:33.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:33.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:33.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:33.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:33.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:33.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:33.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:33.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:33.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:33.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:33.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:33.700 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:33.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:33.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:33.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:33.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:33.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:33.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:33.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:33.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:34.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:34.093 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:34.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:34.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:34.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:34.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:33 smithi137 ceph-mon[90634]: pgmap v1029: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 45072/269937 objects degraded (16.697%); 1/269937 objects misplaced (0.000%) 2024-06-14T23:07:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 45072/269937 objects degraded (16.697%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:34.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:33 smithi086 ceph-mon[97913]: pgmap v1029: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 45072/269937 objects degraded (16.697%); 1/269937 objects misplaced (0.000%) 2024-06-14T23:07:34.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 45072/269937 objects degraded (16.697%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:34.292 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:33 smithi086 ceph-mon[102754]: pgmap v1029: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 45072/269937 objects degraded (16.697%); 1/269937 objects misplaced (0.000%) 2024-06-14T23:07:34.292 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 45072/269937 objects degraded (16.697%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:34.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:34.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:34.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:34.520 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:34.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:34.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:34.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:34.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:34.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:34.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:34.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:34.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:34.958 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:34.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:35.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:35.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:07:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:07:34] "GET /metrics HTTP/1.1" 200 36622 "" "Prometheus/2.43.0" 2024-06-14T23:07:35.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:35.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:35.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:35.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:35.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:35.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:35.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:35.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:35.429 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-06-14T23:07:35.429 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:07:35.429 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-06-14T23:07:35.430 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:07:35.430 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:07:35.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-06-14T23:07:35.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:35.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:35.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:35.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:35.628 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:35.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:35.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:35.737 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:07:35.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:35.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:35.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:35.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:35.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:35.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:36.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:36.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:36.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:36.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:36.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:36.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:36.195 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:35 smithi137 ceph-mon[90634]: pgmap v1030: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 43590/263820 objects degraded (16.523%); 1/263820 objects misplaced (0.000%) 2024-06-14T23:07:36.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:36.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:36.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:35 smithi086 ceph-mon[102754]: pgmap v1030: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 43590/263820 objects degraded (16.523%); 1/263820 objects misplaced (0.000%) 2024-06-14T23:07:36.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:35 smithi086 ceph-mon[97913]: pgmap v1030: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 43590/263820 objects degraded (16.523%); 1/263820 objects misplaced (0.000%) 2024-06-14T23:07:36.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:36.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:36.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:36.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:36.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:36.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:36.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:36.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:36.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:36.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:36.847 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:36.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:36.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:36.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:36.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:37.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:37.112 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:37.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:37.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:37.240 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:37.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:37.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:37.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:37.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:37.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:37.506 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:37.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:37.573 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:07:37.573 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "" 2024-06-14T23:07:37.573 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:07:37.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:37.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:37.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:37.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:37.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:37.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:37.786 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:07:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:07:37.797 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:37.821 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:07:37.821 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:07:37.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:37.861 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:07:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:07:37.898 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:07:37.899 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:07:37.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:37.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:37.933 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:07:37.933 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:07:37.967 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:07:37.967 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:07:37.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-14T23:07:38.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:38.018 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:07:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:07:38.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:38.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:38.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:38.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:38.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:38.185 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:37 smithi137 ceph-mon[90634]: pgmap v1031: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 42739/255813 objects degraded (16.707%); 1/255813 objects misplaced (0.000%) 2024-06-14T23:07:38.185 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/252565781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:38.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:38.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:38.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:38.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:37 smithi086 ceph-mon[102754]: pgmap v1031: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 42739/255813 objects degraded (16.707%); 1/255813 objects misplaced (0.000%) 2024-06-14T23:07:38.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/252565781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:38.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:37 smithi086 ceph-mon[97913]: pgmap v1031: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 918 op/s; 42739/255813 objects degraded (16.707%); 1/255813 objects misplaced (0.000%) 2024-06-14T23:07:38.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/252565781' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:38.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:38.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:38.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:38.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:38.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:38.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:38.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:38.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:38.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:38.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:38.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:38.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:38.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:38.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:07:38.984 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:07:39.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:39.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:39.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:39.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:39.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:39.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:39.221 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 42739/255813 objects degraded (16.707%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:39.221 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 42739/255813 objects degraded (16.707%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:39.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 42739/255813 objects degraded (16.707%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:39.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:39.343 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:39.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:39.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:39.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:39.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:39.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:39.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:39.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:39.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:39.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:39.732 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:39.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:39.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:39.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:39.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:39.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:40.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:40.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:40.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:40.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:40.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:39 smithi137 ceph-mon[90634]: pgmap v1032: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 877 op/s; 42739/255813 objects degraded (16.707%); 1/255813 objects misplaced (0.000%) 2024-06-14T23:07:40.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:40.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:40.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:40.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:40.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:39 smithi086 ceph-mon[102754]: pgmap v1032: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 877 op/s; 42739/255813 objects degraded (16.707%); 1/255813 objects misplaced (0.000%) 2024-06-14T23:07:40.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:39 smithi086 ceph-mon[97913]: pgmap v1032: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 877 op/s; 42739/255813 objects degraded (16.707%); 1/255813 objects misplaced (0.000%) 2024-06-14T23:07:40.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:40.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:40.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:40.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:40.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:40.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:40.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:40.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:40.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:40.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:40.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:40.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:40.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:40.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:40.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:40.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:07:41.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:41.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:41.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:41.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:41.228 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:41.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:41.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:41.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:41.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:41.451 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:07:41.452 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:07:41.452 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:07:41.452 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:07:41.452 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:07:41.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:41.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:41.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:41.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:41.633 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:41.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:41.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:41.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:41.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:41.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:41.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:41.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:42.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:42.020 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:07:42.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:42.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:42.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:41 smithi086 ceph-mon[102754]: pgmap v1033: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s; 40391/241602 objects degraded (16.718%); 1/241602 objects misplaced (0.000%) 2024-06-14T23:07:42.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1758487675' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:07:42.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:41 smithi086 ceph-mon[97913]: pgmap v1033: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s; 40391/241602 objects degraded (16.718%); 1/241602 objects misplaced (0.000%) 2024-06-14T23:07:42.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1758487675' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:07:42.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:42.165 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:42.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:42.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:42.305 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:42.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:42.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:42.443 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:42.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:42.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:42.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:41 smithi137 ceph-mon[90634]: pgmap v1033: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s; 40391/241602 objects degraded (16.718%); 1/241602 objects misplaced (0.000%) 2024-06-14T23:07:42.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1758487675' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:07:42.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:42.569 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:42.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:42.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:42.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:42.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:42.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:42.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:42.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:42.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:42.995 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:42.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:43.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-14T23:07:43.095 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:43.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:43.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:43.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:43.211 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-06-14T23:07:43.211 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:07:43.212 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-06-14T23:07:43.212 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-06-14T23:07:43.212 DEBUG:teuthology.orchestra.run.smithi137:osd.6> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.6.service 2024-06-14T23:07:43.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl start ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.6 2024-06-14T23:07:43.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:43.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:43.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1758487675' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:07:43.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[97913]: osdmap e269: 8 total, 7 up, 8 in 2024-06-14T23:07:43.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:43.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:43.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:43.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1758487675' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:07:43.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[102754]: osdmap e269: 8 total, 7 up, 8 in 2024-06-14T23:07:43.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:43.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:43.277 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:43 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:43.278 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1758487675' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:07:43.278 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:43 smithi137 ceph-mon[90634]: osdmap e269: 8 total, 7 up, 8 in 2024-06-14T23:07:43.278 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:43.278 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:43.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:43.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:43.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:43.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:43.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:43.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:43.568 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:43 smithi137 systemd[1]: Starting Ceph osd.6 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:07:43.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:43.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:43.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:43.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:43.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:43.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:43.852 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:43.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:43.865 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:43 smithi137 podman[161673]: 2024-06-14 23:07:43.469379038 +0000 UTC m=+0.020035690 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:07:43.865 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:43 smithi137 podman[161673]: 2024-06-14 23:07:43.610612093 +0000 UTC m=+0.161268732 container create 6d6a27be83766c0ab454e4f5e2ebc7afa8ab49f64de19e50be7eba2764979d96 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux ) 2024-06-14T23:07:43.865 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:43 smithi137 podman[161673]: 2024-06-14 23:07:43.86405415 +0000 UTC m=+0.414710805 container init 6d6a27be83766c0ab454e4f5e2ebc7afa8ab49f64de19e50be7eba2764979d96 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-06-14T23:07:43.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:43.972 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:44.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:44.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:44.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:44.118 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:44 smithi086 ceph-mon[102754]: pgmap v1035: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 40391/241602 objects degraded (16.718%); 1/241602 objects misplaced (0.000%) 2024-06-14T23:07:44.118 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:44 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 40391/241602 objects degraded (16.718%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:44.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:44.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:44 smithi086 ceph-mon[97913]: pgmap v1035: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 40391/241602 objects degraded (16.718%); 1/241602 objects misplaced (0.000%) 2024-06-14T23:07:44.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:44 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 40391/241602 objects degraded (16.718%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:44.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:44.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:44.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:44.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:44 smithi137 ceph-mon[90634]: pgmap v1035: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 40391/241602 objects degraded (16.718%); 1/241602 objects misplaced (0.000%) 2024-06-14T23:07:44.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:44 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 40391/241602 objects degraded (16.718%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:44.245 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:43 smithi137 podman[161673]: 2024-06-14 23:07:43.871452449 +0000 UTC m=+0.422109093 container start 6d6a27be83766c0ab454e4f5e2ebc7afa8ab49f64de19e50be7eba2764979d96 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-14T23:07:44.245 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:43 smithi137 podman[161673]: 2024-06-14 23:07:43.963909352 +0000 UTC m=+0.514565989 container attach 6d6a27be83766c0ab454e4f5e2ebc7afa8ab49f64de19e50be7eba2764979d96 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate, GIT_CLEAN=True, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-06-14T23:07:44.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:44.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:44.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:44.420 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:44.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:44.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:44.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:44.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:44.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:44.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:44.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:44.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate[161705]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-14T23:07:44.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 bash[161673]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-14T23:07:44.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate[161705]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-14T23:07:44.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 bash[161673]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-14T23:07:44.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate[161705]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 bash[161673]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate[161705]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 bash[161673]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate[161705]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 bash[161673]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate[161705]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 bash[161673]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate[161705]: --> ceph-volume raw activate successful for osd ID: 6 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 bash[161673]: --> ceph-volume raw activate successful for osd ID: 6 2024-06-14T23:07:44.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:44 smithi137 podman[161673]: 2024-06-14 23:07:44.370288109 +0000 UTC m=+0.920944754 container died 6d6a27be83766c0ab454e4f5e2ebc7afa8ab49f64de19e50be7eba2764979d96 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612) 2024-06-14T23:07:44.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:44.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:44.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:44.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:44.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:44.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:45.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:45.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:45.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:45.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:45.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:45.210 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:45.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:45.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:45.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:07:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:07:44] "GET /metrics HTTP/1.1" 200 36615 "" "Prometheus/2.43.0" 2024-06-14T23:07:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:45.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:45.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:45.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:45.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:45.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:45.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:45.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:45.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:45.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:07:45.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:45.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:45.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:45.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:45.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:45.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:45.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:45.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:46.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:46.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:46.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:46.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:46.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:46.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:46.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:46.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:46 smithi086 ceph-mon[97913]: pgmap v1036: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 948 op/s; 38892/235362 objects degraded (16.524%); 1/235362 objects misplaced (0.000%) 2024-06-14T23:07:46.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:46 smithi086 ceph-mon[102754]: pgmap v1036: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 948 op/s; 38892/235362 objects degraded (16.524%); 1/235362 objects misplaced (0.000%) 2024-06-14T23:07:46.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:46.425 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:46 smithi137 ceph-mon[90634]: pgmap v1036: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 948 op/s; 38892/235362 objects degraded (16.524%); 1/235362 objects misplaced (0.000%) 2024-06-14T23:07:46.425 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:46 smithi137 podman[161864]: 2024-06-14 23:07:46.179047304 +0000 UTC m=+1.802315130 container remove 6d6a27be83766c0ab454e4f5e2ebc7afa8ab49f64de19e50be7eba2764979d96 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2) 2024-06-14T23:07:46.450 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:46.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:46.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:46.599 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:46.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:46.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:46.710 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:46 smithi137 podman[162039]: 2024-06-14 23:07:46.327513832 +0000 UTC m=+0.026121558 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:07:46.710 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:46 smithi137 podman[162039]: 2024-06-14 23:07:46.455248101 +0000 UTC m=+0.153855820 container create b31f6edf3d3c82e1db7f0bf5d932fece97d6d88da144e3640c43a4848b87b4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-e47ff4b) 2024-06-14T23:07:46.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:46.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:46.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:46.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T23:07:46.865 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:46.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:46.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:46.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:46.995 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:46 smithi137 podman[162039]: 2024-06-14 23:07:46.709198866 +0000 UTC m=+0.407806599 container init b31f6edf3d3c82e1db7f0bf5d932fece97d6d88da144e3640c43a4848b87b4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-14T23:07:46.995 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:46 smithi137 podman[162039]: 2024-06-14 23:07:46.716242804 +0000 UTC m=+0.414850518 container start b31f6edf3d3c82e1db7f0bf5d932fece97d6d88da144e3640c43a4848b87b4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS) 2024-06-14T23:07:46.995 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:46 smithi137 ceph-osd[162097]: -- 172.21.15.137:0/2491005314 <== mon.1 v2:172.21.15.137:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b03cf021a0 con 0x55b03ced5000 2024-06-14T23:07:46.995 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:46 smithi137 bash[162039]: b31f6edf3d3c82e1db7f0bf5d932fece97d6d88da144e3640c43a4848b87b4f4 2024-06-14T23:07:46.995 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:46 smithi137 systemd[1]: Started Ceph osd.6 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T23:07:47.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:47.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:47.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:47.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:47.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:47.177 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:07:47.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:47.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:47.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:47.303 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:07:47.303 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:07:47.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:47.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:47.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:47.480 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:07:47.303+0000 7ff3f8a7b2c0 -1 Falling back to public interface 2024-06-14T23:07:47.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:47.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:47.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:47.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:47.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:47.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:47.769 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:07:47.769 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:07:47.769 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:07:47.769 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:07:47.769 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-06-14T23:07:47.770 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:07:47.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:47.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:47.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:47.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:47.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:47.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:47.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:48.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:48.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:48 smithi086 ceph-mon[97913]: pgmap v1037: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 945 op/s; 38012/227430 objects degraded (16.714%); 1/227430 objects misplaced (0.000%) 2024-06-14T23:07:48.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2366359693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:48.065 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:48 smithi086 ceph-mon[102754]: pgmap v1037: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 945 op/s; 38012/227430 objects degraded (16.714%); 1/227430 objects misplaced (0.000%) 2024-06-14T23:07:48.066 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2366359693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:48.078 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:48.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:48.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:48.216 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:48.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:48.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:48.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:48.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:48.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:48.488 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:48.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:48.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:48 smithi137 ceph-mon[90634]: pgmap v1037: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 945 op/s; 38012/227430 objects degraded (16.714%); 1/227430 objects misplaced (0.000%) 2024-06-14T23:07:48.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2366359693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:48.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:48.624 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:48.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:48.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:48.763 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:48.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:48.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:48.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:48.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:48.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-14T23:07:49.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:07:49.080 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:07:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:49.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:49.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:49.109 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:49.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:49.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:49 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 38012/227430 objects degraded (16.714%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:49.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:49 smithi137 ceph-mon[90634]: pgmap v1038: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 945 op/s; 38012/227430 objects degraded (16.714%); 1/227430 objects misplaced (0.000%) 2024-06-14T23:07:49.254 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:49 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 38012/227430 objects degraded (16.714%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:49.254 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:49 smithi086 ceph-mon[102754]: pgmap v1038: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 945 op/s; 38012/227430 objects degraded (16.714%); 1/227430 objects misplaced (0.000%) 2024-06-14T23:07:49.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:49 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 38012/227430 objects degraded (16.714%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:49.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:49 smithi086 ceph-mon[97913]: pgmap v1038: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 945 op/s; 38012/227430 objects degraded (16.714%); 1/227430 objects misplaced (0.000%) 2024-06-14T23:07:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:49.273 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:49.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:49.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:49.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:49.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:49.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:49.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:49.575 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:config diff 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:config diff get 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:config get 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:config help [] 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:config set ... 2024-06-14T23:07:49.576 INFO:teuthology.orchestra.run.smithi137.stderr:admin_socket: invalid command 2024-06-14T23:07:49.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:49.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:49.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:49.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:49.814 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:49.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:49.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:49.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:49.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:49.987 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:50.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:50.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:50.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:50.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:50.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:50.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-14T23:07:50.316 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-06-14T23:07:50.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:50.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:50.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:50.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:50.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:50.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:50.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:50.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:50.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:50.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:50.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:50.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:50.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:50.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:50.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:50.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:50.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:51.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:51.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:51.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:07:51.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:51.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:51.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:51.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:51.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:51.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:51.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:51.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:51.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:51.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:51.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:51.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:51.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:51.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:51.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:51.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:51.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:51.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:51.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:51.952 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:07:51.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:51.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:51 smithi137 ceph-mon[90634]: pgmap v1039: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 35662/213321 objects degraded (16.718%); 1/213321 objects misplaced (0.000%) 2024-06-14T23:07:51.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1718524011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:07:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:52.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:52.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:52.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:52.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:52.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:52.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:52.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:52.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:52.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:51 smithi086 ceph-mon[97913]: pgmap v1039: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 35662/213321 objects degraded (16.718%); 1/213321 objects misplaced (0.000%) 2024-06-14T23:07:52.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1718524011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:07:52.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:51 smithi086 ceph-mon[102754]: pgmap v1039: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 35662/213321 objects degraded (16.718%); 1/213321 objects misplaced (0.000%) 2024-06-14T23:07:52.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1718524011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:07:52.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:52.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:52.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:52.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:52.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:52.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:52.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:52.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:52.670 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:52.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:52.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:52.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:52.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:52.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:52.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:52.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:53.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:53.078 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:53.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:53.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:52 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:53.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1718524011' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:07:53.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:52 smithi086 ceph-mon[97913]: osdmap e270: 8 total, 7 up, 8 in 2024-06-14T23:07:53.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:52 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:53.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1718524011' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:07:53.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:52 smithi086 ceph-mon[102754]: osdmap e270: 8 total, 7 up, 8 in 2024-06-14T23:07:53.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:53.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:53.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:53.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:52 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:07:53.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1718524011' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:07:53.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:52 smithi137 ceph-mon[90634]: osdmap e270: 8 total, 7 up, 8 in 2024-06-14T23:07:53.251 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:53.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:53.346 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:53.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:53.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:53.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:53.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:53.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:53.612 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:53.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:53.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:53.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:53.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:53.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:53.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:53.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:53.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:53.993 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:07:53.993 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:07:54.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:54.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:54.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:53 smithi086 ceph-mon[102754]: pgmap v1041: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 35662/213321 objects degraded (16.718%); 1/213321 objects misplaced (0.000%) 2024-06-14T23:07:54.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 35662/213321 objects degraded (16.718%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:54.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:53 smithi086 ceph-mon[97913]: pgmap v1041: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 35662/213321 objects degraded (16.718%); 1/213321 objects misplaced (0.000%) 2024-06-14T23:07:54.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 35662/213321 objects degraded (16.718%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:54.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:54.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:54.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:54.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:53 smithi137 ceph-mon[90634]: pgmap v1041: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 35662/213321 objects degraded (16.718%); 1/213321 objects misplaced (0.000%) 2024-06-14T23:07:54.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 35662/213321 objects degraded (16.718%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:54.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:54.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:54.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:54.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:54.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:54.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:54.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:54.559 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:54.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:54.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:54.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-14T23:07:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:54.709 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:54.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:54.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:54.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:54.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:54.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:54.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:54.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:55.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:55.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:55.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:55.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:07:55.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:55.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:54 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:07:55.245 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:55 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:07:55.117+0000 7ff3f8a7b2c0 -1 osd.6 263 log_to_monitors true 2024-06-14T23:07:55.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:55.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:55.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:55.300 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:54 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:07:55.300 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:54 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:07:55.300 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:07:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:07:54] "GET /metrics HTTP/1.1" 200 36615 "" "Prometheus/2.43.0" 2024-06-14T23:07:55.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T23:07:55.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:55.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:55.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:55.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:55.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:55.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:55.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:55.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:55.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:55.881 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:55.898 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:55.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:55.915 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:07:55.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:56.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:56.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:56.047 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:55 smithi086 ceph-mon[102754]: pgmap v1042: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 34144/207069 objects degraded (16.489%); 1/207069 objects misplaced (0.000%) 2024-06-14T23:07:56.047 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:55 smithi086 ceph-mon[102754]: from='osd.6 [v2:172.21.15.137:6816/2783214843,v1:172.21.15.137:6817/2783214843]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T23:07:56.047 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:55 smithi086 ceph-mon[102754]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T23:07:56.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:55 smithi086 ceph-mon[97913]: pgmap v1042: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 34144/207069 objects degraded (16.489%); 1/207069 objects misplaced (0.000%) 2024-06-14T23:07:56.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:55 smithi086 ceph-mon[97913]: from='osd.6 [v2:172.21.15.137:6816/2783214843,v1:172.21.15.137:6817/2783214843]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T23:07:56.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:55 smithi086 ceph-mon[97913]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T23:07:56.086 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:07:56.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:56.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:56.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:56.235 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:07:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:07:56.007+0000 7ff3ef024640 -1 osd.6 263 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T23:07:56.235 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:55 smithi137 ceph-mon[90634]: pgmap v1042: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 942 op/s; 34144/207069 objects degraded (16.489%); 1/207069 objects misplaced (0.000%) 2024-06-14T23:07:56.235 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:55 smithi137 ceph-mon[90634]: from='osd.6 [v2:172.21.15.137:6816/2783214843,v1:172.21.15.137:6817/2783214843]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T23:07:56.235 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:55 smithi137 ceph-mon[90634]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-14T23:07:56.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:56.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:56.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:56.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:56.456 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:56.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:56.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:56.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:56.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:56.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:56.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:07:56.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:56.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:07:56.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:56.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:56.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:56.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:57.024 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:57.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:57.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:07:57.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:56 smithi086 ceph-mon[97913]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-14T23:07:57.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:56 smithi086 ceph-mon[97913]: osdmap e271: 8 total, 7 up, 8 in 2024-06-14T23:07:57.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:56 smithi086 ceph-mon[97913]: from='osd.6 [v2:172.21.15.137:6816/2783214843,v1:172.21.15.137:6817/2783214843]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:07:57.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:56 smithi086 ceph-mon[97913]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:07:57.153 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:56 smithi086 ceph-mon[102754]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-14T23:07:57.153 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:56 smithi086 ceph-mon[102754]: osdmap e271: 8 total, 7 up, 8 in 2024-06-14T23:07:57.153 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:56 smithi086 ceph-mon[102754]: from='osd.6 [v2:172.21.15.137:6816/2783214843,v1:172.21.15.137:6817/2783214843]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:07:57.153 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:56 smithi086 ceph-mon[102754]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:07:57.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:57.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:07:57.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:56 smithi137 ceph-mon[90634]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-14T23:07:57.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:56 smithi137 ceph-mon[90634]: osdmap e271: 8 total, 7 up, 8 in 2024-06-14T23:07:57.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:56 smithi137 ceph-mon[90634]: from='osd.6 [v2:172.21.15.137:6816/2783214843,v1:172.21.15.137:6817/2783214843]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:07:57.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:56 smithi137 ceph-mon[90634]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:07:57.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:57.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:57.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:57.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:57.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:57.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:57.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:57.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:57.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:57.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:57.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:07:57.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:57.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:57.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:57.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:57.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:57.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:58.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:58.054 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:58.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:58.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[97913]: pgmap v1044: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 33314/199365 objects degraded (16.710%); 1/199365 objects misplaced (0.001%) 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[97913]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[97913]: osd.6 [v2:172.21.15.137:6816/2783214843,v1:172.21.15.137:6817/2783214843] boot 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[97913]: osdmap e272: 8 total, 8 up, 8 in 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[102754]: pgmap v1044: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 33314/199365 objects degraded (16.710%); 1/199365 objects misplaced (0.001%) 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[102754]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T23:07:58.176 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[102754]: osd.6 [v2:172.21.15.137:6816/2783214843,v1:172.21.15.137:6817/2783214843] boot 2024-06-14T23:07:58.177 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[102754]: osdmap e272: 8 total, 8 up, 8 in 2024-06-14T23:07:58.177 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T23:07:58.177 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:58.177 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:58.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:58.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:58.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:58.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:07:58.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:07:58.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:07:58.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:07:58.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:07:58.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:07:58.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:07:58.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:07:58.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:07:58.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:07:58.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:07:58.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:07:58.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:07:58.234 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:07:58.235 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:07:58.235 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:07:58.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:58.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:58.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:58.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:07:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:58.426 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:58.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:07:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:58 smithi137 ceph-mon[90634]: pgmap v1044: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 33314/199365 objects degraded (16.710%); 1/199365 objects misplaced (0.001%) 2024-06-14T23:07:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:58 smithi137 ceph-mon[90634]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T23:07:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:58 smithi137 ceph-mon[90634]: osd.6 [v2:172.21.15.137:6816/2783214843,v1:172.21.15.137:6817/2783214843] boot 2024-06-14T23:07:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:58 smithi137 ceph-mon[90634]: osdmap e272: 8 total, 8 up, 8 in 2024-06-14T23:07:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-14T23:07:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:07:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:07:58.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:07:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:58.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:58.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:07:58.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:58.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:58.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:58.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:58.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:58.791 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:07:58.791 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:07:58.791 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-06-14T23:07:58.791 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:07:58.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:58.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:58.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:58.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:58.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:59.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:59 smithi086 ceph-mon[97913]: osdmap e273: 8 total, 8 up, 8 in 2024-06-14T23:07:59.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:59 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3426294000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:59.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:59.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:59.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:59.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:59.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:59.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:59.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:59.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:59.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:59.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:59.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:59 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 33314/199365 objects degraded (16.710%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:59.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:07:59 smithi086 ceph-mon[97913]: osdmap e274: 8 total, 8 up, 8 in 2024-06-14T23:07:59.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:59 smithi086 ceph-mon[102754]: osdmap e273: 8 total, 8 up, 8 in 2024-06-14T23:07:59.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:59 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3426294000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:59.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:59 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 33314/199365 objects degraded (16.710%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:59.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:07:59 smithi086 ceph-mon[102754]: osdmap e274: 8 total, 8 up, 8 in 2024-06-14T23:07:59.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:59.426 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:59 smithi137 ceph-mon[90634]: osdmap e273: 8 total, 8 up, 8 in 2024-06-14T23:07:59.426 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:59 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3426294000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:07:59.427 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:59 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 33314/199365 objects degraded (16.710%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:07:59.427 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:07:59 smithi137 ceph-mon[90634]: osdmap e274: 8 total, 8 up, 8 in 2024-06-14T23:07:59.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:59.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:59.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:59.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:59.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:59.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:07:59.726 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:59.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:07:59.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:07:59.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:07:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:59.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:07:59.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:07:59.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:07:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:07:59.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:00.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:00.137 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:00.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:00.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:00.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:00.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:00.328 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:00 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:08:00.328 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:00 smithi137 ceph-mon[90634]: 1.5 scrub ok 2024-06-14T23:08:00.328 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:00 smithi137 ceph-mon[90634]: pgmap v1047: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 774 op/s; 33314/199365 objects degraded (16.710%); 1/199365 objects misplaced (0.001%) 2024-06-14T23:08:00.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:00 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:08:00.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:00 smithi086 ceph-mon[102754]: 1.5 scrub ok 2024-06-14T23:08:00.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:00 smithi086 ceph-mon[102754]: pgmap v1047: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 774 op/s; 33314/199365 objects degraded (16.710%); 1/199365 objects misplaced (0.001%) 2024-06-14T23:08:00.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:00 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:08:00.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:00 smithi086 ceph-mon[97913]: 1.5 scrub ok 2024-06-14T23:08:00.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:00 smithi086 ceph-mon[97913]: pgmap v1047: 109 pgs: 4 active+clean+remapped, 28 active+undersized, 12 active+undersized+degraded, 65 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 774 op/s; 33314/199365 objects degraded (16.710%); 1/199365 objects misplaced (0.001%) 2024-06-14T23:08:00.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:00.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:00.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:00.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:00.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:00.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:00.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:00.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:00.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:00.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:00.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:00.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:00.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:00.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:00.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:01.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:01.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:01.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:01.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:01.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:01.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:01.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:01.408 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:01.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:01.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:01.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:01.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:01.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:01.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:01.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:01.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:08:01.785 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:08:01.785 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:08:01.786 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:08:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:08:01.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:01.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:01.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:01.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:01.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:01.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:02.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:02.113 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:02.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:02.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:02.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:02.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:02.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:02.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:02 smithi086 ceph-mon[102754]: pgmap v1049: 109 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+clean+remapped, 6 active+undersized, 2 active+undersized+degraded, 93 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 465 op/s; 24366/192510 objects degraded (12.657%); 0 B/s, 10 objects/s recovering 2024-06-14T23:08:02.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:02 smithi086 ceph-mon[97913]: pgmap v1049: 109 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+clean+remapped, 6 active+undersized, 2 active+undersized+degraded, 93 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 465 op/s; 24366/192510 objects degraded (12.657%); 0 B/s, 10 objects/s recovering 2024-06-14T23:08:02.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:02.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:02.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:02.414 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:02 smithi137 ceph-mon[90634]: pgmap v1049: 109 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+clean+remapped, 6 active+undersized, 2 active+undersized+degraded, 93 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 465 op/s; 24366/192510 objects degraded (12.657%); 0 B/s, 10 objects/s recovering 2024-06-14T23:08:02.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:02.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:02.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:02.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:02.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:02.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:02.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:02.807 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:02.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:02.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:02.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:02.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:03.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:03.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:03.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:03.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:03.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:03.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:03.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:03.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1122250685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:03.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:03 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:03.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:03 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:03.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:03 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:03.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1122250685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:03.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:03 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:03.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:03 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:03.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:03 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:03.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:03.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:03.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:03.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:08:03.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1122250685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:03.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:03 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:03.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:03 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:03.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:03 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:03.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:03.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:03.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:03.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:03.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:03.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:03.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:03.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:03.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:03.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:03.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:03.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:04.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:04.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:04.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:04.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:04.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:04.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:04.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:04.299 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[97913]: pgmap v1050: 109 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 101 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 436 op/s; 23572/191598 objects degraded (12.303%); 13 KiB/s, 8 objects/s recovering 2024-06-14T23:08:04.299 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:08:04.299 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:08:04.299 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[97913]: osdmap e275: 8 total, 8 up, 8 in 2024-06-14T23:08:04.299 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 23572/191598 objects degraded (12.303%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:04.300 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:08:04.300 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[102754]: pgmap v1050: 109 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 101 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 436 op/s; 23572/191598 objects degraded (12.303%); 13 KiB/s, 8 objects/s recovering 2024-06-14T23:08:04.300 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:08:04.300 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:08:04.300 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[102754]: osdmap e275: 8 total, 8 up, 8 in 2024-06-14T23:08:04.300 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 23572/191598 objects degraded (12.303%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:04.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:04.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:04.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:04.443 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:04.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:04.478 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-06-14T23:08:04.479 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:08:04.479 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_6 2024-06-14T23:08:04.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:08:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:04 smithi137 ceph-mon[90634]: pgmap v1050: 109 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 101 active+clean; 3.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 436 op/s; 23572/191598 objects degraded (12.303%); 13 KiB/s, 8 objects/s recovering 2024-06-14T23:08:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:04 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:08:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:04 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:08:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:04 smithi137 ceph-mon[90634]: osdmap e275: 8 total, 8 up, 8 in 2024-06-14T23:08:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:04 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 23572/191598 objects degraded (12.303%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:04.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:04.582 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:04.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:04.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:04.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:04.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:04.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:04.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:04.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:04.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:04.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:04.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:05.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:05.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:05.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:05.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:05.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:05.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:05.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:05 smithi086 ceph-mon[97913]: pgmap v1052: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 6.1 KiB/s rd, 16 KiB/s wr, 548 op/s; 23567/188133 objects degraded (12.527%); 11 KiB/s, 7 objects/s recovering 2024-06-14T23:08:05.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:08:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:08:04] "GET /metrics HTTP/1.1" 200 36471 "" "Prometheus/2.43.0" 2024-06-14T23:08:05.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:05 smithi086 ceph-mon[102754]: pgmap v1052: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 6.1 KiB/s rd, 16 KiB/s wr, 548 op/s; 23567/188133 objects degraded (12.527%); 11 KiB/s, 7 objects/s recovering 2024-06-14T23:08:05.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:05.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:05.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:05.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:05 smithi137 ceph-mon[90634]: pgmap v1052: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 6.1 KiB/s rd, 16 KiB/s wr, 548 op/s; 23567/188133 objects degraded (12.527%); 11 KiB/s, 7 objects/s recovering 2024-06-14T23:08:05.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:05.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:05.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:05.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:05.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:05.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:05.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:05.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:05.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:05.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:05.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:06.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:06.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:06.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:06.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:06.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:06.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:06.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:06.321 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:06.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:06.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:06.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:06.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:06.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:06.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:06.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:06.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:06.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:06.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:06.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:06.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:06.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:06.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:06.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:06.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:07.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:07.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:07.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:07.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:07.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:07.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:07.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:07.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:07.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:07.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:07.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:07.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:07.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:07.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:07.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:07.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:07.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:07.878 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:07.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:07.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:07 smithi086 ceph-mon[97913]: pgmap v1053: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 KiB/s wr, 768 op/s; 23567/181008 objects degraded (13.020%); 9.8 KiB/s, 25 objects/s recovering 2024-06-14T23:08:07.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-14T23:08:07.972 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:07.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:07.981 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:08.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:08.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:08.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:08.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:08.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:08.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:08.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:07 smithi137 ceph-mon[90634]: pgmap v1053: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 KiB/s wr, 768 op/s; 23567/181008 objects degraded (13.020%); 9.8 KiB/s, 25 objects/s recovering 2024-06-14T23:08:08.252 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:07 smithi086 ceph-mon[102754]: pgmap v1053: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 KiB/s wr, 768 op/s; 23567/181008 objects degraded (13.020%); 9.8 KiB/s, 25 objects/s recovering 2024-06-14T23:08:08.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:08.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:08.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:08.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:08.477 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:08.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:08.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:08.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:08.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:08.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:08.774 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:08.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:08.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:08.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:08.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:08.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:09.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:09.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:09.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:09.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:09.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:09.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:09.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:09.184 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 23567/181008 objects degraded (13.020%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:09.184 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 23567/181008 objects degraded (13.020%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:09.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 23567/181008 objects degraded (13.020%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:09.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:09.325 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:09.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:09.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:09.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:09.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:09.569 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:09.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:09.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:09.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:08:09.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:09.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:09.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:09.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:09.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:09.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:09.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:10.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:09 smithi086 ceph-mon[97913]: pgmap v1054: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 4.2 KiB/s rd, 11 KiB/s wr, 647 op/s; 23567/180264 objects degraded (13.074%); 7.9 KiB/s, 21 objects/s recovering 2024-06-14T23:08:10.047 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:09 smithi086 ceph-mon[102754]: pgmap v1054: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 4.2 KiB/s rd, 11 KiB/s wr, 647 op/s; 23567/180264 objects degraded (13.074%); 7.9 KiB/s, 21 objects/s recovering 2024-06-14T23:08:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:10.048 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:10.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:10.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:10.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:10.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:10.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:09 smithi137 ceph-mon[90634]: pgmap v1054: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 4.2 KiB/s rd, 11 KiB/s wr, 647 op/s; 23567/180264 objects degraded (13.074%); 7.9 KiB/s, 21 objects/s recovering 2024-06-14T23:08:10.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:10.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:10.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:10.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:10.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:10.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:10.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:10.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:10.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:10.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:10.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:10.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:10.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:10.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:10.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:10.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:10.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:11.013 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:11.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:11.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:11.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:11.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:11.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:11.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:11.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:11.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:11.460 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:11.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:11.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:11.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:11.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:11.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:11.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:11.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:11.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:11.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:11.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:12.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:12.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:12.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:12.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:12.176 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:12.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:12.235 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:11 smithi086 ceph-mon[102754]: pgmap v1055: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 4.2 KiB/s rd, 11 KiB/s wr, 789 op/s; 23567/168906 objects degraded (13.953%); 7.9 KiB/s, 30 objects/s recovering 2024-06-14T23:08:12.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:11 smithi086 ceph-mon[97913]: pgmap v1055: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 4.2 KiB/s rd, 11 KiB/s wr, 789 op/s; 23567/168906 objects degraded (13.953%); 7.9 KiB/s, 30 objects/s recovering 2024-06-14T23:08:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:11 smithi137 ceph-mon[90634]: pgmap v1055: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 4.2 KiB/s rd, 11 KiB/s wr, 789 op/s; 23567/168906 objects degraded (13.953%); 7.9 KiB/s, 30 objects/s recovering 2024-06-14T23:08:12.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:12.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:12.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:12.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:12.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:12.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:12.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:12.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:12.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:12.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:12.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:12.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:12.815 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:08:12.817 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:08:12.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:12.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:13.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:13.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:13.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:13.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:13.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:13.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:13.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:13.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:13.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:13.443 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:13.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:13.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:13.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:13.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:13.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:13.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:13.733 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:13.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:13 smithi137 ceph-mon[90634]: from='client.36396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:08:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:13 smithi137 ceph-mon[90634]: pgmap v1056: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 778 op/s; 23567/168234 objects degraded (14.008%); 0 B/s, 30 objects/s recovering 2024-06-14T23:08:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 23567/168906 objects degraded (13.953%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:13.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:13.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:13.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:13.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[97913]: from='client.36396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:08:13.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[97913]: pgmap v1056: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 778 op/s; 23567/168234 objects degraded (14.008%); 0 B/s, 30 objects/s recovering 2024-06-14T23:08:13.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 23567/168906 objects degraded (13.953%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:13.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:13.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:13.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[102754]: from='client.36396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:08:13.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[102754]: pgmap v1056: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 778 op/s; 23567/168234 objects degraded (14.008%); 0 B/s, 30 objects/s recovering 2024-06-14T23:08:13.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 23567/168906 objects degraded (13.953%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:13.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:13.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:13.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:13.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:14.049 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:14.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:14.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:14.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:14.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:14.255 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:14.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:14.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:14.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:14.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:14.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:14.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:14.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:14.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:14.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:14.698 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:14.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:14.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:14.832 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1055,"stamp":"2024-06-14T23:08:10.882047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3688845911,"num_objects":56302,"num_object_clones":0,"num_object_copies":168906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56302,"num_whiteouts":0,"num_read":67,"num_read_kb":125,"num_write":198454,"num_write_kb":8148712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":832493,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126074,"ondisk_log_size":126074,"up":327,"acting":319,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18209764,"kb_used_data":12738656,"kb_used_omap":35,"kb_used_meta":5470876,"kb_avail":731751452,"statfs":{"total":767960285184,"available":749313486848,"internally_reserved":0,"allocated":13044383744,"data_stored":13033380454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36089,"internal_metadata":5602177799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,8,10,13,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-515610413,"num_objects":-7868,"num_object_clones":0,"num_object_copies":-23604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7868,"num_whiteouts":0,"num_read":22,"num_read_kb":42,"num_write":7879,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006428"},"pg_stats":[{"pgid":"1.11","version":"77'161","reported_seq":650,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995019+0000","last_change":"2024-06-14T23:07:12.090120+0000","last_active":"2024-06-14T23:07:58.995019+0000","last_peered":"2024-06-14T23:07:58.995019+0000","last_clean":"2024-06-14T23:07:58.995019+0000","last_became_active":"2024-06-14T23:07:00.939305+0000","last_became_peered":"2024-06-14T23:07:00.939305+0000","last_unstale":"2024-06-14T23:07:58.995019+0000","last_undegraded":"2024-06-14T23:07:58.995019+0000","last_fullsized":"2024-06-14T23:07:58.995019+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:15:19.613254+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":208,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995315+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:07:58.995315+0000","last_peered":"2024-06-14T23:07:58.995315+0000","last_clean":"2024-06-14T23:07:58.995315+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:07:58.995315+0000","last_undegraded":"2024-06-14T23:07:58.995315+0000","last_fullsized":"2024-06-14T23:07:58.995315+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":270,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.993814+0000","last_change":"2024-06-14T23:04:34.895560+0000","last_active":"2024-06-14T23:07:58.993814+0000","last_peered":"2024-06-14T23:07:58.993814+0000","last_clean":"2024-06-14T23:07:58.993814+0000","last_became_active":"2024-06-14T23:04:34.887055+0000","last_became_peered":"2024-06-14T23:04:34.887055+0000","last_unstale":"2024-06-14T23:07:58.993814+0000","last_undegraded":"2024-06-14T23:07:58.993814+0000","last_fullsized":"2024-06-14T23:07:58.993814+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:26:36.259241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":594,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.005812+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:07:59.005812+0000","last_peered":"2024-06-14T23:07:59.005812+0000","last_clean":"2024-06-14T23:07:59.005812+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:07:59.005812+0000","last_undegraded":"2024-06-14T23:07:59.005812+0000","last_fullsized":"2024-06-14T23:07:59.005812+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":235,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.018924+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:07:59.018924+0000","last_peered":"2024-06-14T23:07:59.018924+0000","last_clean":"2024-06-14T23:07:59.018924+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:07:59.018924+0000","last_undegraded":"2024-06-14T23:07:59.018924+0000","last_fullsized":"2024-06-14T23:07:59.018924+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":229,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.997888+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:07:58.997888+0000","last_peered":"2024-06-14T23:07:58.997888+0000","last_clean":"2024-06-14T23:07:58.997888+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:07:58.997888+0000","last_undegraded":"2024-06-14T23:07:58.997888+0000","last_fullsized":"2024-06-14T23:07:58.997888+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":614,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.013977+0000","last_change":"2024-06-14T23:07:09.705052+0000","last_active":"2024-06-14T23:07:59.013977+0000","last_peered":"2024-06-14T23:07:59.013977+0000","last_clean":"2024-06-14T23:07:59.013977+0000","last_became_active":"2024-06-14T23:07:00.938267+0000","last_became_peered":"2024-06-14T23:07:00.938267+0000","last_unstale":"2024-06-14T23:07:59.013977+0000","last_undegraded":"2024-06-14T23:07:59.013977+0000","last_fullsized":"2024-06-14T23:07:59.013977+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.704172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.704172+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:44:55.998829+0000","scrub_duration":0.0024350880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":112,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.001425+0000","last_change":"2024-06-14T23:06:18.999778+0000","last_active":"2024-06-14T23:07:59.001425+0000","last_peered":"2024-06-14T23:07:59.001425+0000","last_clean":"2024-06-14T23:07:59.001425+0000","last_became_active":"2024-06-14T23:06:18.999306+0000","last_became_peered":"2024-06-14T23:06:18.999306+0000","last_unstale":"2024-06-14T23:07:59.001425+0000","last_undegraded":"2024-06-14T23:07:59.001425+0000","last_fullsized":"2024-06-14T23:07:59.001425+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:18:01.226932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":159,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.999829+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:07:58.999829+0000","last_peered":"2024-06-14T23:07:58.999829+0000","last_clean":"2024-06-14T23:07:58.999829+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:07:58.999829+0000","last_undegraded":"2024-06-14T23:07:58.999829+0000","last_fullsized":"2024-06-14T23:07:58.999829+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":610,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.024177+0000","last_change":"2024-06-14T23:07:59.023708+0000","last_active":"2024-06-14T23:07:59.024177+0000","last_peered":"2024-06-14T23:07:59.024177+0000","last_clean":"2024-06-14T23:07:59.024177+0000","last_became_active":"2024-06-14T23:07:59.023129+0000","last_became_peered":"2024-06-14T23:07:59.023129+0000","last_unstale":"2024-06-14T23:07:59.024177+0000","last_undegraded":"2024-06-14T23:07:59.024177+0000","last_fullsized":"2024-06-14T23:07:59.024177+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:45:29.886582+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":231,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.001847+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:07:59.001847+0000","last_peered":"2024-06-14T23:07:59.001847+0000","last_clean":"2024-06-14T23:07:59.001847+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:07:59.001847+0000","last_undegraded":"2024-06-14T23:07:59.001847+0000","last_fullsized":"2024-06-14T23:07:59.001847+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":241,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.997557+0000","last_change":"2024-06-14T23:07:58.008693+0000","last_active":"2024-06-14T23:07:58.997557+0000","last_peered":"2024-06-14T23:07:58.997557+0000","last_clean":"2024-06-14T23:07:58.997557+0000","last_became_active":"2024-06-14T23:07:58.008150+0000","last_became_peered":"2024-06-14T23:07:58.008150+0000","last_unstale":"2024-06-14T23:07:58.997557+0000","last_undegraded":"2024-06-14T23:07:58.997557+0000","last_fullsized":"2024-06-14T23:07:58.997557+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:38:32.621234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":651,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.001251+0000","last_change":"2024-06-14T23:07:21.981837+0000","last_active":"2024-06-14T23:07:59.001251+0000","last_peered":"2024-06-14T23:07:59.001251+0000","last_clean":"2024-06-14T23:07:59.001251+0000","last_became_active":"2024-06-14T23:07:00.947390+0000","last_became_peered":"2024-06-14T23:07:00.947390+0000","last_unstale":"2024-06-14T23:07:59.001251+0000","last_undegraded":"2024-06-14T23:07:59.001251+0000","last_fullsized":"2024-06-14T23:07:59.001251+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:21.981787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:21.981787+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:47:43.919655+0000","scrub_duration":0.0025287560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"275'12468","reported_seq":12514,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:09.555574+0000","last_change":"2024-06-14T23:05:03.621489+0000","last_active":"2024-06-14T23:08:09.555574+0000","last_peered":"2024-06-14T23:08:09.555574+0000","last_clean":"2024-06-14T23:08:09.555574+0000","last_became_active":"2024-06-14T23:05:03.621114+0000","last_became_peered":"2024-06-14T23:05:03.621114+0000","last_unstale":"2024-06-14T23:08:09.555574+0000","last_undegraded":"2024-06-14T23:08:09.555574+0000","last_fullsized":"2024-06-14T23:08:09.555574+0000","mapping_epoch":237,"log_start":"247'3820","ondisk_log_start":"247'3820","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":8648,"log_dups_size":0,"ondisk_log_size":8648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:33.811591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230293504,"num_objects":3514,"num_object_clones":0,"num_object_copies":10542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12468,"num_write_kb":511424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":225,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.994535+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:07:58.994535+0000","last_peered":"2024-06-14T23:07:58.994535+0000","last_clean":"2024-06-14T23:07:58.994535+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:07:58.994535+0000","last_undegraded":"2024-06-14T23:07:58.994535+0000","last_fullsized":"2024-06-14T23:07:58.994535+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":235,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995688+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:07:58.995688+0000","last_peered":"2024-06-14T23:07:58.995688+0000","last_clean":"2024-06-14T23:07:58.995688+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:07:58.995688+0000","last_undegraded":"2024-06-14T23:07:58.995688+0000","last_fullsized":"2024-06-14T23:07:58.995688+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":637,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.029024+0000","last_change":"2024-06-14T23:07:59.028296+0000","last_active":"2024-06-14T23:07:59.029024+0000","last_peered":"2024-06-14T23:07:59.029024+0000","last_clean":"2024-06-14T23:07:59.029024+0000","last_became_active":"2024-06-14T23:07:59.003433+0000","last_became_peered":"2024-06-14T23:07:59.003433+0000","last_unstale":"2024-06-14T23:07:59.029024+0000","last_undegraded":"2024-06-14T23:07:59.029024+0000","last_fullsized":"2024-06-14T23:07:59.029024+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.188207+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:21:33.569950+0000","scrub_duration":0.019682448000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"275'12487","reported_seq":12545,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:10.784283+0000","last_change":"2024-06-14T23:06:19.064148+0000","last_active":"2024-06-14T23:08:10.784283+0000","last_peered":"2024-06-14T23:08:10.784283+0000","last_clean":"2024-06-14T23:08:10.784283+0000","last_became_active":"2024-06-14T23:06:19.007544+0000","last_became_peered":"2024-06-14T23:06:19.007544+0000","last_unstale":"2024-06-14T23:08:10.784283+0000","last_undegraded":"2024-06-14T23:08:10.784283+0000","last_fullsized":"2024-06-14T23:08:10.784283+0000","mapping_epoch":253,"log_start":"244'2400","ondisk_log_start":"244'2400","created":237,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:47.143283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230490112,"num_objects":3517,"num_object_clones":0,"num_object_copies":10551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12487,"num_write_kb":512128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":226,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.994127+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:07:58.994127+0000","last_peered":"2024-06-14T23:07:58.994127+0000","last_clean":"2024-06-14T23:07:58.994127+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:07:58.994127+0000","last_undegraded":"2024-06-14T23:07:58.994127+0000","last_fullsized":"2024-06-14T23:07:58.994127+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":252,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000258+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:07:59.000258+0000","last_peered":"2024-06-14T23:07:59.000258+0000","last_clean":"2024-06-14T23:07:59.000258+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:07:59.000258+0000","last_undegraded":"2024-06-14T23:07:59.000258+0000","last_fullsized":"2024-06-14T23:07:59.000258+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":544,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.004155+0000","last_change":"2024-06-14T23:07:59.003744+0000","last_active":"2024-06-14T23:07:59.004155+0000","last_peered":"2024-06-14T23:07:59.004155+0000","last_clean":"2024-06-14T23:07:59.004155+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:07:59.004155+0000","last_undegraded":"2024-06-14T23:07:59.004155+0000","last_fullsized":"2024-06-14T23:07:59.004155+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:17.057842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:17.057842+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":42,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"275'12504","reported_seq":12567,"reported_epoch":275,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:08:09.561278+0000","last_change":"2024-06-14T23:07:59.229371+0000","last_active":"2024-06-14T23:08:09.561278+0000","last_peered":"2024-06-14T23:08:09.561278+0000","last_clean":"2024-06-14T23:06:59.932215+0000","last_became_active":"2024-06-14T23:07:59.191870+0000","last_became_peered":"2024-06-14T23:07:59.191870+0000","last_unstale":"2024-06-14T23:08:09.561278+0000","last_undegraded":"2024-06-14T23:07:59.018018+0000","last_fullsized":"2024-06-14T23:07:58.995777+0000","mapping_epoch":273,"log_start":"248'3931","ondisk_log_start":"248'3931","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:45.161702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231342080,"num_objects":3530,"num_object_clones":0,"num_object_copies":10590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12504,"num_write_kb":513088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":3530}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":170,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.018849+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:07:59.018849+0000","last_peered":"2024-06-14T23:07:59.018849+0000","last_clean":"2024-06-14T23:07:59.018849+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:07:59.018849+0000","last_undegraded":"2024-06-14T23:07:59.018849+0000","last_fullsized":"2024-06-14T23:07:59.018849+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":216,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.998208+0000","last_change":"2024-06-14T23:07:58.028433+0000","last_active":"2024-06-14T23:07:58.998208+0000","last_peered":"2024-06-14T23:07:58.998208+0000","last_clean":"2024-06-14T23:07:58.998208+0000","last_became_active":"2024-06-14T23:07:58.023997+0000","last_became_peered":"2024-06-14T23:07:58.023997+0000","last_unstale":"2024-06-14T23:07:58.998208+0000","last_undegraded":"2024-06-14T23:07:58.998208+0000","last_fullsized":"2024-06-14T23:07:58.998208+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:36:30.743683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":493,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000963+0000","last_change":"2024-06-14T23:07:22.996051+0000","last_active":"2024-06-14T23:07:59.000963+0000","last_peered":"2024-06-14T23:07:59.000963+0000","last_clean":"2024-06-14T23:07:59.000963+0000","last_became_active":"2024-06-14T23:07:00.943273+0000","last_became_peered":"2024-06-14T23:07:00.943273+0000","last_unstale":"2024-06-14T23:07:59.000963+0000","last_undegraded":"2024-06-14T23:07:59.000963+0000","last_fullsized":"2024-06-14T23:07:59.000963+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:22.995968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:22.995968+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:08:40.160818+0000","scrub_duration":0.0055045409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"275'12551","reported_seq":12631,"reported_epoch":275,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:08:09.184435+0000","last_change":"2024-06-14T23:07:59.277412+0000","last_active":"2024-06-14T23:08:09.184435+0000","last_peered":"2024-06-14T23:08:09.184435+0000","last_clean":"2024-06-14T23:06:59.927883+0000","last_became_active":"2024-06-14T23:07:59.177730+0000","last_became_peered":"2024-06-14T23:07:59.177730+0000","last_unstale":"2024-06-14T23:08:09.184435+0000","last_undegraded":"2024-06-14T23:07:59.276970+0000","last_fullsized":"2024-06-14T23:07:58.993975+0000","mapping_epoch":273,"log_start":"251'5724","ondisk_log_start":"251'5724","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:29:20.153552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235339776,"num_objects":3591,"num_object_clones":0,"num_object_copies":10773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12551,"num_write_kb":516544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":3591}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":253,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995587+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:07:58.995587+0000","last_peered":"2024-06-14T23:07:58.995587+0000","last_clean":"2024-06-14T23:07:58.995587+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:07:58.995587+0000","last_undegraded":"2024-06-14T23:07:58.995587+0000","last_fullsized":"2024-06-14T23:07:58.995587+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":89,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.014312+0000","last_change":"2024-06-14T23:06:19.002672+0000","last_active":"2024-06-14T23:07:59.014312+0000","last_peered":"2024-06-14T23:07:59.014312+0000","last_clean":"2024-06-14T23:07:59.014312+0000","last_became_active":"2024-06-14T23:06:19.002175+0000","last_became_peered":"2024-06-14T23:06:19.002175+0000","last_unstale":"2024-06-14T23:07:59.014312+0000","last_undegraded":"2024-06-14T23:07:59.014312+0000","last_fullsized":"2024-06-14T23:07:59.014312+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:50.841812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"275'12341","reported_seq":12398,"reported_epoch":275,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:08:10.450388+0000","last_change":"2024-06-14T23:07:59.211041+0000","last_active":"2024-06-14T23:08:10.450388+0000","last_peered":"2024-06-14T23:08:10.450388+0000","last_clean":"2024-06-14T23:06:59.881114+0000","last_became_active":"2024-06-14T23:07:59.173333+0000","last_became_peered":"2024-06-14T23:07:59.173333+0000","last_unstale":"2024-06-14T23:08:10.450388+0000","last_undegraded":"2024-06-14T23:07:59.013445+0000","last_fullsized":"2024-06-14T23:07:58.994235+0000","mapping_epoch":273,"log_start":"251'5628","ondisk_log_start":"251'5628","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:06:29.100040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222494720,"num_objects":3395,"num_object_clones":0,"num_object_copies":10185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12341,"num_write_kb":503552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":3395}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":528,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.996543+0000","last_change":"2024-06-14T23:04:40.403447+0000","last_active":"2024-06-14T23:07:58.996543+0000","last_peered":"2024-06-14T23:07:58.996543+0000","last_clean":"2024-06-14T23:07:58.996543+0000","last_became_active":"2024-06-14T23:04:37.904456+0000","last_became_peered":"2024-06-14T23:04:37.904456+0000","last_unstale":"2024-06-14T23:07:58.996543+0000","last_undegraded":"2024-06-14T23:07:58.996543+0000","last_fullsized":"2024-06-14T23:07:58.996543+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:04:40.403365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:04:40.403365+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:45.174359+0000","scrub_duration":0.0014330689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":45,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000548+0000","last_change":"2024-06-14T23:06:19.002737+0000","last_active":"2024-06-14T23:07:59.000548+0000","last_peered":"2024-06-14T23:07:59.000548+0000","last_clean":"2024-06-14T23:07:59.000548+0000","last_became_active":"2024-06-14T23:06:19.002078+0000","last_became_peered":"2024-06-14T23:06:19.002078+0000","last_unstale":"2024-06-14T23:07:59.000548+0000","last_undegraded":"2024-06-14T23:07:59.000548+0000","last_fullsized":"2024-06-14T23:07:59.000548+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:00:16.317592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":252,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000863+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:07:59.000863+0000","last_peered":"2024-06-14T23:07:59.000863+0000","last_clean":"2024-06-14T23:07:59.000863+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:07:59.000863+0000","last_undegraded":"2024-06-14T23:07:59.000863+0000","last_fullsized":"2024-06-14T23:07:59.000863+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"275'12230","reported_seq":12278,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:10.748389+0000","last_change":"2024-06-14T23:05:03.807700+0000","last_active":"2024-06-14T23:08:10.748389+0000","last_peered":"2024-06-14T23:08:10.748389+0000","last_clean":"2024-06-14T23:08:10.748389+0000","last_became_active":"2024-06-14T23:05:03.807400+0000","last_became_peered":"2024-06-14T23:05:03.807400+0000","last_unstale":"2024-06-14T23:08:10.748389+0000","last_undegraded":"2024-06-14T23:08:10.748389+0000","last_fullsized":"2024-06-14T23:08:10.748389+0000","mapping_epoch":237,"log_start":"244'2200","ondisk_log_start":"244'2200","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:56:35.097613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226623488,"num_objects":3458,"num_object_clones":0,"num_object_copies":10374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":502016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":620,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995517+0000","last_change":"2024-06-14T23:04:37.902544+0000","last_active":"2024-06-14T23:07:58.995517+0000","last_peered":"2024-06-14T23:07:58.995517+0000","last_clean":"2024-06-14T23:07:58.995517+0000","last_became_active":"2024-06-14T23:04:37.902191+0000","last_became_peered":"2024-06-14T23:04:37.902191+0000","last_unstale":"2024-06-14T23:07:58.995517+0000","last_undegraded":"2024-06-14T23:07:58.995517+0000","last_fullsized":"2024-06-14T23:07:58.995517+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:05:09.672062+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":133,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000279+0000","last_change":"2024-06-14T23:07:58.023693+0000","last_active":"2024-06-14T23:07:59.000279+0000","last_peered":"2024-06-14T23:07:59.000279+0000","last_clean":"2024-06-14T23:07:59.000279+0000","last_became_active":"2024-06-14T23:07:58.023441+0000","last_became_peered":"2024-06-14T23:07:58.023441+0000","last_unstale":"2024-06-14T23:07:59.000279+0000","last_undegraded":"2024-06-14T23:07:59.000279+0000","last_fullsized":"2024-06-14T23:07:59.000279+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:08:47.037322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":265,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.001578+0000","last_change":"2024-06-14T23:04:34.887791+0000","last_active":"2024-06-14T23:07:59.001578+0000","last_peered":"2024-06-14T23:07:59.001578+0000","last_clean":"2024-06-14T23:07:59.001578+0000","last_became_active":"2024-06-14T23:04:34.887099+0000","last_became_peered":"2024-06-14T23:04:34.887099+0000","last_unstale":"2024-06-14T23:07:59.001578+0000","last_undegraded":"2024-06-14T23:07:59.001578+0000","last_fullsized":"2024-06-14T23:07:59.001578+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:46:07.470301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":614,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.014114+0000","last_change":"2024-06-14T23:07:13.730354+0000","last_active":"2024-06-14T23:07:59.014114+0000","last_peered":"2024-06-14T23:07:59.014114+0000","last_clean":"2024-06-14T23:07:59.014114+0000","last_became_active":"2024-06-14T23:07:00.933661+0000","last_became_peered":"2024-06-14T23:07:00.933661+0000","last_unstale":"2024-06-14T23:07:59.014114+0000","last_undegraded":"2024-06-14T23:07:59.014114+0000","last_fullsized":"2024-06-14T23:07:59.014114+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:13.729891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:13.729891+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:56:55.375260+0000","scrub_duration":0.015960974999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"275'12600","reported_seq":12655,"reported_epoch":275,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:08:07.480077+0000","last_change":"2024-06-14T23:07:59.257392+0000","last_active":"2024-06-14T23:08:07.480077+0000","last_peered":"2024-06-14T23:08:07.480077+0000","last_clean":"2024-06-14T23:06:59.893668+0000","last_became_active":"2024-06-14T23:07:59.217568+0000","last_became_peered":"2024-06-14T23:07:59.217568+0000","last_unstale":"2024-06-14T23:08:07.480077+0000","last_undegraded":"2024-06-14T23:07:59.015649+0000","last_fullsized":"2024-06-14T23:07:58.997047+0000","mapping_epoch":273,"log_start":"252'5846","ondisk_log_start":"252'5846","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6754,"log_dups_size":0,"ondisk_log_size":6754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:18:35.755111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240254976,"num_objects":3666,"num_object_clones":0,"num_object_copies":10998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12600,"num_write_kb":520512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":3666}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":256,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000329+0000","last_change":"2024-06-14T23:04:34.896257+0000","last_active":"2024-06-14T23:07:59.000329+0000","last_peered":"2024-06-14T23:07:59.000329+0000","last_clean":"2024-06-14T23:07:59.000329+0000","last_became_active":"2024-06-14T23:04:34.895926+0000","last_became_peered":"2024-06-14T23:04:34.895926+0000","last_unstale":"2024-06-14T23:07:59.000329+0000","last_undegraded":"2024-06-14T23:07:59.000329+0000","last_fullsized":"2024-06-14T23:07:59.000329+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:17.800902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":252,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.001214+0000","last_change":"2024-06-14T23:07:58.023766+0000","last_active":"2024-06-14T23:07:59.001214+0000","last_peered":"2024-06-14T23:07:59.001214+0000","last_clean":"2024-06-14T23:07:59.001214+0000","last_became_active":"2024-06-14T23:07:58.023506+0000","last_became_peered":"2024-06-14T23:07:58.023506+0000","last_unstale":"2024-06-14T23:07:59.001214+0000","last_undegraded":"2024-06-14T23:07:59.001214+0000","last_fullsized":"2024-06-14T23:07:59.001214+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:38.473906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"275'12146","reported_seq":12194,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:10.419165+0000","last_change":"2024-06-14T23:05:03.810157+0000","last_active":"2024-06-14T23:08:10.419165+0000","last_peered":"2024-06-14T23:08:10.419165+0000","last_clean":"2024-06-14T23:08:10.419165+0000","last_became_active":"2024-06-14T23:05:03.809591+0000","last_became_peered":"2024-06-14T23:05:03.809591+0000","last_unstale":"2024-06-14T23:08:10.419165+0000","last_undegraded":"2024-06-14T23:08:10.419165+0000","last_fullsized":"2024-06-14T23:08:10.419165+0000","mapping_epoch":237,"log_start":"251'5403","ondisk_log_start":"251'5403","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6743,"log_dups_size":0,"ondisk_log_size":6743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:04:46.257184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":3456,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12146,"num_write_kb":499264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":629,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995313+0000","last_change":"2024-06-14T23:07:58.017607+0000","last_active":"2024-06-14T23:07:58.995313+0000","last_peered":"2024-06-14T23:07:58.995313+0000","last_clean":"2024-06-14T23:07:58.995313+0000","last_became_active":"2024-06-14T23:07:58.015201+0000","last_became_peered":"2024-06-14T23:07:58.015201+0000","last_unstale":"2024-06-14T23:07:58.995313+0000","last_undegraded":"2024-06-14T23:07:58.995313+0000","last_fullsized":"2024-06-14T23:07:58.995313+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:37:24.915537+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":233,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.027604+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:07:59.027604+0000","last_peered":"2024-06-14T23:07:59.027604+0000","last_clean":"2024-06-14T23:07:59.027604+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:07:59.027604+0000","last_undegraded":"2024-06-14T23:07:59.027604+0000","last_fullsized":"2024-06-14T23:07:59.027604+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":246,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995070+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:07:58.995070+0000","last_peered":"2024-06-14T23:07:58.995070+0000","last_clean":"2024-06-14T23:07:58.995070+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:07:58.995070+0000","last_undegraded":"2024-06-14T23:07:58.995070+0000","last_fullsized":"2024-06-14T23:07:58.995070+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"275'12752","reported_seq":12817,"reported_epoch":275,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:08:09.595059+0000","last_change":"2024-06-14T23:07:59.254316+0000","last_active":"2024-06-14T23:08:09.595059+0000","last_peered":"2024-06-14T23:08:09.595059+0000","last_clean":"2024-06-14T23:06:59.923360+0000","last_became_active":"2024-06-14T23:07:59.217456+0000","last_became_peered":"2024-06-14T23:07:59.217456+0000","last_unstale":"2024-06-14T23:08:09.595059+0000","last_undegraded":"2024-06-14T23:07:59.022562+0000","last_fullsized":"2024-06-14T23:07:58.996112+0000","mapping_epoch":273,"log_start":"248'4102","ondisk_log_start":"248'4102","created":237,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":8650,"log_dups_size":0,"ondisk_log_size":8650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:38.233859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239468544,"num_objects":3654,"num_object_clones":0,"num_object_copies":10962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12752,"num_write_kb":524992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":3654}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":555,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.993754+0000","last_change":"2024-06-14T23:07:58.009471+0000","last_active":"2024-06-14T23:07:58.993754+0000","last_peered":"2024-06-14T23:07:58.993754+0000","last_clean":"2024-06-14T23:07:58.993754+0000","last_became_active":"2024-06-14T23:07:58.009127+0000","last_became_peered":"2024-06-14T23:07:58.009127+0000","last_unstale":"2024-06-14T23:07:58.993754+0000","last_undegraded":"2024-06-14T23:07:58.993754+0000","last_fullsized":"2024-06-14T23:07:58.993754+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:06:21.536637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:06:21.536637+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035987950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":260,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995551+0000","last_change":"2024-06-14T23:04:34.889456+0000","last_active":"2024-06-14T23:07:58.995551+0000","last_peered":"2024-06-14T23:07:58.995551+0000","last_clean":"2024-06-14T23:07:58.995551+0000","last_became_active":"2024-06-14T23:04:34.888908+0000","last_became_peered":"2024-06-14T23:04:34.888908+0000","last_unstale":"2024-06-14T23:07:58.995551+0000","last_undegraded":"2024-06-14T23:07:58.995551+0000","last_fullsized":"2024-06-14T23:07:58.995551+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:00:24.632734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":196,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.997026+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:07:58.997026+0000","last_peered":"2024-06-14T23:07:58.997026+0000","last_clean":"2024-06-14T23:07:58.997026+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:07:58.997026+0000","last_undegraded":"2024-06-14T23:07:58.997026+0000","last_fullsized":"2024-06-14T23:07:58.997026+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":190,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.998557+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:07:58.998557+0000","last_peered":"2024-06-14T23:07:58.998557+0000","last_clean":"2024-06-14T23:07:58.998557+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:07:58.998557+0000","last_undegraded":"2024-06-14T23:07:58.998557+0000","last_fullsized":"2024-06-14T23:07:58.998557+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"275'12225","reported_seq":12294,"reported_epoch":275,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:08:10.030066+0000","last_change":"2024-06-14T23:07:59.240747+0000","last_active":"2024-06-14T23:08:10.030066+0000","last_peered":"2024-06-14T23:08:10.030066+0000","last_clean":"2024-06-14T23:06:59.934530+0000","last_became_active":"2024-06-14T23:07:59.205704+0000","last_became_peered":"2024-06-14T23:07:59.205704+0000","last_unstale":"2024-06-14T23:08:10.030066+0000","last_undegraded":"2024-06-14T23:07:59.025674+0000","last_fullsized":"2024-06-14T23:07:59.000428+0000","mapping_epoch":273,"log_start":"252'5770","ondisk_log_start":"252'5770","created":237,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:56:18.854000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231669760,"num_objects":3535,"num_object_clones":0,"num_object_copies":10605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12225,"num_write_kb":504320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":3535}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":600,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.014234+0000","last_change":"2024-06-14T23:07:58.009966+0000","last_active":"2024-06-14T23:07:59.014234+0000","last_peered":"2024-06-14T23:07:59.014234+0000","last_clean":"2024-06-14T23:07:59.014234+0000","last_became_active":"2024-06-14T23:07:58.009493+0000","last_became_peered":"2024-06-14T23:07:58.009493+0000","last_unstale":"2024-06-14T23:07:59.014234+0000","last_undegraded":"2024-06-14T23:07:59.014234+0000","last_fullsized":"2024-06-14T23:07:59.014234+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":237,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.994319+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:07:58.994319+0000","last_peered":"2024-06-14T23:07:58.994319+0000","last_clean":"2024-06-14T23:07:58.994319+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:07:58.994319+0000","last_undegraded":"2024-06-14T23:07:58.994319+0000","last_fullsized":"2024-06-14T23:07:58.994319+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":113,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.999747+0000","last_change":"2024-06-14T23:06:19.001589+0000","last_active":"2024-06-14T23:07:58.999747+0000","last_peered":"2024-06-14T23:07:58.999747+0000","last_clean":"2024-06-14T23:07:58.999747+0000","last_became_active":"2024-06-14T23:06:19.001260+0000","last_became_peered":"2024-06-14T23:06:19.001260+0000","last_unstale":"2024-06-14T23:07:58.999747+0000","last_undegraded":"2024-06-14T23:07:58.999747+0000","last_fullsized":"2024-06-14T23:07:58.999747+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:54:46.316751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":241,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995828+0000","last_change":"2024-06-14T23:06:18.998828+0000","last_active":"2024-06-14T23:07:58.995828+0000","last_peered":"2024-06-14T23:07:58.995828+0000","last_clean":"2024-06-14T23:07:58.995828+0000","last_became_active":"2024-06-14T23:06:18.998461+0000","last_became_peered":"2024-06-14T23:06:18.998461+0000","last_unstale":"2024-06-14T23:07:58.995828+0000","last_undegraded":"2024-06-14T23:07:58.995828+0000","last_fullsized":"2024-06-14T23:07:58.995828+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:18:21.826501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"275'12176","reported_seq":12242,"reported_epoch":275,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:08:10.026998+0000","last_change":"2024-06-14T23:07:59.236118+0000","last_active":"2024-06-14T23:08:10.026998+0000","last_peered":"2024-06-14T23:08:10.026998+0000","last_clean":"2024-06-14T23:06:59.877480+0000","last_became_active":"2024-06-14T23:07:59.205631+0000","last_became_peered":"2024-06-14T23:07:59.205631+0000","last_unstale":"2024-06-14T23:08:10.026998+0000","last_undegraded":"2024-06-14T23:07:59.026782+0000","last_fullsized":"2024-06-14T23:07:58.999813+0000","mapping_epoch":273,"log_start":"252'5696","ondisk_log_start":"252'5696","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6480,"log_dups_size":0,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:38:36.711829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228720640,"num_objects":3490,"num_object_clones":0,"num_object_copies":10470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12176,"num_write_kb":501312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":3490}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":525,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000152+0000","last_change":"2024-06-14T23:07:10.877003+0000","last_active":"2024-06-14T23:07:59.000152+0000","last_peered":"2024-06-14T23:07:59.000152+0000","last_clean":"2024-06-14T23:07:59.000152+0000","last_became_active":"2024-06-14T23:06:52.047842+0000","last_became_peered":"2024-06-14T23:06:52.047842+0000","last_unstale":"2024-06-14T23:07:59.000152+0000","last_undegraded":"2024-06-14T23:07:59.000152+0000","last_fullsized":"2024-06-14T23:07:59.000152+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:09:38.064292+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":57,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.997558+0000","last_change":"2024-06-14T23:04:34.879251+0000","last_active":"2024-06-14T23:07:58.997558+0000","last_peered":"2024-06-14T23:07:58.997558+0000","last_clean":"2024-06-14T23:07:58.997558+0000","last_became_active":"2024-06-14T23:04:34.878470+0000","last_became_peered":"2024-06-14T23:04:34.878470+0000","last_unstale":"2024-06-14T23:07:58.997558+0000","last_undegraded":"2024-06-14T23:07:58.997558+0000","last_fullsized":"2024-06-14T23:07:58.997558+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:55.949794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":280,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.994514+0000","last_change":"2024-06-14T23:07:58.010456+0000","last_active":"2024-06-14T23:07:58.994514+0000","last_peered":"2024-06-14T23:07:58.994514+0000","last_clean":"2024-06-14T23:07:58.994514+0000","last_became_active":"2024-06-14T23:07:58.010157+0000","last_became_peered":"2024-06-14T23:07:58.010157+0000","last_unstale":"2024-06-14T23:07:58.994514+0000","last_undegraded":"2024-06-14T23:07:58.994514+0000","last_fullsized":"2024-06-14T23:07:58.994514+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:44:17.382159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":231,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.994873+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:07:58.994873+0000","last_peered":"2024-06-14T23:07:58.994873+0000","last_clean":"2024-06-14T23:07:58.994873+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:07:58.994873+0000","last_undegraded":"2024-06-14T23:07:58.994873+0000","last_fullsized":"2024-06-14T23:07:58.994873+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"275'12497","reported_seq":12550,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:09.993291+0000","last_change":"2024-06-14T23:06:04.928960+0000","last_active":"2024-06-14T23:08:09.993291+0000","last_peered":"2024-06-14T23:08:09.993291+0000","last_clean":"2024-06-14T23:08:09.993291+0000","last_became_active":"2024-06-14T23:06:04.923775+0000","last_became_peered":"2024-06-14T23:06:04.923775+0000","last_unstale":"2024-06-14T23:08:09.993291+0000","last_undegraded":"2024-06-14T23:08:09.993291+0000","last_fullsized":"2024-06-14T23:08:09.993291+0000","mapping_epoch":250,"log_start":"252'6052","ondisk_log_start":"252'6052","created":237,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:10:46.076558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232849408,"num_objects":3553,"num_object_clones":0,"num_object_copies":10659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12497,"num_write_kb":513600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":631,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995105+0000","last_change":"2024-06-14T23:07:11.050751+0000","last_active":"2024-06-14T23:07:58.995105+0000","last_peered":"2024-06-14T23:07:58.995105+0000","last_clean":"2024-06-14T23:07:58.995105+0000","last_became_active":"2024-06-14T23:06:52.051996+0000","last_became_peered":"2024-06-14T23:06:52.051996+0000","last_unstale":"2024-06-14T23:07:58.995105+0000","last_undegraded":"2024-06-14T23:07:58.995105+0000","last_fullsized":"2024-06-14T23:07:58.995105+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:52:38.070062+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":257,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.997106+0000","last_change":"2024-06-14T23:07:58.015646+0000","last_active":"2024-06-14T23:07:58.997106+0000","last_peered":"2024-06-14T23:07:58.997106+0000","last_clean":"2024-06-14T23:07:58.997106+0000","last_became_active":"2024-06-14T23:07:58.014865+0000","last_became_peered":"2024-06-14T23:07:58.014865+0000","last_unstale":"2024-06-14T23:07:58.997106+0000","last_undegraded":"2024-06-14T23:07:58.997106+0000","last_fullsized":"2024-06-14T23:07:58.997106+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:52:48.798683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":253,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000870+0000","last_change":"2024-06-14T23:07:58.030089+0000","last_active":"2024-06-14T23:07:59.000870+0000","last_peered":"2024-06-14T23:07:59.000870+0000","last_clean":"2024-06-14T23:07:59.000870+0000","last_became_active":"2024-06-14T23:07:58.029700+0000","last_became_peered":"2024-06-14T23:07:58.029700+0000","last_unstale":"2024-06-14T23:07:59.000870+0000","last_undegraded":"2024-06-14T23:07:59.000870+0000","last_fullsized":"2024-06-14T23:07:59.000870+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:38.204728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":280,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:03.049304+0000","last_change":"2024-06-14T23:04:34.880836+0000","last_active":"2024-06-14T23:08:03.049304+0000","last_peered":"2024-06-14T23:08:03.049304+0000","last_clean":"2024-06-14T23:08:03.049304+0000","last_became_active":"2024-06-14T23:04:34.880540+0000","last_became_peered":"2024-06-14T23:04:34.880540+0000","last_unstale":"2024-06-14T23:08:03.049304+0000","last_undegraded":"2024-06-14T23:08:03.049304+0000","last_fullsized":"2024-06-14T23:08:03.049304+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:06.981511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"275'12392","reported_seq":13157,"reported_epoch":275,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-14T23:08:10.813168+0000","last_change":"2024-06-14T23:07:59.283112+0000","last_active":"2024-06-14T23:08:10.813168+0000","last_peered":"2024-06-14T23:08:10.813168+0000","last_clean":"2024-06-14T23:06:59.906458+0000","last_became_active":"2024-06-14T23:07:59.205602+0000","last_became_peered":"2024-06-14T23:07:59.205602+0000","last_unstale":"2024-06-14T23:08:10.813168+0000","last_undegraded":"2024-06-14T23:08:10.813168+0000","last_fullsized":"2024-06-14T23:07:58.998628+0000","mapping_epoch":273,"log_start":"243'2300","ondisk_log_start":"243'2300","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:13:39.377006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220659736,"num_objects":3368,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12392,"num_write_kb":504257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":3368}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":534,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.999325+0000","last_change":"2024-06-14T23:07:58.020956+0000","last_active":"2024-06-14T23:07:58.999325+0000","last_peered":"2024-06-14T23:07:58.999325+0000","last_clean":"2024-06-14T23:07:58.999325+0000","last_became_active":"2024-06-14T23:07:58.020638+0000","last_became_peered":"2024-06-14T23:07:58.020638+0000","last_unstale":"2024-06-14T23:07:58.999325+0000","last_undegraded":"2024-06-14T23:07:58.999325+0000","last_fullsized":"2024-06-14T23:07:58.999325+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":273,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.019066+0000","last_change":"2024-06-14T23:04:34.880181+0000","last_active":"2024-06-14T23:07:59.019066+0000","last_peered":"2024-06-14T23:07:59.019066+0000","last_clean":"2024-06-14T23:07:59.019066+0000","last_became_active":"2024-06-14T23:04:34.879842+0000","last_became_peered":"2024-06-14T23:04:34.879842+0000","last_unstale":"2024-06-14T23:07:59.019066+0000","last_undegraded":"2024-06-14T23:07:59.019066+0000","last_fullsized":"2024-06-14T23:07:59.019066+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:14:18.433225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":237,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:03.049503+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:08:03.049503+0000","last_peered":"2024-06-14T23:08:03.049503+0000","last_clean":"2024-06-14T23:08:03.049503+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:08:03.049503+0000","last_undegraded":"2024-06-14T23:08:03.049503+0000","last_fullsized":"2024-06-14T23:08:03.049503+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":232,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:03.049176+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:08:03.049176+0000","last_peered":"2024-06-14T23:08:03.049176+0000","last_clean":"2024-06-14T23:08:03.049176+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:08:03.049176+0000","last_undegraded":"2024-06-14T23:08:03.049176+0000","last_fullsized":"2024-06-14T23:08:03.049176+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"275'12585","reported_seq":12633,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:10.411835+0000","last_change":"2024-06-14T23:05:03.621987+0000","last_active":"2024-06-14T23:08:10.411835+0000","last_peered":"2024-06-14T23:08:10.411835+0000","last_clean":"2024-06-14T23:08:10.411835+0000","last_became_active":"2024-06-14T23:05:03.621724+0000","last_became_peered":"2024-06-14T23:05:03.621724+0000","last_unstale":"2024-06-14T23:08:10.411835+0000","last_undegraded":"2024-06-14T23:08:10.411835+0000","last_fullsized":"2024-06-14T23:08:10.411835+0000","mapping_epoch":237,"log_start":"252'5896","ondisk_log_start":"252'5896","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:12.558177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229572608,"num_objects":3503,"num_object_clones":0,"num_object_copies":10509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12585,"num_write_kb":514816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":534,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.998959+0000","last_change":"2024-06-14T23:07:58.008816+0000","last_active":"2024-06-14T23:07:58.998959+0000","last_peered":"2024-06-14T23:07:58.998959+0000","last_clean":"2024-06-14T23:07:58.998959+0000","last_became_active":"2024-06-14T23:07:58.008508+0000","last_became_peered":"2024-06-14T23:07:58.008508+0000","last_unstale":"2024-06-14T23:07:58.998959+0000","last_undegraded":"2024-06-14T23:07:58.998959+0000","last_fullsized":"2024-06-14T23:07:58.998959+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":265,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:03.049442+0000","last_change":"2024-06-14T23:04:34.880684+0000","last_active":"2024-06-14T23:08:03.049442+0000","last_peered":"2024-06-14T23:08:03.049442+0000","last_clean":"2024-06-14T23:08:03.049442+0000","last_became_active":"2024-06-14T23:04:34.880127+0000","last_became_peered":"2024-06-14T23:04:34.880127+0000","last_unstale":"2024-06-14T23:08:03.049442+0000","last_undegraded":"2024-06-14T23:08:03.049442+0000","last_fullsized":"2024-06-14T23:08:03.049442+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:53:52.966928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":240,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.994209+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:07:58.994209+0000","last_peered":"2024-06-14T23:07:58.994209+0000","last_clean":"2024-06-14T23:07:58.994209+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:07:58.994209+0000","last_undegraded":"2024-06-14T23:07:58.994209+0000","last_fullsized":"2024-06-14T23:07:58.994209+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":49,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995478+0000","last_change":"2024-06-14T23:06:19.003881+0000","last_active":"2024-06-14T23:07:58.995478+0000","last_peered":"2024-06-14T23:07:58.995478+0000","last_clean":"2024-06-14T23:07:58.995478+0000","last_became_active":"2024-06-14T23:06:19.003589+0000","last_became_peered":"2024-06-14T23:06:19.003589+0000","last_unstale":"2024-06-14T23:07:58.995478+0000","last_undegraded":"2024-06-14T23:07:58.995478+0000","last_fullsized":"2024-06-14T23:07:58.995478+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:52.332299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"275'12269","reported_seq":12317,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:10.423550+0000","last_change":"2024-06-14T23:05:03.621656+0000","last_active":"2024-06-14T23:08:10.423550+0000","last_peered":"2024-06-14T23:08:10.423550+0000","last_clean":"2024-06-14T23:08:10.423550+0000","last_became_active":"2024-06-14T23:05:03.620834+0000","last_became_peered":"2024-06-14T23:05:03.620834+0000","last_unstale":"2024-06-14T23:08:10.423550+0000","last_undegraded":"2024-06-14T23:08:10.423550+0000","last_fullsized":"2024-06-14T23:08:10.423550+0000","mapping_epoch":237,"log_start":"251'5525","ondisk_log_start":"251'5525","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:59:26.118782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231669760,"num_objects":3535,"num_object_clones":0,"num_object_copies":10605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12269,"num_write_kb":505728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":649,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.001371+0000","last_change":"2024-06-14T23:07:58.803271+0000","last_active":"2024-06-14T23:07:59.001371+0000","last_peered":"2024-06-14T23:07:59.001371+0000","last_clean":"2024-06-14T23:07:59.001371+0000","last_became_active":"2024-06-14T23:07:58.048267+0000","last_became_peered":"2024-06-14T23:07:58.048267+0000","last_unstale":"2024-06-14T23:07:59.001371+0000","last_undegraded":"2024-06-14T23:07:59.001371+0000","last_fullsized":"2024-06-14T23:07:59.001371+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:58.803217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:58.803217+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:20.167126+0000","scrub_duration":0.001285706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":166,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.002034+0000","last_change":"2024-06-14T23:06:19.000734+0000","last_active":"2024-06-14T23:07:59.002034+0000","last_peered":"2024-06-14T23:07:59.002034+0000","last_clean":"2024-06-14T23:07:59.002034+0000","last_became_active":"2024-06-14T23:06:19.000432+0000","last_became_peered":"2024-06-14T23:06:19.000432+0000","last_unstale":"2024-06-14T23:07:59.002034+0000","last_undegraded":"2024-06-14T23:07:59.002034+0000","last_fullsized":"2024-06-14T23:07:59.002034+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:48.268127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":245,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995139+0000","last_change":"2024-06-14T23:07:58.011279+0000","last_active":"2024-06-14T23:07:58.995139+0000","last_peered":"2024-06-14T23:07:58.995139+0000","last_clean":"2024-06-14T23:07:58.995139+0000","last_became_active":"2024-06-14T23:07:58.009870+0000","last_became_peered":"2024-06-14T23:07:58.009870+0000","last_unstale":"2024-06-14T23:07:58.995139+0000","last_undegraded":"2024-06-14T23:07:58.995139+0000","last_fullsized":"2024-06-14T23:07:58.995139+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:59:55.353132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":168,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:03.044695+0000","last_change":"2024-06-14T23:06:04.932485+0000","last_active":"2024-06-14T23:08:03.044695+0000","last_peered":"2024-06-14T23:08:03.044695+0000","last_clean":"2024-06-14T23:08:03.044695+0000","last_became_active":"2024-06-14T23:06:04.932166+0000","last_became_peered":"2024-06-14T23:06:04.932166+0000","last_unstale":"2024-06-14T23:08:03.044695+0000","last_undegraded":"2024-06-14T23:08:03.044695+0000","last_fullsized":"2024-06-14T23:08:03.044695+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:46.149456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"275'12137","reported_seq":12183,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:09.136193+0000","last_change":"2024-06-14T23:05:03.623137+0000","last_active":"2024-06-14T23:08:09.136193+0000","last_peered":"2024-06-14T23:08:09.136193+0000","last_clean":"2024-06-14T23:08:09.136193+0000","last_became_active":"2024-06-14T23:05:03.622833+0000","last_became_peered":"2024-06-14T23:05:03.622833+0000","last_unstale":"2024-06-14T23:08:09.136193+0000","last_undegraded":"2024-06-14T23:08:09.136193+0000","last_fullsized":"2024-06-14T23:08:09.136193+0000","mapping_epoch":237,"log_start":"251'5430","ondisk_log_start":"251'5430","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:53:45.610358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230227968,"num_objects":3513,"num_object_clones":0,"num_object_copies":10539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":500800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":597,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.001533+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:07:59.001533+0000","last_peered":"2024-06-14T23:07:59.001533+0000","last_clean":"2024-06-14T23:07:59.001533+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:07:59.001533+0000","last_undegraded":"2024-06-14T23:07:59.001533+0000","last_fullsized":"2024-06-14T23:07:59.001533+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":58,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.994774+0000","last_change":"2024-06-14T23:06:19.000517+0000","last_active":"2024-06-14T23:07:58.994774+0000","last_peered":"2024-06-14T23:07:58.994774+0000","last_clean":"2024-06-14T23:07:58.994774+0000","last_became_active":"2024-06-14T23:06:19.000141+0000","last_became_peered":"2024-06-14T23:06:19.000141+0000","last_unstale":"2024-06-14T23:07:58.994774+0000","last_undegraded":"2024-06-14T23:07:58.994774+0000","last_fullsized":"2024-06-14T23:07:58.994774+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:57:42.599655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":230,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.999253+0000","last_change":"2024-06-14T23:07:58.008658+0000","last_active":"2024-06-14T23:07:58.999253+0000","last_peered":"2024-06-14T23:07:58.999253+0000","last_clean":"2024-06-14T23:07:58.999253+0000","last_became_active":"2024-06-14T23:07:58.008400+0000","last_became_peered":"2024-06-14T23:07:58.008400+0000","last_unstale":"2024-06-14T23:07:58.999253+0000","last_undegraded":"2024-06-14T23:07:58.999253+0000","last_fullsized":"2024-06-14T23:07:58.999253+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:24:42.013441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":235,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000130+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:07:59.000130+0000","last_peered":"2024-06-14T23:07:59.000130+0000","last_clean":"2024-06-14T23:07:59.000130+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:07:59.000130+0000","last_undegraded":"2024-06-14T23:07:59.000130+0000","last_fullsized":"2024-06-14T23:07:59.000130+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":252,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000804+0000","last_change":"2024-06-14T23:07:58.030396+0000","last_active":"2024-06-14T23:07:59.000804+0000","last_peered":"2024-06-14T23:07:59.000804+0000","last_clean":"2024-06-14T23:07:59.000804+0000","last_became_active":"2024-06-14T23:07:58.030106+0000","last_became_peered":"2024-06-14T23:07:58.030106+0000","last_unstale":"2024-06-14T23:07:59.000804+0000","last_undegraded":"2024-06-14T23:07:59.000804+0000","last_fullsized":"2024-06-14T23:07:59.000804+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:31:52.585282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":258,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000949+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:07:59.000949+0000","last_peered":"2024-06-14T23:07:59.000949+0000","last_clean":"2024-06-14T23:07:59.000949+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:07:59.000949+0000","last_undegraded":"2024-06-14T23:07:59.000949+0000","last_fullsized":"2024-06-14T23:07:59.000949+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":279,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:03.049017+0000","last_change":"2024-06-14T23:07:58.010642+0000","last_active":"2024-06-14T23:08:03.049017+0000","last_peered":"2024-06-14T23:08:03.049017+0000","last_clean":"2024-06-14T23:08:03.049017+0000","last_became_active":"2024-06-14T23:07:58.010358+0000","last_became_peered":"2024-06-14T23:07:58.010358+0000","last_unstale":"2024-06-14T23:08:03.049017+0000","last_undegraded":"2024-06-14T23:08:03.049017+0000","last_fullsized":"2024-06-14T23:08:03.049017+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:42.212299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":157,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:08:00.030500+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:08:00.030500+0000","last_peered":"2024-06-14T23:08:00.030500+0000","last_clean":"2024-06-14T23:08:00.030500+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:08:00.030500+0000","last_undegraded":"2024-06-14T23:08:00.030500+0000","last_fullsized":"2024-06-14T23:08:00.030500+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":606,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.996991+0000","last_change":"2024-06-14T23:07:09.356684+0000","last_active":"2024-06-14T23:07:58.996991+0000","last_peered":"2024-06-14T23:07:58.996991+0000","last_clean":"2024-06-14T23:07:58.996991+0000","last_became_active":"2024-06-14T23:07:00.942504+0000","last_became_peered":"2024-06-14T23:07:00.942504+0000","last_unstale":"2024-06-14T23:07:58.996991+0000","last_undegraded":"2024-06-14T23:07:58.996991+0000","last_fullsized":"2024-06-14T23:07:58.996991+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:18:54.289960+0000","scrub_duration":0.013653241999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":236,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995298+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:07:58.995298+0000","last_peered":"2024-06-14T23:07:58.995298+0000","last_clean":"2024-06-14T23:07:58.995298+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:07:58.995298+0000","last_undegraded":"2024-06-14T23:07:58.995298+0000","last_fullsized":"2024-06-14T23:07:58.995298+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":138,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.026441+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:07:59.026441+0000","last_peered":"2024-06-14T23:07:59.026441+0000","last_clean":"2024-06-14T23:07:59.026441+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:07:59.026441+0000","last_undegraded":"2024-06-14T23:07:59.026441+0000","last_fullsized":"2024-06-14T23:07:59.026441+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":112,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995772+0000","last_change":"2024-06-14T23:06:19.000252+0000","last_active":"2024-06-14T23:07:58.995772+0000","last_peered":"2024-06-14T23:07:58.995772+0000","last_clean":"2024-06-14T23:07:58.995772+0000","last_became_active":"2024-06-14T23:06:18.999939+0000","last_became_peered":"2024-06-14T23:06:18.999939+0000","last_unstale":"2024-06-14T23:07:58.995772+0000","last_undegraded":"2024-06-14T23:07:58.995772+0000","last_fullsized":"2024-06-14T23:07:58.995772+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:50.509140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":247,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-14T23:08:03.044629+0000","last_change":"2024-06-14T23:07:58.028129+0000","last_active":"2024-06-14T23:08:03.044629+0000","last_peered":"2024-06-14T23:08:03.044629+0000","last_clean":"2024-06-14T23:08:03.044629+0000","last_became_active":"2024-06-14T23:07:58.027376+0000","last_became_peered":"2024-06-14T23:07:58.027376+0000","last_unstale":"2024-06-14T23:08:03.044629+0000","last_undegraded":"2024-06-14T23:08:03.044629+0000","last_fullsized":"2024-06-14T23:08:03.044629+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:25:25.805140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":216,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.999420+0000","last_change":"2024-06-14T23:07:58.010437+0000","last_active":"2024-06-14T23:07:58.999420+0000","last_peered":"2024-06-14T23:07:58.999420+0000","last_clean":"2024-06-14T23:07:58.999420+0000","last_became_active":"2024-06-14T23:07:58.010143+0000","last_became_peered":"2024-06-14T23:07:58.010143+0000","last_unstale":"2024-06-14T23:07:58.999420+0000","last_undegraded":"2024-06-14T23:07:58.999420+0000","last_fullsized":"2024-06-14T23:07:58.999420+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:46:46.681752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":134,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.994361+0000","last_change":"2024-06-14T23:06:18.998767+0000","last_active":"2024-06-14T23:07:58.994361+0000","last_peered":"2024-06-14T23:07:58.994361+0000","last_clean":"2024-06-14T23:07:58.994361+0000","last_became_active":"2024-06-14T23:06:18.998481+0000","last_became_peered":"2024-06-14T23:06:18.998481+0000","last_unstale":"2024-06-14T23:07:58.994361+0000","last_undegraded":"2024-06-14T23:07:58.994361+0000","last_fullsized":"2024-06-14T23:07:58.994361+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:58:13.282478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":113,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.001358+0000","last_change":"2024-06-14T23:06:18.999600+0000","last_active":"2024-06-14T23:07:59.001358+0000","last_peered":"2024-06-14T23:07:59.001358+0000","last_clean":"2024-06-14T23:07:59.001358+0000","last_became_active":"2024-06-14T23:06:18.999293+0000","last_became_peered":"2024-06-14T23:06:18.999293+0000","last_unstale":"2024-06-14T23:07:59.001358+0000","last_undegraded":"2024-06-14T23:07:59.001358+0000","last_fullsized":"2024-06-14T23:07:59.001358+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:28:12.373005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":245,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.994381+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:07:58.994381+0000","last_peered":"2024-06-14T23:07:58.994381+0000","last_clean":"2024-06-14T23:07:58.994381+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:07:58.994381+0000","last_undegraded":"2024-06-14T23:07:58.994381+0000","last_fullsized":"2024-06-14T23:07:58.994381+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":40,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.998419+0000","last_change":"2024-06-14T23:05:43.964532+0000","last_active":"2024-06-14T23:07:58.998419+0000","last_peered":"2024-06-14T23:07:58.998419+0000","last_clean":"2024-06-14T23:07:58.998419+0000","last_became_active":"2024-06-14T23:05:43.964255+0000","last_became_peered":"2024-06-14T23:05:43.964255+0000","last_unstale":"2024-06-14T23:07:58.998419+0000","last_undegraded":"2024-06-14T23:07:58.998419+0000","last_fullsized":"2024-06-14T23:07:58.998419+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:06.098124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":274,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000431+0000","last_change":"2024-06-14T23:04:34.883863+0000","last_active":"2024-06-14T23:07:59.000431+0000","last_peered":"2024-06-14T23:07:59.000431+0000","last_clean":"2024-06-14T23:07:59.000431+0000","last_became_active":"2024-06-14T23:04:34.881668+0000","last_became_peered":"2024-06-14T23:04:34.881668+0000","last_unstale":"2024-06-14T23:07:59.000431+0000","last_undegraded":"2024-06-14T23:07:59.000431+0000","last_fullsized":"2024-06-14T23:07:59.000431+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:44:45.737416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":57,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.997394+0000","last_change":"2024-06-14T23:04:34.879498+0000","last_active":"2024-06-14T23:07:58.997394+0000","last_peered":"2024-06-14T23:07:58.997394+0000","last_clean":"2024-06-14T23:07:58.997394+0000","last_became_active":"2024-06-14T23:04:34.878932+0000","last_became_peered":"2024-06-14T23:04:34.878932+0000","last_unstale":"2024-06-14T23:07:58.997394+0000","last_undegraded":"2024-06-14T23:07:58.997394+0000","last_fullsized":"2024-06-14T23:07:58.997394+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:56:50.201189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":48,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000626+0000","last_change":"2024-06-14T23:06:19.003121+0000","last_active":"2024-06-14T23:07:59.000626+0000","last_peered":"2024-06-14T23:07:59.000626+0000","last_clean":"2024-06-14T23:07:59.000626+0000","last_became_active":"2024-06-14T23:06:19.002731+0000","last_became_peered":"2024-06-14T23:06:19.002731+0000","last_unstale":"2024-06-14T23:07:59.000626+0000","last_undegraded":"2024-06-14T23:07:59.000626+0000","last_fullsized":"2024-06-14T23:07:59.000626+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:31:04.851974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":252,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000744+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:07:59.000744+0000","last_peered":"2024-06-14T23:07:59.000744+0000","last_clean":"2024-06-14T23:07:59.000744+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:07:59.000744+0000","last_undegraded":"2024-06-14T23:07:59.000744+0000","last_fullsized":"2024-06-14T23:07:59.000744+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":238,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995503+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:07:58.995503+0000","last_peered":"2024-06-14T23:07:58.995503+0000","last_clean":"2024-06-14T23:07:58.995503+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:07:58.995503+0000","last_undegraded":"2024-06-14T23:07:58.995503+0000","last_fullsized":"2024-06-14T23:07:58.995503+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":252,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.000352+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:07:59.000352+0000","last_peered":"2024-06-14T23:07:59.000352+0000","last_clean":"2024-06-14T23:07:59.000352+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:07:59.000352+0000","last_undegraded":"2024-06-14T23:07:59.000352+0000","last_fullsized":"2024-06-14T23:07:59.000352+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":238,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:59.018999+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:07:59.018999+0000","last_peered":"2024-06-14T23:07:59.018999+0000","last_clean":"2024-06-14T23:07:59.018999+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:07:59.018999+0000","last_undegraded":"2024-06-14T23:07:59.018999+0000","last_fullsized":"2024-06-14T23:07:59.018999+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":57,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.997627+0000","last_change":"2024-06-14T23:04:34.879700+0000","last_active":"2024-06-14T23:07:58.997627+0000","last_peered":"2024-06-14T23:07:58.997627+0000","last_clean":"2024-06-14T23:07:58.997627+0000","last_became_active":"2024-06-14T23:04:34.878924+0000","last_became_peered":"2024-06-14T23:04:34.878924+0000","last_unstale":"2024-06-14T23:07:58.997627+0000","last_undegraded":"2024-06-14T23:07:58.997627+0000","last_fullsized":"2024-06-14T23:07:58.997627+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:28.826486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":162,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.996728+0000","last_change":"2024-06-14T23:07:58.010033+0000","last_active":"2024-06-14T23:07:58.996728+0000","last_peered":"2024-06-14T23:07:58.996728+0000","last_clean":"2024-06-14T23:07:58.996728+0000","last_became_active":"2024-06-14T23:07:58.009712+0000","last_became_peered":"2024-06-14T23:07:58.009712+0000","last_unstale":"2024-06-14T23:07:58.996728+0000","last_undegraded":"2024-06-14T23:07:58.996728+0000","last_fullsized":"2024-06-14T23:07:58.996728+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:30:02.772328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":252,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.999745+0000","last_change":"2024-06-14T23:07:58.016608+0000","last_active":"2024-06-14T23:07:58.999745+0000","last_peered":"2024-06-14T23:07:58.999745+0000","last_clean":"2024-06-14T23:07:58.999745+0000","last_became_active":"2024-06-14T23:07:58.016290+0000","last_became_peered":"2024-06-14T23:07:58.016290+0000","last_unstale":"2024-06-14T23:07:58.999745+0000","last_undegraded":"2024-06-14T23:07:58.999745+0000","last_fullsized":"2024-06-14T23:07:58.999745+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:12:50.936457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":59,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-14T23:07:58.995695+0000","last_change":"2024-06-14T23:06:19.002447+0000","last_active":"2024-06-14T23:07:58.995695+0000","last_peered":"2024-06-14T23:07:58.995695+0000","last_clean":"2024-06-14T23:07:58.995695+0000","last_became_active":"2024-06-14T23:06:19.002029+0000","last_became_peered":"2024-06-14T23:06:19.002029+0000","last_unstale":"2024-06-14T23:07:58.995695+0000","last_undegraded":"2024-06-14T23:07:58.995695+0000","last_fullsized":"2024-06-14T23:07:58.995695+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:27.132531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3688169496,"num_objects":56278,"num_object_clones":0,"num_object_copies":168834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198360,"num_write_kb":8148353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12946386944,"data_stored":12946374728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122637,"ondisk_log_size":122637,"up":48,"acting":40,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19646,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":670873,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":124,"num_write":33,"num_write_kb":330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":766889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3382,"ondisk_log_size":3382,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234409,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2269420,"kb_used_data":1366308,"kb_used_omap":1,"kb_used_meta":903038,"kb_avail":91475732,"statfs":{"total":95995035648,"available":93671149568,"internally_reserved":0,"allocated":1399099392,"data_stored":1397710753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924711256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.91300000000000003,"15min":0.95999999999999996},"min":{"1min":0.223,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6499999999999999,"5min":3.1699999999999999,"15min":6.9539999999999997},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.1739999999999999,"15min":1.0740000000000001},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.20300000000000001},"max":{"1min":3.2839999999999998,"5min":6.9550000000000001,"15min":6.9550000000000001},"last":0.28599999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:08:06 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":1.2869999999999999,"15min":1.6950000000000001},"min":{"1min":0.33600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":12.010999999999999,"5min":12.010999999999999,"15min":83.295000000000002},"last":12.010999999999999},{"interface":"front","average":{"1min":2.532,"5min":1.5209999999999999,"15min":1.716},"min":{"1min":0.27500000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":13.531000000000001,"5min":13.531000000000001,"15min":82.972999999999999},"last":13.531000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:08:06 2024","interfaces":[{"interface":"back","average":{"1min":2.8490000000000002,"5min":1.379,"15min":1.115},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":20.390000000000001,"5min":20.390000000000001,"15min":20.390000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":2.7200000000000002,"5min":1.427,"15min":1.1830000000000001},"min":{"1min":0.28100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":20.327999999999999,"5min":20.327999999999999,"15min":20.327999999999999},"last":0.45800000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.2729999999999999,"15min":1.3129999999999999},"min":{"1min":0.28699999999999998,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":2.8250000000000002,"5min":7.8609999999999998,"15min":55.683999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":2.1349999999999998,"5min":1.5369999999999999,"15min":1.405},"min":{"1min":0.40500000000000003,"5min":0.30099999999999999,"15min":0.29799999999999999},"max":{"1min":6.3010000000000002,"5min":7.7670000000000003,"15min":55.533999999999999},"last":0.35499999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.1000000000000001,"15min":0.93400000000000005},"min":{"1min":0.248,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.3279999999999998,"5min":11.342000000000001,"15min":11.342000000000001},"last":0.624},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.1459999999999999,"15min":0.97699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.407,"5min":12.651,"15min":16.879000000000001},"last":0.59199999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.96999999999999997,"15min":1.01},"min":{"1min":0.26400000000000001,"5min":0.252,"15min":0.23100000000000001},"max":{"1min":2.161,"5min":4.6619999999999999,"15min":6.9000000000000004},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.0920000000000001,"15min":1.0309999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":3.2509999999999999,"5min":6.2809999999999997,"15min":6.8529999999999998},"last":0.48599999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":1.367,"15min":1.0449999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6320000000000001,"5min":8.952,"15min":8.952},"last":2.0430000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.2989999999999999,"15min":1.008},"min":{"1min":0.34000000000000002,"5min":0.29599999999999999,"15min":0.28299999999999997},"max":{"1min":1.4850000000000001,"5min":7.0629999999999997,"15min":12.951000000000001},"last":0.81599999999999995}]}]},{"osd":6,"up_from":272,"seq":1168231104515,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4028836,"kb_used_data":3635864,"kb_used_omap":7,"kb_used_meta":392952,"kb_avail":89716316,"statfs":{"total":95995035648,"available":91869507584,"internally_reserved":0,"allocated":3723124736,"data_stored":3721747940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":402383290},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,5,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":5,"up_from":38,"seq":163208757684,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1810072,"kb_used_data":914832,"kb_used_omap":1,"kb_used_meta":895230,"kb_avail":91935080,"statfs":{"total":95995035648,"available":94141521920,"internally_reserved":0,"allocated":936787968,"data_stored":935415698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916715864},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:07:44 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.6950000000000001,"15min":1.371},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":6.1950000000000003,"5min":14.94,"15min":14.94},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.619,"5min":1.673,"15min":1.335},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":6.2130000000000001,"5min":9.3819999999999997,"15min":9.3819999999999997},"last":3.7509999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.1699999999999999,"15min":1.0069999999999999},"min":{"1min":0.28399999999999997,"5min":0.252,"15min":0.23999999999999999},"max":{"1min":6.9480000000000004,"5min":7.4370000000000003,"15min":7.4370000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.345,"5min":1.365,"15min":1.2290000000000001},"min":{"1min":0.28199999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.7380000000000004,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":1.956}]},{"osd":2,"last update":"Fri Jun 14 23:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.494,"15min":1.1799999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":3.367,"5min":14.891,"15min":14.891},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.343,"5min":1.478,"15min":1.198},"min":{"1min":0.27200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.2130000000000001,"5min":9.532,"15min":9.532},"last":0.441}]},{"osd":3,"last update":"Fri Jun 14 23:07:30 2024","interfaces":[{"interface":"back","average":{"1min":1.712,"5min":1.579,"15min":1.4830000000000001},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":7.2160000000000002,"5min":9.2149999999999999,"15min":45.744},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.77,"5min":1.6519999999999999,"15min":1.5589999999999999},"min":{"1min":0.34899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.2320000000000002,"5min":14.914,"15min":45.838000000000001},"last":1.7310000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:07:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.101,"15min":0.85699999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.24399999999999999},"max":{"1min":6.9119999999999999,"5min":14.352,"15min":14.352},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.3049999999999999,"15min":1.0029999999999999},"min":{"1min":0.29299999999999998,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":2.4380000000000002,"5min":9.5980000000000008,"15min":9.5980000000000008},"last":1.1970000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.458,"15min":1.0960000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":4.6719999999999997,"5min":15.382999999999999,"15min":15.382999999999999},"last":3.1230000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.417,"15min":1.071},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.6020000000000003,"5min":14.855,"15min":14.855},"last":4.3300000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":1.3779999999999999,"15min":1.083},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.7669999999999999,"5min":14.818,"15min":14.818},"last":3.153},{"interface":"front","average":{"1min":1.641,"5min":1.2110000000000001,"15min":1.0369999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.25600000000000001},"max":{"1min":6.9779999999999998,"5min":14.429,"15min":14.429},"last":3.0880000000000001}]}]},{"osd":4,"up_from":30,"seq":128849019320,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1806888,"kb_used_data":908000,"kb_used_omap":1,"kb_used_meta":898878,"kb_avail":91938264,"statfs":{"total":95995035648,"available":94144782336,"internally_reserved":0,"allocated":929792000,"data_stored":928407146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920451416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:07:44 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.286,"15min":1.1120000000000001},"min":{"1min":0.23300000000000001,"5min":0.221,"15min":0.17100000000000001},"max":{"1min":6.8440000000000003,"5min":6.8440000000000003,"15min":10.567},"last":1.254},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.3129999999999999,"15min":1.117},"min":{"1min":0.251,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":4.1319999999999997,"5min":7.4900000000000002,"15min":8.3390000000000004},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:07:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0629999999999999,"15min":1.3140000000000001},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":3.1600000000000001,"5min":5.0430000000000001,"15min":56.917000000000002},"last":3.4489999999999998},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.1499999999999999,"15min":1.419},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.3010000000000002,"5min":6.2130000000000001,"15min":56.789999999999999},"last":1.907}]},{"osd":2,"last update":"Fri Jun 14 23:07:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":2.0030000000000001,"15min":1.2270000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.23000000000000001},"max":{"1min":4.4020000000000001,"5min":46.866,"15min":46.866},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.381,"5min":1.875,"15min":1.272},"min":{"1min":0.312,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.3920000000000003,"5min":46.851999999999997,"15min":46.851999999999997},"last":1.3620000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.048,"15min":1.032},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.6439999999999999,"5min":5.3239999999999998,"15min":6.617},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.292,"15min":1.1599999999999999},"min":{"1min":0.32500000000000001,"5min":0.28499999999999998,"15min":0.27500000000000002},"max":{"1min":3.77,"5min":6.1440000000000001,"15min":10.510999999999999},"last":3.5150000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:07:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":1.1419999999999999,"15min":1.5129999999999999},"min":{"1min":0.28899999999999998,"5min":0.217,"15min":0.217},"max":{"1min":2.0270000000000001,"5min":12.315,"15min":139.13200000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.623,"5min":1.175,"15min":1.542},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26100000000000001},"max":{"1min":2.8570000000000002,"5min":10.959,"15min":138.86000000000001},"last":1.216}]},{"osd":6,"last update":"Fri Jun 14 23:06:40 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.075,"15min":0.88300000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.7360000000000002,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":1.282},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.0509999999999999,"15min":0.83699999999999997},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.254},"max":{"1min":2.9009999999999998,"5min":5.8680000000000003,"15min":8.8819999999999997},"last":3.5579999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.02,"15min":0.90500000000000003},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.04,"5min":6.0890000000000004,"15min":8.9489999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.753,"5min":1.0369999999999999,"15min":1.018},"min":{"1min":0.30199999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.988,"5min":5.3479999999999999,"15min":8.4730000000000008},"last":0.47099999999999997}]}]},{"osd":3,"up_from":138,"seq":592705487019,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2132056,"kb_used_data":1605136,"kb_used_omap":4,"kb_used_meta":526907,"kb_avail":91613096,"statfs":{"total":95995035648,"available":93811810304,"internally_reserved":0,"allocated":1643659264,"data_stored":1642286767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":539553246},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:07:26 2024","interfaces":[{"interface":"back","average":{"1min":2.6219999999999999,"5min":1.502,"15min":1.1739999999999999},"min":{"1min":0.214,"5min":0.187,"15min":0.184},"max":{"1min":10.602,"5min":11.071999999999999,"15min":11.071999999999999},"last":2.3769999999999998},{"interface":"front","average":{"1min":2.6800000000000002,"5min":1.607,"15min":1.125},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":11.518000000000001,"5min":11.518000000000001,"15min":11.518000000000001},"last":4.2220000000000004}]},{"osd":1,"last update":"Fri Jun 14 23:07:26 2024","interfaces":[{"interface":"back","average":{"1min":2.5950000000000002,"5min":1.5669999999999999,"15min":1.2350000000000001},"min":{"1min":0.48899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":11.596,"5min":11.596,"15min":24.097999999999999},"last":4.1609999999999996},{"interface":"front","average":{"1min":3.1360000000000001,"5min":1.752,"15min":1.3149999999999999},"min":{"1min":0.47099999999999997,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":11.574999999999999,"5min":11.574999999999999,"15min":24.132999999999999},"last":4.133}]},{"osd":2,"last update":"Fri Jun 14 23:07:26 2024","interfaces":[{"interface":"back","average":{"1min":2.2890000000000001,"5min":1.756,"15min":1.1719999999999999},"min":{"1min":0.51000000000000001,"5min":0.26900000000000002,"15min":0.23599999999999999},"max":{"1min":5.093,"5min":9.4730000000000008,"15min":9.4730000000000008},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.891,"5min":1.4370000000000001,"15min":1.0760000000000001},"min":{"1min":0.24399999999999999,"5min":0.23799999999999999,"15min":0.20899999999999999},"max":{"1min":5.8799999999999999,"5min":9.1929999999999996,"15min":10.369},"last":2.1819999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:07:26 2024","interfaces":[{"interface":"back","average":{"1min":6.1970000000000001,"5min":2.4279999999999999,"15min":1.476},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27900000000000003},"max":{"1min":73.263000000000005,"5min":73.263000000000005,"15min":73.263000000000005},"last":2.218},{"interface":"front","average":{"1min":6.2610000000000001,"5min":2.2320000000000002,"15min":1.462},"min":{"1min":0.36499999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":73.5,"5min":73.5,"15min":73.5},"last":0.84699999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:07:26 2024","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":2.298,"15min":1.52},"min":{"1min":0.38200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.0419999999999998,"5min":81.438999999999993,"15min":81.438999999999993},"last":2.355},{"interface":"front","average":{"1min":1.353,"5min":2.3999999999999999,"15min":1.5169999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.0599999999999996,"5min":81.626999999999995,"15min":81.626999999999995},"last":4.29}]},{"osd":6,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":6.7469999999999999,"5min":2.5579999999999998,"15min":1.4750000000000001},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.32400000000000001},"max":{"1min":82.248999999999995,"5min":82.248999999999995,"15min":82.248999999999995},"last":4.3170000000000002},{"interface":"front","average":{"1min":6.5620000000000003,"5min":2.71,"15min":1.528},"min":{"1min":0.44900000000000001,"5min":0.31,"15min":0.31},"max":{"1min":81.585999999999999,"5min":81.585999999999999,"15min":81.585999999999999},"last":2.2850000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:07:26 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.4690000000000001,"15min":1.2689999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.8559999999999999,"5min":5.8559999999999999,"15min":7.7240000000000002},"last":2.2589999999999999},{"interface":"front","average":{"1min":1.792,"5min":1.177,"15min":1.1639999999999999},"min":{"1min":0.41399999999999998,"5min":0.36299999999999999,"15min":0.32400000000000001},"max":{"1min":5.9820000000000002,"5min":5.9820000000000002,"15min":5.9820000000000002},"last":0.876}]}]},{"osd":2,"up_from":133,"seq":571230650546,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1698524,"kb_used_data":1343700,"kb_used_omap":6,"kb_used_meta":354745,"kb_avail":92046628,"statfs":{"total":95995035648,"available":94255747072,"internally_reserved":0,"allocated":1375948800,"data_stored":1374572445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":363259381},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":3.177,"5min":2.2890000000000001,"15min":1.226},"min":{"1min":0.18099999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":10.147,"5min":55.851999999999997,"15min":55.851999999999997},"last":5.1879999999999997},{"interface":"front","average":{"1min":2.085,"5min":2.0459999999999998,"15min":1.3149999999999999},"min":{"1min":0.27500000000000002,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":6.5590000000000002,"5min":56.171999999999997,"15min":56.171999999999997},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":3.2959999999999998,"5min":1.6220000000000001,"15min":1.2190000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8689999999999998,"5min":9.8140000000000001,"15min":21.920000000000002},"last":0.78900000000000003},{"interface":"front","average":{"1min":2.7029999999999998,"5min":1.637,"15min":1.194},"min":{"1min":0.36599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":8.0760000000000005,"5min":14.416,"15min":21.573},"last":5.0880000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:07:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.5109999999999999,"15min":1.2130000000000001},"min":{"1min":0.38100000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.1289999999999996,"5min":6.516,"15min":21.66},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.976,"5min":1.4850000000000001,"15min":1.236},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":6.2679999999999998,"5min":9.9060000000000006,"15min":22.341000000000001},"last":0.48499999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.8600000000000001,"15min":1.4199999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.6349999999999998,"5min":31.928000000000001,"15min":31.928000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.2090000000000001,"5min":1.7969999999999999,"15min":1.349},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":6.8150000000000004,"5min":31.954999999999998,"15min":31.954999999999998},"last":0.54800000000000004}]},{"osd":5,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":1.4710000000000001,"15min":1.234},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.28299999999999997},"max":{"1min":9.6310000000000002,"5min":9.6310000000000002,"15min":31.539999999999999},"last":5.0529999999999999},{"interface":"front","average":{"1min":1.8320000000000001,"5min":1.5129999999999999,"15min":1.258},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":8.8140000000000001,"5min":8.8140000000000001,"15min":31.481999999999999},"last":0.73899999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:06:56 2024","interfaces":[{"interface":"back","average":{"1min":2.25,"5min":1.907,"15min":1.2310000000000001},"min":{"1min":0.35699999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.1440000000000001,"5min":32.005000000000003,"15min":32.005000000000003},"last":4.2880000000000003},{"interface":"front","average":{"1min":2.3460000000000001,"5min":1.905,"15min":1.22},"min":{"1min":0.41199999999999998,"5min":0.34999999999999998,"15min":0.30399999999999999},"max":{"1min":7.7409999999999997,"5min":32.079999999999998,"15min":32.079999999999998},"last":0.59199999999999997}]},{"osd":7,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":2.5139999999999998,"5min":1.5760000000000001,"15min":1.2649999999999999},"min":{"1min":0.35899999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":6.8470000000000004,"5min":9.7539999999999996,"15min":10.614000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.958,"5min":1.4650000000000001,"15min":1.0800000000000001},"min":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":9.5850000000000009,"5min":9.5850000000000009,"15min":9.5850000000000009},"last":5.1150000000000002}]}]},{"osd":1,"up_from":116,"seq":498216206563,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2431056,"kb_used_data":1789576,"kb_used_omap":5,"kb_used_meta":641466,"kb_avail":91314096,"statfs":{"total":95995035648,"available":93505634304,"internally_reserved":0,"allocated":1832525824,"data_stored":1831160885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":656861464},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":1.696,"15min":1.3580000000000001},"min":{"1min":0.20200000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":8.5060000000000002,"5min":9.5350000000000001,"15min":11.257999999999999},"last":2.7349999999999999},{"interface":"front","average":{"1min":2.927,"5min":1.8100000000000001,"15min":1.27},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":13.331,"5min":13.672000000000001,"15min":13.672000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":3.254,"5min":1.8620000000000001,"15min":1.4339999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19500000000000001},"max":{"1min":14.597,"5min":14.597,"15min":14.597},"last":5.6749999999999998},{"interface":"front","average":{"1min":2.4319999999999999,"5min":1.5089999999999999,"15min":1.1060000000000001},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.711,"5min":14.711,"15min":14.711},"last":0.60099999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:07:26 2024","interfaces":[{"interface":"back","average":{"1min":2.5289999999999999,"5min":1.591,"15min":1.3169999999999999},"min":{"1min":0.50700000000000001,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":5.5430000000000001,"5min":7.5679999999999996,"15min":21.422000000000001},"last":1.2869999999999999},{"interface":"front","average":{"1min":2.464,"5min":1.6859999999999999,"15min":1.389},"min":{"1min":0.34300000000000003,"5min":0.24099999999999999,"15min":0.214},"max":{"1min":8.2059999999999995,"5min":8.2059999999999995,"15min":21.016999999999999},"last":2.702}]},{"osd":4,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.2350000000000001,"15min":1.212},"min":{"1min":0.29299999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.8370000000000002,"5min":6.2939999999999996,"15min":7.673},"last":0.66000000000000003},{"interface":"front","average":{"1min":2.488,"5min":1.4550000000000001,"15min":1.393},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.29199999999999998},"max":{"1min":8.2430000000000003,"5min":8.2430000000000003,"15min":9.4169999999999998},"last":5.5960000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.5249999999999999,"15min":1.359},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":9.4489999999999998,"5min":19.132999999999999,"15min":19.132999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4419999999999999,"15min":1.25},"min":{"1min":0.308,"5min":0.29399999999999998,"15min":0.28599999999999998},"max":{"1min":3.806,"5min":19.123000000000001,"15min":19.123000000000001},"last":0.63300000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:05:58 2024","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":2.2349999999999999,"15min":1.669},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":24.856000000000002,"5min":63.795999999999999,"15min":63.795999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":2.548,"5min":2.1760000000000002,"15min":1.571},"min":{"1min":0.39900000000000002,"5min":0.32900000000000001,"15min":0.28499999999999998},"max":{"1min":24.904,"5min":63.811,"15min":63.811},"last":5.6509999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":2.5209999999999999,"5min":1.419,"15min":1.3919999999999999},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":8.266,"5min":8.266,"15min":9.4429999999999996},"last":5.6289999999999996},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.204,"15min":1.1970000000000001},"min":{"1min":0.378,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":9.4939999999999998,"5min":9.4939999999999998,"15min":9.4939999999999998},"last":0.55500000000000005}]}]},{"osd":0,"up_from":91,"seq":390842024217,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2032912,"kb_used_data":1175240,"kb_used_omap":5,"kb_used_meta":857658,"kb_avail":91712240,"statfs":{"total":95995035648,"available":93913333760,"internally_reserved":0,"allocated":1203445760,"data_stored":1202078820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":878241882},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:07:49 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.359,"15min":1.306},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.16700000000000001},"max":{"1min":7.218,"5min":7.952,"15min":16.305},"last":34.448},{"interface":"front","average":{"1min":1.984,"5min":1.5189999999999999,"15min":1.423},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":9.3499999999999996,"5min":9.9990000000000006,"15min":16.184999999999999},"last":34.354999999999997}]},{"osd":2,"last update":"Fri Jun 14 23:08:05 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.4350000000000001,"15min":1.2050000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.8639999999999999,"5min":9.9789999999999992,"15min":16.757000000000001},"last":0.186},{"interface":"front","average":{"1min":2.0099999999999998,"5min":1.4419999999999999,"15min":1.2470000000000001},"min":{"1min":0.22700000000000001,"5min":0.223,"15min":0.222},"max":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":16.055},"last":0.22700000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:07:37 2024","interfaces":[{"interface":"back","average":{"1min":1.8979999999999999,"5min":1.774,"15min":1.306},"min":{"1min":0.26000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":9.391,"5min":11.083,"15min":16.873999999999999},"last":35.094999999999999},{"interface":"front","average":{"1min":2.6989999999999998,"5min":1.778,"15min":1.3340000000000001},"min":{"1min":0.38100000000000001,"5min":0.23999999999999999,"15min":0.224},"max":{"1min":8.0259999999999998,"5min":11.375,"15min":16.085999999999999},"last":38.444000000000003}]},{"osd":4,"last update":"Fri Jun 14 23:07:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3700000000000001,"15min":1.371},"min":{"1min":0.317,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.492,"5min":11.162000000000001,"15min":16.527000000000001},"last":0.371},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.3440000000000001,"15min":1.3169999999999999},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":9.423,"5min":9.423,"15min":15.976000000000001},"last":0.63}]},{"osd":5,"last update":"Fri Jun 14 23:07:37 2024","interfaces":[{"interface":"back","average":{"1min":1.7969999999999999,"5min":1.6599999999999999,"15min":1.5149999999999999},"min":{"1min":0.32400000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":9.4510000000000005,"5min":10.462,"15min":16.108000000000001},"last":0.501},{"interface":"front","average":{"1min":0.876,"5min":1.3999999999999999,"15min":1.4119999999999999},"min":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.31900000000000001},"max":{"1min":4.3769999999999998,"5min":8.0129999999999999,"15min":16.006},"last":0.45200000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:06:36 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.5109999999999999,"15min":1.4419999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30599999999999999},"max":{"1min":3.367,"5min":6.1669999999999998,"15min":16.152000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.7130000000000001,"15min":1.466},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29399999999999998},"max":{"1min":4.274,"5min":9.3130000000000006,"15min":16.227},"last":0.55600000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:07:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.337,"15min":1.351},"min":{"1min":0.36599999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.3970000000000002,"5min":8.0389999999999997,"15min":16.199999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.4379999999999999,"15min":1.417},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":5.5270000000000001,"5min":11.195,"15min":16.259},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1198129152,"data_stored":1198129152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1825505280,"data_stored":1825505280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1367281664,"data_stored":1367277592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1620967424,"data_stored":1620967424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":920522752,"data_stored":920518680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":931004416,"data_stored":931004416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3692695552,"data_stored":3692691480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1390280704,"data_stored":1390280704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:08:14.838 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-14T23:08:14.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set unique_pool_6 pg_num 26 2024-06-14T23:08:14.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:14.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:14.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:14.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:14.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:14.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:15.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:15.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:08:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:08:14] "GET /metrics HTTP/1.1" 200 36627 "" "Prometheus/2.43.0" 2024-06-14T23:08:15.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:15.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:15.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:15.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:15.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:15.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:15.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:15.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:15.369 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:15.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:15.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:15.517 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:15.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:15.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:15.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:15.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:15.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:15.801 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:15.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:15.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:15.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:15.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:15.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:16.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:16.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:16.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:16.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:16.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:16.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:16.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:15 smithi137 ceph-mon[90634]: pgmap v1057: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 761 op/s; 23567/164544 objects degraded (14.323%); 0 B/s, 26 objects/s recovering 2024-06-14T23:08:16.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:15 smithi086 ceph-mon[97913]: pgmap v1057: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 761 op/s; 23567/164544 objects degraded (14.323%); 0 B/s, 26 objects/s recovering 2024-06-14T23:08:16.262 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:15 smithi086 ceph-mon[102754]: pgmap v1057: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 761 op/s; 23567/164544 objects degraded (14.323%); 0 B/s, 26 objects/s recovering 2024-06-14T23:08:16.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:16.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:16.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:16.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:16.480 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:16.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:16.566 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:08:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:08:16.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:16.606 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:08:16.606 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:08:16.638 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:08:16.638 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:08:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:16.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:16.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:16.672 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:08:16.672 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:08:16.699 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:08:16.700 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:08:16.737 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:08:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:08:16.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:16.772 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:08:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:08:16.780 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:16.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:16.808 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:08:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:08:16.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:16.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:16.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:16.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:17.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:17.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:17.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/732194891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:17.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:16 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/732194891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:17.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:17.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:17.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:16 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/732194891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:17.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:17.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:17.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:17.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:17.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:17.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:17.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:17.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:17.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:17.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:17.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:17.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:17.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:17.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:17.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:17.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:17.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:18.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:17 smithi086 ceph-mon[97913]: pgmap v1058: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 23567/157158 objects degraded (14.996%); 0 B/s, 38 objects/s recovering 2024-06-14T23:08:18.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:17 smithi086 ceph-mon[102754]: pgmap v1058: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 23567/157158 objects degraded (14.996%); 0 B/s, 38 objects/s recovering 2024-06-14T23:08:18.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:18.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:18.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:18.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:18.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:17 smithi137 ceph-mon[90634]: pgmap v1058: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 23567/157158 objects degraded (14.996%); 0 B/s, 38 objects/s recovering 2024-06-14T23:08:18.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:18.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:18.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:18.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:18.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:18.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:18.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:18.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:18.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:18.600 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:18.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:18.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:18.702 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:18.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:18.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:18.838 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:18.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:18.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:18.960 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 23567/157158 objects degraded (14.996%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:18.960 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:18 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:18.960 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:18 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:18.973 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:18.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:19.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:19.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:19.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:19.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:19.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 23567/157158 objects degraded (14.996%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:19.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:18 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:19.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:18 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:19.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:19.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:19.307 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 23567/157158 objects degraded (14.996%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:19.307 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:18 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:19.307 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:18 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:19.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:19.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:19.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:19.537 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:19.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:19.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:19.680 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:19.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:19.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:19.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:08:19.814 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:19.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:19.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:19.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:19.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:20.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[97913]: pgmap v1059: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 23567/156465 objects degraded (15.062%); 0 B/s, 26 objects/s recovering 2024-06-14T23:08:20.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:08:20.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:08:20.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:20.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3587484864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:08:20.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:08:20.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[102754]: pgmap v1059: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 23567/156465 objects degraded (15.062%); 0 B/s, 26 objects/s recovering 2024-06-14T23:08:20.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:08:20.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:08:20.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:20.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3587484864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:08:20.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:19 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:08:20.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:20.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:20.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:20.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:20.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:19 smithi137 ceph-mon[90634]: pgmap v1059: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 23567/156465 objects degraded (15.062%); 0 B/s, 26 objects/s recovering 2024-06-14T23:08:20.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:19 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:08:20.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:19 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:08:20.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:19 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:20.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:19 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3587484864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:08:20.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:19 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:08:20.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:20.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:20.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:20.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:20.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:20.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:20.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:20.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:20.659 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:08:20.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:20.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:20.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:20.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:20.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:20.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:20.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:20.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:21.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:21.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:21.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:21.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:21.243 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:21.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:20 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:08:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:20 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:08:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:20 smithi137 ceph-mon[90634]: osdmap e276: 8 total, 8 up, 8 in 2024-06-14T23:08:21.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:20 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:08:21.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:20 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:08:21.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:20 smithi086 ceph-mon[97913]: osdmap e276: 8 total, 8 up, 8 in 2024-06-14T23:08:21.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:20 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:08:21.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:20 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:08:21.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:20 smithi086 ceph-mon[102754]: osdmap e276: 8 total, 8 up, 8 in 2024-06-14T23:08:21.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:21.381 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:21.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:21.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:21.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:21.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:21.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:21.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:21.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:21.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:21.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:21.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:21.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:21.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:21.920 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:08:21.920 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:08:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:21.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:21.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:22.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:21 smithi086 ceph-mon[97913]: pgmap v1061: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 23567/145215 objects degraded (16.229%); 0 B/s, 34 objects/s recovering 2024-06-14T23:08:22.050 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:21 smithi086 ceph-mon[102754]: pgmap v1061: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 23567/145215 objects degraded (16.229%); 0 B/s, 34 objects/s recovering 2024-06-14T23:08:22.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:22.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:22.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:22.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:22.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:22.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:22.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:22.244 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:21 smithi137 ceph-mon[90634]: pgmap v1061: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 23567/145215 objects degraded (16.229%); 0 B/s, 34 objects/s recovering 2024-06-14T23:08:22.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:22.378 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:22.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:22.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:22.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:22.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:22.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:22.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:22.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:22.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:22.792 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:22.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:22.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:22.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:22.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:23.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:23.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:23.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:23.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:23.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:23.227 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:23.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:23.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:23.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:23.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:23.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:23.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:23.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:23.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:23.637 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:23.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:23.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:23.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:23.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:23.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:23.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:23.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:24.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:24.051 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:23 smithi086 ceph-mon[97913]: pgmap v1062: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 793 op/s; 23567/144432 objects degraded (16.317%); 0 B/s, 34 objects/s recovering 2024-06-14T23:08:24.051 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 23567/144432 objects degraded (16.317%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:24.051 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:23 smithi086 ceph-mon[102754]: pgmap v1062: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 793 op/s; 23567/144432 objects degraded (16.317%); 0 B/s, 34 objects/s recovering 2024-06-14T23:08:24.052 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 23567/144432 objects degraded (16.317%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:24.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:24.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:24.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:24.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:24.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:24.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:24.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:23 smithi137 ceph-mon[90634]: pgmap v1062: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 793 op/s; 23567/144432 objects degraded (16.317%); 0 B/s, 34 objects/s recovering 2024-06-14T23:08:24.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 23567/144432 objects degraded (16.317%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:24.245 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:08:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=cleanup t=2024-06-14T23:08:23.769449465Z level=info msg="Completed cleanup jobs" duration=14.718676ms 2024-06-14T23:08:24.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:24.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:24.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:24.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:24.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:24.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:24.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:24.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:24.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:08:24.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:24.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:24.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:24.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T23:08:24.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:24.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:24.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:24.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:24.905 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:24.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:24.993 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 11 pg_num to 26 2024-06-14T23:08:25.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:25.033 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:25.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:25.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:24 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1986412565' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-14T23:08:25.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:08:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:08:24] "GET /metrics HTTP/1.1" 200 36627 "" "Prometheus/2.43.0" 2024-06-14T23:08:25.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:24 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1986412565' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-14T23:08:25.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:25.166 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:25.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:25.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:24 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1986412565' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-14T23:08:25.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:25.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:25.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:25.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:25.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:25.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:25.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:25.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:25.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:25.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:25.709 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:25.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:25.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:25.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:25.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:25.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:25.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:25.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:25.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:26.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:26.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:26.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:25 smithi137 ceph-mon[90634]: pgmap v1063: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 23567/140628 objects degraded (16.758%); 0 B/s, 34 objects/s recovering 2024-06-14T23:08:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1986412565' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-06-14T23:08:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:25 smithi137 ceph-mon[90634]: osdmap e277: 8 total, 8 up, 8 in 2024-06-14T23:08:26.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:26.290 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:25 smithi086 ceph-mon[102754]: pgmap v1063: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 23567/140628 objects degraded (16.758%); 0 B/s, 34 objects/s recovering 2024-06-14T23:08:26.291 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1986412565' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-06-14T23:08:26.291 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:25 smithi086 ceph-mon[102754]: osdmap e277: 8 total, 8 up, 8 in 2024-06-14T23:08:26.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:25 smithi086 ceph-mon[97913]: pgmap v1063: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 796 op/s; 23567/140628 objects degraded (16.758%); 0 B/s, 34 objects/s recovering 2024-06-14T23:08:26.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1986412565' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-06-14T23:08:26.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:25 smithi086 ceph-mon[97913]: osdmap e277: 8 total, 8 up, 8 in 2024-06-14T23:08:26.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:26.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:26.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:26.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:26.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:26.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:26.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:26.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:26.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:26.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:26.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:26.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:26.908 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:26.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:26.968 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:26.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:27.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:27.101 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:27.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:27.127 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-14T23:08:27.127 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:26 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-14T23:08:27.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:27.256 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:27.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:27.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:27.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:27.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:27.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:27.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:26 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-14T23:08:27.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:27.562 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:27.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:27.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:27.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:27.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:27.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:27.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:27.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:27.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:27.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:27.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:28.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:28.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:28.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:28.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:28.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[102754]: pgmap v1065: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 23567/133275 objects degraded (17.683%); 0 B/s, 46 objects/s recovering 2024-06-14T23:08:28.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-14T23:08:28.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[102754]: osdmap e278: 8 total, 8 up, 8 in 2024-06-14T23:08:28.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:28.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:28.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1221538978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:28.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[97913]: pgmap v1065: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 23567/133275 objects degraded (17.683%); 0 B/s, 46 objects/s recovering 2024-06-14T23:08:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-14T23:08:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[97913]: osdmap e278: 8 total, 8 up, 8 in 2024-06-14T23:08:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:28.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1221538978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:28.089 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:08:28.089 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:08:28.089 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:08:28.089 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:08:28.090 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:08:28.090 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:08:28.090 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:08:28.090 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:08:28.090 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:08:28.090 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:08:28.090 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:08:28.091 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:08:28.091 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:08:28.091 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:08:28.091 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:08:28.091 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:08:28.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:28.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:28.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:28.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:28.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:28.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:28.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:28.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:28 smithi137 ceph-mon[90634]: pgmap v1065: 109 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 23567/133275 objects degraded (17.683%); 0 B/s, 46 objects/s recovering 2024-06-14T23:08:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-14T23:08:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:28 smithi137 ceph-mon[90634]: osdmap e278: 8 total, 8 up, 8 in 2024-06-14T23:08:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:08:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1221538978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:28.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:28.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:28.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:28.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:28.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:28.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:28.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:28.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:28.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:28.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:28.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:29.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:29.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:29.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:29.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:29.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:29.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:29.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:29.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:29 smithi086 ceph-mon[97913]: osdmap e279: 8 total, 8 up, 8 in 2024-06-14T23:08:29.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:29 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 23567/133275 objects degraded (17.683%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:29.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:29 smithi086 ceph-mon[97913]: osdmap e280: 8 total, 8 up, 8 in 2024-06-14T23:08:29.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:29 smithi086 ceph-mon[102754]: osdmap e279: 8 total, 8 up, 8 in 2024-06-14T23:08:29.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:29 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 23567/133275 objects degraded (17.683%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:29.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:29 smithi086 ceph-mon[102754]: osdmap e280: 8 total, 8 up, 8 in 2024-06-14T23:08:29.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:29.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:29.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:29.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:29.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:29.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:29.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:29 smithi137 ceph-mon[90634]: osdmap e279: 8 total, 8 up, 8 in 2024-06-14T23:08:29.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:29 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 23567/133275 objects degraded (17.683%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:29.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:29 smithi137 ceph-mon[90634]: osdmap e280: 8 total, 8 up, 8 in 2024-06-14T23:08:29.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:29.612 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:29.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:29.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:29.748 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:29.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:29.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:29.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:29.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:29.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:30.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:30.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:30.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:30.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:30.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:30.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:30.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:30.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:30.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:30.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:30 smithi086 ceph-mon[102754]: pgmap v1068: 119 pgs: 1 remapped+peering, 10 unknown, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 20139/128499 objects degraded (15.672%); 0 B/s, 31 objects/s recovering 2024-06-14T23:08:30.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:30 smithi086 ceph-mon[97913]: pgmap v1068: 119 pgs: 1 remapped+peering, 10 unknown, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 20139/128499 objects degraded (15.672%); 0 B/s, 31 objects/s recovering 2024-06-14T23:08:30.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:30.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:30.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:30.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:30 smithi137 ceph-mon[90634]: pgmap v1068: 119 pgs: 1 remapped+peering, 10 unknown, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 20139/128499 objects degraded (15.672%); 0 B/s, 31 objects/s recovering 2024-06-14T23:08:30.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:30.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:30.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:30.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:30.709 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:30.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:30.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:30.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:30.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:30.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:30.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:31.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:31.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:31.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:31.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:31.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:31.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:31.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:31.362 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:31.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:31.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:31.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:31.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:31.630 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:31.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:31.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:31.648 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-06-14T23:08:31.648 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:08:31.648 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-14T23:08:31.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:08:31.732 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:31.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:31.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:31.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:31.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:31.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:31.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:32.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:32.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:32.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:32.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:32.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:32.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:32.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:32.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:32.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:32 smithi086 ceph-mon[102754]: pgmap v1070: 119 pgs: 6 active+recovery_wait+degraded, 1 remapped+peering, 3 unknown, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 20139/122610 objects degraded (16.425%); 0 B/s, 93 objects/s recovering 2024-06-14T23:08:32.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:32 smithi086 ceph-mon[97913]: pgmap v1070: 119 pgs: 6 active+recovery_wait+degraded, 1 remapped+peering, 3 unknown, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 20139/122610 objects degraded (16.425%); 0 B/s, 93 objects/s recovering 2024-06-14T23:08:32.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:32.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:32.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:32.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:32 smithi137 ceph-mon[90634]: pgmap v1070: 119 pgs: 6 active+recovery_wait+degraded, 1 remapped+peering, 3 unknown, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 20139/122610 objects degraded (16.425%); 0 B/s, 93 objects/s recovering 2024-06-14T23:08:32.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:32.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:32.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:32.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:32.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:32.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:32.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:32.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:32.868 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:32.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:32.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:33.006 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:33.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:33.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:33.137 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:33.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:33.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:33.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:33.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:33.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:33.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:33.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:33.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:33.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:33.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:33.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:33.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:33.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:33.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:33.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:33.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:33.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:33.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:33.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:33.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:34.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:34.090 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:34.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:34.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:34.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:34.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:34.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:34.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:34.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:34 smithi086 ceph-mon[102754]: pgmap v1071: 119 pgs: 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 104 active+clean; 2.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 464 op/s; 21863/124911 objects degraded (17.503%); 0 B/s, 102 objects/s recovering 2024-06-14T23:08:34.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:34 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 21863/124911 objects degraded (17.503%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:34.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:34 smithi086 ceph-mon[97913]: pgmap v1071: 119 pgs: 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 104 active+clean; 2.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 464 op/s; 21863/124911 objects degraded (17.503%); 0 B/s, 102 objects/s recovering 2024-06-14T23:08:34.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:34 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 21863/124911 objects degraded (17.503%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:34.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:34.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:34.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:34.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:34 smithi137 ceph-mon[90634]: pgmap v1071: 119 pgs: 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 104 active+clean; 2.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 464 op/s; 21863/124911 objects degraded (17.503%); 0 B/s, 102 objects/s recovering 2024-06-14T23:08:34.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:34 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 21863/124911 objects degraded (17.503%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:34.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:34.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:34.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:34.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:34.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:34.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:34.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:34.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:34.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:34.948 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:34.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:35.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:35.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:35.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:35.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:35.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:35.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:35.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:35.327 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:35.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:35.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:35 smithi086 ceph-mon[97913]: pgmap v1072: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovering, 104 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 580 op/s; 21863/119520 objects degraded (18.292%); 0 B/s, 78 objects/s recovering 2024-06-14T23:08:35.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:08:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:08:34] "GET /metrics HTTP/1.1" 200 36623 "" "Prometheus/2.43.0" 2024-06-14T23:08:35.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:35 smithi086 ceph-mon[102754]: pgmap v1072: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovering, 104 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 580 op/s; 21863/119520 objects degraded (18.292%); 0 B/s, 78 objects/s recovering 2024-06-14T23:08:35.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:35.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:35.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:35.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:35.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:35 smithi137 ceph-mon[90634]: pgmap v1072: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovering, 104 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 580 op/s; 21863/119520 objects degraded (18.292%); 0 B/s, 78 objects/s recovering 2024-06-14T23:08:35.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:35.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:35.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:35.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:35.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:35.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:35.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:35.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:35.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:35.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:35.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:36.019 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:36.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:36.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:36.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:36.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:36.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:36.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:36.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:36.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:36.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:36.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:36.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:36.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:36.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:36.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:08:36.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:36.745 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:36.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:36.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:36.889 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:36.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:36.991 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:37.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:37.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:37.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:37.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:37.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:37.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:37.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:37.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:37.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:37.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:37.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:37.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:37.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:37.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:37.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:37.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:37.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:37.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:37.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:37.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:37.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:37.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:38.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:38.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:38.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:38.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:38.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:38.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:38.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:37 smithi137 ceph-mon[90634]: pgmap v1073: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 104 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 29 MiB/s wr, 1.67k op/s; 21863/108267 objects degraded (20.194%); 0 B/s, 175 objects/s recovering 2024-06-14T23:08:38.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:37 smithi086 ceph-mon[97913]: pgmap v1073: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 104 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 29 MiB/s wr, 1.67k op/s; 21863/108267 objects degraded (20.194%); 0 B/s, 175 objects/s recovering 2024-06-14T23:08:38.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:37 smithi086 ceph-mon[102754]: pgmap v1073: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 104 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 29 MiB/s wr, 1.67k op/s; 21863/108267 objects degraded (20.194%); 0 B/s, 175 objects/s recovering 2024-06-14T23:08:38.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:38.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:38.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:38.436 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:38.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:38.539 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:38.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:38.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:38.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:38.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:38.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:38.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:38.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:38.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:38.952 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:38.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:39.053 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 21863/108267 objects degraded (20.194%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:39.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 21863/108267 objects degraded (20.194%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:39.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:39.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:39.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:39.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:39.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:39.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 21863/108267 objects degraded (20.194%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:39.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:39.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:39.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:39.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:39.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:39.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:39.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:39.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:39.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:39.772 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:39.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:39.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:39.906 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:39.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:39.975 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:08:40.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:40.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:40.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:40.095 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:40.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:40.196 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:40.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:40.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:39 smithi137 ceph-mon[90634]: pgmap v1074: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 104 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 25 MiB/s wr, 1.49k op/s; 21863/108267 objects degraded (20.194%); 0 B/s, 156 objects/s recovering 2024-06-14T23:08:40.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:39 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3053565486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:40.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:39 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:40.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:40.326 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:40.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:40.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:39 smithi086 ceph-mon[97913]: pgmap v1074: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 104 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 25 MiB/s wr, 1.49k op/s; 21863/108267 objects degraded (20.194%); 0 B/s, 156 objects/s recovering 2024-06-14T23:08:40.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:39 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3053565486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:40.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:39 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:40.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:39 smithi086 ceph-mon[102754]: pgmap v1074: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 104 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 25 MiB/s wr, 1.49k op/s; 21863/108267 objects degraded (20.194%); 0 B/s, 156 objects/s recovering 2024-06-14T23:08:40.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:39 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3053565486' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:40.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:39 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:08:40.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:40.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:40.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:40.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:40.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:40.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:40.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:40.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:40.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:40.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:40.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:40.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:40.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:40.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:41.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:41.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:41.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:41.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:41.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:41.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:40 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:08:41.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:40 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:08:41.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:40 smithi137 ceph-mon[90634]: osdmap e281: 8 total, 8 up, 8 in 2024-06-14T23:08:41.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:41.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:41.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:41.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:40 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:08:41.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:40 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:08:41.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:40 smithi086 ceph-mon[97913]: osdmap e281: 8 total, 8 up, 8 in 2024-06-14T23:08:41.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:40 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:08:41.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:40 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:08:41.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:40 smithi086 ceph-mon[102754]: osdmap e281: 8 total, 8 up, 8 in 2024-06-14T23:08:41.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:41.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:41.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:41.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:41.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:41.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:41.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:41.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:41.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:41.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:41.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:41.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:41.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:41.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:41.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:41.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:42.008 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:42.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:42.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:42.148 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:42.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.36237.0:225053 11.14 e281/278 11:2ab3089e:::benchmark_data_smithi086_226760_object97732:head v 281'13871, mlcod=262'8672)", 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:08:41.007325+0000", 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.000163568, 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019199700000000001, 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.007325+0000", 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.007325+0000", 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.007323+0000", 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:08:42.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.007330+0000", 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4710000000000001e-06 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.007347+0000", 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6495e-05 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.007355+0000", 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8189999999999992e-06 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.007453+0000", 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8752999999999995e-05 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.007477+0000", 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.387e-05 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:225042 11.d 11:bab91e7a:::benchmark_data_smithi086_226760_object97721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-06-14T23:08:42.192 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:08:40.993226+0000", 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.014262657, 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014358154999999999, 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 225042 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:40.993226+0000", 2024-06-14T23:08:42.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:40.993226+0000", 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:40.993224+0000", 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:08:42.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:40.993231+0000", 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4500000000000001e-06 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:40.993246+0000", 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5529999999999999e-05 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:40.993254+0000", 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7409999999999992e-06 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:40.996260+0000", 2024-06-14T23:08:42.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0030064720000000001 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:40.997117+0000", 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00085639199999999998 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,6", 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:40.997269+0000", 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015233299999999999 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.002875+0000", 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056055970000000004 2024-06-14T23:08:42.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:41.007434+0000", 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045593200000000004 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 2 2024-06-14T23:08:42.197 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:08:42.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:42.265 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:42 smithi137 ceph-mon[90634]: pgmap v1076: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 104 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 25 MiB/s wr, 1.75k op/s; 21863/94545 objects degraded (23.124%); 0 B/s, 139 objects/s recovering 2024-06-14T23:08:42.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:42.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:42.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:42 smithi086 ceph-mon[97913]: pgmap v1076: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 104 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 25 MiB/s wr, 1.75k op/s; 21863/94545 objects degraded (23.124%); 0 B/s, 139 objects/s recovering 2024-06-14T23:08:42.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:42 smithi086 ceph-mon[102754]: pgmap v1076: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 104 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 25 MiB/s wr, 1.75k op/s; 21863/94545 objects degraded (23.124%); 0 B/s, 139 objects/s recovering 2024-06-14T23:08:42.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:42.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:42.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:42.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:42.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:42.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:42.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:42.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:42.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:42.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:42.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:42.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:42.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:42.991 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:42.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:42.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:43.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-14T23:08:43.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:43.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:43.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:43.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:43.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:43.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:43.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:43.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:43.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:43.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:43.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:43.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:43.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:43.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:43.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:43.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:43.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:43.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:43.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:43.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:43.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:43.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:43.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:43.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:43.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:44.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:44.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:44.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:44.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:44.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:44.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:44.272 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:44 smithi086 ceph-mon[102754]: pgmap v1077: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20246/91500 objects degraded (22.127%); 0 B/s, 197 objects/s recovering 2024-06-14T23:08:44.272 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:44 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 20246/91500 objects degraded (22.127%), 11 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:44.272 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:44 smithi086 ceph-mon[97913]: pgmap v1077: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20246/91500 objects degraded (22.127%); 0 B/s, 197 objects/s recovering 2024-06-14T23:08:44.272 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:44 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 20246/91500 objects degraded (22.127%), 11 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:44.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:44.364 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:44.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:44.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:44.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:44 smithi137 ceph-mon[90634]: pgmap v1077: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20246/91500 objects degraded (22.127%); 0 B/s, 197 objects/s recovering 2024-06-14T23:08:44.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:44 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 20246/91500 objects degraded (22.127%), 11 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:44.499 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:44.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:44.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:44.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:44.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:44.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:44.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:44.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:44.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:44.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:44.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:44.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:45.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:45.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:45.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:08:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:08:44] "GET /metrics HTTP/1.1" 200 36622 "" "Prometheus/2.43.0" 2024-06-14T23:08:45.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:45.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:45.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:45.228 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:45.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:45.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:45.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:45.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:45.441 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:45.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:45.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:45.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:45.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:45.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:45.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:08:45.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:45.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:45.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:45.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:45.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:45.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:45.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:46.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:46.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:46.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:46 smithi086 ceph-mon[102754]: pgmap v1078: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovering, 105 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 20246/86472 objects degraded (23.413%); 0 B/s, 197 objects/s recovering 2024-06-14T23:08:46.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:46 smithi086 ceph-mon[97913]: pgmap v1078: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovering, 105 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 20246/86472 objects degraded (23.413%); 0 B/s, 197 objects/s recovering 2024-06-14T23:08:46.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:46.133 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:08:46.134 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:08:46.134 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:08:46.134 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:08:46.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:46.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:46.157 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:08:46.157 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:08:46.198 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:08:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:08:46.230 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:08:46.230 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:08:46.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:46.260 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:08:46.260 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:08:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:46.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:46.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:46.283 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:08:46.283 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:08:46.310 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:08:46.310 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:08:46.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:46.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:46.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:46.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:46 smithi137 ceph-mon[90634]: pgmap v1078: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovering, 105 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 20246/86472 objects degraded (23.413%); 0 B/s, 197 objects/s recovering 2024-06-14T23:08:46.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:46.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:46.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:46.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:46.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:46.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:46.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:46.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:46.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:46.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:46.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:46.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:47.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:47.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:47.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:47.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:47.225 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:47.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:47.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4111147141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:47.236 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4111147141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:47.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:47.363 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:47.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:47.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:47.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4111147141' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:47.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:47.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:47.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:47.633 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:47.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:47.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:47.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:47.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:47.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:47.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:47.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:48.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:48.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:48.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:48.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:48.193 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:48.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:48.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:48.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:48.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:48.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:48 smithi086 ceph-mon[97913]: pgmap v1079: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20246/74874 objects degraded (27.040%); 0 B/s, 213 objects/s recovering 2024-06-14T23:08:48.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:48 smithi086 ceph-mon[102754]: pgmap v1079: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20246/74874 objects degraded (27.040%); 0 B/s, 213 objects/s recovering 2024-06-14T23:08:48.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:48.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:48.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:48.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:48 smithi137 ceph-mon[90634]: pgmap v1079: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20246/74874 objects degraded (27.040%); 0 B/s, 213 objects/s recovering 2024-06-14T23:08:48.514 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:08:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:08:48.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:48.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:48.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:48.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:48.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:48.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:48.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:48.866 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:48.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:48.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:48.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:49.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:49 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 20246/74874 objects degraded (27.040%), 11 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:49.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:49.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:49.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:49.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:49.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:49.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:49.327 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:49 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 20246/74874 objects degraded (27.040%), 11 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:49.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:49.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:49.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:49.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:49 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 20246/74874 objects degraded (27.040%), 11 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:49.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:49.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:49.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:49.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:49.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:49.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:49.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:49.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:49.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:49.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:49.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:50.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:50.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:50.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:50.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:50.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:50.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:50.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:50 smithi086 ceph-mon[97913]: from='client.46367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:08:50.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:50 smithi086 ceph-mon[97913]: pgmap v1080: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20246/74874 objects degraded (27.040%); 0 B/s, 213 objects/s recovering 2024-06-14T23:08:50.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:50 smithi086 ceph-mon[102754]: from='client.46367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:08:50.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:50 smithi086 ceph-mon[102754]: pgmap v1080: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20246/74874 objects degraded (27.040%); 0 B/s, 213 objects/s recovering 2024-06-14T23:08:50.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:50.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:50.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:50.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:50.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:50 smithi137 ceph-mon[90634]: from='client.46367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:08:50.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:50 smithi137 ceph-mon[90634]: pgmap v1080: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 20246/74874 objects degraded (27.040%); 0 B/s, 213 objects/s recovering 2024-06-14T23:08:50.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:50.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:50.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:50.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:50.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:50.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:50.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:08:50.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:50.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:50.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:50.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:50.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:51.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:51.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:51 smithi086 ceph-mon[97913]: pgmap v1081: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.27k op/s; 20246/61128 objects degraded (33.121%); 0 B/s, 217 objects/s recovering 2024-06-14T23:08:51.078 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:51 smithi086 ceph-mon[102754]: pgmap v1081: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.27k op/s; 20246/61128 objects degraded (33.121%); 0 B/s, 217 objects/s recovering 2024-06-14T23:08:51.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:51.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:51.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:51.227 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:08:51.227 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:08:51.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:51.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:51.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:51.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:51.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:51.413 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:51 smithi137 ceph-mon[90634]: pgmap v1081: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 105 active+clean; 1.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.27k op/s; 20246/61128 objects degraded (33.121%); 0 B/s, 217 objects/s recovering 2024-06-14T23:08:51.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:51.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:51.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:51.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:51.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:51.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:51.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:51.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:51.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:51.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:51.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:51.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:52.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:52.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:52.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:52.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:52.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:52.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:52.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:52.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:52.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-14T23:08:52.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:52.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:52.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:52.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:52.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:52.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:52.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:52.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:52.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:52.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:52.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:52.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:53.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:53.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:53.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:53.135 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1079,"stamp":"2024-06-14T23:08:46.893451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1634685527,"num_objects":24958,"num_object_clones":0,"num_object_copies":74874,"num_objects_missing_on_primary":7337,"num_objects_missing":7337,"num_objects_degraded":20246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24958,"num_whiteouts":0,"num_read":67,"num_read_kb":125,"num_write":229798,"num_write_kb":8148712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4600,"num_bytes_recovered":832493,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218899,"ondisk_log_size":218899,"up":357,"acting":355,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12012352,"kb_used_data":6410368,"kb_used_omap":35,"kb_used_meta":5601884,"kb_avail":737948864,"statfs":{"total":767960285184,"available":755659636736,"internally_reserved":0,"allocated":6564216832,"data_stored":6552939558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36089,"internal_metadata":5736329991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,6,5,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-729481216,"num_objects":-11131,"num_object_clones":0,"num_object_copies":-33393,"num_objects_missing_on_primary":-1729,"num_objects_missing":-1729,"num_objects_degraded":-1617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2139,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003105"},"pg_stats":[{"pgid":"1.11","version":"77'161","reported_seq":656,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.008387+0000","last_change":"2024-06-14T23:07:12.090120+0000","last_active":"2024-06-14T23:08:29.008387+0000","last_peered":"2024-06-14T23:08:29.008387+0000","last_clean":"2024-06-14T23:08:29.008387+0000","last_became_active":"2024-06-14T23:07:00.939305+0000","last_became_peered":"2024-06-14T23:07:00.939305+0000","last_unstale":"2024-06-14T23:08:29.008387+0000","last_undegraded":"2024-06-14T23:08:29.008387+0000","last_fullsized":"2024-06-14T23:08:29.008387+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:15:19.613254+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":214,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.015181+0000","last_change":"2024-06-14T22:58:09.757730+0000","last_active":"2024-06-14T23:08:29.015181+0000","last_peered":"2024-06-14T23:08:29.015181+0000","last_clean":"2024-06-14T23:08:29.015181+0000","last_became_active":"2024-06-14T22:58:09.757280+0000","last_became_peered":"2024-06-14T22:58:09.757280+0000","last_unstale":"2024-06-14T23:08:29.015181+0000","last_undegraded":"2024-06-14T23:08:29.015181+0000","last_fullsized":"2024-06-14T23:08:29.015181+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:37:34.446318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":276,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006956+0000","last_change":"2024-06-14T23:04:34.895560+0000","last_active":"2024-06-14T23:08:29.006956+0000","last_peered":"2024-06-14T23:08:29.006956+0000","last_clean":"2024-06-14T23:08:29.006956+0000","last_became_active":"2024-06-14T23:04:34.887055+0000","last_became_peered":"2024-06-14T23:04:34.887055+0000","last_unstale":"2024-06-14T23:08:29.006956+0000","last_undegraded":"2024-06-14T23:08:29.006956+0000","last_fullsized":"2024-06-14T23:08:29.006956+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:26:36.259241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":600,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.007811+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:08:29.007811+0000","last_peered":"2024-06-14T23:08:29.007811+0000","last_clean":"2024-06-14T23:08:29.007811+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:08:29.007811+0000","last_undegraded":"2024-06-14T23:08:29.007811+0000","last_fullsized":"2024-06-14T23:08:29.007811+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":241,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.010040+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:08:29.010040+0000","last_peered":"2024-06-14T23:08:29.010040+0000","last_clean":"2024-06-14T23:08:29.010040+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:08:29.010040+0000","last_undegraded":"2024-06-14T23:08:29.010040+0000","last_fullsized":"2024-06-14T23:08:29.010040+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":236,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.008446+0000","last_change":"2024-06-14T22:54:06.277540+0000","last_active":"2024-06-14T23:08:29.008446+0000","last_peered":"2024-06-14T23:08:29.008446+0000","last_clean":"2024-06-14T23:08:29.008446+0000","last_became_active":"2024-06-14T22:54:06.277191+0000","last_became_peered":"2024-06-14T22:54:06.277191+0000","last_unstale":"2024-06-14T23:08:29.008446+0000","last_undegraded":"2024-06-14T23:08:29.008446+0000","last_fullsized":"2024-06-14T23:08:29.008446+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:20.335321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.19","version":"281'13622","reported_seq":13705,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:45.021277+0000","last_change":"2024-06-14T23:08:28.123567+0000","last_active":"2024-06-14T23:08:45.021277+0000","last_peered":"2024-06-14T23:08:45.021277+0000","last_clean":"2024-06-14T23:06:59.934530+0000","last_became_active":"2024-06-14T23:08:28.064979+0000","last_became_peered":"2024-06-14T23:08:28.064979+0000","last_unstale":"2024-06-14T23:08:45.021277+0000","last_undegraded":"2024-06-14T23:08:28.051223+0000","last_fullsized":"2024-06-14T23:08:45.021277+0000","mapping_epoch":278,"log_start":"255'6670","ondisk_log_start":"255'6670","created":278,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6952,"log_dups_size":0,"ondisk_log_size":6952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52723712,"num_objects":805,"num_object_clones":0,"num_object_copies":2415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7076,"num_write_kb":252160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3,6","objects":805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":620,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.015729+0000","last_change":"2024-06-14T23:07:09.705052+0000","last_active":"2024-06-14T23:08:29.015729+0000","last_peered":"2024-06-14T23:08:29.015729+0000","last_clean":"2024-06-14T23:08:29.015729+0000","last_became_active":"2024-06-14T23:07:00.938267+0000","last_became_peered":"2024-06-14T23:07:00.938267+0000","last_unstale":"2024-06-14T23:08:29.015729+0000","last_undegraded":"2024-06-14T23:08:29.015729+0000","last_fullsized":"2024-06-14T23:08:29.015729+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.704172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.704172+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:44:55.998829+0000","scrub_duration":0.0024350880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":119,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:39.971568+0000","last_change":"2024-06-14T23:06:18.999778+0000","last_active":"2024-06-14T23:08:39.971568+0000","last_peered":"2024-06-14T23:08:39.971568+0000","last_clean":"2024-06-14T23:08:39.971568+0000","last_became_active":"2024-06-14T23:06:18.999306+0000","last_became_peered":"2024-06-14T23:06:18.999306+0000","last_unstale":"2024-06-14T23:08:39.971568+0000","last_undegraded":"2024-06-14T23:08:39.971568+0000","last_fullsized":"2024-06-14T23:08:39.971568+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:18:01.226932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":165,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003253+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:08:29.003253+0000","last_peered":"2024-06-14T23:08:29.003253+0000","last_clean":"2024-06-14T23:08:29.003253+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:08:29.003253+0000","last_undegraded":"2024-06-14T23:08:29.003253+0000","last_fullsized":"2024-06-14T23:08:29.003253+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.18","version":"281'13628","reported_seq":15760,"reported_epoch":281,"state":"active+recovering","last_fresh":"2024-06-14T23:08:46.243378+0000","last_change":"2024-06-14T23:08:40.179861+0000","last_active":"2024-06-14T23:08:46.243378+0000","last_peered":"2024-06-14T23:08:46.243378+0000","last_clean":"2024-06-14T23:06:59.877480+0000","last_became_active":"2024-06-14T23:08:29.015052+0000","last_became_peered":"2024-06-14T23:08:29.015052+0000","last_unstale":"2024-06-14T23:08:46.243378+0000","last_undegraded":"2024-06-14T23:08:46.243378+0000","last_fullsized":"2024-06-14T23:08:46.243378+0000","mapping_epoch":278,"log_start":"255'6596","ondisk_log_start":"255'6596","created":278,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7032,"log_dups_size":0,"ondisk_log_size":7032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47480832,"num_objects":725,"num_object_clones":0,"num_object_copies":2175,"num_objects_missing_on_primary":590,"num_objects_missing":590,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7109,"num_write_kb":250656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1029,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5,6","objects":725}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":616,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.007891+0000","last_change":"2024-06-14T23:07:59.023708+0000","last_active":"2024-06-14T23:08:29.007891+0000","last_peered":"2024-06-14T23:08:29.007891+0000","last_clean":"2024-06-14T23:08:29.007891+0000","last_became_active":"2024-06-14T23:07:59.023129+0000","last_became_peered":"2024-06-14T23:07:59.023129+0000","last_unstale":"2024-06-14T23:08:29.007891+0000","last_undegraded":"2024-06-14T23:08:29.007891+0000","last_fullsized":"2024-06-14T23:08:29.007891+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:45:29.886582+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":238,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:35.136724+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:08:35.136724+0000","last_peered":"2024-06-14T23:08:35.136724+0000","last_clean":"2024-06-14T23:08:35.136724+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:08:35.136724+0000","last_undegraded":"2024-06-14T23:08:35.136724+0000","last_fullsized":"2024-06-14T23:08:35.136724+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":247,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003364+0000","last_change":"2024-06-14T23:07:58.008693+0000","last_active":"2024-06-14T23:08:29.003364+0000","last_peered":"2024-06-14T23:08:29.003364+0000","last_clean":"2024-06-14T23:08:29.003364+0000","last_became_active":"2024-06-14T23:07:58.008150+0000","last_became_peered":"2024-06-14T23:07:58.008150+0000","last_unstale":"2024-06-14T23:08:29.003364+0000","last_undegraded":"2024-06-14T23:08:29.003364+0000","last_fullsized":"2024-06-14T23:08:29.003364+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:38:32.621234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":657,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003663+0000","last_change":"2024-06-14T23:07:21.981837+0000","last_active":"2024-06-14T23:08:29.003663+0000","last_peered":"2024-06-14T23:08:29.003663+0000","last_clean":"2024-06-14T23:08:29.003663+0000","last_became_active":"2024-06-14T23:07:00.947390+0000","last_became_peered":"2024-06-14T23:07:00.947390+0000","last_unstale":"2024-06-14T23:08:29.003663+0000","last_undegraded":"2024-06-14T23:08:29.003663+0000","last_fullsized":"2024-06-14T23:08:29.003663+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:21.981787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:21.981787+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:47:43.919655+0000","scrub_duration":0.0025287560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"281'13842","reported_seq":13904,"reported_epoch":281,"state":"active+recovery_wait","last_fresh":"2024-06-14T23:08:44.606088+0000","last_change":"2024-06-14T23:08:29.009400+0000","last_active":"2024-06-14T23:08:44.606088+0000","last_peered":"2024-06-14T23:08:44.606088+0000","last_clean":"2024-06-14T23:08:26.990904+0000","last_became_active":"2024-06-14T23:08:28.063933+0000","last_became_peered":"2024-06-14T23:08:28.063933+0000","last_unstale":"2024-06-14T23:08:44.606088+0000","last_undegraded":"2024-06-14T23:08:44.606088+0000","last_fullsized":"2024-06-14T23:08:44.606088+0000","mapping_epoch":237,"log_start":"248'4720","ondisk_log_start":"248'4720","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":9122,"log_dups_size":0,"ondisk_log_size":9122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53379072,"num_objects":814,"num_object_clones":0,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":255712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":814}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":231,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006049+0000","last_change":"2024-06-14T22:58:32.753082+0000","last_active":"2024-06-14T23:08:29.006049+0000","last_peered":"2024-06-14T23:08:29.006049+0000","last_clean":"2024-06-14T23:08:29.006049+0000","last_became_active":"2024-06-14T22:58:32.752695+0000","last_became_peered":"2024-06-14T22:58:32.752695+0000","last_unstale":"2024-06-14T23:08:29.006049+0000","last_undegraded":"2024-06-14T23:08:29.006049+0000","last_fullsized":"2024-06-14T23:08:29.006049+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:10:00.159170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":241,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.008288+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:08:29.008288+0000","last_peered":"2024-06-14T23:08:29.008288+0000","last_clean":"2024-06-14T23:08:29.008288+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:08:29.008288+0000","last_undegraded":"2024-06-14T23:08:29.008288+0000","last_fullsized":"2024-06-14T23:08:29.008288+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":643,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006421+0000","last_change":"2024-06-14T23:07:59.028296+0000","last_active":"2024-06-14T23:08:29.006421+0000","last_peered":"2024-06-14T23:08:29.006421+0000","last_clean":"2024-06-14T23:08:29.006421+0000","last_became_active":"2024-06-14T23:07:59.003433+0000","last_became_peered":"2024-06-14T23:07:59.003433+0000","last_unstale":"2024-06-14T23:08:29.006421+0000","last_undegraded":"2024-06-14T23:08:29.006421+0000","last_fullsized":"2024-06-14T23:08:29.006421+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.188207+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:21:33.569950+0000","scrub_duration":0.019682448000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"281'13852","reported_seq":13929,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:45.079425+0000","last_change":"2024-06-14T23:08:28.083896+0000","last_active":"2024-06-14T23:08:45.079425+0000","last_peered":"2024-06-14T23:08:45.079425+0000","last_clean":"2024-06-14T23:08:45.079425+0000","last_became_active":"2024-06-14T23:08:28.051610+0000","last_became_peered":"2024-06-14T23:08:28.051610+0000","last_unstale":"2024-06-14T23:08:45.079425+0000","last_undegraded":"2024-06-14T23:08:45.079425+0000","last_fullsized":"2024-06-14T23:08:45.079425+0000","mapping_epoch":253,"log_start":"244'3200","ondisk_log_start":"244'3200","created":237,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":10652,"log_dups_size":0,"ondisk_log_size":10652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52428800,"num_objects":800,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7202,"num_write_kb":256064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":32768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":232,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.007547+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:08:29.007547+0000","last_peered":"2024-06-14T23:08:29.007547+0000","last_clean":"2024-06-14T23:08:29.007547+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:08:29.007547+0000","last_undegraded":"2024-06-14T23:08:29.007547+0000","last_fullsized":"2024-06-14T23:08:29.007547+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":258,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003407+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:08:29.003407+0000","last_peered":"2024-06-14T23:08:29.003407+0000","last_clean":"2024-06-14T23:08:29.003407+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:08:29.003407+0000","last_undegraded":"2024-06-14T23:08:29.003407+0000","last_fullsized":"2024-06-14T23:08:29.003407+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":550,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006000+0000","last_change":"2024-06-14T23:07:59.003744+0000","last_active":"2024-06-14T23:08:29.006000+0000","last_peered":"2024-06-14T23:08:29.006000+0000","last_clean":"2024-06-14T23:08:29.006000+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:08:29.006000+0000","last_undegraded":"2024-06-14T23:08:29.006000+0000","last_fullsized":"2024-06-14T23:08:29.006000+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:17.057842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:17.057842+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":42,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"281'13875","reported_seq":13955,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:44.532161+0000","last_change":"2024-06-14T23:08:29.043322+0000","last_active":"2024-06-14T23:08:44.532161+0000","last_peered":"2024-06-14T23:08:44.532161+0000","last_clean":"2024-06-14T23:06:59.932215+0000","last_became_active":"2024-06-14T23:08:29.016536+0000","last_became_peered":"2024-06-14T23:08:29.016536+0000","last_unstale":"2024-06-14T23:08:44.532161+0000","last_undegraded":"2024-06-14T23:08:29.009290+0000","last_fullsized":"2024-06-14T23:08:44.532161+0000","mapping_epoch":279,"log_start":"248'4731","ondisk_log_start":"248'4731","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":9144,"log_dups_size":0,"ondisk_log_size":9144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53575680,"num_objects":817,"num_object_clones":0,"num_object_copies":2451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7199,"num_write_kb":256544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,6,7","objects":817}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":176,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.009646+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:08:29.009646+0000","last_peered":"2024-06-14T23:08:29.009646+0000","last_clean":"2024-06-14T23:08:29.009646+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:08:29.009646+0000","last_undegraded":"2024-06-14T23:08:29.009646+0000","last_fullsized":"2024-06-14T23:08:29.009646+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":222,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.004122+0000","last_change":"2024-06-14T23:07:58.028433+0000","last_active":"2024-06-14T23:08:29.004122+0000","last_peered":"2024-06-14T23:08:29.004122+0000","last_clean":"2024-06-14T23:08:29.004122+0000","last_became_active":"2024-06-14T23:07:58.023997+0000","last_became_peered":"2024-06-14T23:07:58.023997+0000","last_unstale":"2024-06-14T23:08:29.004122+0000","last_undegraded":"2024-06-14T23:08:29.004122+0000","last_fullsized":"2024-06-14T23:08:29.004122+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:36:30.743683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":499,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003975+0000","last_change":"2024-06-14T23:07:22.996051+0000","last_active":"2024-06-14T23:08:29.003975+0000","last_peered":"2024-06-14T23:08:29.003975+0000","last_clean":"2024-06-14T23:08:29.003975+0000","last_became_active":"2024-06-14T23:07:00.943273+0000","last_became_peered":"2024-06-14T23:07:00.943273+0000","last_unstale":"2024-06-14T23:08:29.003975+0000","last_undegraded":"2024-06-14T23:08:29.003975+0000","last_fullsized":"2024-06-14T23:08:29.003975+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:22.995968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:22.995968+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:08:40.160818+0000","scrub_duration":0.0055045409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"281'13962","reported_seq":14059,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:44.179343+0000","last_change":"2024-06-14T23:08:29.036301+0000","last_active":"2024-06-14T23:08:44.179343+0000","last_peered":"2024-06-14T23:08:44.179343+0000","last_clean":"2024-06-14T23:06:59.927883+0000","last_became_active":"2024-06-14T23:08:29.014052+0000","last_became_peered":"2024-06-14T23:08:29.014052+0000","last_unstale":"2024-06-14T23:08:44.179343+0000","last_undegraded":"2024-06-14T23:08:29.009046+0000","last_fullsized":"2024-06-14T23:08:44.179343+0000","mapping_epoch":279,"log_start":"254'6624","ondisk_log_start":"254'6624","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54493184,"num_objects":831,"num_object_clones":0,"num_object_copies":2493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7239,"num_write_kb":258272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,6,7","objects":831}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":259,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.009128+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:08:29.009128+0000","last_peered":"2024-06-14T23:08:29.009128+0000","last_clean":"2024-06-14T23:08:29.009128+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:08:29.009128+0000","last_undegraded":"2024-06-14T23:08:29.009128+0000","last_fullsized":"2024-06-14T23:08:29.009128+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":96,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:35.513735+0000","last_change":"2024-06-14T23:06:19.002672+0000","last_active":"2024-06-14T23:08:35.513735+0000","last_peered":"2024-06-14T23:08:35.513735+0000","last_clean":"2024-06-14T23:08:35.513735+0000","last_became_active":"2024-06-14T23:06:19.002175+0000","last_became_peered":"2024-06-14T23:06:19.002175+0000","last_unstale":"2024-06-14T23:08:35.513735+0000","last_undegraded":"2024-06-14T23:08:35.513735+0000","last_fullsized":"2024-06-14T23:08:35.513735+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:50.841812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"281'13729","reported_seq":13799,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:46.210784+0000","last_change":"2024-06-14T23:08:29.074517+0000","last_active":"2024-06-14T23:08:46.210784+0000","last_peered":"2024-06-14T23:08:46.210784+0000","last_clean":"2024-06-14T23:06:59.881114+0000","last_became_active":"2024-06-14T23:08:29.021153+0000","last_became_peered":"2024-06-14T23:08:29.021153+0000","last_unstale":"2024-06-14T23:08:46.210784+0000","last_undegraded":"2024-06-14T23:08:29.011082+0000","last_fullsized":"2024-06-14T23:08:46.210784+0000","mapping_epoch":279,"log_start":"254'6428","ondisk_log_start":"254'6428","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7301,"log_dups_size":0,"ondisk_log_size":7301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47841280,"num_objects":730,"num_object_clones":0,"num_object_copies":2190,"num_objects_missing_on_primary":1660,"num_objects_missing":1660,"num_objects_degraded":1660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7138,"num_write_kb":251776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,6,7","objects":730}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":534,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.011685+0000","last_change":"2024-06-14T23:04:40.403447+0000","last_active":"2024-06-14T23:08:29.011685+0000","last_peered":"2024-06-14T23:08:29.011685+0000","last_clean":"2024-06-14T23:08:29.011685+0000","last_became_active":"2024-06-14T23:04:37.904456+0000","last_became_peered":"2024-06-14T23:04:37.904456+0000","last_unstale":"2024-06-14T23:08:29.011685+0000","last_undegraded":"2024-06-14T23:08:29.011685+0000","last_fullsized":"2024-06-14T23:08:29.011685+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:04:40.403365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:04:40.403365+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:45.174359+0000","scrub_duration":0.0014330689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":51,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003402+0000","last_change":"2024-06-14T23:06:19.002737+0000","last_active":"2024-06-14T23:08:29.003402+0000","last_peered":"2024-06-14T23:08:29.003402+0000","last_clean":"2024-06-14T23:08:29.003402+0000","last_became_active":"2024-06-14T23:06:19.002078+0000","last_became_peered":"2024-06-14T23:06:19.002078+0000","last_unstale":"2024-06-14T23:08:29.003402+0000","last_undegraded":"2024-06-14T23:08:29.003402+0000","last_fullsized":"2024-06-14T23:08:29.003402+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:00:16.317592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":258,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003438+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:08:29.003438+0000","last_peered":"2024-06-14T23:08:29.003438+0000","last_clean":"2024-06-14T23:08:29.003438+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:08:29.003438+0000","last_undegraded":"2024-06-14T23:08:29.003438+0000","last_fullsized":"2024-06-14T23:08:29.003438+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"281'13558","reported_seq":13621,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:45.134579+0000","last_change":"2024-06-14T23:08:28.092688+0000","last_active":"2024-06-14T23:08:45.134579+0000","last_peered":"2024-06-14T23:08:45.134579+0000","last_clean":"2024-06-14T23:08:45.134579+0000","last_became_active":"2024-06-14T23:08:28.067946+0000","last_became_peered":"2024-06-14T23:08:28.067946+0000","last_unstale":"2024-06-14T23:08:45.134579+0000","last_undegraded":"2024-06-14T23:08:45.134579+0000","last_fullsized":"2024-06-14T23:08:45.134579+0000","mapping_epoch":237,"log_start":"244'3000","ondisk_log_start":"244'3000","created":237,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":10558,"log_dups_size":0,"ondisk_log_size":10558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52330496,"num_objects":798,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7045,"num_write_kb":251008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":626,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.008064+0000","last_change":"2024-06-14T23:04:37.902544+0000","last_active":"2024-06-14T23:08:29.008064+0000","last_peered":"2024-06-14T23:08:29.008064+0000","last_clean":"2024-06-14T23:08:29.008064+0000","last_became_active":"2024-06-14T23:04:37.902191+0000","last_became_peered":"2024-06-14T23:04:37.902191+0000","last_unstale":"2024-06-14T23:08:29.008064+0000","last_undegraded":"2024-06-14T23:08:29.008064+0000","last_fullsized":"2024-06-14T23:08:29.008064+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:33.815704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:33.815704+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:05:09.672062+0000","scrub_duration":0.001826647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":139,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003699+0000","last_change":"2024-06-14T23:07:58.023693+0000","last_active":"2024-06-14T23:08:29.003699+0000","last_peered":"2024-06-14T23:08:29.003699+0000","last_clean":"2024-06-14T23:08:29.003699+0000","last_became_active":"2024-06-14T23:07:58.023441+0000","last_became_peered":"2024-06-14T23:07:58.023441+0000","last_unstale":"2024-06-14T23:08:29.003699+0000","last_undegraded":"2024-06-14T23:08:29.003699+0000","last_fullsized":"2024-06-14T23:08:29.003699+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:08:47.037322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":271,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.004004+0000","last_change":"2024-06-14T23:04:34.887791+0000","last_active":"2024-06-14T23:08:29.004004+0000","last_peered":"2024-06-14T23:08:29.004004+0000","last_clean":"2024-06-14T23:08:29.004004+0000","last_became_active":"2024-06-14T23:04:34.887099+0000","last_became_peered":"2024-06-14T23:04:34.887099+0000","last_unstale":"2024-06-14T23:08:29.004004+0000","last_undegraded":"2024-06-14T23:08:29.004004+0000","last_fullsized":"2024-06-14T23:08:29.004004+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:46:07.470301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":620,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.015811+0000","last_change":"2024-06-14T23:07:13.730354+0000","last_active":"2024-06-14T23:08:29.015811+0000","last_peered":"2024-06-14T23:08:29.015811+0000","last_clean":"2024-06-14T23:08:29.015811+0000","last_became_active":"2024-06-14T23:07:00.933661+0000","last_became_peered":"2024-06-14T23:07:00.933661+0000","last_unstale":"2024-06-14T23:08:29.015811+0000","last_undegraded":"2024-06-14T23:08:29.015811+0000","last_fullsized":"2024-06-14T23:08:29.015811+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:13.729891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:13.729891+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:56:55.375260+0000","scrub_duration":0.015960974999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"281'14140","reported_seq":14207,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:46.212502+0000","last_change":"2024-06-14T23:08:29.236933+0000","last_active":"2024-06-14T23:08:46.212502+0000","last_peered":"2024-06-14T23:08:46.212502+0000","last_clean":"2024-06-14T23:06:59.893668+0000","last_became_active":"2024-06-14T23:08:29.192732+0000","last_became_peered":"2024-06-14T23:08:29.192732+0000","last_unstale":"2024-06-14T23:08:46.212502+0000","last_undegraded":"2024-06-14T23:08:29.013553+0000","last_fullsized":"2024-06-14T23:08:46.212502+0000","mapping_epoch":279,"log_start":"244'2519","ondisk_log_start":"244'2519","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":11621,"log_dups_size":0,"ondisk_log_size":11621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50298880,"num_objects":767,"num_object_clones":0,"num_object_copies":2301,"num_objects_missing_on_primary":1724,"num_objects_missing":1724,"num_objects_degraded":1724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7365,"num_write_kb":260256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3,6","objects":767}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":262,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003491+0000","last_change":"2024-06-14T23:04:34.896257+0000","last_active":"2024-06-14T23:08:29.003491+0000","last_peered":"2024-06-14T23:08:29.003491+0000","last_clean":"2024-06-14T23:08:29.003491+0000","last_became_active":"2024-06-14T23:04:34.895926+0000","last_became_peered":"2024-06-14T23:04:34.895926+0000","last_unstale":"2024-06-14T23:08:29.003491+0000","last_undegraded":"2024-06-14T23:08:29.003491+0000","last_fullsized":"2024-06-14T23:08:29.003491+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:17.800902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":258,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.005439+0000","last_change":"2024-06-14T23:07:58.023766+0000","last_active":"2024-06-14T23:08:29.005439+0000","last_peered":"2024-06-14T23:08:29.005439+0000","last_clean":"2024-06-14T23:08:29.005439+0000","last_became_active":"2024-06-14T23:07:58.023506+0000","last_became_peered":"2024-06-14T23:07:58.023506+0000","last_unstale":"2024-06-14T23:08:29.005439+0000","last_undegraded":"2024-06-14T23:08:29.005439+0000","last_fullsized":"2024-06-14T23:08:29.005439+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:38.473906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"281'13519","reported_seq":13583,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:45.469089+0000","last_change":"2024-06-14T23:08:28.067952+0000","last_active":"2024-06-14T23:08:45.469089+0000","last_peered":"2024-06-14T23:08:45.469089+0000","last_clean":"2024-06-14T23:08:45.469089+0000","last_became_active":"2024-06-14T23:08:28.034856+0000","last_became_peered":"2024-06-14T23:08:28.034856+0000","last_unstale":"2024-06-14T23:08:45.469089+0000","last_undegraded":"2024-06-14T23:08:45.469089+0000","last_fullsized":"2024-06-14T23:08:45.469089+0000","mapping_epoch":237,"log_start":"254'6203","ondisk_log_start":"254'6203","created":237,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49446912,"num_objects":754,"num_object_clones":0,"num_object_copies":2262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":249632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":635,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.007807+0000","last_change":"2024-06-14T23:07:58.017607+0000","last_active":"2024-06-14T23:08:29.007807+0000","last_peered":"2024-06-14T23:08:29.007807+0000","last_clean":"2024-06-14T23:08:29.007807+0000","last_became_active":"2024-06-14T23:07:58.015201+0000","last_became_peered":"2024-06-14T23:07:58.015201+0000","last_unstale":"2024-06-14T23:08:29.007807+0000","last_undegraded":"2024-06-14T23:08:29.007807+0000","last_fullsized":"2024-06-14T23:08:29.007807+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:37:24.915537+0000","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":239,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006246+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:08:29.006246+0000","last_peered":"2024-06-14T23:08:29.006246+0000","last_clean":"2024-06-14T23:08:29.006246+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:08:29.006246+0000","last_undegraded":"2024-06-14T23:08:29.006246+0000","last_fullsized":"2024-06-14T23:08:29.006246+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":252,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.015683+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:08:29.015683+0000","last_peered":"2024-06-14T23:08:29.015683+0000","last_clean":"2024-06-14T23:08:29.015683+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:08:29.015683+0000","last_undegraded":"2024-06-14T23:08:29.015683+0000","last_fullsized":"2024-06-14T23:08:29.015683+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"281'14806","reported_seq":14876,"reported_epoch":281,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:08:44.601432+0000","last_change":"2024-06-14T23:07:59.254316+0000","last_active":"2024-06-14T23:08:44.601432+0000","last_peered":"2024-06-14T23:08:44.601432+0000","last_clean":"2024-06-14T23:06:59.923360+0000","last_became_active":"2024-06-14T23:07:59.217456+0000","last_became_peered":"2024-06-14T23:07:59.217456+0000","last_unstale":"2024-06-14T23:08:44.601432+0000","last_undegraded":"2024-06-14T23:07:59.022562+0000","last_fullsized":"2024-06-14T23:07:58.996112+0000","mapping_epoch":273,"log_start":"255'6863","ondisk_log_start":"255'6863","created":237,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:38.233859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":1600,"num_object_clones":0,"num_object_copies":4800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14806,"num_write_kb":524992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1600}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":561,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.007185+0000","last_change":"2024-06-14T23:07:58.009471+0000","last_active":"2024-06-14T23:08:29.007185+0000","last_peered":"2024-06-14T23:08:29.007185+0000","last_clean":"2024-06-14T23:08:29.007185+0000","last_became_active":"2024-06-14T23:07:58.009127+0000","last_became_peered":"2024-06-14T23:07:58.009127+0000","last_unstale":"2024-06-14T23:08:29.007185+0000","last_undegraded":"2024-06-14T23:08:29.007185+0000","last_fullsized":"2024-06-14T23:08:29.007185+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:06:21.536637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:06:21.536637+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035987950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":266,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.008874+0000","last_change":"2024-06-14T23:04:34.889456+0000","last_active":"2024-06-14T23:08:29.008874+0000","last_peered":"2024-06-14T23:08:29.008874+0000","last_clean":"2024-06-14T23:08:29.008874+0000","last_became_active":"2024-06-14T23:04:34.888908+0000","last_became_peered":"2024-06-14T23:04:34.888908+0000","last_unstale":"2024-06-14T23:08:29.008874+0000","last_undegraded":"2024-06-14T23:08:29.008874+0000","last_fullsized":"2024-06-14T23:08:29.008874+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:00:24.632734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":202,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.009534+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:08:29.009534+0000","last_peered":"2024-06-14T23:08:29.009534+0000","last_clean":"2024-06-14T23:08:29.009534+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:08:29.009534+0000","last_undegraded":"2024-06-14T23:08:29.009534+0000","last_fullsized":"2024-06-14T23:08:29.009534+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":196,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006713+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:08:29.006713+0000","last_peered":"2024-06-14T23:08:29.006713+0000","last_clean":"2024-06-14T23:08:29.006713+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:08:29.006713+0000","last_undegraded":"2024-06-14T23:08:29.006713+0000","last_fullsized":"2024-06-14T23:08:29.006713+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"281'13699","reported_seq":13783,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:45.034313+0000","last_change":"2024-06-14T23:08:29.204894+0000","last_active":"2024-06-14T23:08:45.034313+0000","last_peered":"2024-06-14T23:08:45.034313+0000","last_clean":"2024-06-14T23:06:59.934530+0000","last_became_active":"2024-06-14T23:08:29.192760+0000","last_became_peered":"2024-06-14T23:08:29.192760+0000","last_unstale":"2024-06-14T23:08:45.034313+0000","last_undegraded":"2024-06-14T23:08:29.007100+0000","last_fullsized":"2024-06-14T23:08:45.034313+0000","mapping_epoch":279,"log_start":"255'6670","ondisk_log_start":"255'6670","created":237,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47677440,"num_objects":727,"num_object_clones":0,"num_object_copies":2181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7152,"num_write_kb":252160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3,6","objects":727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":607,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:39.967630+0000","last_change":"2024-06-14T23:07:58.009966+0000","last_active":"2024-06-14T23:08:39.967630+0000","last_peered":"2024-06-14T23:08:39.967630+0000","last_clean":"2024-06-14T23:08:39.967630+0000","last_became_active":"2024-06-14T23:07:58.009493+0000","last_became_peered":"2024-06-14T23:07:58.009493+0000","last_unstale":"2024-06-14T23:08:39.967630+0000","last_undegraded":"2024-06-14T23:08:39.967630+0000","last_fullsized":"2024-06-14T23:08:39.967630+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":243,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.014570+0000","last_change":"2024-06-14T22:55:53.664029+0000","last_active":"2024-06-14T23:08:29.014570+0000","last_peered":"2024-06-14T23:08:29.014570+0000","last_clean":"2024-06-14T23:08:29.014570+0000","last_became_active":"2024-06-14T22:55:53.663512+0000","last_became_peered":"2024-06-14T22:55:53.663512+0000","last_unstale":"2024-06-14T23:08:29.014570+0000","last_undegraded":"2024-06-14T23:08:29.014570+0000","last_fullsized":"2024-06-14T23:08:29.014570+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:37.504263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":119,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003161+0000","last_change":"2024-06-14T23:06:19.001589+0000","last_active":"2024-06-14T23:08:29.003161+0000","last_peered":"2024-06-14T23:08:29.003161+0000","last_clean":"2024-06-14T23:08:29.003161+0000","last_became_active":"2024-06-14T23:06:19.001260+0000","last_became_peered":"2024-06-14T23:06:19.001260+0000","last_unstale":"2024-06-14T23:08:29.003161+0000","last_undegraded":"2024-06-14T23:08:29.003161+0000","last_fullsized":"2024-06-14T23:08:29.003161+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:54:46.316751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":247,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.009214+0000","last_change":"2024-06-14T23:06:18.998828+0000","last_active":"2024-06-14T23:08:29.009214+0000","last_peered":"2024-06-14T23:08:29.009214+0000","last_clean":"2024-06-14T23:08:29.009214+0000","last_became_active":"2024-06-14T23:06:18.998461+0000","last_became_peered":"2024-06-14T23:06:18.998461+0000","last_unstale":"2024-06-14T23:08:29.009214+0000","last_undegraded":"2024-06-14T23:08:29.009214+0000","last_fullsized":"2024-06-14T23:08:29.009214+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:18:21.826501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"281'13610","reported_seq":13688,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:46.214550+0000","last_change":"2024-06-14T23:08:29.070148+0000","last_active":"2024-06-14T23:08:46.214550+0000","last_peered":"2024-06-14T23:08:46.214550+0000","last_clean":"2024-06-14T23:06:59.877480+0000","last_became_active":"2024-06-14T23:08:29.015703+0000","last_became_peered":"2024-06-14T23:08:29.015703+0000","last_unstale":"2024-06-14T23:08:46.214550+0000","last_undegraded":"2024-06-14T23:08:29.011409+0000","last_fullsized":"2024-06-14T23:08:46.214550+0000","mapping_epoch":279,"log_start":"255'6596","ondisk_log_start":"255'6596","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7014,"log_dups_size":0,"ondisk_log_size":7014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48660480,"num_objects":742,"num_object_clones":0,"num_object_copies":2226,"num_objects_missing_on_primary":1641,"num_objects_missing":1641,"num_objects_degraded":1641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":250656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5,6","objects":742}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":531,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003585+0000","last_change":"2024-06-14T23:07:10.877003+0000","last_active":"2024-06-14T23:08:29.003585+0000","last_peered":"2024-06-14T23:08:29.003585+0000","last_clean":"2024-06-14T23:08:29.003585+0000","last_became_active":"2024-06-14T23:06:52.047842+0000","last_became_peered":"2024-06-14T23:06:52.047842+0000","last_unstale":"2024-06-14T23:08:29.003585+0000","last_undegraded":"2024-06-14T23:08:29.003585+0000","last_fullsized":"2024-06-14T23:08:29.003585+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:09:38.064292+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":64,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:39.976672+0000","last_change":"2024-06-14T23:04:34.879251+0000","last_active":"2024-06-14T23:08:39.976672+0000","last_peered":"2024-06-14T23:08:39.976672+0000","last_clean":"2024-06-14T23:08:39.976672+0000","last_became_active":"2024-06-14T23:04:34.878470+0000","last_became_peered":"2024-06-14T23:04:34.878470+0000","last_unstale":"2024-06-14T23:08:39.976672+0000","last_undegraded":"2024-06-14T23:08:39.976672+0000","last_fullsized":"2024-06-14T23:08:39.976672+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:55.949794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":286,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.014922+0000","last_change":"2024-06-14T23:07:58.010456+0000","last_active":"2024-06-14T23:08:29.014922+0000","last_peered":"2024-06-14T23:08:29.014922+0000","last_clean":"2024-06-14T23:08:29.014922+0000","last_became_active":"2024-06-14T23:07:58.010157+0000","last_became_peered":"2024-06-14T23:07:58.010157+0000","last_unstale":"2024-06-14T23:08:29.014922+0000","last_undegraded":"2024-06-14T23:08:29.014922+0000","last_fullsized":"2024-06-14T23:08:29.014922+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:44:17.382159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":237,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.015546+0000","last_change":"2024-06-14T22:49:25.486323+0000","last_active":"2024-06-14T23:08:29.015546+0000","last_peered":"2024-06-14T23:08:29.015546+0000","last_clean":"2024-06-14T23:08:29.015546+0000","last_became_active":"2024-06-14T22:49:25.486019+0000","last_became_peered":"2024-06-14T22:49:25.486019+0000","last_unstale":"2024-06-14T23:08:29.015546+0000","last_undegraded":"2024-06-14T23:08:29.015546+0000","last_fullsized":"2024-06-14T23:08:29.015546+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:59:13.147368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"281'14477","reported_seq":14538,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:45.123503+0000","last_change":"2024-06-14T23:06:04.928960+0000","last_active":"2024-06-14T23:08:45.123503+0000","last_peered":"2024-06-14T23:08:45.123503+0000","last_clean":"2024-06-14T23:08:45.123503+0000","last_became_active":"2024-06-14T23:06:04.923775+0000","last_became_peered":"2024-06-14T23:06:04.923775+0000","last_unstale":"2024-06-14T23:08:45.123503+0000","last_undegraded":"2024-06-14T23:08:45.123503+0000","last_fullsized":"2024-06-14T23:08:45.123503+0000","mapping_epoch":250,"log_start":"264'8669","ondisk_log_start":"264'8669","created":237,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:10:46.076558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103088128,"num_objects":1573,"num_object_clones":0,"num_object_copies":4719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14477,"num_write_kb":513600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":637,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.008466+0000","last_change":"2024-06-14T23:07:11.050751+0000","last_active":"2024-06-14T23:08:29.008466+0000","last_peered":"2024-06-14T23:08:29.008466+0000","last_clean":"2024-06-14T23:08:29.008466+0000","last_became_active":"2024-06-14T23:06:52.051996+0000","last_became_peered":"2024-06-14T23:06:52.051996+0000","last_unstale":"2024-06-14T23:08:29.008466+0000","last_undegraded":"2024-06-14T23:08:29.008466+0000","last_fullsized":"2024-06-14T23:08:29.008466+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:52:38.070062+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":263,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.009608+0000","last_change":"2024-06-14T23:07:58.015646+0000","last_active":"2024-06-14T23:08:29.009608+0000","last_peered":"2024-06-14T23:08:29.009608+0000","last_clean":"2024-06-14T23:08:29.009608+0000","last_became_active":"2024-06-14T23:07:58.014865+0000","last_became_peered":"2024-06-14T23:07:58.014865+0000","last_unstale":"2024-06-14T23:08:29.009608+0000","last_undegraded":"2024-06-14T23:08:29.009608+0000","last_fullsized":"2024-06-14T23:08:29.009608+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:52:48.798683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":259,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003883+0000","last_change":"2024-06-14T23:07:58.030089+0000","last_active":"2024-06-14T23:08:29.003883+0000","last_peered":"2024-06-14T23:08:29.003883+0000","last_clean":"2024-06-14T23:08:29.003883+0000","last_became_active":"2024-06-14T23:07:58.029700+0000","last_became_peered":"2024-06-14T23:07:58.029700+0000","last_unstale":"2024-06-14T23:08:29.003883+0000","last_undegraded":"2024-06-14T23:08:29.003883+0000","last_fullsized":"2024-06-14T23:08:29.003883+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:40:38.204728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":286,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.015022+0000","last_change":"2024-06-14T23:04:34.880836+0000","last_active":"2024-06-14T23:08:29.015022+0000","last_peered":"2024-06-14T23:08:29.015022+0000","last_clean":"2024-06-14T23:08:29.015022+0000","last_became_active":"2024-06-14T23:04:34.880540+0000","last_became_peered":"2024-06-14T23:04:34.880540+0000","last_unstale":"2024-06-14T23:08:29.015022+0000","last_undegraded":"2024-06-14T23:08:29.015022+0000","last_fullsized":"2024-06-14T23:08:29.015022+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:40:06.981511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"281'14272","reported_seq":18041,"reported_epoch":281,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-14T23:08:45.147460+0000","last_change":"2024-06-14T23:07:59.283112+0000","last_active":"2024-06-14T23:08:45.147460+0000","last_peered":"2024-06-14T23:08:45.147460+0000","last_clean":"2024-06-14T23:06:59.906458+0000","last_became_active":"2024-06-14T23:07:59.205602+0000","last_became_peered":"2024-06-14T23:07:59.205602+0000","last_unstale":"2024-06-14T23:08:45.147460+0000","last_undegraded":"2024-06-14T23:08:45.147460+0000","last_fullsized":"2024-06-14T23:07:58.998628+0000","mapping_epoch":273,"log_start":"251'5177","ondisk_log_start":"251'5177","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":9095,"log_dups_size":0,"ondisk_log_size":9095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:13:39.377006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97452056,"num_objects":1488,"num_object_clones":0,"num_object_copies":4464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14272,"num_write_kb":504257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1849,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":1488}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":540,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.005906+0000","last_change":"2024-06-14T23:07:58.020956+0000","last_active":"2024-06-14T23:08:29.005906+0000","last_peered":"2024-06-14T23:08:29.005906+0000","last_clean":"2024-06-14T23:08:29.005906+0000","last_became_active":"2024-06-14T23:07:58.020638+0000","last_became_peered":"2024-06-14T23:07:58.020638+0000","last_unstale":"2024-06-14T23:08:29.005906+0000","last_undegraded":"2024-06-14T23:08:29.005906+0000","last_fullsized":"2024-06-14T23:08:29.005906+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":279,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.010180+0000","last_change":"2024-06-14T23:04:34.880181+0000","last_active":"2024-06-14T23:08:29.010180+0000","last_peered":"2024-06-14T23:08:29.010180+0000","last_clean":"2024-06-14T23:08:29.010180+0000","last_became_active":"2024-06-14T23:04:34.879842+0000","last_became_peered":"2024-06-14T23:04:34.879842+0000","last_unstale":"2024-06-14T23:08:29.010180+0000","last_undegraded":"2024-06-14T23:08:29.010180+0000","last_fullsized":"2024-06-14T23:08:29.010180+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:14:18.433225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":243,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.015129+0000","last_change":"2024-06-14T22:49:25.489181+0000","last_active":"2024-06-14T23:08:29.015129+0000","last_peered":"2024-06-14T23:08:29.015129+0000","last_clean":"2024-06-14T23:08:29.015129+0000","last_became_active":"2024-06-14T22:49:25.485702+0000","last_became_peered":"2024-06-14T22:49:25.485702+0000","last_unstale":"2024-06-14T23:08:29.015129+0000","last_undegraded":"2024-06-14T23:08:29.015129+0000","last_fullsized":"2024-06-14T23:08:29.015129+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:05:23.872054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":238,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.014688+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:08:29.014688+0000","last_peered":"2024-06-14T23:08:29.014688+0000","last_clean":"2024-06-14T23:08:29.014688+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:08:29.014688+0000","last_undegraded":"2024-06-14T23:08:29.014688+0000","last_fullsized":"2024-06-14T23:08:29.014688+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"281'14547","reported_seq":14601,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:45.505046+0000","last_change":"2024-06-14T23:05:03.621987+0000","last_active":"2024-06-14T23:08:45.505046+0000","last_peered":"2024-06-14T23:08:45.505046+0000","last_clean":"2024-06-14T23:08:45.505046+0000","last_became_active":"2024-06-14T23:05:03.621724+0000","last_became_peered":"2024-06-14T23:05:03.621724+0000","last_unstale":"2024-06-14T23:08:45.505046+0000","last_undegraded":"2024-06-14T23:08:45.505046+0000","last_fullsized":"2024-06-14T23:08:45.505046+0000","mapping_epoch":237,"log_start":"259'8379","ondisk_log_start":"259'8379","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6168,"log_dups_size":0,"ondisk_log_size":6168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:12.558177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100990976,"num_objects":1541,"num_object_clones":0,"num_object_copies":4623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14547,"num_write_kb":514816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":540,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.005866+0000","last_change":"2024-06-14T23:07:58.008816+0000","last_active":"2024-06-14T23:08:29.005866+0000","last_peered":"2024-06-14T23:08:29.005866+0000","last_clean":"2024-06-14T23:08:29.005866+0000","last_became_active":"2024-06-14T23:07:58.008508+0000","last_became_peered":"2024-06-14T23:07:58.008508+0000","last_unstale":"2024-06-14T23:08:29.005866+0000","last_undegraded":"2024-06-14T23:08:29.005866+0000","last_fullsized":"2024-06-14T23:08:29.005866+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.555271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.555271+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":271,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.015107+0000","last_change":"2024-06-14T23:04:34.880684+0000","last_active":"2024-06-14T23:08:29.015107+0000","last_peered":"2024-06-14T23:08:29.015107+0000","last_clean":"2024-06-14T23:08:29.015107+0000","last_became_active":"2024-06-14T23:04:34.880127+0000","last_became_peered":"2024-06-14T23:04:34.880127+0000","last_unstale":"2024-06-14T23:08:29.015107+0000","last_undegraded":"2024-06-14T23:08:29.015107+0000","last_fullsized":"2024-06-14T23:08:29.015107+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:53:52.966928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":246,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006666+0000","last_change":"2024-06-14T22:44:51.803053+0000","last_active":"2024-06-14T23:08:29.006666+0000","last_peered":"2024-06-14T23:08:29.006666+0000","last_clean":"2024-06-14T23:08:29.006666+0000","last_became_active":"2024-06-14T22:44:51.801838+0000","last_became_peered":"2024-06-14T22:44:51.801838+0000","last_unstale":"2024-06-14T23:08:29.006666+0000","last_undegraded":"2024-06-14T23:08:29.006666+0000","last_fullsized":"2024-06-14T23:08:29.006666+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:19:01.102066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":55,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.008806+0000","last_change":"2024-06-14T23:06:19.003881+0000","last_active":"2024-06-14T23:08:29.008806+0000","last_peered":"2024-06-14T23:08:29.008806+0000","last_clean":"2024-06-14T23:08:29.008806+0000","last_became_active":"2024-06-14T23:06:19.003589+0000","last_became_peered":"2024-06-14T23:06:19.003589+0000","last_unstale":"2024-06-14T23:08:29.008806+0000","last_undegraded":"2024-06-14T23:08:29.008806+0000","last_fullsized":"2024-06-14T23:08:29.008806+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:16:52.332299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"281'14264","reported_seq":14317,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:45.479972+0000","last_change":"2024-06-14T23:05:03.621656+0000","last_active":"2024-06-14T23:08:45.479972+0000","last_peered":"2024-06-14T23:08:45.479972+0000","last_clean":"2024-06-14T23:08:45.479972+0000","last_became_active":"2024-06-14T23:05:03.620834+0000","last_became_peered":"2024-06-14T23:05:03.620834+0000","last_unstale":"2024-06-14T23:08:45.479972+0000","last_undegraded":"2024-06-14T23:08:45.479972+0000","last_fullsized":"2024-06-14T23:08:45.479972+0000","mapping_epoch":237,"log_start":"259'8084","ondisk_log_start":"259'8084","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6180,"log_dups_size":0,"ondisk_log_size":6180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:59:26.118782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100925440,"num_objects":1540,"num_object_clones":0,"num_object_copies":4620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14264,"num_write_kb":505728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":655,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003801+0000","last_change":"2024-06-14T23:07:58.803271+0000","last_active":"2024-06-14T23:08:29.003801+0000","last_peered":"2024-06-14T23:08:29.003801+0000","last_clean":"2024-06-14T23:08:29.003801+0000","last_became_active":"2024-06-14T23:07:58.048267+0000","last_became_peered":"2024-06-14T23:07:58.048267+0000","last_unstale":"2024-06-14T23:08:29.003801+0000","last_undegraded":"2024-06-14T23:08:29.003801+0000","last_fullsized":"2024-06-14T23:08:29.003801+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:58.803217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:58.803217+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:57:20.167126+0000","scrub_duration":0.001285706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":173,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:30.069928+0000","last_change":"2024-06-14T23:06:19.000734+0000","last_active":"2024-06-14T23:08:30.069928+0000","last_peered":"2024-06-14T23:08:30.069928+0000","last_clean":"2024-06-14T23:08:30.069928+0000","last_became_active":"2024-06-14T23:06:19.000432+0000","last_became_peered":"2024-06-14T23:06:19.000432+0000","last_unstale":"2024-06-14T23:08:30.069928+0000","last_undegraded":"2024-06-14T23:08:30.069928+0000","last_fullsized":"2024-06-14T23:08:30.069928+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:48.268127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":251,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.015461+0000","last_change":"2024-06-14T23:07:58.011279+0000","last_active":"2024-06-14T23:08:29.015461+0000","last_peered":"2024-06-14T23:08:29.015461+0000","last_clean":"2024-06-14T23:08:29.015461+0000","last_became_active":"2024-06-14T23:07:58.009870+0000","last_became_peered":"2024-06-14T23:07:58.009870+0000","last_unstale":"2024-06-14T23:08:29.015461+0000","last_undegraded":"2024-06-14T23:08:29.015461+0000","last_fullsized":"2024-06-14T23:08:29.015461+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:59:55.353132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":174,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006093+0000","last_change":"2024-06-14T23:06:04.932485+0000","last_active":"2024-06-14T23:08:29.006093+0000","last_peered":"2024-06-14T23:08:29.006093+0000","last_clean":"2024-06-14T23:08:29.006093+0000","last_became_active":"2024-06-14T23:06:04.932166+0000","last_became_peered":"2024-06-14T23:06:04.932166+0000","last_unstale":"2024-06-14T23:08:29.006093+0000","last_undegraded":"2024-06-14T23:08:29.006093+0000","last_fullsized":"2024-06-14T23:08:29.006093+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:19:46.149456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"281'14006","reported_seq":14058,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:44.150778+0000","last_change":"2024-06-14T23:05:03.623137+0000","last_active":"2024-06-14T23:08:44.150778+0000","last_peered":"2024-06-14T23:08:44.150778+0000","last_clean":"2024-06-14T23:08:44.150778+0000","last_became_active":"2024-06-14T23:05:03.622833+0000","last_became_peered":"2024-06-14T23:05:03.622833+0000","last_unstale":"2024-06-14T23:08:44.150778+0000","last_undegraded":"2024-06-14T23:08:44.150778+0000","last_fullsized":"2024-06-14T23:08:44.150778+0000","mapping_epoch":237,"log_start":"256'7456","ondisk_log_start":"256'7456","created":237,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:53:45.610358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107741184,"num_objects":1644,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14006,"num_write_kb":500800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":604,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:39.971348+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:08:39.971348+0000","last_peered":"2024-06-14T23:08:39.971348+0000","last_clean":"2024-06-14T23:08:39.971348+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:08:39.971348+0000","last_undegraded":"2024-06-14T23:08:39.971348+0000","last_fullsized":"2024-06-14T23:08:39.971348+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":65,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:30.509069+0000","last_change":"2024-06-14T23:06:19.000517+0000","last_active":"2024-06-14T23:08:30.509069+0000","last_peered":"2024-06-14T23:08:30.509069+0000","last_clean":"2024-06-14T23:08:30.509069+0000","last_became_active":"2024-06-14T23:06:19.000141+0000","last_became_peered":"2024-06-14T23:06:19.000141+0000","last_unstale":"2024-06-14T23:08:30.509069+0000","last_undegraded":"2024-06-14T23:08:30.509069+0000","last_fullsized":"2024-06-14T23:08:30.509069+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:57:42.599655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":236,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.005773+0000","last_change":"2024-06-14T23:07:58.008658+0000","last_active":"2024-06-14T23:08:29.005773+0000","last_peered":"2024-06-14T23:08:29.005773+0000","last_clean":"2024-06-14T23:08:29.005773+0000","last_became_active":"2024-06-14T23:07:58.008400+0000","last_became_peered":"2024-06-14T23:07:58.008400+0000","last_unstale":"2024-06-14T23:08:29.005773+0000","last_undegraded":"2024-06-14T23:08:29.005773+0000","last_fullsized":"2024-06-14T23:08:29.005773+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:24:42.013441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":241,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003266+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:08:29.003266+0000","last_peered":"2024-06-14T23:08:29.003266+0000","last_clean":"2024-06-14T23:08:29.003266+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:08:29.003266+0000","last_undegraded":"2024-06-14T23:08:29.003266+0000","last_fullsized":"2024-06-14T23:08:29.003266+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":258,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003705+0000","last_change":"2024-06-14T23:07:58.030396+0000","last_active":"2024-06-14T23:08:29.003705+0000","last_peered":"2024-06-14T23:08:29.003705+0000","last_clean":"2024-06-14T23:08:29.003705+0000","last_became_active":"2024-06-14T23:07:58.030106+0000","last_became_peered":"2024-06-14T23:07:58.030106+0000","last_unstale":"2024-06-14T23:08:29.003705+0000","last_undegraded":"2024-06-14T23:08:29.003705+0000","last_fullsized":"2024-06-14T23:08:29.003705+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:31:52.585282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":264,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003531+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:08:29.003531+0000","last_peered":"2024-06-14T23:08:29.003531+0000","last_clean":"2024-06-14T23:08:29.003531+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:08:29.003531+0000","last_undegraded":"2024-06-14T23:08:29.003531+0000","last_fullsized":"2024-06-14T23:08:29.003531+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":285,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.014613+0000","last_change":"2024-06-14T23:07:58.010642+0000","last_active":"2024-06-14T23:08:29.014613+0000","last_peered":"2024-06-14T23:08:29.014613+0000","last_clean":"2024-06-14T23:08:29.014613+0000","last_became_active":"2024-06-14T23:07:58.010358+0000","last_became_peered":"2024-06-14T23:07:58.010358+0000","last_unstale":"2024-06-14T23:08:29.014613+0000","last_undegraded":"2024-06-14T23:08:29.014613+0000","last_fullsized":"2024-06-14T23:08:29.014613+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:42.212299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":163,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006750+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:08:29.006750+0000","last_peered":"2024-06-14T23:08:29.006750+0000","last_clean":"2024-06-14T23:08:29.006750+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:08:29.006750+0000","last_undegraded":"2024-06-14T23:08:29.006750+0000","last_fullsized":"2024-06-14T23:08:29.006750+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":612,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.011875+0000","last_change":"2024-06-14T23:07:09.356684+0000","last_active":"2024-06-14T23:08:29.011875+0000","last_peered":"2024-06-14T23:08:29.011875+0000","last_clean":"2024-06-14T23:08:29.011875+0000","last_became_active":"2024-06-14T23:07:00.942504+0000","last_became_peered":"2024-06-14T23:07:00.942504+0000","last_unstale":"2024-06-14T23:08:29.011875+0000","last_undegraded":"2024-06-14T23:08:29.011875+0000","last_fullsized":"2024-06-14T23:08:29.011875+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:18:54.289960+0000","scrub_duration":0.013653241999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":242,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.009276+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:08:29.009276+0000","last_peered":"2024-06-14T23:08:29.009276+0000","last_clean":"2024-06-14T23:08:29.009276+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:08:29.009276+0000","last_undegraded":"2024-06-14T23:08:29.009276+0000","last_fullsized":"2024-06-14T23:08:29.009276+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":145,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006520+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:08:29.006520+0000","last_peered":"2024-06-14T23:08:29.006520+0000","last_clean":"2024-06-14T23:08:29.006520+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:08:29.006520+0000","last_undegraded":"2024-06-14T23:08:29.006520+0000","last_fullsized":"2024-06-14T23:08:29.006520+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":118,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.009944+0000","last_change":"2024-06-14T23:06:19.000252+0000","last_active":"2024-06-14T23:08:29.009944+0000","last_peered":"2024-06-14T23:08:29.009944+0000","last_clean":"2024-06-14T23:08:29.009944+0000","last_became_active":"2024-06-14T23:06:18.999939+0000","last_became_peered":"2024-06-14T23:06:18.999939+0000","last_unstale":"2024-06-14T23:08:29.009944+0000","last_undegraded":"2024-06-14T23:08:29.009944+0000","last_fullsized":"2024-06-14T23:08:29.009944+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:50.509140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":253,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006061+0000","last_change":"2024-06-14T23:07:58.028129+0000","last_active":"2024-06-14T23:08:29.006061+0000","last_peered":"2024-06-14T23:08:29.006061+0000","last_clean":"2024-06-14T23:08:29.006061+0000","last_became_active":"2024-06-14T23:07:58.027376+0000","last_became_peered":"2024-06-14T23:07:58.027376+0000","last_unstale":"2024-06-14T23:08:29.006061+0000","last_undegraded":"2024-06-14T23:08:29.006061+0000","last_fullsized":"2024-06-14T23:08:29.006061+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:25:25.805140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.10","version":"281'13494","reported_seq":13558,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:45.516394+0000","last_change":"2024-06-14T23:08:28.089675+0000","last_active":"2024-06-14T23:08:45.516394+0000","last_peered":"2024-06-14T23:08:45.516394+0000","last_clean":"2024-06-14T23:08:45.516394+0000","last_became_active":"2024-06-14T23:08:28.063796+0000","last_became_peered":"2024-06-14T23:08:28.063796+0000","last_unstale":"2024-06-14T23:08:45.516394+0000","last_undegraded":"2024-06-14T23:08:45.516394+0000","last_fullsized":"2024-06-14T23:08:45.516394+0000","mapping_epoch":278,"log_start":"254'6203","ondisk_log_start":"254'6203","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7291,"log_dups_size":0,"ondisk_log_size":7291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51085312,"num_objects":780,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7022,"num_write_kb":249632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":222,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006076+0000","last_change":"2024-06-14T23:07:58.010437+0000","last_active":"2024-06-14T23:08:29.006076+0000","last_peered":"2024-06-14T23:08:29.006076+0000","last_clean":"2024-06-14T23:08:29.006076+0000","last_became_active":"2024-06-14T23:07:58.010143+0000","last_became_peered":"2024-06-14T23:07:58.010143+0000","last_unstale":"2024-06-14T23:08:29.006076+0000","last_undegraded":"2024-06-14T23:08:29.006076+0000","last_fullsized":"2024-06-14T23:08:29.006076+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:46:46.681752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":141,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:39.967621+0000","last_change":"2024-06-14T23:06:18.998767+0000","last_active":"2024-06-14T23:08:39.967621+0000","last_peered":"2024-06-14T23:08:39.967621+0000","last_clean":"2024-06-14T23:08:39.967621+0000","last_became_active":"2024-06-14T23:06:18.998481+0000","last_became_peered":"2024-06-14T23:06:18.998481+0000","last_unstale":"2024-06-14T23:08:39.967621+0000","last_undegraded":"2024-06-14T23:08:39.967621+0000","last_fullsized":"2024-06-14T23:08:39.967621+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:58:13.282478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"281'14153","reported_seq":17633,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:46.218276+0000","last_change":"2024-06-14T23:08:40.171703+0000","last_active":"2024-06-14T23:08:46.218276+0000","last_peered":"2024-06-14T23:08:46.218276+0000","last_clean":"2024-06-14T23:08:46.218276+0000","last_became_active":"2024-06-14T23:08:29.192753+0000","last_became_peered":"2024-06-14T23:08:29.192753+0000","last_unstale":"2024-06-14T23:08:46.218276+0000","last_undegraded":"2024-06-14T23:08:46.218276+0000","last_fullsized":"2024-06-14T23:08:46.218276+0000","mapping_epoch":278,"log_start":"244'3011","ondisk_log_start":"244'3011","created":278,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":11142,"log_dups_size":0,"ondisk_log_size":11142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49446912,"num_objects":755,"num_object_clones":0,"num_object_copies":2265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7379,"num_write_kb":260256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":120,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.006539+0000","last_change":"2024-06-14T23:06:18.999600+0000","last_active":"2024-06-14T23:08:29.006539+0000","last_peered":"2024-06-14T23:08:29.006539+0000","last_clean":"2024-06-14T23:08:29.006539+0000","last_became_active":"2024-06-14T23:06:18.999293+0000","last_became_peered":"2024-06-14T23:06:18.999293+0000","last_unstale":"2024-06-14T23:08:29.006539+0000","last_undegraded":"2024-06-14T23:08:29.006539+0000","last_fullsized":"2024-06-14T23:08:29.006539+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:28:12.373005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":252,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:39.967100+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:08:39.967100+0000","last_peered":"2024-06-14T23:08:39.967100+0000","last_clean":"2024-06-14T23:08:39.967100+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:08:39.967100+0000","last_undegraded":"2024-06-14T23:08:39.967100+0000","last_fullsized":"2024-06-14T23:08:39.967100+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"281'13582","reported_seq":13645,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:45.077436+0000","last_change":"2024-06-14T23:08:28.113357+0000","last_active":"2024-06-14T23:08:45.077436+0000","last_peered":"2024-06-14T23:08:45.077436+0000","last_clean":"2024-06-14T23:08:45.077436+0000","last_became_active":"2024-06-14T23:08:28.092875+0000","last_became_peered":"2024-06-14T23:08:28.092875+0000","last_unstale":"2024-06-14T23:08:45.077436+0000","last_undegraded":"2024-06-14T23:08:45.077436+0000","last_fullsized":"2024-06-14T23:08:45.077436+0000","mapping_epoch":278,"log_start":"244'3000","ondisk_log_start":"244'3000","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":10582,"log_dups_size":0,"ondisk_log_size":10582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50757632,"num_objects":775,"num_object_clones":0,"num_object_copies":2325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7070,"num_write_kb":251008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":47,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.011778+0000","last_change":"2024-06-14T23:05:43.964532+0000","last_active":"2024-06-14T23:08:29.011778+0000","last_peered":"2024-06-14T23:08:29.011778+0000","last_clean":"2024-06-14T23:08:29.011778+0000","last_became_active":"2024-06-14T23:05:43.964255+0000","last_became_peered":"2024-06-14T23:05:43.964255+0000","last_unstale":"2024-06-14T23:08:29.011778+0000","last_undegraded":"2024-06-14T23:08:29.011778+0000","last_fullsized":"2024-06-14T23:08:29.011778+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:06.098124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":280,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003842+0000","last_change":"2024-06-14T23:04:34.883863+0000","last_active":"2024-06-14T23:08:29.003842+0000","last_peered":"2024-06-14T23:08:29.003842+0000","last_clean":"2024-06-14T23:08:29.003842+0000","last_became_active":"2024-06-14T23:04:34.881668+0000","last_became_peered":"2024-06-14T23:04:34.881668+0000","last_unstale":"2024-06-14T23:08:29.003842+0000","last_undegraded":"2024-06-14T23:08:29.003842+0000","last_fullsized":"2024-06-14T23:08:29.003842+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:44:45.737416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.13","version":"281'13721","reported_seq":13790,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:46.211043+0000","last_change":"2024-06-14T23:08:29.073970+0000","last_active":"2024-06-14T23:08:46.211043+0000","last_peered":"2024-06-14T23:08:46.211043+0000","last_clean":"2024-06-14T23:06:59.881114+0000","last_became_active":"2024-06-14T23:08:29.021163+0000","last_became_peered":"2024-06-14T23:08:29.021163+0000","last_unstale":"2024-06-14T23:08:46.211043+0000","last_undegraded":"2024-06-14T23:08:29.012542+0000","last_fullsized":"2024-06-14T23:08:46.211043+0000","mapping_epoch":278,"log_start":"254'6428","ondisk_log_start":"254'6428","created":278,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7293,"log_dups_size":0,"ondisk_log_size":7293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48365568,"num_objects":738,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":1722,"num_objects_missing":1722,"num_objects_degraded":1722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":251776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,6,7","objects":738}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":63,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.012142+0000","last_change":"2024-06-14T23:04:34.879498+0000","last_active":"2024-06-14T23:08:29.012142+0000","last_peered":"2024-06-14T23:08:29.012142+0000","last_clean":"2024-06-14T23:08:29.012142+0000","last_became_active":"2024-06-14T23:04:34.878932+0000","last_became_peered":"2024-06-14T23:04:34.878932+0000","last_unstale":"2024-06-14T23:08:29.012142+0000","last_undegraded":"2024-06-14T23:08:29.012142+0000","last_fullsized":"2024-06-14T23:08:29.012142+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:56:50.201189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":54,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003570+0000","last_change":"2024-06-14T23:06:19.003121+0000","last_active":"2024-06-14T23:08:29.003570+0000","last_peered":"2024-06-14T23:08:29.003570+0000","last_clean":"2024-06-14T23:08:29.003570+0000","last_became_active":"2024-06-14T23:06:19.002731+0000","last_became_peered":"2024-06-14T23:06:19.002731+0000","last_unstale":"2024-06-14T23:08:29.003570+0000","last_undegraded":"2024-06-14T23:08:29.003570+0000","last_fullsized":"2024-06-14T23:08:29.003570+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:31:04.851974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.14","version":"281'13984","reported_seq":14078,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:44.145659+0000","last_change":"2024-06-14T23:08:28.081785+0000","last_active":"2024-06-14T23:08:44.145659+0000","last_peered":"2024-06-14T23:08:44.145659+0000","last_clean":"2024-06-14T23:06:59.927883+0000","last_became_active":"2024-06-14T23:08:28.059104+0000","last_became_peered":"2024-06-14T23:08:28.059104+0000","last_unstale":"2024-06-14T23:08:44.145659+0000","last_undegraded":"2024-06-14T23:08:28.041066+0000","last_fullsized":"2024-06-14T23:08:44.145659+0000","mapping_epoch":278,"log_start":"254'6624","ondisk_log_start":"254'6624","created":278,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53051392,"num_objects":810,"num_object_clones":0,"num_object_copies":2430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7262,"num_write_kb":258272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,6,7","objects":810}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":258,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003639+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:08:29.003639+0000","last_peered":"2024-06-14T23:08:29.003639+0000","last_clean":"2024-06-14T23:08:29.003639+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:08:29.003639+0000","last_undegraded":"2024-06-14T23:08:29.003639+0000","last_fullsized":"2024-06-14T23:08:29.003639+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":244,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.009063+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:08:29.009063+0000","last_peered":"2024-06-14T23:08:29.009063+0000","last_clean":"2024-06-14T23:08:29.009063+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:08:29.009063+0000","last_undegraded":"2024-06-14T23:08:29.009063+0000","last_fullsized":"2024-06-14T23:08:29.009063+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.15","version":"281'13897","reported_seq":13974,"reported_epoch":281,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:08:44.540601+0000","last_change":"2024-06-14T23:08:28.083780+0000","last_active":"2024-06-14T23:08:44.540601+0000","last_peered":"2024-06-14T23:08:44.540601+0000","last_clean":"2024-06-14T23:06:59.932215+0000","last_became_active":"2024-06-14T23:08:28.055162+0000","last_became_peered":"2024-06-14T23:08:28.055162+0000","last_unstale":"2024-06-14T23:08:44.540601+0000","last_undegraded":"2024-06-14T23:08:28.039194+0000","last_fullsized":"2024-06-14T23:08:44.540601+0000","mapping_epoch":278,"log_start":"248'4731","ondisk_log_start":"248'4731","created":278,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":9166,"log_dups_size":0,"ondisk_log_size":9166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52133888,"num_objects":796,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":256544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,6,7","objects":796}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":258,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003765+0000","last_change":"2024-06-14T22:54:06.280576+0000","last_active":"2024-06-14T23:08:29.003765+0000","last_peered":"2024-06-14T23:08:29.003765+0000","last_clean":"2024-06-14T23:08:29.003765+0000","last_became_active":"2024-06-14T22:54:06.280288+0000","last_became_peered":"2024-06-14T22:54:06.280288+0000","last_unstale":"2024-06-14T23:08:29.003765+0000","last_undegraded":"2024-06-14T23:08:29.003765+0000","last_fullsized":"2024-06-14T23:08:29.003765+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:33.502244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":244,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.010111+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:08:29.010111+0000","last_peered":"2024-06-14T23:08:29.010111+0000","last_clean":"2024-06-14T23:08:29.010111+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:08:29.010111+0000","last_undegraded":"2024-06-14T23:08:29.010111+0000","last_fullsized":"2024-06-14T23:08:29.010111+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"281'13842","reported_seq":13918,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:45.050213+0000","last_change":"2024-06-14T23:08:28.105453+0000","last_active":"2024-06-14T23:08:45.050213+0000","last_peered":"2024-06-14T23:08:45.050213+0000","last_clean":"2024-06-14T23:08:45.050213+0000","last_became_active":"2024-06-14T23:08:28.082744+0000","last_became_peered":"2024-06-14T23:08:28.082744+0000","last_unstale":"2024-06-14T23:08:45.050213+0000","last_undegraded":"2024-06-14T23:08:45.050213+0000","last_fullsized":"2024-06-14T23:08:45.050213+0000","mapping_epoch":278,"log_start":"244'3200","ondisk_log_start":"244'3200","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":10642,"log_dups_size":0,"ondisk_log_size":10642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53084160,"num_objects":810,"num_object_clones":0,"num_object_copies":2430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7192,"num_write_kb":256064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":32768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":63,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.011769+0000","last_change":"2024-06-14T23:04:34.879700+0000","last_active":"2024-06-14T23:08:29.011769+0000","last_peered":"2024-06-14T23:08:29.011769+0000","last_clean":"2024-06-14T23:08:29.011769+0000","last_became_active":"2024-06-14T23:04:34.878924+0000","last_became_peered":"2024-06-14T23:04:34.878924+0000","last_unstale":"2024-06-14T23:08:29.011769+0000","last_undegraded":"2024-06-14T23:08:29.011769+0000","last_fullsized":"2024-06-14T23:08:29.011769+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:47:28.826486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":168,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.012100+0000","last_change":"2024-06-14T23:07:58.010033+0000","last_active":"2024-06-14T23:08:29.012100+0000","last_peered":"2024-06-14T23:08:29.012100+0000","last_clean":"2024-06-14T23:08:29.012100+0000","last_became_active":"2024-06-14T23:07:58.009712+0000","last_became_peered":"2024-06-14T23:07:58.009712+0000","last_unstale":"2024-06-14T23:08:29.012100+0000","last_undegraded":"2024-06-14T23:08:29.012100+0000","last_fullsized":"2024-06-14T23:08:29.012100+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:30:02.772328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.17","version":"281'13883","reported_seq":13943,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-14T23:08:44.599149+0000","last_change":"2024-06-14T23:08:28.104761+0000","last_active":"2024-06-14T23:08:44.599149+0000","last_peered":"2024-06-14T23:08:44.599149+0000","last_clean":"2024-06-14T23:08:44.599149+0000","last_became_active":"2024-06-14T23:08:28.087843+0000","last_became_peered":"2024-06-14T23:08:28.087843+0000","last_unstale":"2024-06-14T23:08:44.599149+0000","last_undegraded":"2024-06-14T23:08:44.599149+0000","last_fullsized":"2024-06-14T23:08:44.599149+0000","mapping_epoch":278,"log_start":"248'4720","ondisk_log_start":"248'4720","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:05:02.609871+0000","last_clean_scrub_stamp":"2024-06-14T23:05:02.609871+0000","objects_scrubbed":0,"log_size":9163,"log_dups_size":0,"ondisk_log_size":9163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50692096,"num_objects":774,"num_object_clones":0,"num_object_copies":2322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7218,"num_write_kb":255712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":258,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.003152+0000","last_change":"2024-06-14T23:07:58.016608+0000","last_active":"2024-06-14T23:08:29.003152+0000","last_peered":"2024-06-14T23:08:29.003152+0000","last_clean":"2024-06-14T23:08:29.003152+0000","last_became_active":"2024-06-14T23:07:58.016290+0000","last_became_peered":"2024-06-14T23:07:58.016290+0000","last_unstale":"2024-06-14T23:08:29.003152+0000","last_undegraded":"2024-06-14T23:08:29.003152+0000","last_fullsized":"2024-06-14T23:08:29.003152+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:12:50.936457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":65,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-06-14T23:08:29.009863+0000","last_change":"2024-06-14T23:06:19.002447+0000","last_active":"2024-06-14T23:08:29.009863+0000","last_peered":"2024-06-14T23:08:29.009863+0000","last_clean":"2024-06-14T23:08:29.009863+0000","last_became_active":"2024-06-14T23:06:19.002029+0000","last_became_peered":"2024-06-14T23:06:19.002029+0000","last_unstale":"2024-06-14T23:08:29.009863+0000","last_undegraded":"2024-06-14T23:08:29.009863+0000","last_fullsized":"2024-06-14T23:08:29.009863+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:36:27.132531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":26,"stat_sum":{"num_bytes":1634009112,"num_objects":24934,"num_object_clones":0,"num_object_copies":74802,"num_objects_missing_on_primary":7337,"num_objects_missing":7337,"num_objects_degraded":20246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229704,"num_write_kb":8148353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4593,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6432108544,"data_stored":6432096328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215462,"ondisk_log_size":215462,"up":78,"acting":76,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19646,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":670873,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":124,"num_write":33,"num_write_kb":330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":766889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3382,"ondisk_log_size":3382,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234416,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1543296,"kb_used_data":632056,"kb_used_omap":1,"kb_used_meta":911230,"kb_avail":92201856,"statfs":{"total":95995035648,"available":94414700544,"internally_reserved":0,"allocated":647225344,"data_stored":645802457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":933099864},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:08:22 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.81799999999999995,"15min":0.92200000000000004},"min":{"1min":0.218,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":3.2799999999999998,"5min":3.2799999999999998,"15min":6.9539999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":0.96199999999999997,"15min":1.054},"min":{"1min":0.224,"5min":0.224,"15min":0.20300000000000001},"max":{"1min":3.8530000000000002,"5min":4.3550000000000004,"15min":6.9550000000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:08:06 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":1.2869999999999999,"15min":1.6950000000000001},"min":{"1min":0.33600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":12.010999999999999,"5min":12.010999999999999,"15min":83.295000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":2.532,"5min":1.5209999999999999,"15min":1.716},"min":{"1min":0.27500000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":13.531000000000001,"5min":13.531000000000001,"15min":82.972999999999999},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:08:06 2024","interfaces":[{"interface":"back","average":{"1min":2.8490000000000002,"5min":1.379,"15min":1.115},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":20.390000000000001,"5min":20.390000000000001,"15min":20.390000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":2.7200000000000002,"5min":1.427,"15min":1.1830000000000001},"min":{"1min":0.28100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":20.327999999999999,"5min":20.327999999999999,"15min":20.327999999999999},"last":0.35499999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:08:22 2024","interfaces":[{"interface":"back","average":{"1min":2.8540000000000001,"5min":1.4770000000000001,"15min":1.4670000000000001},"min":{"1min":0.29099999999999998,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":20.645,"5min":20.645,"15min":55.683999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":2.8420000000000001,"5min":1.708,"15min":1.554},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":22.042999999999999,"5min":22.042999999999999,"15min":55.533999999999999},"last":0.374}]},{"osd":4,"last update":"Fri Jun 14 23:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.77600000000000002,"15min":0.91900000000000004},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.8170000000000002,"5min":5.3810000000000002,"15min":11.342000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.876,"15min":0.94699999999999995},"min":{"1min":0.25700000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.2830000000000004,"5min":7.0960000000000001,"15min":16.879000000000001},"last":0.95399999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.84699999999999998,"15min":1.0069999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23100000000000001},"max":{"1min":3.3290000000000002,"5min":4.383,"15min":6.9000000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.93200000000000005,"15min":0.98799999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.9209999999999998,"5min":5.4240000000000004,"15min":6.8529999999999998},"last":0.68600000000000005}]},{"osd":6,"last update":"Fri Jun 14 23:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":1.367,"15min":1.0449999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.6320000000000001,"5min":8.952,"15min":8.952},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.2989999999999999,"15min":1.008},"min":{"1min":0.34000000000000002,"5min":0.29599999999999999,"15min":0.28299999999999997},"max":{"1min":1.4850000000000001,"5min":7.0629999999999997,"15min":12.951000000000001},"last":0.55600000000000005}]}]},{"osd":6,"up_from":272,"seq":1168231104523,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2583540,"kb_used_data":2190572,"kb_used_omap":7,"kb_used_meta":392952,"kb_avail":91161612,"statfs":{"total":95995035648,"available":93349490688,"internally_reserved":0,"allocated":2243145728,"data_stored":2241734684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":402383290},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.1029999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5990000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9430000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.2649999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757691,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304940,"kb_used_data":413796,"kb_used_omap":1,"kb_used_meta":891134,"kb_avail":92440212,"statfs":{"total":95995035648,"available":94658777088,"internally_reserved":0,"allocated":423727104,"data_stored":422320586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912521560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:07:44 2024","interfaces":[{"interface":"back","average":{"1min":1.5620000000000001,"5min":1.6950000000000001,"15min":1.371},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":6.1950000000000003,"5min":14.94,"15min":14.94},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.619,"5min":1.673,"15min":1.335},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":6.2130000000000001,"5min":9.3819999999999997,"15min":9.3819999999999997},"last":0.33800000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.1699999999999999,"15min":1.0069999999999999},"min":{"1min":0.28399999999999997,"5min":0.252,"15min":0.23999999999999999},"max":{"1min":6.9480000000000004,"5min":7.4370000000000003,"15min":7.4370000000000003},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.345,"5min":1.365,"15min":1.2290000000000001},"min":{"1min":0.28199999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.7380000000000004,"5min":9.5039999999999996,"15min":9.5039999999999996},"last":0.82399999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.494,"15min":1.1799999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":3.367,"5min":14.891,"15min":14.891},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.343,"5min":1.478,"15min":1.198},"min":{"1min":0.27200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.2130000000000001,"5min":9.532,"15min":9.532},"last":0.84799999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.2970000000000002,"5min":1.5820000000000001,"15min":1.6000000000000001},"min":{"1min":0.40400000000000003,"5min":0.249,"15min":0.249},"max":{"1min":8.4350000000000005,"5min":8.4350000000000005,"15min":45.744},"last":4.2350000000000003},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.456,"15min":1.623},"min":{"1min":0.318,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.4290000000000003,"5min":7.4290000000000003,"15min":45.838000000000001},"last":2.2669999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.93999999999999995,"15min":0.82399999999999995},"min":{"1min":0.25900000000000001,"5min":0.255,"15min":0.255},"max":{"1min":1.353,"5min":14.352,"15min":14.352},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.091,"5min":1.111,"15min":0.996},"min":{"1min":0.28799999999999998,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":8.4039999999999999,"5min":9.5980000000000008,"15min":9.5980000000000008},"last":0.54800000000000004}]},{"osd":6,"last update":"Fri Jun 14 23:06:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.458,"15min":1.0960000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":4.6719999999999997,"5min":15.382999999999999,"15min":15.382999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.417,"15min":1.071},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.6020000000000003,"5min":14.855,"15min":14.855},"last":0.46000000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:08:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.0740000000000001,"15min":1.0860000000000001},"min":{"1min":0.314,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.153,"5min":6.29,"15min":14.818},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":1.21,"15min":1.0469999999999999},"min":{"1min":0.30299999999999999,"5min":0.28699999999999998,"15min":0.27000000000000002},"max":{"1min":3.0880000000000001,"5min":14.429,"15min":14.429},"last":0.48399999999999999}]}]},{"osd":4,"up_from":30,"seq":128849019327,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1310652,"kb_used_data":415860,"kb_used_omap":1,"kb_used_meta":894782,"kb_avail":92434500,"statfs":{"total":95995035648,"available":94652928000,"internally_reserved":0,"allocated":425840640,"data_stored":424421538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916257112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:07:44 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.286,"15min":1.1120000000000001},"min":{"1min":0.23300000000000001,"5min":0.221,"15min":0.17100000000000001},"max":{"1min":6.8440000000000003,"5min":6.8440000000000003,"15min":10.567},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.3129999999999999,"15min":1.117},"min":{"1min":0.251,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":4.1319999999999997,"5min":7.4900000000000002,"15min":8.3390000000000004},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:07:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0629999999999999,"15min":1.3140000000000001},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":3.1600000000000001,"5min":5.0430000000000001,"15min":56.917000000000002},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.1499999999999999,"15min":1.419},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.3010000000000002,"5min":6.2130000000000001,"15min":56.789999999999999},"last":0.46300000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:07:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":2.0030000000000001,"15min":1.2270000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.23000000000000001},"max":{"1min":4.4020000000000001,"5min":46.866,"15min":46.866},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.381,"5min":1.875,"15min":1.272},"min":{"1min":0.312,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.3920000000000003,"5min":46.851999999999997,"15min":46.851999999999997},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.048,"15min":1.032},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.6439999999999999,"5min":5.3239999999999998,"15min":6.617},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.292,"15min":1.1599999999999999},"min":{"1min":0.32500000000000001,"5min":0.28499999999999998,"15min":0.27500000000000002},"max":{"1min":3.77,"5min":6.1440000000000001,"15min":10.510999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Fri Jun 14 23:08:23 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.96999999999999997,"15min":1.4890000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.3759999999999999,"5min":12.315,"15min":139.13200000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.91600000000000004,"15min":1.5209999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.9569999999999999,"5min":10.959,"15min":138.86000000000001},"last":0.628}]},{"osd":6,"last update":"Fri Jun 14 23:06:40 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.075,"15min":0.88300000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.7360000000000002,"5min":6.8049999999999997,"15min":6.8049999999999997},"last":0.748},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.0509999999999999,"15min":0.83699999999999997},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.254},"max":{"1min":2.9009999999999998,"5min":5.8680000000000003,"15min":8.8819999999999997},"last":4.9169999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:08:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.89100000000000001,"15min":0.89600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.698,"5min":6.0890000000000004,"15min":8.9489999999999998},"last":0.433},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.86299999999999999,"15min":1.004},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.611,"5min":5.1779999999999999,"15min":8.4730000000000008},"last":0.59299999999999997}]}]},{"osd":3,"up_from":138,"seq":592705487026,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286444,"kb_used_data":737892,"kb_used_omap":4,"kb_used_meta":548539,"kb_avail":92458708,"statfs":{"total":95995035648,"available":94677716992,"internally_reserved":0,"allocated":755601408,"data_stored":754194663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":561704414},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:08:32 2024","interfaces":[{"interface":"back","average":{"1min":2.1589999999999998,"5min":1.6519999999999999,"15min":1.296},"min":{"1min":0.19400000000000001,"5min":0.187,"15min":0.184},"max":{"1min":7.9870000000000001,"5min":11.071999999999999,"15min":11.071999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.454,"15min":1.206},"min":{"1min":0.219,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.2030000000000003,"5min":11.518000000000001,"15min":11.518000000000001},"last":0.248}]},{"osd":1,"last update":"Fri Jun 14 23:08:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3420000000000001,"15min":1.29},"min":{"1min":0.28999999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.1609999999999996,"5min":11.596,"15min":24.097999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.1720000000000002,"5min":1.7290000000000001,"15min":1.4339999999999999},"min":{"1min":0.251,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":12.946,"5min":12.946,"15min":24.132999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:08:32 2024","interfaces":[{"interface":"back","average":{"1min":2.044,"5min":1.5780000000000001,"15min":1.2809999999999999},"min":{"1min":0.32000000000000001,"5min":0.26900000000000002,"15min":0.23599999999999999},"max":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":9.4730000000000008},"last":0.498},{"interface":"front","average":{"1min":1.95,"5min":1.4610000000000001,"15min":1.1819999999999999},"min":{"1min":0.245,"5min":0.23799999999999999,"15min":0.20899999999999999},"max":{"1min":4.7119999999999997,"5min":5.8799999999999999,"15min":10.369},"last":0.34699999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:08:32 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":2.5059999999999998,"15min":1.532},"min":{"1min":0.29299999999999998,"5min":0.28100000000000003,"15min":0.27900000000000003},"max":{"1min":4.6479999999999997,"5min":73.263000000000005,"15min":73.263000000000005},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.5900000000000001,"5min":2.2919999999999998,"15min":1.526},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.8100000000000005,"5min":73.5,"15min":73.5},"last":0.48099999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:08:32 2024","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":2.371,"15min":1.6140000000000001},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.9290000000000003,"5min":81.438999999999993,"15min":81.438999999999993},"last":0.38},{"interface":"front","average":{"1min":1.571,"5min":2.3500000000000001,"15min":1.581},"min":{"1min":0.32800000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.1040000000000001,"5min":81.626999999999995,"15min":81.626999999999995},"last":1.4379999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:06:23 2024","interfaces":[{"interface":"back","average":{"1min":6.7469999999999999,"5min":2.5579999999999998,"15min":1.4750000000000001},"min":{"1min":0.35799999999999998,"5min":0.34499999999999997,"15min":0.32400000000000001},"max":{"1min":82.248999999999995,"5min":82.248999999999995,"15min":82.248999999999995},"last":1.4099999999999999},{"interface":"front","average":{"1min":6.5620000000000003,"5min":2.71,"15min":1.528},"min":{"1min":0.44900000000000001,"5min":0.31,"15min":0.31},"max":{"1min":81.585999999999999,"5min":81.585999999999999,"15min":81.585999999999999},"last":1.145}]},{"osd":7,"last update":"Fri Jun 14 23:08:32 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.514,"15min":1.3340000000000001},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.4210000000000003,"5min":5.8559999999999999,"15min":7.7240000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.732,"5min":1.3089999999999999,"15min":1.2370000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":7.9649999999999999,"5min":7.9649999999999999,"15min":7.9649999999999999},"last":1.0169999999999999}]}]},{"osd":2,"up_from":133,"seq":571230650553,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016048,"kb_used_data":617000,"kb_used_omap":6,"kb_used_meta":399033,"kb_avail":92729104,"statfs":{"total":95995035648,"available":94954602496,"internally_reserved":0,"allocated":631808000,"data_stored":630397397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":408610293},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":3.177,"5min":2.2890000000000001,"15min":1.226},"min":{"1min":0.18099999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":10.147,"5min":55.851999999999997,"15min":55.851999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.085,"5min":2.0459999999999998,"15min":1.3149999999999999},"min":{"1min":0.27500000000000002,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":6.5590000000000002,"5min":56.171999999999997,"15min":56.171999999999997},"last":0.255}]},{"osd":1,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":3.2959999999999998,"5min":1.6220000000000001,"15min":1.2190000000000001},"min":{"1min":0.22700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8689999999999998,"5min":9.8140000000000001,"15min":21.920000000000002},"last":0.308},{"interface":"front","average":{"1min":2.7029999999999998,"5min":1.637,"15min":1.194},"min":{"1min":0.36599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":8.0760000000000005,"5min":14.416,"15min":21.573},"last":0.35499999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:08:35 2024","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":1.5620000000000001,"15min":1.3580000000000001},"min":{"1min":0.33300000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":21.66},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.5779999999999998,"5min":1.6200000000000001,"15min":1.3819999999999999},"min":{"1min":0.433,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":10.297000000000001,"5min":10.297000000000001,"15min":22.341000000000001},"last":0.40500000000000003}]},{"osd":4,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.8600000000000001,"15min":1.4199999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.6349999999999998,"5min":31.928000000000001,"15min":31.928000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":2.2090000000000001,"5min":1.7969999999999999,"15min":1.349},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":6.8150000000000004,"5min":31.954999999999998,"15min":31.954999999999998},"last":0.36499999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":1.4710000000000001,"15min":1.234},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.28299999999999997},"max":{"1min":9.6310000000000002,"5min":9.6310000000000002,"15min":31.539999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.8320000000000001,"5min":1.5129999999999999,"15min":1.258},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":8.8140000000000001,"5min":8.8140000000000001,"15min":31.481999999999999},"last":0.76100000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:06:56 2024","interfaces":[{"interface":"back","average":{"1min":2.25,"5min":1.907,"15min":1.2310000000000001},"min":{"1min":0.35699999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.1440000000000001,"5min":32.005000000000003,"15min":32.005000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":2.3460000000000001,"5min":1.905,"15min":1.22},"min":{"1min":0.41199999999999998,"5min":0.34999999999999998,"15min":0.30399999999999999},"max":{"1min":7.7409999999999997,"5min":32.079999999999998,"15min":32.079999999999998},"last":3.1110000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:08:01 2024","interfaces":[{"interface":"back","average":{"1min":2.5139999999999998,"5min":1.5760000000000001,"15min":1.2649999999999999},"min":{"1min":0.35899999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":6.8470000000000004,"5min":9.7539999999999996,"15min":10.614000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.958,"5min":1.4650000000000001,"15min":1.0800000000000001},"min":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":9.5850000000000009,"5min":9.5850000000000009,"15min":9.5850000000000009},"last":1.893}]}]},{"osd":1,"up_from":116,"seq":498216206570,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1478756,"kb_used_data":812380,"kb_used_omap":5,"kb_used_meta":666362,"kb_avail":92266396,"statfs":{"total":95995035648,"available":94480789504,"internally_reserved":0,"allocated":831877120,"data_stored":830477933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":682354968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":1.696,"15min":1.3580000000000001},"min":{"1min":0.20200000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":8.5060000000000002,"5min":9.5350000000000001,"15min":11.257999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":2.927,"5min":1.8100000000000001,"15min":1.27},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":13.331,"5min":13.672000000000001,"15min":13.672000000000001},"last":6.7539999999999996}]},{"osd":2,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":3.254,"5min":1.8620000000000001,"15min":1.4339999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19500000000000001},"max":{"1min":14.597,"5min":14.597,"15min":14.597},"last":0.42999999999999999},{"interface":"front","average":{"1min":2.4319999999999999,"5min":1.5089999999999999,"15min":1.1060000000000001},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.711,"5min":14.711,"15min":14.711},"last":6.665}]},{"osd":3,"last update":"Fri Jun 14 23:08:29 2024","interfaces":[{"interface":"back","average":{"1min":2.1030000000000002,"5min":1.5249999999999999,"15min":1.4350000000000001},"min":{"1min":0.246,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":11.454000000000001,"5min":11.454000000000001,"15min":21.422000000000001},"last":1.71},{"interface":"front","average":{"1min":2.6989999999999998,"5min":1.7669999999999999,"15min":1.544},"min":{"1min":0.35599999999999998,"5min":0.24099999999999999,"15min":0.214},"max":{"1min":10.584,"5min":10.584,"15min":21.016999999999999},"last":3.3260000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.2350000000000001,"15min":1.212},"min":{"1min":0.29299999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.8370000000000002,"5min":6.2939999999999996,"15min":7.673},"last":1.7390000000000001},{"interface":"front","average":{"1min":2.488,"5min":1.4550000000000001,"15min":1.393},"min":{"1min":0.34100000000000003,"5min":0.313,"15min":0.29199999999999998},"max":{"1min":8.2430000000000003,"5min":8.2430000000000003,"15min":9.4169999999999998},"last":0.47299999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.5249999999999999,"15min":1.359},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.27500000000000002},"max":{"1min":9.4489999999999998,"5min":19.132999999999999,"15min":19.132999999999999},"last":6.7169999999999996},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4419999999999999,"15min":1.25},"min":{"1min":0.308,"5min":0.29399999999999998,"15min":0.28599999999999998},"max":{"1min":3.806,"5min":19.123000000000001,"15min":19.123000000000001},"last":1.772}]},{"osd":6,"last update":"Fri Jun 14 23:05:58 2024","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":2.2349999999999999,"15min":1.669},"min":{"1min":0.38700000000000001,"5min":0.33500000000000002,"15min":0.29999999999999999},"max":{"1min":24.856000000000002,"5min":63.795999999999999,"15min":63.795999999999999},"last":6.5510000000000002},{"interface":"front","average":{"1min":2.548,"5min":2.1760000000000002,"15min":1.571},"min":{"1min":0.39900000000000002,"5min":0.32900000000000001,"15min":0.28499999999999998},"max":{"1min":24.904,"5min":63.811,"15min":63.811},"last":3.1920000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:08:03 2024","interfaces":[{"interface":"back","average":{"1min":2.5209999999999999,"5min":1.419,"15min":1.3919999999999999},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":8.266,"5min":8.266,"15min":9.4429999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.204,"15min":1.1970000000000001},"min":{"1min":0.378,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":9.4939999999999998,"5min":9.4939999999999998,"15min":9.4939999999999998},"last":6.7770000000000001}]}]},{"osd":0,"up_from":91,"seq":390842024224,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1488676,"kb_used_data":590812,"kb_used_omap":5,"kb_used_meta":897850,"kb_avail":92256476,"statfs":{"total":95995035648,"available":94470631424,"internally_reserved":0,"allocated":604991488,"data_stored":603590300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":919398490},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:07:49 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.359,"15min":1.306},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.16700000000000001},"max":{"1min":7.218,"5min":7.952,"15min":16.305},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.984,"5min":1.5189999999999999,"15min":1.423},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":9.3499999999999996,"5min":9.9990000000000006,"15min":16.184999999999999},"last":0.254}]},{"osd":2,"last update":"Fri Jun 14 23:08:05 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.4350000000000001,"15min":1.2050000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":6.8639999999999999,"5min":9.9789999999999992,"15min":16.757000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.0099999999999998,"5min":1.4419999999999999,"15min":1.2470000000000001},"min":{"1min":0.22700000000000001,"5min":0.223,"15min":0.222},"max":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":16.055},"last":0.33100000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:08:38 2024","interfaces":[{"interface":"back","average":{"1min":3.6909999999999998,"5min":2.0449999999999999,"15min":1.528},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":35.094999999999999,"5min":35.094999999999999,"15min":35.094999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.9980000000000002,"5min":2.177,"15min":1.577},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.224},"max":{"1min":38.444000000000003,"5min":38.444000000000003,"15min":38.444000000000003},"last":0.41099999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:08:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.419,"15min":1.3080000000000001},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.0970000000000004,"5min":11.162000000000001,"15min":16.527000000000001},"last":0.316},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.1799999999999999,"15min":1.238},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.7839999999999998,"5min":9.423,"15min":15.976000000000001},"last":2.6259999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.401,"15min":1.415},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.754,"5min":10.462,"15min":16.108000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1559999999999999,"15min":1.3500000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.31900000000000001},"max":{"1min":6.8019999999999996,"5min":6.8019999999999996,"15min":16.006},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:06:36 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.5109999999999999,"15min":1.4419999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30599999999999999},"max":{"1min":3.367,"5min":6.1669999999999998,"15min":16.152000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.7130000000000001,"15min":1.466},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.29399999999999998},"max":{"1min":4.274,"5min":9.3130000000000006,"15min":16.227},"last":0.69099999999999995}]},{"osd":7,"last update":"Fri Jun 14 23:08:38 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1439999999999999,"15min":1.3009999999999999},"min":{"1min":0.38600000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":6.8369999999999997,"5min":6.8369999999999997,"15min":16.199999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.641,"5min":1.3799999999999999,"15min":1.3580000000000001},"min":{"1min":0.35999999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":6.1600000000000001,"5min":11.195,"15min":16.259},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":580780032,"data_stored":580780032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":809893888,"data_stored":809893888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":613814272,"data_stored":613810200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":716111872,"data_stored":716111872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":421466112,"data_stored":421462040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":417792000,"data_stored":417792000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2231963648,"data_stored":2231959576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":640286720,"data_stored":640286720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:08:53.141 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-14T23:08:53.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-06-14T23:08:53.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:53.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:53.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:53.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:53.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:53.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:53.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:53.528 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:53.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:53.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:53.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:53.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:53.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:53.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:53.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:53.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:53.896 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:53.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:53.961 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:53 smithi086 ceph-mon[97913]: pgmap v1082: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 18605/57951 objects degraded (32.105%); 0 B/s, 262 objects/s recovering 2024-06-14T23:08:53.961 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 18605/57951 objects degraded (32.105%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:53.961 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1791652873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:08:53.961 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:53 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:08:53.976 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:08:53.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:54.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:54.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:54.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:54.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:54.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:54.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:53 smithi137 ceph-mon[90634]: pgmap v1082: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 18605/57951 objects degraded (32.105%); 0 B/s, 262 objects/s recovering 2024-06-14T23:08:54.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 18605/57951 objects degraded (32.105%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:54.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1791652873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:08:54.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:53 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:08:54.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:54.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:54.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:54.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:53 smithi086 ceph-mon[102754]: pgmap v1082: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 18605/57951 objects degraded (32.105%); 0 B/s, 262 objects/s recovering 2024-06-14T23:08:54.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 18605/57951 objects degraded (32.105%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:54.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1791652873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:08:54.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:53 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:08:54.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:54.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:54.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:54.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:54.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:54.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:54.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:54.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:54.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:54.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:54.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:54.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:54.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:54.979 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:54.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:54.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:55.076 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:08:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:08:54] "GET /metrics HTTP/1.1" 200 36622 "" "Prometheus/2.43.0" 2024-06-14T23:08:55.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:54 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:08:55.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:54 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:08:55.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:54 smithi086 ceph-mon[97913]: osdmap e282: 8 total, 8 up, 8 in 2024-06-14T23:08:55.077 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:54 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:08:55.077 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:54 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:08:55.077 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:54 smithi086 ceph-mon[102754]: osdmap e282: 8 total, 8 up, 8 in 2024-06-14T23:08:55.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:55.113 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:55.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:55.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:55.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:54 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:08:55.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:54 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:08:55.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:54 smithi137 ceph-mon[90634]: osdmap e282: 8 total, 8 up, 8 in 2024-06-14T23:08:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:55.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:55.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:55.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:55.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:55.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:55.501 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:55.505 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:08:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:55.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:55.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:55.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:55.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:55.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:55.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:55.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:55.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:55.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:55.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:55.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:56.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:56.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:56.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:56.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:56.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:55 smithi086 ceph-mon[97913]: pgmap v1084: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovering, 106 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 18605/53046 objects degraded (35.073%); 0 B/s, 232 objects/s recovering 2024-06-14T23:08:56.212 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:55 smithi086 ceph-mon[102754]: pgmap v1084: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovering, 106 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 18605/53046 objects degraded (35.073%); 0 B/s, 232 objects/s recovering 2024-06-14T23:08:56.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:56.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:56.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:56.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:55 smithi137 ceph-mon[90634]: pgmap v1084: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovering, 106 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 18605/53046 objects degraded (35.073%); 0 B/s, 232 objects/s recovering 2024-06-14T23:08:56.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:56.361 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:56.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:56.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:56.496 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:56.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:56.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:56.629 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:56.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:56.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:56.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:56.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:56.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:56.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:56.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:57.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:57.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:57.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:57.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:57.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:57.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:57.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:57.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:57.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:57.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:57.443 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:57.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:57.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:57.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:57.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:57.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:57.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:57.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:57.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:08:57.849 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:57.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:08:57.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:08:57.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:57.986 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:57.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:58.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:08:58.119 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:08:58.119 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:08:58.119 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:08:58.120 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:08:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:58.122 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:58.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:08:58.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:58.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:57 smithi137 ceph-mon[90634]: pgmap v1085: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 870 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 18605/41796 objects degraded (44.514%); 0 B/s, 247 objects/s recovering 2024-06-14T23:08:58.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4180875333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:58.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:58.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:58.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:58.302 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:57 smithi086 ceph-mon[102754]: pgmap v1085: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 870 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 18605/41796 objects degraded (44.514%); 0 B/s, 247 objects/s recovering 2024-06-14T23:08:58.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4180875333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:58.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:58.303 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:57 smithi086 ceph-mon[97913]: pgmap v1085: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 870 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 18605/41796 objects degraded (44.514%); 0 B/s, 247 objects/s recovering 2024-06-14T23:08:58.303 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4180875333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:08:58.303 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:08:58.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:58.392 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:58.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:58.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:58.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:58.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:58.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:58.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:58.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:58.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:08:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:58.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:58.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:08:58.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:08:58.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:58.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:08:59.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:08:59.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:59.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:08:59.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:08:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:59.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:08:59.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:08:59.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 18605/41796 objects degraded (44.514%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:59.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:59.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 18605/41796 objects degraded (44.514%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:59.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 18605/41796 objects degraded (44.514%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T23:08:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:59.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:59.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:59.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:59.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:59.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:59.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:59.609 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:59.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:59.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:08:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:59.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:59.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:08:59.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:08:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:08:59.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:08:59.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:08:59.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:00.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:00.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:00.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:00.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:00.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:00.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:00.197 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:08:59 smithi086 ceph-mon[97913]: pgmap v1086: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 870 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 18605/41796 objects degraded (44.514%); 0 B/s, 247 objects/s recovering 2024-06-14T23:09:00.197 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:08:59 smithi086 ceph-mon[102754]: pgmap v1086: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 870 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 18605/41796 objects degraded (44.514%); 0 B/s, 247 objects/s recovering 2024-06-14T23:09:00.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:00.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:00.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:00.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:00.448 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:00.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:00.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:08:59 smithi137 ceph-mon[90634]: pgmap v1086: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 870 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 18605/41796 objects degraded (44.514%); 0 B/s, 247 objects/s recovering 2024-06-14T23:09:00.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:00.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:00.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:00.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:00.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:00.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:00.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:00.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:00.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:00.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:00.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:00.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:01.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:01.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:01.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:01.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:01.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:01.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:01.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:01.394 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:01.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:01.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:01.468 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:01.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:01.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:01.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:01.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:01.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:01.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:01.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:01.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:01.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:01.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:01.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:01.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:02.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:02.089 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:02.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:02.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:02.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:02.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:02.290 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:01 smithi086 ceph-mon[97913]: pgmap v1087: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 594 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 18605/28575 objects degraded (65.109%); 0 B/s, 251 objects/s recovering 2024-06-14T23:09:02.290 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:01 smithi086 ceph-mon[102754]: pgmap v1087: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 594 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 18605/28575 objects degraded (65.109%); 0 B/s, 251 objects/s recovering 2024-06-14T23:09:02.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:02.353 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:02.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:02.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:02.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:02.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:01 smithi137 ceph-mon[90634]: pgmap v1087: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovering, 106 active+clean; 594 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 18605/28575 objects degraded (65.109%); 0 B/s, 251 objects/s recovering 2024-06-14T23:09:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:02.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:02.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:02.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:02.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:02.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:02.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:02.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:02.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:02.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:03.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:03.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:03.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:03.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:03.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:03.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:03.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:03.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:03.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:03.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:03.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:09:03.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:03.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:03.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:03.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:03.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:03.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:03.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:03.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:03.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:03.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:03.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:04.010 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:04.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:04.111 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:04.146 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:04.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:04.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:04.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:04.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:04.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:04 smithi086 ceph-mon[97913]: pgmap v1088: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 107 active+clean; 533 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 18605/25650 objects degraded (72.534%); 0 B/s, 176 objects/s recovering 2024-06-14T23:09:04.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:04 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 18605/25650 objects degraded (72.534%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:09:04.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:04 smithi086 ceph-mon[102754]: pgmap v1088: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 107 active+clean; 533 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 18605/25650 objects degraded (72.534%); 0 B/s, 176 objects/s recovering 2024-06-14T23:09:04.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:04 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 18605/25650 objects degraded (72.534%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:09:04.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:04.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:04.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:04 smithi137 ceph-mon[90634]: pgmap v1088: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 107 active+clean; 533 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 18605/25650 objects degraded (72.534%); 0 B/s, 176 objects/s recovering 2024-06-14T23:09:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:04 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 18605/25650 objects degraded (72.534%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:09:04.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:04.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:04.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:04.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:04.705 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:04.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:04.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:04.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:04.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:04.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:04.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:05.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:05.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:05.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:05.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:05.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:05.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:09:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:09:04] "GET /metrics HTTP/1.1" 200 36617 "" "Prometheus/2.43.0" 2024-06-14T23:09:05.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:05.365 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:05.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:05.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:05.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:05.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:05.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:05.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:05.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:05.779 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:05.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:05.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:05.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:05.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:06.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:06 smithi086 ceph-mon[97913]: pgmap v1089: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 107 active+clean; 417 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 18605/20076 objects degraded (92.673%); 0 B/s, 161 objects/s recovering 2024-06-14T23:09:06.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:06.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:06.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:06.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:06.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:06.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:06.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:06.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:06.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:06.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:06 smithi086 ceph-mon[102754]: pgmap v1089: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 107 active+clean; 417 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 18605/20076 objects degraded (92.673%); 0 B/s, 161 objects/s recovering 2024-06-14T23:09:06.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:06.448 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:06.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:06.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:06.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:06 smithi137 ceph-mon[90634]: pgmap v1089: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 107 active+clean; 417 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 18605/20076 objects degraded (92.673%); 0 B/s, 161 objects/s recovering 2024-06-14T23:09:06.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:06.599 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:06.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:06.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:06.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:06.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:06.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:06.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:06.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:06.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:06.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:07.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:07.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:07.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:07.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:07.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:07.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:07.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:07.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:07.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:07.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:07.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:07.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:07.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:07.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:07.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:07.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:07.606 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 127318 objects 2024-06-14T23:09:07.606 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :137.313 2024-06-14T23:09:07.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:07.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:07.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:07.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:07.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:07.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:07.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:07.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:07.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:07.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:08.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:08.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:08.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:08.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:08.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:08.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:08.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:08.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:08.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:08.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:08.316 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:08 smithi086 ceph-mon[97913]: pgmap v1090: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 107 active+clean; 169 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s; 18605/8157 objects degraded (228.086%); 0 B/s, 172 objects/s recovering 2024-06-14T23:09:08.317 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:08 smithi086 ceph-mon[102754]: pgmap v1090: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 107 active+clean; 169 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s; 18605/8157 objects degraded (228.086%); 0 B/s, 172 objects/s recovering 2024-06-14T23:09:08.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:08.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:08.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:08.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:08 smithi137 ceph-mon[90634]: pgmap v1090: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 107 active+clean; 169 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s; 18605/8157 objects degraded (228.086%); 0 B/s, 172 objects/s recovering 2024-06-14T23:09:08.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:08.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:08.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:08.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:08.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:08.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:08.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:08.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:08.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:08.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:08.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:08.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:09.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:09.053 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:09.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:09.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:09.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:09.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:09.264 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:09:09.264 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1073452377' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:09.264 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:09.264 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 18605/8157 objects degraded (228.086%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:09:09.264 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:09:09.264 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:09:09.264 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[97913]: osdmap e283: 8 total, 8 up, 8 in 2024-06-14T23:09:09.265 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/682443621' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-14T23:09:09.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1073452377' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:09.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:09.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 18605/8157 objects degraded (228.086%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:09:09.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:09:09.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:09:09.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[102754]: osdmap e283: 8 total, 8 up, 8 in 2024-06-14T23:09:09.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/682443621' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-14T23:09:09.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:09.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:09.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 1 pg_num to 31 2024-06-14T23:09:09.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:09.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:09.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:09.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:09.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1073452377' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:09.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:09 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:09.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:09 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 18605/8157 objects degraded (228.086%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:09:09.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:09 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:09:09.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:09 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:09:09.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:09 smithi137 ceph-mon[90634]: osdmap e283: 8 total, 8 up, 8 in 2024-06-14T23:09:09.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/682443621' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-14T23:09:09.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:09.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:09.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:09.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:09.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:09.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:09.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:09.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:09.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:09.925 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:09.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:10.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:10.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:10.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:10.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:10.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:10.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:10.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:10.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:10.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:10.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:10 smithi086 ceph-mon[102754]: pgmap v1092: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 107 active+clean; 169 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 18605/8157 objects degraded (228.086%); 0 B/s, 81 objects/s recovering 2024-06-14T23:09:10.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/682443621' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-06-14T23:09:10.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:10 smithi086 ceph-mon[102754]: osdmap e284: 8 total, 8 up, 8 in 2024-06-14T23:09:10.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:10 smithi086 ceph-mon[97913]: pgmap v1092: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 107 active+clean; 169 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 18605/8157 objects degraded (228.086%); 0 B/s, 81 objects/s recovering 2024-06-14T23:09:10.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/682443621' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-06-14T23:09:10.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:10 smithi086 ceph-mon[97913]: osdmap e284: 8 total, 8 up, 8 in 2024-06-14T23:09:10.396 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:10 smithi137 ceph-mon[90634]: pgmap v1092: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 107 active+clean; 169 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 18605/8157 objects degraded (228.086%); 0 B/s, 81 objects/s recovering 2024-06-14T23:09:10.396 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/682443621' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-06-14T23:09:10.396 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:10 smithi137 ceph-mon[90634]: osdmap e284: 8 total, 8 up, 8 in 2024-06-14T23:09:10.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:10.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:10.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:10.503 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:09:10.504 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:09:10.511 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:09:10.511 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:09:10.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:10.528 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:09:10.528 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:09:10.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:10.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:10.547 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:09:10.547 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:09:10.563 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:09:10.563 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:09:10.580 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:09:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:09:10.596 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:09:10.596 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:09:10.612 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:09:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:09:10.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:10.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:10.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:10.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:10.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:10.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:10.914 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:10.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:11.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:11.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:11.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:11.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:11.165 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:11.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:11.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:11.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:11.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:11.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2360012255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:11.335 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-14T23:09:11.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2360012255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:11.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-14T23:09:11.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:11.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:11.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:11.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2360012255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:11.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-14T23:09:11.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:11.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:11.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:11.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:11.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:11.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:11.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:11.785 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:11.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:11.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:11.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:12.003 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:09:12.003 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:09:12.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:12.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:09:12.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:12.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:12.059 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:12 smithi086 ceph-mon[102754]: pgmap v1094: 119 pgs: 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 109 active+clean; 30 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 15111/1482 objects degraded (1019.636%); 0 B/s, 121 objects/s recovering 2024-06-14T23:09:12.059 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-06-14T23:09:12.059 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:12 smithi086 ceph-mon[102754]: osdmap e285: 8 total, 8 up, 8 in 2024-06-14T23:09:12.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:12 smithi086 ceph-mon[97913]: pgmap v1094: 119 pgs: 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 109 active+clean; 30 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 15111/1482 objects degraded (1019.636%); 0 B/s, 121 objects/s recovering 2024-06-14T23:09:12.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-06-14T23:09:12.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:12 smithi086 ceph-mon[97913]: osdmap e285: 8 total, 8 up, 8 in 2024-06-14T23:09:12.067 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-06-14T23:09:12.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-06-14T23:09:12.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:12.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:12.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:12.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:12.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:12.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:12.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:12.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:12.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:12.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:12 smithi137 ceph-mon[90634]: pgmap v1094: 119 pgs: 9 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 109 active+clean; 30 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 15111/1482 objects degraded (1019.636%); 0 B/s, 121 objects/s recovering 2024-06-14T23:09:12.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-06-14T23:09:12.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:12 smithi137 ceph-mon[90634]: osdmap e285: 8 total, 8 up, 8 in 2024-06-14T23:09:12.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:12.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:12.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:12.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:12.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:12.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:12.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:12.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-14T23:09:12.745 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:12.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:12.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:12.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:12.798 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:12.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:12.911 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:12.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:13.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:13.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:13.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:13.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:13.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:13.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:13.197 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:13 smithi086 ceph-mon[102754]: osdmap e286: 8 total, 8 up, 8 in 2024-06-14T23:09:13.197 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:13.198 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:13.198 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:13 smithi086 ceph-mon[97913]: osdmap e286: 8 total, 8 up, 8 in 2024-06-14T23:09:13.198 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:13.198 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:13.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:13.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:13.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:13.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:13.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:13.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:13 smithi137 ceph-mon[90634]: osdmap e286: 8 total, 8 up, 8 in 2024-06-14T23:09:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:13.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:13.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:13.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:13.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:13.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:13.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:13.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:13.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:13.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:13.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:09:13.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:13.905 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:13.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:13.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:14.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:14.029 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:14.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:14.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:14.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:14.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:14.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:14.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:14.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:14.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:14 smithi086 ceph-mon[97913]: pgmap v1097: 129 pgs: 2 peering, 10 unknown, 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovering, 107 active+clean; 20 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 13389/984 objects degraded (1360.671%); 0 B/s, 431 objects/s recovering 2024-06-14T23:09:14.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:14 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 15111/1482 objects degraded (1019.636%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-14T23:09:14.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:14 smithi086 ceph-mon[102754]: pgmap v1097: 129 pgs: 2 peering, 10 unknown, 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovering, 107 active+clean; 20 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 13389/984 objects degraded (1360.671%); 0 B/s, 431 objects/s recovering 2024-06-14T23:09:14.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:14 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 15111/1482 objects degraded (1019.636%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-14T23:09:14.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:14.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:14.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:14.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:14.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:14 smithi137 ceph-mon[90634]: pgmap v1097: 129 pgs: 2 peering, 10 unknown, 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovering, 107 active+clean; 20 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 13389/984 objects degraded (1360.671%); 0 B/s, 431 objects/s recovering 2024-06-14T23:09:14.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:14 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 15111/1482 objects degraded (1019.636%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-14T23:09:14.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:14.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:14.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:14.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:14.649 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:14.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:14.700 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:14.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:14.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:14.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:14.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:14.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:14.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:15.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:15.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:15.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:15.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:09:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:09:14] "GET /metrics HTTP/1.1" 200 36633 "" "Prometheus/2.43.0" 2024-06-14T23:09:15.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:15.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:15.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:15.173 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:15.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:15.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:15.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:15.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:15.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:15.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:15.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:15.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:15.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:15.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:15.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:15.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:15.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:15.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:15.773 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:15.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:15.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:15.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:15.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:15.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:16.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:16.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:16.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:16.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:16.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:16.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:16.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:16.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:16.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:16.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:16 smithi086 ceph-mon[102754]: pgmap v1098: 129 pgs: 1 active+recovering+undersized+remapped, 2 peering, 5 unknown, 8 active+recovery_wait+degraded, 1 active+recovering, 112 active+clean; 20 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 446 op/s; 13389/984 objects degraded (1360.671%); 0 B/s, 524 objects/s recovering 2024-06-14T23:09:16.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:16 smithi086 ceph-mon[97913]: pgmap v1098: 129 pgs: 1 active+recovering+undersized+remapped, 2 peering, 5 unknown, 8 active+recovery_wait+degraded, 1 active+recovering, 112 active+clean; 20 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 446 op/s; 13389/984 objects degraded (1360.671%); 0 B/s, 524 objects/s recovering 2024-06-14T23:09:16.359 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-06-14T23:09:16.359 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:09:16.359 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T23:09:16.359 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.353356 2024-06-14T23:09:16.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 1 0.35335574513149703 2024-06-14T23:09:16.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:16.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:16.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:16.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:16 smithi137 ceph-mon[90634]: pgmap v1098: 129 pgs: 1 active+recovering+undersized+remapped, 2 peering, 5 unknown, 8 active+recovery_wait+degraded, 1 active+recovering, 112 active+clean; 20 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 446 op/s; 13389/984 objects degraded (1360.671%); 0 B/s, 524 objects/s recovering 2024-06-14T23:09:16.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:16.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:16.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:16.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:16.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:16.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:16.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:16.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:16.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:16.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:16.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:16.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:16.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:17.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:17.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_6' removed 2024-06-14T23:09:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:09:17.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:17.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:17.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:17.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:17.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:17.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[97913]: 1.3 scrub starts 2024-06-14T23:09:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/389579557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:09:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:09:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:09:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/170939056' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:09:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:09:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[102754]: 1.3 scrub starts 2024-06-14T23:09:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/389579557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:09:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:09:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:09:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/170939056' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:09:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:17 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:09:17.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:17.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:17.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:17.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:17 smithi137 ceph-mon[90634]: 1.3 scrub starts 2024-06-14T23:09:17.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/389579557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:09:17.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:17 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:09:17.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:17 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:09:17.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/170939056' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:09:17.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:17 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:09:17.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:17.544 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:17.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:17.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:17.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:17.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:17.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:17.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:17.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:17.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:17.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:17.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:18.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:18.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:18.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:18.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[97913]: pgmap v1099: 129 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded, 1 active+recovering, 121 active+clean; 20 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 293 op/s; 10005/984 objects degraded (1016.768%); 0 B/s, 721 objects/s recovering 2024-06-14T23:09:18.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[97913]: 11.0 scrub starts 2024-06-14T23:09:18.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[97913]: osdmap e287: 8 total, 8 up, 8 in 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[97913]: 1.e deep-scrub starts 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[97913]: 1.e deep-scrub ok 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[102754]: pgmap v1099: 129 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded, 1 active+recovering, 121 active+clean; 20 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 293 op/s; 10005/984 objects degraded (1016.768%); 0 B/s, 721 objects/s recovering 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[102754]: 11.0 scrub starts 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[102754]: osdmap e287: 8 total, 8 up, 8 in 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[102754]: 1.e deep-scrub starts 2024-06-14T23:09:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:18 smithi086 ceph-mon[102754]: 1.e deep-scrub ok 2024-06-14T23:09:18.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:18.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:18.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:18.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:18.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:18.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:18.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:18.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:18.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:18.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:18.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:18.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:18 smithi137 ceph-mon[90634]: pgmap v1099: 129 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+degraded, 1 active+recovering, 121 active+clean; 20 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 293 op/s; 10005/984 objects degraded (1016.768%); 0 B/s, 721 objects/s recovering 2024-06-14T23:09:18.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:18 smithi137 ceph-mon[90634]: 11.0 scrub starts 2024-06-14T23:09:18.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:18 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:09:18.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:18 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:09:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:18 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:09:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:18 smithi137 ceph-mon[90634]: osdmap e287: 8 total, 8 up, 8 in 2024-06-14T23:09:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:18 smithi137 ceph-mon[90634]: 1.e deep-scrub starts 2024-06-14T23:09:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:18 smithi137 ceph-mon[90634]: 1.e deep-scrub ok 2024-06-14T23:09:18.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:18.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:18.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:18.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:18.584 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:09:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:09:18.601 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:09:18.601 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:09:18.617 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:09:18.617 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:09:18.633 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:09:18.634 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:09:18.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:18.650 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:09:18.651 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:09:18.667 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:09:18.668 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:09:18.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:18.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:18.684 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:09:18.684 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:09:18.700 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:09:18.701 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:09:18.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:18.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:18.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:18.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:18.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:18.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:19.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:19.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:19.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:19.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:19.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:19.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:19.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:19.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:19.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:19.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:19 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 10005/984 objects degraded (1016.768%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-14T23:09:19.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:19 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2071409962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:19.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:19 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-14T23:09:19.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:19 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 10005/984 objects degraded (1016.768%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-14T23:09:19.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:19 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2071409962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:19.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:19 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-14T23:09:19.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:19.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:19.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:19 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 10005/984 objects degraded (1016.768%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-14T23:09:19.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:19 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2071409962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:19.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:19 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-14T23:09:19.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:19.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:19.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:19.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:19.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:19.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:19.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:19.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:19.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:19.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:19.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:20.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:20.048 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:20.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:20.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:20.170 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:20.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:20.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:20.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:20.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:20.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:20.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[97913]: pgmap v1101: 103 pgs: 103 active+clean; 661 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:20.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[97913]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10005/984 objects degraded (1016.768%), 6 pgs degraded, 1 pg undersized) 2024-06-14T23:09:20.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-06-14T23:09:20.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[97913]: osdmap e288: 8 total, 8 up, 8 in 2024-06-14T23:09:20.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:09:20.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[102754]: pgmap v1101: 103 pgs: 103 active+clean; 661 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:20.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[102754]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10005/984 objects degraded (1016.768%), 6 pgs degraded, 1 pg undersized) 2024-06-14T23:09:20.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-06-14T23:09:20.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[102754]: osdmap e288: 8 total, 8 up, 8 in 2024-06-14T23:09:20.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:09:20.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:20.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:20.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:20.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:20 smithi137 ceph-mon[90634]: pgmap v1101: 103 pgs: 103 active+clean; 661 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:20.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:20 smithi137 ceph-mon[90634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10005/984 objects degraded (1016.768%), 6 pgs degraded, 1 pg undersized) 2024-06-14T23:09:20.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-06-14T23:09:20.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:20 smithi137 ceph-mon[90634]: osdmap e288: 8 total, 8 up, 8 in 2024-06-14T23:09:20.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:09:20.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:20.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:20.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:20.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:20.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:20.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:20.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:20.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:20.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:20.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:20.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:20.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:20.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:21.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:21.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:21.034 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:21.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:21.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:21.157 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:21.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:21 smithi137 ceph-mon[90634]: osdmap e289: 8 total, 8 up, 8 in 2024-06-14T23:09:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:21 smithi137 ceph-mon[90634]: pgmap v1104: 103 pgs: 6 remapped+peering, 2 peering, 95 active+clean; 674 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 10 KiB/s rd, 5 op/s; 9.0 KiB/s, 0 objects/s recovering 2024-06-14T23:09:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:21 smithi137 ceph-mon[90634]: osdmap e290: 8 total, 8 up, 8 in 2024-06-14T23:09:21.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:21.280 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:21.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:21.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:21 smithi086 ceph-mon[97913]: osdmap e289: 8 total, 8 up, 8 in 2024-06-14T23:09:21.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:21 smithi086 ceph-mon[97913]: pgmap v1104: 103 pgs: 6 remapped+peering, 2 peering, 95 active+clean; 674 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 10 KiB/s rd, 5 op/s; 9.0 KiB/s, 0 objects/s recovering 2024-06-14T23:09:21.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:21 smithi086 ceph-mon[97913]: osdmap e290: 8 total, 8 up, 8 in 2024-06-14T23:09:21.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:21 smithi086 ceph-mon[102754]: osdmap e289: 8 total, 8 up, 8 in 2024-06-14T23:09:21.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:21 smithi086 ceph-mon[102754]: pgmap v1104: 103 pgs: 6 remapped+peering, 2 peering, 95 active+clean; 674 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 10 KiB/s rd, 5 op/s; 9.0 KiB/s, 0 objects/s recovering 2024-06-14T23:09:21.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:21 smithi086 ceph-mon[102754]: osdmap e290: 8 total, 8 up, 8 in 2024-06-14T23:09:21.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:21.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:21.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:21.419 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T23:09:21.419 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T23:09:21.420 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-06-14T23:09:21.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-06-14T23:09:21.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:21.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:21.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:21.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:21.651 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:21.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:21.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:21.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:09:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:21.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:21.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:21.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:21.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:21.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:22.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:22.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:22.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:22.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:22.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:22.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:22.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:22.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:22.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:22 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-14T23:09:22.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:22 smithi086 ceph-mon[97913]: osdmap e291: 8 total, 8 up, 8 in 2024-06-14T23:09:22.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:22 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-14T23:09:22.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:22 smithi086 ceph-mon[102754]: osdmap e291: 8 total, 8 up, 8 in 2024-06-14T23:09:22.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:22.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:22.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:22.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:22 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-14T23:09:22.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:22 smithi137 ceph-mon[90634]: osdmap e291: 8 total, 8 up, 8 in 2024-06-14T23:09:22.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:22.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:22.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:22.532 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:09:22.532 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:09:22.532 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-06-14T23:09:22.532 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:09:22.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:22.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:22.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:22.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:09:22.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:22.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:22.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:22.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:22.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:22.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:22.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:23.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:23.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:23.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:23.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:23.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:23.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:23.243 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:23.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:23.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:23.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-14T23:09:23.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:23.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:23.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:23.503 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:23.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:23.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:23.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:23.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:23.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:23.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:23.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:23.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:23.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:23.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:23.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:23.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:23.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:24.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:24.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[102754]: pgmap v1107: 103 pgs: 6 remapped+peering, 2 peering, 95 active+clean; 674 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 KiB/s wr, 4 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:09:24.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[102754]: osdmap e292: 8 total, 8 up, 8 in 2024-06-14T23:09:24.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:24.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:24.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:24.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[97913]: pgmap v1107: 103 pgs: 6 remapped+peering, 2 peering, 95 active+clean; 674 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 KiB/s wr, 4 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:09:24.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[97913]: osdmap e292: 8 total, 8 up, 8 in 2024-06-14T23:09:24.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:24.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:24.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:24.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:24.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:24.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:24.241 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:24.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:24.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:24.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:23 smithi137 ceph-mon[90634]: pgmap v1107: 103 pgs: 6 remapped+peering, 2 peering, 95 active+clean; 674 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 KiB/s wr, 4 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:09:24.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:23 smithi137 ceph-mon[90634]: osdmap e292: 8 total, 8 up, 8 in 2024-06-14T23:09:24.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:24.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:24.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:24.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:24.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:24.488 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:24.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:24.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:24.612 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:24.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:24.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:24.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:24.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:24.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:24.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:24.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:24.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:24.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:24.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:25.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:25.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:25.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:25.129 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:09:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:09:24] "GET /metrics HTTP/1.1" 200 36633 "" "Prometheus/2.43.0" 2024-06-14T23:09:25.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:25.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:25.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:25.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:25.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:25.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:25.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:25.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:25.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:25.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:25.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:25.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:25.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:25.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:25.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:25.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:25.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:25.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:25.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:25.968 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.1 primary-affinity to 0.353356 (8231572) 2024-06-14T23:09:25.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:25.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:26.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:26.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:26.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:26.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:26.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:26.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:25 smithi137 ceph-mon[90634]: pgmap v1109: 103 pgs: 3 remapped+peering, 100 active+clean; 674 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 4.3 KiB/s rd, 11 KiB/s wr, 3 op/s; 61 KiB/s, 0 objects/s recovering 2024-06-14T23:09:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1259778969' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.35335574513149703}]: dispatch 2024-06-14T23:09:26.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:25 smithi086 ceph-mon[97913]: pgmap v1109: 103 pgs: 3 remapped+peering, 100 active+clean; 674 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 4.3 KiB/s rd, 11 KiB/s wr, 3 op/s; 61 KiB/s, 0 objects/s recovering 2024-06-14T23:09:26.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1259778969' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.35335574513149703}]: dispatch 2024-06-14T23:09:26.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:25 smithi086 ceph-mon[102754]: pgmap v1109: 103 pgs: 3 remapped+peering, 100 active+clean; 674 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 4.3 KiB/s rd, 11 KiB/s wr, 3 op/s; 61 KiB/s, 0 objects/s recovering 2024-06-14T23:09:26.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1259778969' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.35335574513149703}]: dispatch 2024-06-14T23:09:26.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:26.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:26.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:26.483 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:26.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:26.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:26.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:26.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:26.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:26.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:26.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:26.734 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:26.735 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:26.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:26.856 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:26.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:26.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:26.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:26.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:27.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:27.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:27.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:27.173 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1259778969' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.35335574513149703}]': finished 2024-06-14T23:09:27.173 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:26 smithi086 ceph-mon[97913]: osdmap e293: 8 total, 8 up, 8 in 2024-06-14T23:09:27.173 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1259778969' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.35335574513149703}]': finished 2024-06-14T23:09:27.173 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:26 smithi086 ceph-mon[102754]: osdmap e293: 8 total, 8 up, 8 in 2024-06-14T23:09:27.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:27.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:27.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:27.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1259778969' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.35335574513149703}]': finished 2024-06-14T23:09:27.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:26 smithi137 ceph-mon[90634]: osdmap e293: 8 total, 8 up, 8 in 2024-06-14T23:09:27.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:27.353 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:27.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:27.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:27.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:27.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:27.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:27.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:27.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:27.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:27.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:27.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:27.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:27.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:27.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:27.910 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:27.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:27.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:27.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:28.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:28.101 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:28.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:28.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:28.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:28.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:28.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:28.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[97913]: pgmap v1111: 103 pgs: 1 peering, 102 active+clean; 567 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-06-14T23:09:28.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[97913]: osdmap e294: 8 total, 8 up, 8 in 2024-06-14T23:09:28.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[97913]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:09:28.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:28.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:28.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[102754]: pgmap v1111: 103 pgs: 1 peering, 102 active+clean; 567 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-06-14T23:09:28.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[102754]: osdmap e294: 8 total, 8 up, 8 in 2024-06-14T23:09:28.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[102754]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:09:28.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:28.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:28.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:28.401 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_7' created 2024-06-14T23:09:28.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:28.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:28.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:28 smithi137 ceph-mon[90634]: pgmap v1111: 103 pgs: 1 peering, 102 active+clean; 567 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-06-14T23:09:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:28 smithi137 ceph-mon[90634]: osdmap e294: 8 total, 8 up, 8 in 2024-06-14T23:09:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:28 smithi137 ceph-mon[90634]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:09:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:28.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:28.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:28.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:28.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:28.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:28.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:28.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:28.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:28.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:28.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:28.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:28.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:29.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:29.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:29.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:29.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:29.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:29.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:29.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:29.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:29.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:29.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:29.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:29.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:29.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:29 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3377536294' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-14T23:09:29.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:29 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-14T23:09:29.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:29 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-06-14T23:09:29.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:29 smithi137 ceph-mon[90634]: osdmap e295: 8 total, 8 up, 8 in 2024-06-14T23:09:29.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:29 smithi137 ceph-mon[90634]: pgmap v1114: 119 pgs: 16 unknown, 1 peering, 102 active+clean; 567 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-06-14T23:09:29.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3377536294' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-14T23:09:29.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-14T23:09:29.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-06-14T23:09:29.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[97913]: osdmap e295: 8 total, 8 up, 8 in 2024-06-14T23:09:29.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[97913]: pgmap v1114: 119 pgs: 16 unknown, 1 peering, 102 active+clean; 567 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-06-14T23:09:29.549 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3377536294' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-14T23:09:29.549 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-14T23:09:29.549 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-06-14T23:09:29.549 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[102754]: osdmap e295: 8 total, 8 up, 8 in 2024-06-14T23:09:29.549 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:29 smithi086 ceph-mon[102754]: pgmap v1114: 119 pgs: 16 unknown, 1 peering, 102 active+clean; 567 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-06-14T23:09:29.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:29.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:29.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:29.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:29.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:29.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:29.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:29.860 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:29.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:29.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:29.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:29.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:30.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:30.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:30.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:30.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:30.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:30.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:30.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:30.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:30.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:30.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:30.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:30.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:30.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:30.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:30 smithi137 ceph-mon[90634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:09:30.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:30 smithi137 ceph-mon[90634]: osdmap e296: 8 total, 8 up, 8 in 2024-06-14T23:09:30.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:30 smithi086 ceph-mon[97913]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:09:30.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:30 smithi086 ceph-mon[97913]: osdmap e296: 8 total, 8 up, 8 in 2024-06-14T23:09:30.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:30 smithi086 ceph-mon[102754]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:09:30.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:30 smithi086 ceph-mon[102754]: osdmap e296: 8 total, 8 up, 8 in 2024-06-14T23:09:30.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:30.609 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:30.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:30.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:30.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:30.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:30.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:30.860 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:30.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:30.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:30.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:30.987 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:30.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:31.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:31.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:31.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:31.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:31.236 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:31.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:31.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:31.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:31.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:31.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:31.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:31.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:31.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:31.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:31.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:31.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:31.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:31.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:31 smithi137 ceph-mon[90634]: osdmap e297: 8 total, 8 up, 8 in 2024-06-14T23:09:31.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:31 smithi137 ceph-mon[90634]: pgmap v1117: 119 pgs: 6 creating+peering, 4 unknown, 1 peering, 108 active+clean; 584 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 6.7 KiB/s rd, 18 KiB/s wr, 5 op/s; 13 KiB/s, 0 objects/s recovering 2024-06-14T23:09:31.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:31 smithi086 ceph-mon[97913]: osdmap e297: 8 total, 8 up, 8 in 2024-06-14T23:09:31.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:31 smithi086 ceph-mon[97913]: pgmap v1117: 119 pgs: 6 creating+peering, 4 unknown, 1 peering, 108 active+clean; 584 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 6.7 KiB/s rd, 18 KiB/s wr, 5 op/s; 13 KiB/s, 0 objects/s recovering 2024-06-14T23:09:31.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:31 smithi086 ceph-mon[102754]: osdmap e297: 8 total, 8 up, 8 in 2024-06-14T23:09:31.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:31 smithi086 ceph-mon[102754]: pgmap v1117: 119 pgs: 6 creating+peering, 4 unknown, 1 peering, 108 active+clean; 584 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 6.7 KiB/s rd, 18 KiB/s wr, 5 op/s; 13 KiB/s, 0 objects/s recovering 2024-06-14T23:09:31.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:31.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:31.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:31.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:31.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:31.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:31.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:32.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:32.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:32.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:32.197 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-06-14T23:09:32.197 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:09:32.197 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 3, 4, 5, 6, 7, 2] 2024-06-14T23:09:32.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd out 0 2024-06-14T23:09:32.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:32.264 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:32.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:32.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:32.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:32.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:32.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:32.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:32.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:32.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:32.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:32.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:32.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:32.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:32.763 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:32.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:32.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:32.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:32.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:32.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:33.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:33.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:33.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:33.137 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:33.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:33.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:33.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:33.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:33.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:33.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:33.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:33.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:33.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:33.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:33.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:33.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:33.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:33.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:33.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:33.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:33.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:33.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:33.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:34.013 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:34.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:34.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:33 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:34.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:33 smithi086 ceph-mon[97913]: pgmap v1118: 119 pgs: 6 creating+peering, 113 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 4.5 KiB/s rd, 12 KiB/s wr, 3 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-06-14T23:09:34.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:33 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:34.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:33 smithi086 ceph-mon[102754]: pgmap v1118: 119 pgs: 6 creating+peering, 113 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 4.5 KiB/s rd, 12 KiB/s wr, 3 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-06-14T23:09:34.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:34.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:34.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:34.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:33 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:33 smithi137 ceph-mon[90634]: pgmap v1118: 119 pgs: 6 creating+peering, 113 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 4.5 KiB/s rd, 12 KiB/s wr, 3 op/s; 8.8 KiB/s, 0 objects/s recovering 2024-06-14T23:09:34.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:34.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:34.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:34.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:34.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:34.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:34.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-06-14T23:09:34.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:34.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:34.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:34.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:34.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:34.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:34.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:34.762 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:34.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:34.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:34.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:34.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:34.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:34.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:35.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:35.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:35.113 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:09:35.113 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:09:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:09:34] "GET /metrics HTTP/1.1" 200 34204 "" "Prometheus/2.43.0" 2024-06-14T23:09:35.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:34 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T23:09:35.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2870746046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:35.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:34 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:35.114 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:34 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T23:09:35.114 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2870746046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:35.114 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:34 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:35.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:35.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:35.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:35.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:34 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-14T23:09:35.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2870746046' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:35.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:34 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:09:35.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:35.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:35.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:35.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:35.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:35.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:35.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:35.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:35.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:35.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:35.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:35.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:35.750 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:09:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:09:35.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:35.771 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:09:35.771 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:09:35.788 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:35.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:35.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:35.791 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:09:35.792 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:09:35.821 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:09:35.821 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:09:35.840 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:09:35.840 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:09:35.858 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:09:35.858 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:09:35.874 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:09:35.875 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:09:35.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:35.892 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:09:35.892 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:09:35.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:35.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:36.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:36.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:36.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:36.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:09:36.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:09:36.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[97913]: osdmap e298: 8 total, 8 up, 8 in 2024-06-14T23:09:36.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[97913]: pgmap v1120: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 4.5 KiB/s rd, 12 KiB/s wr, 3 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-06-14T23:09:36.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3942191081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:36.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:09:36.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:09:36.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[102754]: osdmap e298: 8 total, 8 up, 8 in 2024-06-14T23:09:36.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[102754]: pgmap v1120: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 4.5 KiB/s rd, 12 KiB/s wr, 3 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-06-14T23:09:36.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3942191081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:36.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:36.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:36.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:36.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:35 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:09:36.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:35 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:09:36.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:35 smithi137 ceph-mon[90634]: osdmap e298: 8 total, 8 up, 8 in 2024-06-14T23:09:36.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:35 smithi137 ceph-mon[90634]: pgmap v1120: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 4.5 KiB/s rd, 12 KiB/s wr, 3 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-06-14T23:09:36.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3942191081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:36.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:36.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:36.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:36.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:36.414 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:36.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:36.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:36.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:36.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:36.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:36.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:36.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:36.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:36.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:36.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:36.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:36.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:36.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:37.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:37.041 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:37.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:37.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:37.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:37.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:37.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:37.292 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:37.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:37.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:37.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:37.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:37.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:37.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:37.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:37.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:37.670 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:37.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:37.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:37.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:37.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:37.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:37.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:37.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:38.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:38.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:38.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:38.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:38.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:38.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:38.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:37 smithi137 ceph-mon[90634]: pgmap v1121: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:38.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:38.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:38.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:38.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:37 smithi086 ceph-mon[102754]: pgmap v1121: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:38.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:37 smithi086 ceph-mon[97913]: pgmap v1121: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:38.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:38.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:09:38.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:38.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:38.519 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:38.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:38.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:38.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:38.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:38.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:38.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:38.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:38.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:38.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:38.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:38.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:38.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:39.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:39.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:09:39.053 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:39.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:39.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:39.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:39.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:39.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:39.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:39.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:39.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:39.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:39.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:39.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:39.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:39.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:39.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:39.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:39.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:39.743 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:09:39.744 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:09:39.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:39.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:39.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:39.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:39.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:39.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:39.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:39 smithi086 ceph-mon[97913]: pgmap v1122: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:40.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:40.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:40.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:40.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:40.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:40.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:40.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:40.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:39 smithi137 ceph-mon[90634]: pgmap v1122: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:40.266 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:39 smithi086 ceph-mon[102754]: pgmap v1122: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:40.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:40.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:40.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:40.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:40.437 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:40.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:40.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:40.560 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:40.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:40.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:40 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:09:40.372+0000 7fe0161db640 -1 mon.a@0(leader).osd e298 definitely_dead 0 2024-06-14T23:09:40.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:40.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-14T23:09:40.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:40.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:40.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:40.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:40.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:40.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:40.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:40.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:40.980 INFO:teuthology.orchestra.run.smithi086.stderr:marked out osd.0. 2024-06-14T23:09:41.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:41.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:41.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:41.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:41.175 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:41.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:41.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2671688699' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-14T23:09:41.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:40 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-14T23:09:41.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:40 smithi137 ceph-mon[90634]: Client client.admin marked osd.0 out, while it was still marked up 2024-06-14T23:09:41.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:41.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:41.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:41.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2671688699' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-14T23:09:41.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:40 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-14T23:09:41.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:40 smithi086 ceph-mon[102754]: Client client.admin marked osd.0 out, while it was still marked up 2024-06-14T23:09:41.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2671688699' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-14T23:09:41.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:40 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-06-14T23:09:41.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:40 smithi086 ceph-mon[97913]: Client client.admin marked osd.0 out, while it was still marked up 2024-06-14T23:09:41.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:41.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:41.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:41.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:41.545 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:41.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:41.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:41.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:41.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:41.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:41.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:41.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:41.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:41.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:41.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:41.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:41.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:42.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:42.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:42.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:42.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:42.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:42.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:42.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:41 smithi137 ceph-mon[90634]: pgmap v1123: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:42.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:41 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-06-14T23:09:42.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:41 smithi137 ceph-mon[90634]: osdmap e299: 8 total, 8 up, 7 in 2024-06-14T23:09:42.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:42.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:42.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:42.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:41 smithi086 ceph-mon[97913]: pgmap v1123: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:42.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:41 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-06-14T23:09:42.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:41 smithi086 ceph-mon[97913]: osdmap e299: 8 total, 8 up, 7 in 2024-06-14T23:09:42.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:41 smithi086 ceph-mon[102754]: pgmap v1123: 119 pgs: 119 active+clean; 584 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-06-14T23:09:42.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:41 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-06-14T23:09:42.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:41 smithi086 ceph-mon[102754]: osdmap e299: 8 total, 8 up, 7 in 2024-06-14T23:09:42.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:42.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:42.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:42.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:42.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:42.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:42.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:42.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:42.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:42.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:42.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:42.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:42.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:42.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:42.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:42.957 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:42.957 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:42.988 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[97913]: osdmap e300: 8 total, 8 up, 7 in 2024-06-14T23:09:42.988 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:42.988 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:42.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:42.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:42.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:43.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:43.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:43.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:43.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:43.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:43.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:43.244 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[102754]: osdmap e300: 8 total, 8 up, 7 in 2024-06-14T23:09:43.244 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:43.244 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:43.244 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:43.245 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:43.245 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:43.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:42 smithi137 ceph-mon[90634]: osdmap e300: 8 total, 8 up, 7 in 2024-06-14T23:09:43.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:43.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:43.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:43.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:43.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:43.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:43.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:43.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:43.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:43.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:43.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:43.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:43.506 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:43.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:43.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:43.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:43.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:43.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:43.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:43.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:43.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:43.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:43.892 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:43.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:43.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:44.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:44.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:44.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:44.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:44.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:44.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:44.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:43 smithi137 ceph-mon[90634]: pgmap v1126: 119 pgs: 1 remapped+peering, 3 peering, 115 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-06-14T23:09:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:43 smithi137 ceph-mon[90634]: osdmap e301: 8 total, 8 up, 7 in 2024-06-14T23:09:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:43 smithi137 ceph-mon[90634]: osdmap e302: 8 total, 8 up, 7 in 2024-06-14T23:09:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:09:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:09:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:43 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:09:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:44.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:44.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[97913]: pgmap v1126: 119 pgs: 1 remapped+peering, 3 peering, 115 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[97913]: osdmap e301: 8 total, 8 up, 7 in 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[97913]: osdmap e302: 8 total, 8 up, 7 in 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[102754]: pgmap v1126: 119 pgs: 1 remapped+peering, 3 peering, 115 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[102754]: osdmap e301: 8 total, 8 up, 7 in 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[102754]: osdmap e302: 8 total, 8 up, 7 in 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:09:44.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:09:44.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:44.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:43 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:09:44.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:44.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:44.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:44.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:44.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:44.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:44.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:44.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:44.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:44.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:44.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:44.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:44.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:44.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:44.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:44.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:45.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:45.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:45.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:45.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:45.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:45.222 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:09:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:09:44] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T23:09:45.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:45.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:45.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:45.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:45.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:45.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:45.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:45.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:45.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:45.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:45 smithi086 ceph-mon[97913]: osdmap e303: 8 total, 8 up, 7 in 2024-06-14T23:09:45.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:45 smithi086 ceph-mon[97913]: pgmap v1130: 119 pgs: 6 active+undersized+remapped, 1 remapped+peering, 3 peering, 109 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 150 B/s, 1 objects/s recovering 2024-06-14T23:09:45.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:45 smithi086 ceph-mon[102754]: osdmap e303: 8 total, 8 up, 7 in 2024-06-14T23:09:45.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:45 smithi086 ceph-mon[102754]: pgmap v1130: 119 pgs: 6 active+undersized+remapped, 1 remapped+peering, 3 peering, 109 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 150 B/s, 1 objects/s recovering 2024-06-14T23:09:45.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:45.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:45.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:45.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:45.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:45.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:45.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:45 smithi137 ceph-mon[90634]: osdmap e303: 8 total, 8 up, 7 in 2024-06-14T23:09:45.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:45 smithi137 ceph-mon[90634]: pgmap v1130: 119 pgs: 6 active+undersized+remapped, 1 remapped+peering, 3 peering, 109 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 150 B/s, 1 objects/s recovering 2024-06-14T23:09:45.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:45.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:45.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:45.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:45.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:45.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:46.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:46.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:46.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:46.236 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:46.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:46.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:46.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:46.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:46.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:46.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:46.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:46.490 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:46.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:46.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:46.615 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:46.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:46.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:46.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:46.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:46.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:46.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:46.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:46.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:46.990 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:46.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:47.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:47 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:09:47.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:47 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:09:47.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:47.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:47.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:47.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:47.236 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:47.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:47.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:47.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:47.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:47.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:47.375 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 2] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-06-14T23:09:47.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:09:47.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:47.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:47.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:47.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:47 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:09:47.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:47.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:47.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:47.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:47.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:47.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:47.753 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:09:47.753 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:09:47.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:47.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:47.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:47.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:47.989 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:47.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:48.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:48.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:48.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:48.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:48.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:48.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:48.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:48.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:48 smithi086 ceph-mon[97913]: pgmap v1131: 119 pgs: 6 active+undersized+remapped, 113 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 65 KiB/s, 2 objects/s recovering 2024-06-14T23:09:48.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:48 smithi086 ceph-mon[102754]: pgmap v1131: 119 pgs: 6 active+undersized+remapped, 113 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 65 KiB/s, 2 objects/s recovering 2024-06-14T23:09:48.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:48.365 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:48.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:48.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:48.491 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:48.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:48.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:48 smithi137 ceph-mon[90634]: pgmap v1131: 119 pgs: 6 active+undersized+remapped, 113 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 65 KiB/s, 2 objects/s recovering 2024-06-14T23:09:48.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-14T23:09:48.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:48.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:48.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:48.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:48.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:48.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:48.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:48.782 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:09:48.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:48.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:48.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:48.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:48.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:48.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:48.996 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:09:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:09:49.013 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:09:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:09:49.030 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:09:49.031 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:09:49.047 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:09:49.047 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:09:49.064 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:09:49.064 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:09:49.080 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:09:49.081 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:09:49.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:49.097 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:09:49.098 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:09:49.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:49.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:49.114 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:09:49.114 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:09:49.197 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3241107054' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:49.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:49.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:49.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:49.297 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3241107054' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:49.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3241107054' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:49.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:49.356 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:49.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:49.457 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:49.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:49.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:49.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:49.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:49.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:49.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:49.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:49.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:49.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:49.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:49.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:49.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:49.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:49.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:50.025 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:50 smithi137 ceph-mon[90634]: pgmap v1132: 119 pgs: 119 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 53 KiB/s, 1 objects/s recovering 2024-06-14T23:09:50.025 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:50 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3203414980' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:09:50.025 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:50 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:09:50.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:50.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:50.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:50.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:50.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:50.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:50.272 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:50 smithi086 ceph-mon[102754]: pgmap v1132: 119 pgs: 119 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 53 KiB/s, 1 objects/s recovering 2024-06-14T23:09:50.272 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:50 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3203414980' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:09:50.272 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:50 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:09:50.272 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:50 smithi086 ceph-mon[97913]: pgmap v1132: 119 pgs: 119 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 53 KiB/s, 1 objects/s recovering 2024-06-14T23:09:50.273 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:50 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3203414980' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:09:50.273 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:50 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:09:50.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:50.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:50.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:50.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:50.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:50.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:50.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:50.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:50.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:50.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:50.697 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-06-14T23:09:50.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:50.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:50.769 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:09:50.770 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:09:50.770 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-06-14T23:09:50.770 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:09:50.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:50.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:50.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:50.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:50.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:50.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:51.041 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:09:51.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:51.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:51.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:51.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:51.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:51.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:51.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:51.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:51.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:51.336 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:51.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:51.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:09:51.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:51 smithi086 ceph-mon[102754]: osdmap e304: 8 total, 8 up, 7 in 2024-06-14T23:09:51.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1379915929' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:09:51.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:09:51.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:09:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:51 smithi086 ceph-mon[97913]: osdmap e304: 8 total, 8 up, 7 in 2024-06-14T23:09:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1379915929' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:09:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:09:51.406 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:09:51.406 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:51 smithi137 ceph-mon[90634]: osdmap e304: 8 total, 8 up, 7 in 2024-06-14T23:09:51.406 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1379915929' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:09:51.406 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:09:51.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:51.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:51.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:51.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-14T23:09:51.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:51.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:51.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:51.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:51.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:51.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:51.782 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:09:51.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:51.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:51.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:51.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:51.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:51.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:52.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:09:52.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:52.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:52.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:52.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[97913]: pgmap v1134: 119 pgs: 119 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 42 KiB/s, 0 objects/s recovering 2024-06-14T23:09:52.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[97913]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:09:52.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:09:52.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:09:52.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[97913]: osdmap e305: 8 total, 8 up, 7 in 2024-06-14T23:09:52.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:09:52.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[97913]: 1.8 scrub ok 2024-06-14T23:09:52.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[102754]: pgmap v1134: 119 pgs: 119 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 42 KiB/s, 0 objects/s recovering 2024-06-14T23:09:52.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[102754]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:09:52.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:09:52.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:09:52.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[102754]: osdmap e305: 8 total, 8 up, 7 in 2024-06-14T23:09:52.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:09:52.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:52 smithi086 ceph-mon[102754]: 1.8 scrub ok 2024-06-14T23:09:52.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:52.210 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:52.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:52.231 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:52 smithi137 ceph-mon[90634]: pgmap v1134: 119 pgs: 119 active+clean; 584 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 42 KiB/s, 0 objects/s recovering 2024-06-14T23:09:52.231 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:52 smithi137 ceph-mon[90634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:09:52.231 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:52 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:09:52.231 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:52 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:09:52.231 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:52 smithi137 ceph-mon[90634]: osdmap e305: 8 total, 8 up, 7 in 2024-06-14T23:09:52.231 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:52 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:09:52.231 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:52 smithi137 ceph-mon[90634]: 1.8 scrub ok 2024-06-14T23:09:52.260 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:52.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:52.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:52.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:52.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:52.449 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-06-14T23:09:52.450 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-14T23:09:52.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:09:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:52.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:52.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:52.457 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T23:09:52.457 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T23:09:52.502 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T23:09:52.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T23:09:52.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_261492 2024-06-14T23:09:52.504 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:09:52.504 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T23:09:52.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:52.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:52.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:52.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:52.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:52.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:52.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:52.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:52.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:52.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:53.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:53.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:53.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:53.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:53.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:53.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:53.297 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:53.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:53.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:53 smithi086 ceph-mon[97913]: 1.1e scrub starts 2024-06-14T23:09:53.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:53 smithi086 ceph-mon[97913]: 1.1e scrub ok 2024-06-14T23:09:53.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:53 smithi086 ceph-mon[102754]: 1.1e scrub starts 2024-06-14T23:09:53.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:53 smithi086 ceph-mon[102754]: 1.1e scrub ok 2024-06-14T23:09:53.396 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:53 smithi137 ceph-mon[90634]: 1.1e scrub starts 2024-06-14T23:09:53.396 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:53 smithi137 ceph-mon[90634]: 1.1e scrub ok 2024-06-14T23:09:53.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:53.439 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:53.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:53.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:53.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 3985 3969 248.049 248.062 0.0062152 0.0040173 2024-06-14T23:09:53.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:53.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:53.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:53.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:53.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:53.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:53.753 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:53.755 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:09:53.755 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:09:53.756 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:09:53.756 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:09:53.756 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:09:53.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:53.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:53.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:53.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:53.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:53.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:54.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:54.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:54.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:54.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:09:54.198 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[102754]: 1.f scrub starts 2024-06-14T23:09:54.199 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[102754]: 1.f scrub ok 2024-06-14T23:09:54.199 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[102754]: 1.0 scrub starts 2024-06-14T23:09:54.199 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[102754]: 1.0 scrub ok 2024-06-14T23:09:54.199 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[102754]: pgmap v1136: 119 pgs: 119 active+clean; 584 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-06-14T23:09:54.199 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[102754]: 1.a scrub starts 2024-06-14T23:09:54.199 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[102754]: 1.a scrub ok 2024-06-14T23:09:54.199 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[97913]: 1.f scrub starts 2024-06-14T23:09:54.199 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[97913]: 1.f scrub ok 2024-06-14T23:09:54.199 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[97913]: 1.0 scrub starts 2024-06-14T23:09:54.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[97913]: 1.0 scrub ok 2024-06-14T23:09:54.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[97913]: pgmap v1136: 119 pgs: 119 active+clean; 584 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-06-14T23:09:54.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[97913]: 1.a scrub starts 2024-06-14T23:09:54.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-mon[97913]: 1.a scrub ok 2024-06-14T23:09:54.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:54.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:54.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:54.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:54.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:54.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:54.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:54 smithi137 ceph-mon[90634]: 1.f scrub starts 2024-06-14T23:09:54.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:54 smithi137 ceph-mon[90634]: 1.f scrub ok 2024-06-14T23:09:54.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:54 smithi137 ceph-mon[90634]: 1.0 scrub starts 2024-06-14T23:09:54.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:54 smithi137 ceph-mon[90634]: 1.0 scrub ok 2024-06-14T23:09:54.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:54 smithi137 ceph-mon[90634]: pgmap v1136: 119 pgs: 119 active+clean; 584 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-06-14T23:09:54.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:54 smithi137 ceph-mon[90634]: 1.a scrub starts 2024-06-14T23:09:54.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:54 smithi137 ceph-mon[90634]: 1.a scrub ok 2024-06-14T23:09:54.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 16 7606 7590 237.168 226.312 0.00276673 0.00420734 2024-06-14T23:09:54.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:54.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:54.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:54.621 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:09:54.621 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:09:54.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-14T23:09:54.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:54.651 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:09:54.651 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:09:54.684 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:09:54.685 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:09:54.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:54.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:54.715 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:09:54.715 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:09:54.745 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:09:54.745 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:09:54.775 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:09:54.775 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:09:54.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:54.808 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:09:54.809 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:09:54.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:54.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:54.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:54.838 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:09:54.838 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:09:54.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:54.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:54.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:55.063 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:09:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:09:54] "GET /metrics HTTP/1.1" 200 36545 "" "Prometheus/2.43.0" 2024-06-14T23:09:55.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:55.082 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:09:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:55.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:55.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:55.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:55.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:55.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:55.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: 1.7 scrub starts 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: 1.7 scrub ok 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: 1.16 scrub starts 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: 1.16 scrub ok 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: 1.5 scrub ok 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1470646950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: 1.9 scrub starts 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: 1.9 scrub ok 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[97913]: pgmap v1137: 119 pgs: 119 active+clean; 80 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 9.9 MiB/s wr, 158 op/s 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: 1.7 scrub starts 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: 1.7 scrub ok 2024-06-14T23:09:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: 1.16 scrub starts 2024-06-14T23:09:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: 1.16 scrub ok 2024-06-14T23:09:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:09:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: 1.5 scrub ok 2024-06-14T23:09:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1470646950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: 1.9 scrub starts 2024-06-14T23:09:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: 1.9 scrub ok 2024-06-14T23:09:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:55 smithi086 ceph-mon[102754]: pgmap v1137: 119 pgs: 119 active+clean; 80 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 9.9 MiB/s wr, 158 op/s 2024-06-14T23:09:55.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:55.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:55.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:55.435 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: 1.7 scrub starts 2024-06-14T23:09:55.435 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: 1.7 scrub ok 2024-06-14T23:09:55.436 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: 1.16 scrub starts 2024-06-14T23:09:55.436 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: 1.16 scrub ok 2024-06-14T23:09:55.436 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:09:55.436 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: 1.5 scrub ok 2024-06-14T23:09:55.436 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1470646950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:09:55.436 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: 1.9 scrub starts 2024-06-14T23:09:55.436 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: 1.9 scrub ok 2024-06-14T23:09:55.436 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:55 smithi137 ceph-mon[90634]: pgmap v1137: 119 pgs: 119 active+clean; 80 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 9.9 MiB/s wr, 158 op/s 2024-06-14T23:09:55.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:55.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 16 10924 10908 227.234 207.375 0.0039561 0.00439233 2024-06-14T23:09:55.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:55.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:55.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:55.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:55.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:55.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:55.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:55.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:55.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:55.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:55.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:55.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:55.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:56.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:56.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:56.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:56.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:56.198 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:56.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:56.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:56.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:56.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:56.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:56 smithi086 ceph-mon[102754]: 1.15 scrub starts 2024-06-14T23:09:56.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:56 smithi086 ceph-mon[102754]: 1.15 scrub ok 2024-06-14T23:09:56.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:56 smithi086 ceph-mon[102754]: 1.18 scrub starts 2024-06-14T23:09:56.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:56 smithi086 ceph-mon[102754]: 1.18 scrub ok 2024-06-14T23:09:56.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:56 smithi086 ceph-mon[97913]: 1.15 scrub starts 2024-06-14T23:09:56.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:56 smithi086 ceph-mon[97913]: 1.15 scrub ok 2024-06-14T23:09:56.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:56 smithi086 ceph-mon[97913]: 1.18 scrub starts 2024-06-14T23:09:56.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:56 smithi086 ceph-mon[97913]: 1.18 scrub ok 2024-06-14T23:09:56.424 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:56 smithi137 ceph-mon[90634]: 1.15 scrub starts 2024-06-14T23:09:56.425 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:56 smithi137 ceph-mon[90634]: 1.15 scrub ok 2024-06-14T23:09:56.425 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:56 smithi137 ceph-mon[90634]: 1.18 scrub starts 2024-06-14T23:09:56.425 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:56 smithi137 ceph-mon[90634]: 1.18 scrub ok 2024-06-14T23:09:56.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:56.478 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:56.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:56.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 16 14276 14260 222.796 209.5 0.00279146 0.00448315 2024-06-14T23:09:56.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:56.613 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:56.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:56.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:56.722 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:09:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:56.748 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:56.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:56.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:56.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:56.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:56.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:57.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:57.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:57.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:57.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:57.196 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:09:57.197 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:09:57.260 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[97913]: 1.6 scrub starts 2024-06-14T23:09:57.260 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[97913]: 1.6 scrub ok 2024-06-14T23:09:57.260 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:09:57.260 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[97913]: 1.1c scrub ok 2024-06-14T23:09:57.260 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[97913]: pgmap v1138: 119 pgs: 119 active+clean; 488 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 61 MiB/s wr, 971 op/s 2024-06-14T23:09:57.261 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[102754]: 1.6 scrub starts 2024-06-14T23:09:57.261 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[102754]: 1.6 scrub ok 2024-06-14T23:09:57.261 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:09:57.261 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[102754]: 1.1c scrub ok 2024-06-14T23:09:57.261 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:57 smithi086 ceph-mon[102754]: pgmap v1138: 119 pgs: 119 active+clean; 488 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 61 MiB/s wr, 971 op/s 2024-06-14T23:09:57.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:57.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:57.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:57.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:09:57.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:57.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:09:57.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:57 smithi137 ceph-mon[90634]: 1.6 scrub starts 2024-06-14T23:09:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:57 smithi137 ceph-mon[90634]: 1.6 scrub ok 2024-06-14T23:09:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:57 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:09:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:57 smithi137 ceph-mon[90634]: 1.1c scrub ok 2024-06-14T23:09:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:57 smithi137 ceph-mon[90634]: pgmap v1138: 119 pgs: 119 active+clean; 488 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 61 MiB/s wr, 971 op/s 2024-06-14T23:09:57.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 16 17305 17289 216.096 189.312 0.00492868 0.00462145 2024-06-14T23:09:57.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:57.567 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:57.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:57.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:57.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:57.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:57.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:57.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:09:57.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:57.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:09:57.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:57.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:57.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:58.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:58.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:09:58.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:58.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:58.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:58.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:58.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:58.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:58.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:58.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:58.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:58.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:58.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 20087 20071 209.057 173.875 0.0051496 0.00477739 2024-06-14T23:09:58.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:58.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:58.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T23:09:58.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:58.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:58.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:58.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:58.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[97913]: 1.d scrub starts 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[97913]: 1.d scrub ok 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[97913]: 1.b scrub starts 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[97913]: 1.b scrub ok 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[97913]: 1.19 scrub starts 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[97913]: 1.19 scrub ok 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[102754]: 1.d scrub starts 2024-06-14T23:09:58.736 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[102754]: 1.d scrub ok 2024-06-14T23:09:58.737 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[102754]: 1.b scrub starts 2024-06-14T23:09:58.737 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[102754]: 1.b scrub ok 2024-06-14T23:09:58.737 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[102754]: 1.19 scrub starts 2024-06-14T23:09:58.737 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[102754]: 1.19 scrub ok 2024-06-14T23:09:58.737 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:58.737 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:58.737 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:58.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:58 smithi137 ceph-mon[90634]: 1.d scrub starts 2024-06-14T23:09:58.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:58 smithi137 ceph-mon[90634]: 1.d scrub ok 2024-06-14T23:09:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:58 smithi137 ceph-mon[90634]: 1.b scrub starts 2024-06-14T23:09:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:58 smithi137 ceph-mon[90634]: 1.b scrub ok 2024-06-14T23:09:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:58 smithi137 ceph-mon[90634]: 1.19 scrub starts 2024-06-14T23:09:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:58 smithi137 ceph-mon[90634]: 1.19 scrub ok 2024-06-14T23:09:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:09:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:09:58.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:58.806 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:58.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:09:58.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:09:58.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:58.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:09:59.041 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:09:59.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:59.086 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:59.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:59.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:59.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:59.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:59.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:59.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:09:59.349 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:09:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:59.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:09:59.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:09:59.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:59.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:09:59.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:09:59 smithi137 ceph-mon[90634]: pgmap v1139: 119 pgs: 119 active+clean; 936 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-14T23:09:59.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 15 22607 22592 201.698 157.562 0.00370651 0.00494962 2024-06-14T23:09:59.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:09:59.612 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:09:59 smithi086 ceph-mon[102754]: pgmap v1139: 119 pgs: 119 active+clean; 936 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-14T23:09:59.613 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:09:59 smithi086 ceph-mon[97913]: pgmap v1139: 119 pgs: 119 active+clean; 936 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-14T23:09:59.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:59.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:09:59.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:09:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:59.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:09:59.810 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:09:59.812 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:09:59.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:09:59.926 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:09:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:09:59.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:00.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:00.065 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:00.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:00.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:00.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:00.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:00.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:00.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:00.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:00.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:00.418 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:10:00.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:00.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:00.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:00.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 24729 24713 193.055 132.562 0.0120835 0.00517384 2024-06-14T23:10:00.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:00.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[97913]: 1.1d scrub starts 2024-06-14T23:10:00.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[97913]: 1.1d scrub ok 2024-06-14T23:10:00.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[97913]: from='client.46478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:10:00.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[97913]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-06-14T23:10:00.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[97913]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-14T23:10:00.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2075734675' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:10:00.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[102754]: 1.1d scrub starts 2024-06-14T23:10:00.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[102754]: 1.1d scrub ok 2024-06-14T23:10:00.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[102754]: from='client.46478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:10:00.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[102754]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-06-14T23:10:00.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[102754]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-14T23:10:00.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2075734675' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:10:00.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:00.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:00.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:00.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:00.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:00.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:00 smithi137 ceph-mon[90634]: 1.1d scrub starts 2024-06-14T23:10:00.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:00 smithi137 ceph-mon[90634]: 1.1d scrub ok 2024-06-14T23:10:00.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:00 smithi137 ceph-mon[90634]: from='client.46478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:10:00.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:00 smithi137 ceph-mon[90634]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-06-14T23:10:00.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:00 smithi137 ceph-mon[90634]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-14T23:10:00.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2075734675' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:10:00.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:00.880 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:00.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1139,"stamp":"2024-06-14T23:09:58.915624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":981209618,"num_objects":14988,"num_object_clones":0,"num_object_copies":44964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14988,"num_whiteouts":0,"num_read":25,"num_read_kb":48,"num_write":15039,"num_write_kb":957917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":422262,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20013,"ondisk_log_size":20013,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":365,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7103500,"kb_used_data":2367380,"kb_used_omap":29,"kb_used_meta":4736034,"kb_avail":649112564,"statfs":{"total":671965249536,"available":664691265536,"internally_reserved":0,"allocated":2424197120,"data_stored":2413599673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30035,"internal_metadata":4849699501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":980611121,"num_objects":14964,"num_object_clones":0,"num_object_copies":44892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14964,"num_whiteouts":0,"num_read":-66,"num_read_kb":-124,"num_write":14932,"num_write_kb":957430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-1193418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.886768"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":639,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275090+0000","last_change":"2024-06-14T23:09:26.977906+0000","last_active":"2024-06-14T23:09:50.275090+0000","last_peered":"2024-06-14T23:09:50.275090+0000","last_clean":"2024-06-14T23:09:50.275090+0000","last_became_active":"2024-06-14T23:09:26.977581+0000","last_became_peered":"2024-06-14T23:09:26.977581+0000","last_unstale":"2024-06-14T23:09:50.275090+0000","last_undegraded":"2024-06-14T23:09:50.275090+0000","last_fullsized":"2024-06-14T23:09:50.275090+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:21.981787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:21.981787+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016278149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":105,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033943+0000","last_change":"2024-06-14T23:09:41.998405+0000","last_active":"2024-06-14T23:09:50.033943+0000","last_peered":"2024-06-14T23:09:50.033943+0000","last_clean":"2024-06-14T23:09:50.033943+0000","last_became_active":"2024-06-14T23:09:41.991224+0000","last_became_peered":"2024-06-14T23:09:41.991224+0000","last_unstale":"2024-06-14T23:09:50.033943+0000","last_undegraded":"2024-06-14T23:09:50.033943+0000","last_fullsized":"2024-06-14T23:09:50.033943+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:36.846569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":282,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.031723+0000","last_change":"2024-06-14T23:07:58.016608+0000","last_active":"2024-06-14T23:09:50.031723+0000","last_peered":"2024-06-14T23:09:50.031723+0000","last_clean":"2024-06-14T23:09:50.031723+0000","last_became_active":"2024-06-14T23:07:58.016290+0000","last_became_peered":"2024-06-14T23:07:58.016290+0000","last_unstale":"2024-06-14T23:09:50.031723+0000","last_undegraded":"2024-06-14T23:09:50.031723+0000","last_fullsized":"2024-06-14T23:09:50.031723+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:12:50.936457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":193,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.273889+0000","last_change":"2024-06-14T23:07:58.010033+0000","last_active":"2024-06-14T23:09:50.273889+0000","last_peered":"2024-06-14T23:09:50.273889+0000","last_clean":"2024-06-14T23:09:50.273889+0000","last_became_active":"2024-06-14T23:07:58.009712+0000","last_became_peered":"2024-06-14T23:07:58.009712+0000","last_unstale":"2024-06-14T23:09:50.273889+0000","last_undegraded":"2024-06-14T23:09:50.273889+0000","last_fullsized":"2024-06-14T23:09:50.273889+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:30:02.772328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":97,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274260+0000","last_change":"2024-06-14T23:09:41.992023+0000","last_active":"2024-06-14T23:09:50.274260+0000","last_peered":"2024-06-14T23:09:50.274260+0000","last_clean":"2024-06-14T23:09:50.274260+0000","last_became_active":"2024-06-14T23:09:41.991361+0000","last_became_peered":"2024-06-14T23:09:41.991361+0000","last_unstale":"2024-06-14T23:09:50.274260+0000","last_undegraded":"2024-06-14T23:09:50.274260+0000","last_fullsized":"2024-06-14T23:09:50.274260+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:05.345808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":642,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:56.196715+0000","last_change":"2024-06-14T23:09:56.196659+0000","last_active":"2024-06-14T23:09:56.196715+0000","last_peered":"2024-06-14T23:09:56.196715+0000","last_clean":"2024-06-14T23:09:56.196715+0000","last_became_active":"2024-06-14T23:09:26.976247+0000","last_became_peered":"2024-06-14T23:09:26.976247+0000","last_unstale":"2024-06-14T23:09:56.196715+0000","last_undegraded":"2024-06-14T23:09:56.196715+0000","last_fullsized":"2024-06-14T23:09:56.196715+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:09:56.196559+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:09:56.196559+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:26:21.912245+0000","scrub_duration":0.0039154719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":268,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033789+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:09:50.033789+0000","last_peered":"2024-06-14T23:09:50.033789+0000","last_clean":"2024-06-14T23:09:50.033789+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:09:50.033789+0000","last_undegraded":"2024-06-14T23:09:50.033789+0000","last_fullsized":"2024-06-14T23:09:50.033789+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":292,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032062+0000","last_change":"2024-06-14T23:09:41.988017+0000","last_active":"2024-06-14T23:09:50.032062+0000","last_peered":"2024-06-14T23:09:50.032062+0000","last_clean":"2024-06-14T23:09:50.032062+0000","last_became_active":"2024-06-14T23:09:41.987704+0000","last_became_peered":"2024-06-14T23:09:41.987704+0000","last_unstale":"2024-06-14T23:09:50.032062+0000","last_undegraded":"2024-06-14T23:09:50.032062+0000","last_fullsized":"2024-06-14T23:09:50.032062+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:01:08.666023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":454,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:51.419545+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:09:51.419545+0000","last_peered":"2024-06-14T23:09:51.419545+0000","last_clean":"2024-06-14T23:09:51.419545+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:09:51.419545+0000","last_undegraded":"2024-06-14T23:09:51.419545+0000","last_fullsized":"2024-06-14T23:09:51.419545+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":268,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033519+0000","last_change":"2024-06-14T22:49:25.476521+0000","last_active":"2024-06-14T23:09:50.033519+0000","last_peered":"2024-06-14T23:09:50.033519+0000","last_clean":"2024-06-14T23:09:50.033519+0000","last_became_active":"2024-06-14T22:49:25.475838+0000","last_became_peered":"2024-06-14T22:49:25.475838+0000","last_unstale":"2024-06-14T23:09:50.033519+0000","last_undegraded":"2024-06-14T23:09:50.033519+0000","last_fullsized":"2024-06-14T23:09:50.033519+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:17.230004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":282,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032504+0000","last_change":"2024-06-14T22:54:06.281117+0000","last_active":"2024-06-14T23:09:50.032504+0000","last_peered":"2024-06-14T23:09:50.032504+0000","last_clean":"2024-06-14T23:09:50.032504+0000","last_became_active":"2024-06-14T22:54:06.280705+0000","last_became_peered":"2024-06-14T22:54:06.280705+0000","last_unstale":"2024-06-14T23:09:50.032504+0000","last_undegraded":"2024-06-14T23:09:50.032504+0000","last_fullsized":"2024-06-14T23:09:50.032504+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:00:20.733191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":88,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032370+0000","last_change":"2024-06-14T23:09:41.997516+0000","last_active":"2024-06-14T23:09:50.032370+0000","last_peered":"2024-06-14T23:09:50.032370+0000","last_clean":"2024-06-14T23:09:50.032370+0000","last_became_active":"2024-06-14T23:09:41.996805+0000","last_became_peered":"2024-06-14T23:09:41.996805+0000","last_unstale":"2024-06-14T23:09:50.032370+0000","last_undegraded":"2024-06-14T23:09:50.032370+0000","last_fullsized":"2024-06-14T23:09:50.032370+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:33:55.038127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":88,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274753+0000","last_change":"2024-06-14T23:04:34.879498+0000","last_active":"2024-06-14T23:09:50.274753+0000","last_peered":"2024-06-14T23:09:50.274753+0000","last_clean":"2024-06-14T23:09:50.274753+0000","last_became_active":"2024-06-14T23:04:34.878932+0000","last_became_peered":"2024-06-14T23:04:34.878932+0000","last_unstale":"2024-06-14T23:09:50.274753+0000","last_undegraded":"2024-06-14T23:09:50.274753+0000","last_fullsized":"2024-06-14T23:09:50.274753+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:56:50.201189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":598,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033952+0000","last_change":"2024-06-14T23:09:44.364546+0000","last_active":"2024-06-14T23:09:50.033952+0000","last_peered":"2024-06-14T23:09:50.033952+0000","last_clean":"2024-06-14T23:09:50.033952+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:09:50.033952+0000","last_undegraded":"2024-06-14T23:09:50.033952+0000","last_fullsized":"2024-06-14T23:09:50.033952+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:04:40.403365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:04:40.403365+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014330689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":304,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032142+0000","last_change":"2024-06-14T23:04:34.883863+0000","last_active":"2024-06-14T23:09:50.032142+0000","last_peered":"2024-06-14T23:09:50.032142+0000","last_clean":"2024-06-14T23:09:50.032142+0000","last_became_active":"2024-06-14T23:04:34.881668+0000","last_became_peered":"2024-06-14T23:04:34.881668+0000","last_unstale":"2024-06-14T23:09:50.032142+0000","last_undegraded":"2024-06-14T23:09:50.032142+0000","last_fullsized":"2024-06-14T23:09:50.032142+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:44:45.737416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":71,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.273649+0000","last_change":"2024-06-14T23:05:43.964532+0000","last_active":"2024-06-14T23:09:50.273649+0000","last_peered":"2024-06-14T23:09:50.273649+0000","last_clean":"2024-06-14T23:09:50.273649+0000","last_became_active":"2024-06-14T23:05:43.964255+0000","last_became_peered":"2024-06-14T23:05:43.964255+0000","last_unstale":"2024-06-14T23:09:50.273649+0000","last_undegraded":"2024-06-14T23:09:50.273649+0000","last_fullsized":"2024-06-14T23:09:50.273649+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:06.098124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":694,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:55.212657+0000","last_change":"2024-06-14T23:09:55.212605+0000","last_active":"2024-06-14T23:09:55.212657+0000","last_peered":"2024-06-14T23:09:55.212657+0000","last_clean":"2024-06-14T23:09:55.212657+0000","last_became_active":"2024-06-14T23:09:23.059739+0000","last_became_peered":"2024-06-14T23:09:23.059739+0000","last_unstale":"2024-06-14T23:09:55.212657+0000","last_undegraded":"2024-06-14T23:09:55.212657+0000","last_fullsized":"2024-06-14T23:09:55.212657+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:40:33.308373+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":276,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.273544+0000","last_change":"2024-06-14T22:54:06.284406+0000","last_active":"2024-06-14T23:09:50.273544+0000","last_peered":"2024-06-14T23:09:50.273544+0000","last_clean":"2024-06-14T23:09:50.273544+0000","last_became_active":"2024-06-14T22:54:06.284122+0000","last_became_peered":"2024-06-14T22:54:06.284122+0000","last_unstale":"2024-06-14T23:09:50.273544+0000","last_undegraded":"2024-06-14T23:09:50.273544+0000","last_fullsized":"2024-06-14T23:09:50.273544+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:35:37.523381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":153,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275535+0000","last_change":"2024-06-14T23:09:41.991893+0000","last_active":"2024-06-14T23:09:50.275535+0000","last_peered":"2024-06-14T23:09:50.275535+0000","last_clean":"2024-06-14T23:09:50.275535+0000","last_became_active":"2024-06-14T23:09:41.991586+0000","last_became_peered":"2024-06-14T23:09:41.991586+0000","last_unstale":"2024-06-14T23:09:50.275535+0000","last_undegraded":"2024-06-14T23:09:50.275535+0000","last_fullsized":"2024-06-14T23:09:50.275535+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:44:50.715285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":632,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033280+0000","last_change":"2024-06-14T23:09:44.365902+0000","last_active":"2024-06-14T23:09:50.033280+0000","last_peered":"2024-06-14T23:09:50.033280+0000","last_clean":"2024-06-14T23:09:50.033280+0000","last_became_active":"2024-06-14T23:09:44.365563+0000","last_became_peered":"2024-06-14T23:09:44.365563+0000","last_unstale":"2024-06-14T23:09:50.033280+0000","last_undegraded":"2024-06-14T23:09:50.033280+0000","last_fullsized":"2024-06-14T23:09:50.033280+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:13.729891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:13.729891+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.273790+0000","last_change":"2024-06-14T23:09:41.988134+0000","last_active":"2024-06-14T23:09:50.273790+0000","last_peered":"2024-06-14T23:09:50.273790+0000","last_clean":"2024-06-14T23:09:50.273790+0000","last_became_active":"2024-06-14T23:09:41.987844+0000","last_became_peered":"2024-06-14T23:09:41.987844+0000","last_unstale":"2024-06-14T23:09:50.273790+0000","last_undegraded":"2024-06-14T23:09:50.273790+0000","last_fullsized":"2024-06-14T23:09:50.273790+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:54:13.123903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":246,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.034547+0000","last_change":"2024-06-14T23:07:58.010437+0000","last_active":"2024-06-14T23:09:50.034547+0000","last_peered":"2024-06-14T23:09:50.034547+0000","last_clean":"2024-06-14T23:09:50.034547+0000","last_became_active":"2024-06-14T23:07:58.010143+0000","last_became_peered":"2024-06-14T23:07:58.010143+0000","last_unstale":"2024-06-14T23:09:50.034547+0000","last_undegraded":"2024-06-14T23:09:50.034547+0000","last_fullsized":"2024-06-14T23:09:50.034547+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:46:46.681752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":581,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032271+0000","last_change":"2024-06-14T23:09:23.060041+0000","last_active":"2024-06-14T23:09:50.032271+0000","last_peered":"2024-06-14T23:09:50.032271+0000","last_clean":"2024-06-14T23:09:50.032271+0000","last_became_active":"2024-06-14T23:09:23.059594+0000","last_became_peered":"2024-06-14T23:09:23.059594+0000","last_unstale":"2024-06-14T23:09:50.032271+0000","last_undegraded":"2024-06-14T23:09:50.032271+0000","last_fullsized":"2024-06-14T23:09:50.032271+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:31.843380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:31.843380+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001175607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":277,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274896+0000","last_change":"2024-06-14T23:07:58.028129+0000","last_active":"2024-06-14T23:09:50.274896+0000","last_peered":"2024-06-14T23:09:50.274896+0000","last_clean":"2024-06-14T23:09:50.274896+0000","last_became_active":"2024-06-14T23:07:58.027376+0000","last_became_peered":"2024-06-14T23:07:58.027376+0000","last_unstale":"2024-06-14T23:09:50.274896+0000","last_undegraded":"2024-06-14T23:09:50.274896+0000","last_fullsized":"2024-06-14T23:09:50.274896+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:25:25.805140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":152,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.034067+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:09:50.034067+0000","last_peered":"2024-06-14T23:09:50.034067+0000","last_clean":"2024-06-14T23:09:50.034067+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:09:50.034067+0000","last_undegraded":"2024-06-14T23:09:50.034067+0000","last_fullsized":"2024-06-14T23:09:50.034067+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":591,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:55.180673+0000","last_change":"2024-06-14T23:09:55.180612+0000","last_active":"2024-06-14T23:09:55.180673+0000","last_peered":"2024-06-14T23:09:55.180673+0000","last_clean":"2024-06-14T23:09:55.180673+0000","last_became_active":"2024-06-14T23:09:23.060934+0000","last_became_peered":"2024-06-14T23:09:23.060934+0000","last_unstale":"2024-06-14T23:09:55.180673+0000","last_undegraded":"2024-06-14T23:09:55.180673+0000","last_fullsized":"2024-06-14T23:09:55.180673+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:34:55.929893+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":169,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.273605+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:09:50.273605+0000","last_peered":"2024-06-14T23:09:50.273605+0000","last_clean":"2024-06-14T23:09:50.273605+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:09:50.273605+0000","last_undegraded":"2024-06-14T23:09:50.273605+0000","last_fullsized":"2024-06-14T23:09:50.273605+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":266,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033587+0000","last_change":"2024-06-14T22:49:25.475562+0000","last_active":"2024-06-14T23:09:50.033587+0000","last_peered":"2024-06-14T23:09:50.033587+0000","last_clean":"2024-06-14T23:09:50.033587+0000","last_became_active":"2024-06-14T22:49:25.475252+0000","last_became_peered":"2024-06-14T22:49:25.475252+0000","last_unstale":"2024-06-14T23:09:50.033587+0000","last_undegraded":"2024-06-14T23:09:50.033587+0000","last_fullsized":"2024-06-14T23:09:50.033587+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:07.662599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":637,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274475+0000","last_change":"2024-06-14T23:07:09.356684+0000","last_active":"2024-06-14T23:09:50.274475+0000","last_peered":"2024-06-14T23:09:50.274475+0000","last_clean":"2024-06-14T23:09:50.274475+0000","last_became_active":"2024-06-14T23:07:00.942504+0000","last_became_peered":"2024-06-14T23:07:00.942504+0000","last_unstale":"2024-06-14T23:09:50.274475+0000","last_undegraded":"2024-06-14T23:09:50.274475+0000","last_fullsized":"2024-06-14T23:09:50.274475+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:18:54.289960+0000","scrub_duration":0.013653241999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":187,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275285+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:09:50.275285+0000","last_peered":"2024-06-14T23:09:50.275285+0000","last_clean":"2024-06-14T23:09:50.275285+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:09:50.275285+0000","last_undegraded":"2024-06-14T23:09:50.275285+0000","last_fullsized":"2024-06-14T23:09:50.275285+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":291,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.034152+0000","last_change":"2024-06-14T23:09:26.976502+0000","last_active":"2024-06-14T23:09:50.034152+0000","last_peered":"2024-06-14T23:09:50.034152+0000","last_clean":"2024-06-14T23:09:50.034152+0000","last_became_active":"2024-06-14T23:09:26.976195+0000","last_became_peered":"2024-06-14T23:09:26.976195+0000","last_unstale":"2024-06-14T23:09:50.034152+0000","last_undegraded":"2024-06-14T23:09:50.034152+0000","last_fullsized":"2024-06-14T23:09:50.034152+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:52:47.944728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":471,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033637+0000","last_change":"2024-06-14T23:09:44.364336+0000","last_active":"2024-06-14T23:09:50.033637+0000","last_peered":"2024-06-14T23:09:50.033637+0000","last_clean":"2024-06-14T23:09:50.033637+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:09:50.033637+0000","last_undegraded":"2024-06-14T23:09:50.033637+0000","last_fullsized":"2024-06-14T23:09:50.033637+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:42.661567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:42.661567+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":288,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032629+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:09:50.032629+0000","last_peered":"2024-06-14T23:09:50.032629+0000","last_clean":"2024-06-14T23:09:50.032629+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:09:50.032629+0000","last_undegraded":"2024-06-14T23:09:50.032629+0000","last_fullsized":"2024-06-14T23:09:50.032629+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":282,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032675+0000","last_change":"2024-06-14T23:07:58.030396+0000","last_active":"2024-06-14T23:09:50.032675+0000","last_peered":"2024-06-14T23:09:50.032675+0000","last_clean":"2024-06-14T23:09:50.032675+0000","last_became_active":"2024-06-14T23:07:58.030106+0000","last_became_peered":"2024-06-14T23:07:58.030106+0000","last_unstale":"2024-06-14T23:09:50.032675+0000","last_undegraded":"2024-06-14T23:09:50.032675+0000","last_fullsized":"2024-06-14T23:09:50.032675+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:31:52.585282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":528,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:53.735453+0000","last_change":"2024-06-14T23:09:53.735645+0000","last_active":"2024-06-14T23:09:53.735453+0000","last_peered":"2024-06-14T23:09:53.735453+0000","last_clean":"2024-06-14T23:09:53.735453+0000","last_became_active":"2024-06-14T23:09:44.365708+0000","last_became_peered":"2024-06-14T23:09:44.365708+0000","last_unstale":"2024-06-14T23:09:53.735453+0000","last_undegraded":"2024-06-14T23:09:53.735453+0000","last_fullsized":"2024-06-14T23:09:53.735453+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.735431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.735431+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:01:47.605101+0000","scrub_duration":0.0032271439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.9","version":"305'603","reported_seq":622,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:55.175913+0000","last_change":"2024-06-14T23:09:29.408767+0000","last_active":"2024-06-14T23:09:55.175913+0000","last_peered":"2024-06-14T23:09:55.175913+0000","last_clean":"2024-06-14T23:09:55.175913+0000","last_became_active":"2024-06-14T23:09:29.408500+0000","last_became_peered":"2024-06-14T23:09:29.408500+0000","last_unstale":"2024-06-14T23:09:55.175913+0000","last_undegraded":"2024-06-14T23:09:55.175913+0000","last_fullsized":"2024-06-14T23:09:55.175913+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:07:23.834180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":603,"num_object_clones":0,"num_object_copies":1809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":603,"num_write_kb":38592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":265,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.031822+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:09:50.031822+0000","last_peered":"2024-06-14T23:09:50.031822+0000","last_clean":"2024-06-14T23:09:50.031822+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:09:50.031822+0000","last_undegraded":"2024-06-14T23:09:50.031822+0000","last_fullsized":"2024-06-14T23:09:50.031822+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":260,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.034467+0000","last_change":"2024-06-14T23:07:58.008658+0000","last_active":"2024-06-14T23:09:50.034467+0000","last_peered":"2024-06-14T23:09:50.034467+0000","last_clean":"2024-06-14T23:09:50.034467+0000","last_became_active":"2024-06-14T23:07:58.008400+0000","last_became_peered":"2024-06-14T23:07:58.008400+0000","last_unstale":"2024-06-14T23:09:50.034467+0000","last_undegraded":"2024-06-14T23:09:50.034467+0000","last_fullsized":"2024-06-14T23:09:50.034467+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:24:42.013441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":49,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032706+0000","last_change":"2024-06-14T23:09:42.000492+0000","last_active":"2024-06-14T23:09:50.032706+0000","last_peered":"2024-06-14T23:09:50.032706+0000","last_clean":"2024-06-14T23:09:50.032706+0000","last_became_active":"2024-06-14T23:09:41.996356+0000","last_became_peered":"2024-06-14T23:09:41.996356+0000","last_unstale":"2024-06-14T23:09:50.032706+0000","last_undegraded":"2024-06-14T23:09:50.032706+0000","last_fullsized":"2024-06-14T23:09:50.032706+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:23:08.682537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":628,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275128+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:09:50.275128+0000","last_peered":"2024-06-14T23:09:50.275128+0000","last_clean":"2024-06-14T23:09:50.275128+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:09:50.275128+0000","last_undegraded":"2024-06-14T23:09:50.275128+0000","last_fullsized":"2024-06-14T23:09:50.275128+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"305'614","reported_seq":642,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:55.158038+0000","last_change":"2024-06-14T23:09:41.988341+0000","last_active":"2024-06-14T23:09:55.158038+0000","last_peered":"2024-06-14T23:09:55.158038+0000","last_clean":"2024-06-14T23:09:55.158038+0000","last_became_active":"2024-06-14T23:09:41.987981+0000","last_became_peered":"2024-06-14T23:09:41.987981+0000","last_unstale":"2024-06-14T23:09:55.158038+0000","last_undegraded":"2024-06-14T23:09:55.158038+0000","last_fullsized":"2024-06-14T23:09:55.158038+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":614,"log_dups_size":0,"ondisk_log_size":614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:34.132457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40239104,"num_objects":614,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":614,"num_write_kb":39296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":207,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275571+0000","last_change":"2024-06-14T23:09:41.984952+0000","last_active":"2024-06-14T23:09:50.275571+0000","last_peered":"2024-06-14T23:09:50.275571+0000","last_clean":"2024-06-14T23:09:50.275571+0000","last_became_active":"2024-06-14T23:09:41.984620+0000","last_became_peered":"2024-06-14T23:09:41.984620+0000","last_unstale":"2024-06-14T23:09:50.275571+0000","last_undegraded":"2024-06-14T23:09:50.275571+0000","last_fullsized":"2024-06-14T23:09:50.275571+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:39:10.546051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":258,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032888+0000","last_change":"2024-06-14T23:09:26.973214+0000","last_active":"2024-06-14T23:09:50.032888+0000","last_peered":"2024-06-14T23:09:50.032888+0000","last_clean":"2024-06-14T23:09:50.032888+0000","last_became_active":"2024-06-14T23:09:26.972926+0000","last_became_peered":"2024-06-14T23:09:26.972926+0000","last_unstale":"2024-06-14T23:09:50.032888+0000","last_undegraded":"2024-06-14T23:09:50.032888+0000","last_fullsized":"2024-06-14T23:09:50.032888+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:15:22.918314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":19,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033548+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:09:50.033548+0000","last_peered":"2024-06-14T23:09:50.033548+0000","last_clean":"2024-06-14T23:09:50.033548+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:09:50.033548+0000","last_undegraded":"2024-06-14T23:09:50.033548+0000","last_fullsized":"2024-06-14T23:09:50.033548+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":702,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:54.214458+0000","last_change":"2024-06-14T23:09:54.214407+0000","last_active":"2024-06-14T23:09:54.214458+0000","last_peered":"2024-06-14T23:09:54.214458+0000","last_clean":"2024-06-14T23:09:54.214458+0000","last_became_active":"2024-06-14T23:09:12.030962+0000","last_became_peered":"2024-06-14T23:09:12.030962+0000","last_unstale":"2024-06-14T23:09:54.214458+0000","last_undegraded":"2024-06-14T23:09:54.214458+0000","last_fullsized":"2024-06-14T23:09:54.214458+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.214300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.214300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:44:08.135912+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"305'617","reported_seq":636,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:55.168042+0000","last_change":"2024-06-14T23:09:29.408548+0000","last_active":"2024-06-14T23:09:55.168042+0000","last_peered":"2024-06-14T23:09:55.168042+0000","last_clean":"2024-06-14T23:09:55.168042+0000","last_became_active":"2024-06-14T23:09:29.408275+0000","last_became_peered":"2024-06-14T23:09:29.408275+0000","last_unstale":"2024-06-14T23:09:55.168042+0000","last_undegraded":"2024-06-14T23:09:55.168042+0000","last_fullsized":"2024-06-14T23:09:55.168042+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:40:46.741581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40435712,"num_objects":617,"num_object_clones":0,"num_object_copies":1851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":617,"num_write_kb":39488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":89,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033358+0000","last_change":"2024-06-14T23:09:41.994913+0000","last_active":"2024-06-14T23:09:50.033358+0000","last_peered":"2024-06-14T23:09:50.033358+0000","last_clean":"2024-06-14T23:09:50.033358+0000","last_became_active":"2024-06-14T23:09:41.994371+0000","last_became_peered":"2024-06-14T23:09:41.994371+0000","last_unstale":"2024-06-14T23:09:50.033358+0000","last_undegraded":"2024-06-14T23:09:50.033358+0000","last_fullsized":"2024-06-14T23:09:50.033358+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:50:42.271107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":280,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.169040+0000","last_change":"2024-06-14T23:09:41.992195+0000","last_active":"2024-06-14T23:09:50.169040+0000","last_peered":"2024-06-14T23:09:50.169040+0000","last_clean":"2024-06-14T23:09:50.169040+0000","last_became_active":"2024-06-14T23:09:41.991654+0000","last_became_peered":"2024-06-14T23:09:41.991654+0000","last_unstale":"2024-06-14T23:09:50.169040+0000","last_undegraded":"2024-06-14T23:09:50.169040+0000","last_fullsized":"2024-06-14T23:09:50.169040+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:42:26.237016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":232,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274688+0000","last_change":"2024-06-14T23:09:26.973898+0000","last_active":"2024-06-14T23:09:50.274688+0000","last_peered":"2024-06-14T23:09:50.274688+0000","last_clean":"2024-06-14T23:09:50.274688+0000","last_became_active":"2024-06-14T23:09:26.972891+0000","last_became_peered":"2024-06-14T23:09:26.972891+0000","last_unstale":"2024-06-14T23:09:50.274688+0000","last_undegraded":"2024-06-14T23:09:50.274688+0000","last_fullsized":"2024-06-14T23:09:50.274688+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:09:28.624443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":604,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:55.324395+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:09:55.324395+0000","last_peered":"2024-06-14T23:09:55.324395+0000","last_clean":"2024-06-14T23:09:55.324395+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:09:55.324395+0000","last_undegraded":"2024-06-14T23:09:55.324395+0000","last_fullsized":"2024-06-14T23:09:55.324395+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"305'623","reported_seq":651,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:55.164902+0000","last_change":"2024-06-14T23:09:41.993713+0000","last_active":"2024-06-14T23:09:55.164902+0000","last_peered":"2024-06-14T23:09:55.164902+0000","last_clean":"2024-06-14T23:09:55.164902+0000","last_became_active":"2024-06-14T23:09:41.991654+0000","last_became_peered":"2024-06-14T23:09:41.991654+0000","last_unstale":"2024-06-14T23:09:55.164902+0000","last_undegraded":"2024-06-14T23:09:55.164902+0000","last_fullsized":"2024-06-14T23:09:55.164902+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:36:15.718053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40828928,"num_objects":623,"num_object_clones":0,"num_object_copies":1869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":623,"num_write_kb":39872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":262,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274109+0000","last_change":"2024-06-14T22:49:25.494118+0000","last_active":"2024-06-14T23:09:50.274109+0000","last_peered":"2024-06-14T23:09:50.274109+0000","last_clean":"2024-06-14T23:09:50.274109+0000","last_became_active":"2024-06-14T22:49:25.490611+0000","last_became_peered":"2024-06-14T22:49:25.490611+0000","last_unstale":"2024-06-14T23:09:50.274109+0000","last_undegraded":"2024-06-14T23:09:50.274109+0000","last_fullsized":"2024-06-14T23:09:50.274109+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:10.810689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274397+0000","last_change":"2024-06-14T23:09:41.993144+0000","last_active":"2024-06-14T23:09:50.274397+0000","last_peered":"2024-06-14T23:09:50.274397+0000","last_clean":"2024-06-14T23:09:50.274397+0000","last_became_active":"2024-06-14T23:09:41.992708+0000","last_became_peered":"2024-06-14T23:09:41.992708+0000","last_unstale":"2024-06-14T23:09:50.274397+0000","last_undegraded":"2024-06-14T23:09:50.274397+0000","last_fullsized":"2024-06-14T23:09:50.274397+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:35:09.004629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":303,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033870+0000","last_change":"2024-06-14T23:04:34.880181+0000","last_active":"2024-06-14T23:09:50.033870+0000","last_peered":"2024-06-14T23:09:50.033870+0000","last_clean":"2024-06-14T23:09:50.033870+0000","last_became_active":"2024-06-14T23:04:34.879842+0000","last_became_peered":"2024-06-14T23:04:34.879842+0000","last_unstale":"2024-06-14T23:09:50.033870+0000","last_undegraded":"2024-06-14T23:09:50.033870+0000","last_fullsized":"2024-06-14T23:09:50.033870+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:14:18.433225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":584,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:53.304891+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:09:53.304891+0000","last_peered":"2024-06-14T23:09:53.304891+0000","last_clean":"2024-06-14T23:09:53.304891+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:09:53.304891+0000","last_undegraded":"2024-06-14T23:09:53.304891+0000","last_fullsized":"2024-06-14T23:09:53.304891+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"305'1206","reported_seq":1234,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:58.212680+0000","last_change":"2024-06-14T23:09:41.992877+0000","last_active":"2024-06-14T23:09:58.212680+0000","last_peered":"2024-06-14T23:09:58.212680+0000","last_clean":"2024-06-14T23:09:58.212680+0000","last_became_active":"2024-06-14T23:09:41.992525+0000","last_became_peered":"2024-06-14T23:09:41.992525+0000","last_unstale":"2024-06-14T23:09:58.212680+0000","last_undegraded":"2024-06-14T23:09:58.212680+0000","last_fullsized":"2024-06-14T23:09:58.212680+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":1206,"log_dups_size":0,"ondisk_log_size":1206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:03:29.770513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79036416,"num_objects":1206,"num_object_clones":0,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1206,"num_write_kb":77184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":247,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274613+0000","last_change":"2024-06-14T23:09:26.974052+0000","last_active":"2024-06-14T23:09:50.274613+0000","last_peered":"2024-06-14T23:09:50.274613+0000","last_clean":"2024-06-14T23:09:50.274613+0000","last_became_active":"2024-06-14T23:09:26.973342+0000","last_became_peered":"2024-06-14T23:09:26.973342+0000","last_unstale":"2024-06-14T23:09:50.274613+0000","last_undegraded":"2024-06-14T23:09:50.274613+0000","last_fullsized":"2024-06-14T23:09:50.274613+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:29:02.869595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":293,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032693+0000","last_change":"2024-06-14T23:09:41.997524+0000","last_active":"2024-06-14T23:09:50.032693+0000","last_peered":"2024-06-14T23:09:50.032693+0000","last_clean":"2024-06-14T23:09:50.032693+0000","last_became_active":"2024-06-14T23:09:41.996927+0000","last_became_peered":"2024-06-14T23:09:41.996927+0000","last_unstale":"2024-06-14T23:09:50.032693+0000","last_undegraded":"2024-06-14T23:09:50.032693+0000","last_fullsized":"2024-06-14T23:09:50.032693+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:40:58.280799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":287,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033405+0000","last_change":"2024-06-14T23:07:58.015646+0000","last_active":"2024-06-14T23:09:50.033405+0000","last_peered":"2024-06-14T23:09:50.033405+0000","last_clean":"2024-06-14T23:09:50.033405+0000","last_became_active":"2024-06-14T23:07:58.014865+0000","last_became_peered":"2024-06-14T23:07:58.014865+0000","last_unstale":"2024-06-14T23:09:50.033405+0000","last_undegraded":"2024-06-14T23:09:50.033405+0000","last_fullsized":"2024-06-14T23:09:50.033405+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:52:48.798683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":681,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033129+0000","last_change":"2024-06-14T23:09:44.365983+0000","last_active":"2024-06-14T23:09:50.033129+0000","last_peered":"2024-06-14T23:09:50.033129+0000","last_clean":"2024-06-14T23:09:50.033129+0000","last_became_active":"2024-06-14T23:09:44.365555+0000","last_became_peered":"2024-06-14T23:09:44.365555+0000","last_unstale":"2024-06-14T23:09:50.033129+0000","last_undegraded":"2024-06-14T23:09:50.033129+0000","last_fullsized":"2024-06-14T23:09:50.033129+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:17:46.905401+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"305'881","reported_seq":899,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:56.282587+0000","last_change":"2024-06-14T23:09:29.407436+0000","last_active":"2024-06-14T23:09:56.282587+0000","last_peered":"2024-06-14T23:09:56.282587+0000","last_clean":"2024-06-14T23:09:56.282587+0000","last_became_active":"2024-06-14T23:09:29.406969+0000","last_became_peered":"2024-06-14T23:09:29.406969+0000","last_unstale":"2024-06-14T23:09:56.282587+0000","last_undegraded":"2024-06-14T23:09:56.282587+0000","last_fullsized":"2024-06-14T23:09:56.282587+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:50:42.755988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57737216,"num_objects":881,"num_object_clones":0,"num_object_copies":2643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":881,"num_write_kb":56384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":271,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274416+0000","last_change":"2024-06-14T23:09:41.993985+0000","last_active":"2024-06-14T23:09:50.274416+0000","last_peered":"2024-06-14T23:09:50.274416+0000","last_clean":"2024-06-14T23:09:50.274416+0000","last_became_active":"2024-06-14T23:09:41.993295+0000","last_became_peered":"2024-06-14T23:09:41.993295+0000","last_unstale":"2024-06-14T23:09:50.274416+0000","last_undegraded":"2024-06-14T23:09:50.274416+0000","last_fullsized":"2024-06-14T23:09:50.274416+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:21.170363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":292,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274220+0000","last_change":"2024-06-14T23:09:26.976578+0000","last_active":"2024-06-14T23:09:50.274220+0000","last_peered":"2024-06-14T23:09:50.274220+0000","last_clean":"2024-06-14T23:09:50.274220+0000","last_became_active":"2024-06-14T23:09:26.976272+0000","last_became_peered":"2024-06-14T23:09:26.976272+0000","last_unstale":"2024-06-14T23:09:50.274220+0000","last_undegraded":"2024-06-14T23:09:50.274220+0000","last_fullsized":"2024-06-14T23:09:50.274220+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:19.966496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274318+0000","last_change":"2024-06-14T23:09:41.992061+0000","last_active":"2024-06-14T23:09:50.274318+0000","last_peered":"2024-06-14T23:09:50.274318+0000","last_clean":"2024-06-14T23:09:50.274318+0000","last_became_active":"2024-06-14T23:09:41.991474+0000","last_became_peered":"2024-06-14T23:09:41.991474+0000","last_unstale":"2024-06-14T23:09:50.274318+0000","last_undegraded":"2024-06-14T23:09:50.274318+0000","last_fullsized":"2024-06-14T23:09:50.274318+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:04:20.342413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":575,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.031858+0000","last_change":"2024-06-14T23:09:44.365692+0000","last_active":"2024-06-14T23:09:50.031858+0000","last_peered":"2024-06-14T23:09:50.031858+0000","last_clean":"2024-06-14T23:09:50.031858+0000","last_became_active":"2024-06-14T23:09:44.365349+0000","last_became_peered":"2024-06-14T23:09:44.365349+0000","last_unstale":"2024-06-14T23:09:50.031858+0000","last_undegraded":"2024-06-14T23:09:50.031858+0000","last_fullsized":"2024-06-14T23:09:50.031858+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:53.023804+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.e","version":"305'1206","reported_seq":1234,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:58.208430+0000","last_change":"2024-06-14T23:09:41.994118+0000","last_active":"2024-06-14T23:09:58.208430+0000","last_peered":"2024-06-14T23:09:58.208430+0000","last_clean":"2024-06-14T23:09:58.208430+0000","last_became_active":"2024-06-14T23:09:41.993702+0000","last_became_peered":"2024-06-14T23:09:41.993702+0000","last_unstale":"2024-06-14T23:09:58.208430+0000","last_undegraded":"2024-06-14T23:09:58.208430+0000","last_fullsized":"2024-06-14T23:09:58.208430+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":1206,"log_dups_size":0,"ondisk_log_size":1206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:25.994252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79036416,"num_objects":1206,"num_object_clones":0,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1206,"num_write_kb":77184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":281,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033498+0000","last_change":"2024-06-14T23:09:41.989579+0000","last_active":"2024-06-14T23:09:50.033498+0000","last_peered":"2024-06-14T23:09:50.033498+0000","last_clean":"2024-06-14T23:09:50.033498+0000","last_became_active":"2024-06-14T23:09:41.988965+0000","last_became_peered":"2024-06-14T23:09:41.988965+0000","last_unstale":"2024-06-14T23:09:50.033498+0000","last_undegraded":"2024-06-14T23:09:50.033498+0000","last_fullsized":"2024-06-14T23:09:50.033498+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:46:30.422012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":153,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.031742+0000","last_change":"2024-06-14T23:09:41.991652+0000","last_active":"2024-06-14T23:09:50.031742+0000","last_peered":"2024-06-14T23:09:50.031742+0000","last_clean":"2024-06-14T23:09:50.031742+0000","last_became_active":"2024-06-14T23:09:41.991317+0000","last_became_peered":"2024-06-14T23:09:41.991317+0000","last_unstale":"2024-06-14T23:09:50.031742+0000","last_undegraded":"2024-06-14T23:09:50.031742+0000","last_fullsized":"2024-06-14T23:09:50.031742+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:16:18.209329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":277,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:51.039048+0000","last_change":"2024-06-14T23:09:41.990111+0000","last_active":"2024-06-14T23:09:51.039048+0000","last_peered":"2024-06-14T23:09:51.039048+0000","last_clean":"2024-06-14T23:09:51.039048+0000","last_became_active":"2024-06-14T23:09:41.989799+0000","last_became_peered":"2024-06-14T23:09:41.989799+0000","last_unstale":"2024-06-14T23:09:51.039048+0000","last_undegraded":"2024-06-14T23:09:51.039048+0000","last_fullsized":"2024-06-14T23:09:51.039048+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:01:18.659356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":482,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032180+0000","last_change":"2024-06-14T23:09:26.972037+0000","last_active":"2024-06-14T23:09:50.032180+0000","last_peered":"2024-06-14T23:09:50.032180+0000","last_clean":"2024-06-14T23:09:50.032180+0000","last_became_active":"2024-06-14T23:09:26.971702+0000","last_became_peered":"2024-06-14T23:09:26.971702+0000","last_unstale":"2024-06-14T23:09:50.032180+0000","last_undegraded":"2024-06-14T23:09:50.032180+0000","last_fullsized":"2024-06-14T23:09:50.032180+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:00:30.719678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:00:30.719678+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":27,"num_write":7,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"305'1189","reported_seq":1207,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:58.224430+0000","last_change":"2024-06-14T23:09:29.408960+0000","last_active":"2024-06-14T23:09:58.224430+0000","last_peered":"2024-06-14T23:09:58.224430+0000","last_clean":"2024-06-14T23:09:58.224430+0000","last_became_active":"2024-06-14T23:09:29.408598+0000","last_became_peered":"2024-06-14T23:09:29.408598+0000","last_unstale":"2024-06-14T23:09:58.224430+0000","last_undegraded":"2024-06-14T23:09:58.224430+0000","last_fullsized":"2024-06-14T23:09:58.224430+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":1189,"log_dups_size":0,"ondisk_log_size":1189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:11:09.540804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77922304,"num_objects":1189,"num_object_clones":0,"num_object_copies":3567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1189,"num_write_kb":76096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":220,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.034746+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:09:50.034746+0000","last_peered":"2024-06-14T23:09:50.034746+0000","last_clean":"2024-06-14T23:09:50.034746+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:09:50.034746+0000","last_undegraded":"2024-06-14T23:09:50.034746+0000","last_fullsized":"2024-06-14T23:09:50.034746+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":226,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033328+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:09:50.033328+0000","last_peered":"2024-06-14T23:09:50.033328+0000","last_clean":"2024-06-14T23:09:50.033328+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:09:50.033328+0000","last_undegraded":"2024-06-14T23:09:50.033328+0000","last_fullsized":"2024-06-14T23:09:50.033328+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":290,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033440+0000","last_change":"2024-06-14T23:04:34.889456+0000","last_active":"2024-06-14T23:09:50.033440+0000","last_peered":"2024-06-14T23:09:50.033440+0000","last_clean":"2024-06-14T23:09:50.033440+0000","last_became_active":"2024-06-14T23:04:34.888908+0000","last_became_peered":"2024-06-14T23:04:34.888908+0000","last_unstale":"2024-06-14T23:09:50.033440+0000","last_undegraded":"2024-06-14T23:09:50.033440+0000","last_fullsized":"2024-06-14T23:09:50.033440+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:00:24.632734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":573,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:52.751694+0000","last_change":"2024-06-14T23:09:52.751795+0000","last_active":"2024-06-14T23:09:52.751694+0000","last_peered":"2024-06-14T23:09:52.751694+0000","last_clean":"2024-06-14T23:09:52.751694+0000","last_became_active":"2024-06-14T23:09:44.362290+0000","last_became_peered":"2024-06-14T23:09:44.362290+0000","last_unstale":"2024-06-14T23:09:52.751694+0000","last_undegraded":"2024-06-14T23:09:52.751694+0000","last_fullsized":"2024-06-14T23:09:52.751694+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.751667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.751667+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:51:37.289606+0000","scrub_duration":0.0066202860000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.7","version":"305'1253","reported_seq":1271,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:58.216721+0000","last_change":"2024-06-14T23:09:29.407769+0000","last_active":"2024-06-14T23:09:58.216721+0000","last_peered":"2024-06-14T23:09:58.216721+0000","last_clean":"2024-06-14T23:09:58.216721+0000","last_became_active":"2024-06-14T23:09:29.407435+0000","last_became_peered":"2024-06-14T23:09:29.407435+0000","last_unstale":"2024-06-14T23:09:58.216721+0000","last_undegraded":"2024-06-14T23:09:58.216721+0000","last_fullsized":"2024-06-14T23:09:58.216721+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":1253,"log_dups_size":0,"ondisk_log_size":1253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:00:52.235174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82116608,"num_objects":1253,"num_object_clones":0,"num_object_copies":3759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1253,"num_write_kb":80192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":277,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274639+0000","last_change":"2024-06-14T22:54:06.282328+0000","last_active":"2024-06-14T23:09:50.274639+0000","last_peered":"2024-06-14T23:09:50.274639+0000","last_clean":"2024-06-14T23:09:50.274639+0000","last_became_active":"2024-06-14T22:54:06.282019+0000","last_became_peered":"2024-06-14T22:54:06.282019+0000","last_unstale":"2024-06-14T23:09:50.274639+0000","last_undegraded":"2024-06-14T23:09:50.274639+0000","last_fullsized":"2024-06-14T23:09:50.274639+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:37:34.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":264,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275697+0000","last_change":"2024-06-14T22:54:06.281907+0000","last_active":"2024-06-14T23:09:50.275697+0000","last_peered":"2024-06-14T23:09:50.275697+0000","last_clean":"2024-06-14T23:09:50.275697+0000","last_became_active":"2024-06-14T22:54:06.279263+0000","last_became_peered":"2024-06-14T22:54:06.279263+0000","last_unstale":"2024-06-14T23:09:50.275697+0000","last_undegraded":"2024-06-14T23:09:50.275697+0000","last_fullsized":"2024-06-14T23:09:50.275697+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:29:20.328111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":679,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:53.260428+0000","last_change":"2024-06-14T23:09:53.260390+0000","last_active":"2024-06-14T23:09:53.260428+0000","last_peered":"2024-06-14T23:09:53.260428+0000","last_clean":"2024-06-14T23:09:53.260428+0000","last_became_active":"2024-06-14T23:09:12.038891+0000","last_became_peered":"2024-06-14T23:09:12.038891+0000","last_unstale":"2024-06-14T23:09:53.260428+0000","last_undegraded":"2024-06-14T23:09:53.260428+0000","last_fullsized":"2024-06-14T23:09:53.260428+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:10:25.737566+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"305'686","reported_seq":714,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:55.562903+0000","last_change":"2024-06-14T23:09:41.990888+0000","last_active":"2024-06-14T23:09:55.562903+0000","last_peered":"2024-06-14T23:09:55.562903+0000","last_clean":"2024-06-14T23:09:55.562903+0000","last_became_active":"2024-06-14T23:09:41.990527+0000","last_became_peered":"2024-06-14T23:09:41.990527+0000","last_unstale":"2024-06-14T23:09:55.562903+0000","last_undegraded":"2024-06-14T23:09:55.562903+0000","last_fullsized":"2024-06-14T23:09:55.562903+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:54:12.823411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44957696,"num_objects":686,"num_object_clones":0,"num_object_copies":2058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":686,"num_write_kb":43904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":282,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032853+0000","last_change":"2024-06-14T23:07:58.023766+0000","last_active":"2024-06-14T23:09:50.032853+0000","last_peered":"2024-06-14T23:09:50.032853+0000","last_clean":"2024-06-14T23:09:50.032853+0000","last_became_active":"2024-06-14T23:07:58.023506+0000","last_became_peered":"2024-06-14T23:07:58.023506+0000","last_unstale":"2024-06-14T23:09:50.032853+0000","last_undegraded":"2024-06-14T23:09:50.032853+0000","last_fullsized":"2024-06-14T23:09:50.032853+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:46:38.473906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":286,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032044+0000","last_change":"2024-06-14T23:04:34.896257+0000","last_active":"2024-06-14T23:09:50.032044+0000","last_peered":"2024-06-14T23:09:50.032044+0000","last_clean":"2024-06-14T23:09:50.032044+0000","last_became_active":"2024-06-14T23:04:34.895926+0000","last_became_peered":"2024-06-14T23:04:34.895926+0000","last_unstale":"2024-06-14T23:09:50.032044+0000","last_undegraded":"2024-06-14T23:09:50.032044+0000","last_fullsized":"2024-06-14T23:09:50.032044+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:17.800902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":627,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:57.265131+0000","last_change":"2024-06-14T23:09:57.265103+0000","last_active":"2024-06-14T23:09:57.265131+0000","last_peered":"2024-06-14T23:09:57.265131+0000","last_clean":"2024-06-14T23:09:57.265131+0000","last_became_active":"2024-06-14T23:09:44.362891+0000","last_became_peered":"2024-06-14T23:09:44.362891+0000","last_unstale":"2024-06-14T23:09:57.265131+0000","last_undegraded":"2024-06-14T23:09:57.265131+0000","last_fullsized":"2024-06-14T23:09:57.265131+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:31:37.652182+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"305'826","reported_seq":844,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:56.284302+0000","last_change":"2024-06-14T23:09:29.409026+0000","last_active":"2024-06-14T23:09:56.284302+0000","last_peered":"2024-06-14T23:09:56.284302+0000","last_clean":"2024-06-14T23:09:56.284302+0000","last_became_active":"2024-06-14T23:09:29.408554+0000","last_became_peered":"2024-06-14T23:09:29.408554+0000","last_unstale":"2024-06-14T23:09:56.284302+0000","last_undegraded":"2024-06-14T23:09:56.284302+0000","last_fullsized":"2024-06-14T23:09:56.284302+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:52:33.913591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54132736,"num_objects":826,"num_object_clones":0,"num_object_copies":2478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":826,"num_write_kb":52864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":295,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032666+0000","last_change":"2024-06-14T23:04:34.887791+0000","last_active":"2024-06-14T23:09:50.032666+0000","last_peered":"2024-06-14T23:09:50.032666+0000","last_clean":"2024-06-14T23:09:50.032666+0000","last_became_active":"2024-06-14T23:04:34.887099+0000","last_became_peered":"2024-06-14T23:04:34.887099+0000","last_unstale":"2024-06-14T23:09:50.032666+0000","last_undegraded":"2024-06-14T23:09:50.032666+0000","last_fullsized":"2024-06-14T23:09:50.032666+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:46:07.470301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":163,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.031990+0000","last_change":"2024-06-14T23:07:58.023693+0000","last_active":"2024-06-14T23:09:50.031990+0000","last_peered":"2024-06-14T23:09:50.031990+0000","last_clean":"2024-06-14T23:09:50.031990+0000","last_became_active":"2024-06-14T23:07:58.023441+0000","last_became_peered":"2024-06-14T23:07:58.023441+0000","last_unstale":"2024-06-14T23:09:50.031990+0000","last_undegraded":"2024-06-14T23:09:50.031990+0000","last_fullsized":"2024-06-14T23:09:50.031990+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:08:47.037322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":670,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:51.280808+0000","last_change":"2024-06-14T23:09:51.280777+0000","last_active":"2024-06-14T23:09:51.280808+0000","last_peered":"2024-06-14T23:09:51.280808+0000","last_clean":"2024-06-14T23:09:51.280808+0000","last_became_active":"2024-06-14T23:09:12.032943+0000","last_became_peered":"2024-06-14T23:09:12.032943+0000","last_unstale":"2024-06-14T23:09:51.280808+0000","last_undegraded":"2024-06-14T23:09:51.280808+0000","last_fullsized":"2024-06-14T23:09:51.280808+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:21:25.480521+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"305'855","reported_seq":873,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:56.285924+0000","last_change":"2024-06-14T23:09:29.407269+0000","last_active":"2024-06-14T23:09:56.285924+0000","last_peered":"2024-06-14T23:09:56.285924+0000","last_clean":"2024-06-14T23:09:56.285924+0000","last_became_active":"2024-06-14T23:09:29.407017+0000","last_became_peered":"2024-06-14T23:09:29.407017+0000","last_unstale":"2024-06-14T23:09:56.285924+0000","last_undegraded":"2024-06-14T23:09:56.285924+0000","last_fullsized":"2024-06-14T23:09:56.285924+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:13.493974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56033280,"num_objects":855,"num_object_clones":0,"num_object_copies":2565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":855,"num_write_kb":54720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":282,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032555+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:09:50.032555+0000","last_peered":"2024-06-14T23:09:50.032555+0000","last_clean":"2024-06-14T23:09:50.032555+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:09:50.032555+0000","last_undegraded":"2024-06-14T23:09:50.032555+0000","last_fullsized":"2024-06-14T23:09:50.032555+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":89,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032408+0000","last_change":"2024-06-14T23:09:42.004061+0000","last_active":"2024-06-14T23:09:50.032408+0000","last_peered":"2024-06-14T23:09:50.032408+0000","last_clean":"2024-06-14T23:09:50.032408+0000","last_became_active":"2024-06-14T23:09:41.996133+0000","last_became_peered":"2024-06-14T23:09:41.996133+0000","last_unstale":"2024-06-14T23:09:50.032408+0000","last_undegraded":"2024-06-14T23:09:50.032408+0000","last_fullsized":"2024-06-14T23:09:50.032408+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:09:39.715596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":580,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:54.656248+0000","last_change":"2024-06-14T23:09:54.656331+0000","last_active":"2024-06-14T23:09:54.656248+0000","last_peered":"2024-06-14T23:09:54.656248+0000","last_clean":"2024-06-14T23:09:54.656248+0000","last_became_active":"2024-06-14T23:09:12.029935+0000","last_became_peered":"2024-06-14T23:09:12.029935+0000","last_unstale":"2024-06-14T23:09:54.656248+0000","last_undegraded":"2024-06-14T23:09:54.656248+0000","last_fullsized":"2024-06-14T23:09:54.656248+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.656210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.656210+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:34:51.790531+0000","scrub_duration":0.0026539979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"305'1252","reported_seq":1280,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:58.221211+0000","last_change":"2024-06-14T23:09:41.992020+0000","last_active":"2024-06-14T23:09:58.221211+0000","last_peered":"2024-06-14T23:09:58.221211+0000","last_clean":"2024-06-14T23:09:58.221211+0000","last_became_active":"2024-06-14T23:09:41.991722+0000","last_became_peered":"2024-06-14T23:09:41.991722+0000","last_unstale":"2024-06-14T23:09:58.221211+0000","last_undegraded":"2024-06-14T23:09:58.221211+0000","last_fullsized":"2024-06-14T23:09:58.221211+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":1252,"log_dups_size":0,"ondisk_log_size":1252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:35:54.751263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82051072,"num_objects":1252,"num_object_clones":0,"num_object_copies":3756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":80128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":92,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032189+0000","last_change":"2024-06-14T23:09:41.997273+0000","last_active":"2024-06-14T23:09:50.032189+0000","last_peered":"2024-06-14T23:09:50.032189+0000","last_clean":"2024-06-14T23:09:50.032189+0000","last_became_active":"2024-06-14T23:09:41.996543+0000","last_became_peered":"2024-06-14T23:09:41.996543+0000","last_unstale":"2024-06-14T23:09:50.032189+0000","last_undegraded":"2024-06-14T23:09:50.032189+0000","last_fullsized":"2024-06-14T23:09:50.032189+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:31.805297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":283,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033544+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:09:50.033544+0000","last_peered":"2024-06-14T23:09:50.033544+0000","last_clean":"2024-06-14T23:09:50.033544+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:09:50.033544+0000","last_undegraded":"2024-06-14T23:09:50.033544+0000","last_fullsized":"2024-06-14T23:09:50.033544+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":543,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032620+0000","last_change":"2024-06-14T23:09:17.382466+0000","last_active":"2024-06-14T23:09:50.032620+0000","last_peered":"2024-06-14T23:09:50.032620+0000","last_clean":"2024-06-14T23:09:50.032620+0000","last_became_active":"2024-06-14T23:09:12.030205+0000","last_became_peered":"2024-06-14T23:09:12.030205+0000","last_unstale":"2024-06-14T23:09:50.032620+0000","last_undegraded":"2024-06-14T23:09:50.032620+0000","last_fullsized":"2024-06-14T23:09:50.032620+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:09:17.382389+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:33.121472+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"305'1140","reported_seq":1156,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:58.211751+0000","last_change":"2024-06-14T23:09:29.408859+0000","last_active":"2024-06-14T23:09:58.211751+0000","last_peered":"2024-06-14T23:09:58.211751+0000","last_clean":"2024-06-14T23:09:58.211751+0000","last_became_active":"2024-06-14T23:09:29.408548+0000","last_became_peered":"2024-06-14T23:09:29.408548+0000","last_unstale":"2024-06-14T23:09:58.211751+0000","last_undegraded":"2024-06-14T23:09:58.211751+0000","last_fullsized":"2024-06-14T23:09:58.211751+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:19.810173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74711040,"num_objects":1140,"num_object_clones":0,"num_object_copies":3420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1140,"num_write_kb":72960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":246,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.034015+0000","last_change":"2024-06-14T23:07:58.028433+0000","last_active":"2024-06-14T23:09:50.034015+0000","last_peered":"2024-06-14T23:09:50.034015+0000","last_clean":"2024-06-14T23:09:50.034015+0000","last_became_active":"2024-06-14T23:07:58.023997+0000","last_became_peered":"2024-06-14T23:07:58.023997+0000","last_unstale":"2024-06-14T23:09:50.034015+0000","last_undegraded":"2024-06-14T23:09:50.034015+0000","last_fullsized":"2024-06-14T23:09:50.034015+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:36:30.743683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":200,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033699+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:09:50.033699+0000","last_peered":"2024-06-14T23:09:50.033699+0000","last_clean":"2024-06-14T23:09:50.033699+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:09:50.033699+0000","last_undegraded":"2024-06-14T23:09:50.033699+0000","last_fullsized":"2024-06-14T23:09:50.033699+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":584,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:52.265831+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:09:52.265831+0000","last_peered":"2024-06-14T23:09:52.265831+0000","last_clean":"2024-06-14T23:09:52.265831+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:09:52.265831+0000","last_undegraded":"2024-06-14T23:09:52.265831+0000","last_fullsized":"2024-06-14T23:09:52.265831+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"305'837","reported_seq":855,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:56.289517+0000","last_change":"2024-06-14T23:09:29.403408+0000","last_active":"2024-06-14T23:09:56.289517+0000","last_peered":"2024-06-14T23:09:56.289517+0000","last_clean":"2024-06-14T23:09:56.289517+0000","last_became_active":"2024-06-14T23:09:29.403000+0000","last_became_peered":"2024-06-14T23:09:29.403000+0000","last_unstale":"2024-06-14T23:09:56.289517+0000","last_undegraded":"2024-06-14T23:09:56.289517+0000","last_fullsized":"2024-06-14T23:09:56.289517+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:16.838865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54853632,"num_objects":837,"num_object_clones":0,"num_object_copies":2511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":837,"num_write_kb":53568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":282,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.031964+0000","last_change":"2024-06-14T22:54:06.281348+0000","last_active":"2024-06-14T23:09:50.031964+0000","last_peered":"2024-06-14T23:09:50.031964+0000","last_clean":"2024-06-14T23:09:50.031964+0000","last_became_active":"2024-06-14T22:54:06.281052+0000","last_became_peered":"2024-06-14T22:54:06.281052+0000","last_unstale":"2024-06-14T23:09:50.031964+0000","last_undegraded":"2024-06-14T23:09:50.031964+0000","last_fullsized":"2024-06-14T23:09:50.031964+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:35.834762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":256,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.169126+0000","last_change":"2024-06-14T22:58:32.753327+0000","last_active":"2024-06-14T23:09:50.169126+0000","last_peered":"2024-06-14T23:09:50.169126+0000","last_clean":"2024-06-14T23:09:50.169126+0000","last_became_active":"2024-06-14T22:58:32.752970+0000","last_became_peered":"2024-06-14T22:58:32.752970+0000","last_unstale":"2024-06-14T23:09:50.169126+0000","last_undegraded":"2024-06-14T23:09:50.169126+0000","last_fullsized":"2024-06-14T23:09:50.169126+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:20:41.419919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":699,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275216+0000","last_change":"2024-06-14T23:09:44.365846+0000","last_active":"2024-06-14T23:09:50.275216+0000","last_peered":"2024-06-14T23:09:50.275216+0000","last_clean":"2024-06-14T23:09:50.275216+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:09:50.275216+0000","last_undegraded":"2024-06-14T23:09:50.275216+0000","last_fullsized":"2024-06-14T23:09:50.275216+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.188207+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019682448000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"305'1177","reported_seq":1195,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:58.218422+0000","last_change":"2024-06-14T23:09:29.409002+0000","last_active":"2024-06-14T23:09:58.218422+0000","last_peered":"2024-06-14T23:09:58.218422+0000","last_clean":"2024-06-14T23:09:58.218422+0000","last_became_active":"2024-06-14T23:09:29.408690+0000","last_became_peered":"2024-06-14T23:09:29.408690+0000","last_unstale":"2024-06-14T23:09:58.218422+0000","last_undegraded":"2024-06-14T23:09:58.218422+0000","last_fullsized":"2024-06-14T23:09:58.218422+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":1177,"log_dups_size":0,"ondisk_log_size":1177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:12:23.845758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77135872,"num_objects":1177,"num_object_clones":0,"num_object_copies":3531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1177,"num_write_kb":75328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":265,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033047+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:09:50.033047+0000","last_peered":"2024-06-14T23:09:50.033047+0000","last_clean":"2024-06-14T23:09:50.033047+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:09:50.033047+0000","last_undegraded":"2024-06-14T23:09:50.033047+0000","last_fullsized":"2024-06-14T23:09:50.033047+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":269,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.169546+0000","last_change":"2024-06-14T23:09:41.999005+0000","last_active":"2024-06-14T23:09:50.169546+0000","last_peered":"2024-06-14T23:09:50.169546+0000","last_clean":"2024-06-14T23:09:50.169546+0000","last_became_active":"2024-06-14T23:09:41.992436+0000","last_became_peered":"2024-06-14T23:09:41.992436+0000","last_unstale":"2024-06-14T23:09:50.169546+0000","last_undegraded":"2024-06-14T23:09:50.169546+0000","last_fullsized":"2024-06-14T23:09:50.169546+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:49:18.960678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":701,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-06-14T23:09:57.146977+0000","last_change":"2024-06-14T23:09:57.146929+0000","last_active":"2024-06-14T23:09:57.146977+0000","last_peered":"2024-06-14T23:09:57.146977+0000","last_clean":"2024-06-14T23:09:57.146977+0000","last_became_active":"2024-06-14T23:09:12.027671+0000","last_became_peered":"2024-06-14T23:09:12.027671+0000","last_unstale":"2024-06-14T23:09:57.146977+0000","last_undegraded":"2024-06-14T23:09:57.146977+0000","last_fullsized":"2024-06-14T23:09:57.146977+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:43:36.130883+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":291,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033667+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:09:50.033667+0000","last_peered":"2024-06-14T23:09:50.033667+0000","last_clean":"2024-06-14T23:09:50.033667+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:09:50.033667+0000","last_undegraded":"2024-06-14T23:09:50.033667+0000","last_fullsized":"2024-06-14T23:09:50.033667+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":262,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275379+0000","last_change":"2024-06-14T22:54:06.280927+0000","last_active":"2024-06-14T23:09:50.275379+0000","last_peered":"2024-06-14T23:09:50.275379+0000","last_clean":"2024-06-14T23:09:50.275379+0000","last_became_active":"2024-06-14T22:54:06.280587+0000","last_became_peered":"2024-06-14T22:54:06.280587+0000","last_unstale":"2024-06-14T23:09:50.275379+0000","last_undegraded":"2024-06-14T23:09:50.275379+0000","last_fullsized":"2024-06-14T23:09:50.275379+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:24:53.788268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":640,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033170+0000","last_change":"2024-06-14T23:07:59.023708+0000","last_active":"2024-06-14T23:09:50.033170+0000","last_peered":"2024-06-14T23:09:50.033170+0000","last_clean":"2024-06-14T23:09:50.033170+0000","last_became_active":"2024-06-14T23:07:59.023129+0000","last_became_peered":"2024-06-14T23:07:59.023129+0000","last_unstale":"2024-06-14T23:09:50.033170+0000","last_undegraded":"2024-06-14T23:09:50.033170+0000","last_fullsized":"2024-06-14T23:09:50.033170+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:45:29.886582+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":189,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.031766+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:09:50.031766+0000","last_peered":"2024-06-14T23:09:50.031766+0000","last_clean":"2024-06-14T23:09:50.031766+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:09:50.031766+0000","last_undegraded":"2024-06-14T23:09:50.031766+0000","last_fullsized":"2024-06-14T23:09:50.031766+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":152,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275617+0000","last_change":"2024-06-14T23:09:41.985588+0000","last_active":"2024-06-14T23:09:50.275617+0000","last_peered":"2024-06-14T23:09:50.275617+0000","last_clean":"2024-06-14T23:09:50.275617+0000","last_became_active":"2024-06-14T23:09:41.985298+0000","last_became_peered":"2024-06-14T23:09:41.985298+0000","last_unstale":"2024-06-14T23:09:50.275617+0000","last_undegraded":"2024-06-14T23:09:50.275617+0000","last_fullsized":"2024-06-14T23:09:50.275617+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:37:49.953871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":596,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.274849+0000","last_change":"2024-06-14T23:09:26.975673+0000","last_active":"2024-06-14T23:09:50.274849+0000","last_peered":"2024-06-14T23:09:50.274849+0000","last_clean":"2024-06-14T23:09:50.274849+0000","last_became_active":"2024-06-14T23:09:26.974706+0000","last_became_peered":"2024-06-14T23:09:26.974706+0000","last_unstale":"2024-06-14T23:09:50.274849+0000","last_undegraded":"2024-06-14T23:09:50.274849+0000","last_fullsized":"2024-06-14T23:09:50.274849+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.704172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.704172+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001486769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":96,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.168577+0000","last_change":"2024-06-14T23:09:41.990264+0000","last_active":"2024-06-14T23:09:50.168577+0000","last_peered":"2024-06-14T23:09:50.168577+0000","last_clean":"2024-06-14T23:09:50.168577+0000","last_became_active":"2024-06-14T23:09:41.989903+0000","last_became_peered":"2024-06-14T23:09:41.989903+0000","last_unstale":"2024-06-14T23:09:50.168577+0000","last_undegraded":"2024-06-14T23:09:50.168577+0000","last_fullsized":"2024-06-14T23:09:50.168577+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:21:55.165325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":265,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033893+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:09:50.033893+0000","last_peered":"2024-06-14T23:09:50.033893+0000","last_clean":"2024-06-14T23:09:50.033893+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:09:50.033893+0000","last_undegraded":"2024-06-14T23:09:50.033893+0000","last_fullsized":"2024-06-14T23:09:50.033893+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":624,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.032880+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:09:50.032880+0000","last_peered":"2024-06-14T23:09:50.032880+0000","last_clean":"2024-06-14T23:09:50.032880+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:09:50.032880+0000","last_undegraded":"2024-06-14T23:09:50.032880+0000","last_fullsized":"2024-06-14T23:09:50.032880+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":300,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.168440+0000","last_change":"2024-06-14T23:04:34.895560+0000","last_active":"2024-06-14T23:09:50.168440+0000","last_peered":"2024-06-14T23:09:50.168440+0000","last_clean":"2024-06-14T23:09:50.168440+0000","last_became_active":"2024-06-14T23:04:34.887055+0000","last_became_peered":"2024-06-14T23:04:34.887055+0000","last_unstale":"2024-06-14T23:09:50.168440+0000","last_undegraded":"2024-06-14T23:09:50.168440+0000","last_fullsized":"2024-06-14T23:09:50.168440+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:26:36.259241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":136,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.275322+0000","last_change":"2024-06-14T23:09:26.975559+0000","last_active":"2024-06-14T23:09:50.275322+0000","last_peered":"2024-06-14T23:09:50.275322+0000","last_clean":"2024-06-14T23:09:50.275322+0000","last_became_active":"2024-06-14T23:09:26.975226+0000","last_became_peered":"2024-06-14T23:09:26.975226+0000","last_unstale":"2024-06-14T23:09:50.275322+0000","last_undegraded":"2024-06-14T23:09:50.275322+0000","last_fullsized":"2024-06-14T23:09:50.275322+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:03:18.213961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":700,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-06-14T23:09:50.033083+0000","last_change":"2024-06-14T23:09:44.361304+0000","last_active":"2024-06-14T23:09:50.033083+0000","last_peered":"2024-06-14T23:09:50.033083+0000","last_clean":"2024-06-14T23:09:50.033083+0000","last_became_active":"2024-06-14T23:09:44.360985+0000","last_became_peered":"2024-06-14T23:09:44.360985+0000","last_unstale":"2024-06-14T23:09:50.033083+0000","last_undegraded":"2024-06-14T23:09:50.033083+0000","last_fullsized":"2024-06-14T23:09:50.033083+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:50:12.289879+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":980746240,"num_objects":14965,"num_object_clones":0,"num_object_copies":44895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14965,"num_write_kb":957760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2643656704,"data_stored":2643656704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14965,"ondisk_log_size":14965,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":660,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19646,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":457836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":48,"num_write":13,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":421602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234431,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1434032,"kb_used_data":526888,"kb_used_omap":1,"kb_used_meta":907134,"kb_avail":92311120,"statfs":{"total":95995035648,"available":94526586880,"internally_reserved":0,"allocated":539533312,"data_stored":538003162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":928905560},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:09:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.71699999999999997,"15min":0.92900000000000005},"min":{"1min":0.28399999999999997,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":2.6749999999999998,"5min":3.2799999999999998,"15min":6.9539999999999997},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.88600000000000001,"15min":1.0700000000000001},"min":{"1min":0.26600000000000001,"5min":0.224,"15min":0.20300000000000001},"max":{"1min":2.9630000000000001,"5min":4.3550000000000004,"15min":6.9550000000000001},"last":0.76200000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:09:06 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.3109999999999999,"15min":1.714},"min":{"1min":0.23599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.9910000000000001,"5min":12.010999999999999,"15min":83.295000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.175,"5min":1.486,"15min":1.736},"min":{"1min":0.308,"5min":0.27500000000000002,"15min":0.22900000000000001},"max":{"1min":3.3919999999999999,"5min":13.531000000000001,"15min":82.972999999999999},"last":0.68000000000000005}]},{"osd":2,"last update":"Fri Jun 14 23:09:06 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.4299999999999999,"15min":1.1479999999999999},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":2.637,"5min":20.390000000000001,"15min":20.390000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.4219999999999999,"15min":1.234},"min":{"1min":0.29999999999999999,"5min":0.27900000000000003,"15min":0.252},"max":{"1min":3.0779999999999998,"5min":20.327999999999999,"15min":20.327999999999999},"last":0.72499999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:09:26 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":1.508,"15min":1.54},"min":{"1min":0.313,"5min":0.28699999999999998,"15min":0.28199999999999997},"max":{"1min":12.532,"5min":20.645,"15min":55.683999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.091,"5min":1.623,"15min":1.587},"min":{"1min":0.34999999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.1120000000000001,"5min":22.042999999999999,"15min":55.533999999999999},"last":0.81699999999999995}]},{"osd":4,"last update":"Fri Jun 14 23:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.77500000000000002,"15min":0.92700000000000005},"min":{"1min":0.26700000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.4590000000000001,"5min":5.3810000000000002,"15min":11.342000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.995,"5min":0.75600000000000001,"15min":0.98199999999999998},"min":{"1min":0.254,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.1459999999999999,"5min":7.0960000000000001,"15min":16.879000000000001},"last":0.89000000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.75900000000000001,"15min":1.0209999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":3.0150000000000001,"5min":4.383,"15min":6.9000000000000004},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.0800000000000001,"5min":0.89500000000000002,"15min":1.018},"min":{"1min":0.33100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.032,"5min":5.4240000000000004,"15min":6.8529999999999998},"last":1.6040000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:08:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":5.5919999999999996,"5min":5.5919999999999996,"15min":5.5919999999999996},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.1520000000000001},"last":1.637}]}]},{"osd":6,"up_from":272,"seq":1168231104537,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":992736,"kb_used_data":599768,"kb_used_omap":7,"kb_used_meta":392952,"kb_avail":92752416,"statfs":{"total":95995035648,"available":94978473984,"internally_reserved":0,"allocated":614162432,"data_stored":612648223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":402383290},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.877,"15min":1.877},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":6.6440000000000001,"5min":6.6440000000000001,"15min":6.6440000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.925,"5min":1.925,"15min":1.925},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":6.6790000000000003,"5min":6.6790000000000003,"15min":6.6790000000000003},"last":0.85599999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:09:01 2024","interfaces":[{"interface":"back","average":{"1min":2.476,"5min":2.476,"15min":2.476},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":8.173,"5min":8.173,"15min":8.173},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.3759999999999999,"5min":2.3759999999999999,"15min":2.3759999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":10.380000000000001,"5min":10.380000000000001,"15min":10.380000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.1299999999999999,"5min":5.1299999999999999,"15min":5.1299999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":7.9359999999999999,"5min":7.9359999999999999,"15min":7.9359999999999999},"last":0.80800000000000005}]},{"osd":3,"last update":"Fri Jun 14 23:09:01 2024","interfaces":[{"interface":"back","average":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":6.7089999999999996,"5min":6.7089999999999996,"15min":6.7089999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":6.6619999999999999,"5min":6.6619999999999999,"15min":6.6619999999999999},"last":0.877}]},{"osd":4,"last update":"Fri Jun 14 23:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":5.7389999999999999,"5min":5.7389999999999999,"15min":5.7389999999999999},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":5.2089999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":0.438}]},{"osd":5,"last update":"Fri Jun 14 23:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":7.7469999999999999,"5min":7.7469999999999999,"15min":7.7469999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":7.7640000000000002,"5min":7.7640000000000002,"15min":7.7640000000000002},"last":0.73499999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:09:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":5.5049999999999999,"5min":5.5049999999999999,"15min":5.5049999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":7.4619999999999997,"5min":7.4619999999999997,"15min":7.4619999999999997},"last":1.014}]}]},{"osd":5,"up_from":38,"seq":163208757706,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1029656,"kb_used_data":138512,"kb_used_omap":1,"kb_used_meta":891134,"kb_avail":92715496,"statfs":{"total":95995035648,"available":94940667904,"internally_reserved":0,"allocated":141836288,"data_stored":140326059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912521560},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.95499999999999996,"15min":1.353},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.0579999999999998,"5min":6.1950000000000003,"15min":14.94},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.878,"5min":1.232,"15min":1.3899999999999999},"min":{"1min":0.26100000000000001,"5min":0.22800000000000001,"15min":0.219},"max":{"1min":4.181,"5min":8.3520000000000003,"15min":9.3819999999999997},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Jun 14 23:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.238,"15min":1.014},"min":{"1min":0.26900000000000002,"5min":0.26100000000000001,"15min":0.23999999999999999},"max":{"1min":4.1589999999999998,"5min":7.4370000000000003,"15min":7.4370000000000003},"last":0.245},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.1920000000000002,"5min":5.1920000000000002,"15min":9.5039999999999996},"last":0.59299999999999997}]},{"osd":2,"last update":"Fri Jun 14 23:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.222,"15min":1.234},"min":{"1min":0.30199999999999999,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":4.0970000000000004,"5min":4.0970000000000004,"15min":14.891},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.196,"15min":1.2450000000000001},"min":{"1min":0.34100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.4580000000000002,"5min":6.2080000000000002,"15min":9.532},"last":0.624}]},{"osd":3,"last update":"Fri Jun 14 23:09:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.458,"15min":1.643},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":4.3949999999999996,"5min":8.4350000000000005,"15min":45.744},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.3140000000000001,"15min":1.657},"min":{"1min":0.41899999999999998,"5min":0.29299999999999998,"15min":0.28899999999999998},"max":{"1min":4.2560000000000002,"5min":7.4290000000000003,"15min":45.838000000000001},"last":1.153}]},{"osd":4,"last update":"Fri Jun 14 23:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.78900000000000003,"15min":0.85199999999999998},"min":{"1min":0.27900000000000003,"5min":0.255,"15min":0.255},"max":{"1min":3.012,"5min":6.9119999999999999,"15min":14.352},"last":1.726},{"interface":"front","average":{"1min":1.008,"5min":0.94399999999999995,"15min":1.032},"min":{"1min":0.309,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":4.7649999999999997,"5min":8.4039999999999999,"15min":9.5980000000000008},"last":0.308}]},{"osd":6,"last update":"Fri Jun 14 23:08:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":9.4710000000000001,"5min":9.4710000000000001,"15min":9.4710000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":10.632,"5min":10.632,"15min":10.632},"last":0.47199999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":0.97799999999999998,"15min":1.105},"min":{"1min":0.29099999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.2240000000000002,"5min":6.29,"15min":14.818},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.997,"15min":1.0329999999999999},"min":{"1min":0.35299999999999998,"5min":0.28699999999999998,"15min":0.27000000000000002},"max":{"1min":5.1239999999999997,"5min":6.9779999999999998,"15min":14.429},"last":1.3400000000000001}]}]},{"osd":4,"up_from":30,"seq":128849019342,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1194788,"kb_used_data":304092,"kb_used_omap":1,"kb_used_meta":890686,"kb_avail":92550364,"statfs":{"total":95995035648,"available":94771572736,"internally_reserved":0,"allocated":311390208,"data_stored":309871467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912062808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:09:50 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.79900000000000004,"15min":1.093},"min":{"1min":0.31900000000000001,"5min":0.217,"15min":0.17100000000000001},"max":{"1min":2.4049999999999998,"5min":6.8440000000000003,"15min":10.567},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.83999999999999997,"15min":1.089},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":6.1379999999999999,"5min":7.4900000000000002,"15min":8.3390000000000004},"last":0.33900000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:09:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.4079999999999999,"15min":1.48},"min":{"1min":0.32500000000000001,"5min":0.26400000000000001,"15min":0.23100000000000001},"max":{"1min":3.6930000000000001,"5min":42.656999999999996,"15min":56.917000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.363,"15min":1.536},"min":{"1min":0.36299999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.1150000000000002,"5min":42.192999999999998,"15min":56.789999999999999},"last":0.41999999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:09:56 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":2.1459999999999999,"15min":1.4179999999999999},"min":{"1min":0.34200000000000003,"5min":0.248,"15min":0.23000000000000001},"max":{"1min":1.5449999999999999,"5min":46.866,"15min":46.866},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":2.0379999999999998,"15min":1.456},"min":{"1min":0.39300000000000002,"5min":0.28299999999999997,"15min":0.27600000000000002},"max":{"1min":1.5309999999999999,"5min":46.851999999999997,"15min":46.851999999999997},"last":0.39200000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:09:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":1.2869999999999999,"15min":1.2190000000000001},"min":{"1min":0.34699999999999998,"5min":0.25,"15min":0.25},"max":{"1min":1.5669999999999999,"5min":42.445999999999998,"15min":42.445999999999998},"last":1.2689999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.329,"15min":1.3400000000000001},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.27500000000000002},"max":{"1min":2.6589999999999998,"5min":42.151000000000003,"15min":42.151000000000003},"last":0.53600000000000003}]},{"osd":5,"last update":"Fri Jun 14 23:09:24 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.69899999999999995,"15min":1.508},"min":{"1min":0.28199999999999997,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.9670000000000001,"5min":3.9670000000000001,"15min":139.13200000000001},"last":0.373},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.75,"15min":1.5489999999999999},"min":{"1min":0.28100000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.7759999999999998,"5min":4.7759999999999998,"15min":138.86000000000001},"last":0.44400000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":4.665,"5min":4.665,"15min":4.665},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.8890000000000002,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":1.2410000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":0.93899999999999995,"15min":0.96299999999999997},"min":{"1min":0.38300000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.7300000000000004,"5min":6.0890000000000004,"15min":8.9489999999999998},"last":1.984},{"interface":"front","average":{"1min":1.4470000000000001,"5min":0.86299999999999999,"15min":1.028},"min":{"1min":0.42599999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":5.7930000000000001,"5min":5.7930000000000001,"15min":7.25},"last":2.1829999999999998}]}]},{"osd":3,"up_from":138,"seq":592705487040,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":844564,"kb_used_data":269772,"kb_used_omap":4,"kb_used_meta":574779,"kb_avail":92900588,"statfs":{"total":95995035648,"available":95130202112,"internally_reserved":0,"allocated":276246528,"data_stored":274739624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":588574174},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:09:32 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.3839999999999999,"15min":1.329},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.992,"5min":10.602,"15min":11.071999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.427,"15min":1.2649999999999999},"min":{"1min":0.221,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0419999999999998,"5min":11.518000000000001,"15min":11.518000000000001},"last":0.38500000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:09:32 2024","interfaces":[{"interface":"back","average":{"1min":2,"5min":1.5229999999999999,"15min":1.3999999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":22.605,"5min":22.605,"15min":24.097999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":2.3100000000000001,"5min":1.95,"15min":1.5620000000000001},"min":{"1min":0.29799999999999999,"5min":0.25,"15min":0.24099999999999999},"max":{"1min":22.994,"5min":22.994,"15min":24.132999999999999},"last":0.44400000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:09:32 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.502,"15min":1.3140000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":3.5579999999999998,"5min":8.8399999999999999,"15min":9.4730000000000008},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.054,"5min":1.375,"15min":1.2290000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.0720000000000001,"5min":5.8799999999999999,"15min":10.369},"last":0.35699999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:09:32 2024","interfaces":[{"interface":"back","average":{"1min":2.097,"5min":2.383,"15min":1.6299999999999999},"min":{"1min":0.371,"5min":0.28100000000000003,"15min":0.27900000000000003},"max":{"1min":22.643999999999998,"5min":73.263000000000005,"15min":73.263000000000005},"last":0.628},{"interface":"front","average":{"1min":0.77800000000000002,"5min":2.073,"15min":1.536},"min":{"1min":0.27600000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.484,"5min":73.5,"15min":73.5},"last":0.52700000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:09:32 2024","interfaces":[{"interface":"back","average":{"1min":2.202,"5min":2.524,"15min":1.7190000000000001},"min":{"1min":0.33200000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":22.669,"5min":81.438999999999993,"15min":81.438999999999993},"last":0.47399999999999998},{"interface":"front","average":{"1min":2.456,"5min":2.5299999999999998,"15min":1.704},"min":{"1min":0.38700000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":21.846,"5min":81.626999999999995,"15min":81.626999999999995},"last":0.46200000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:08:57 2024","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":5.1749999999999998,"5min":5.1749999999999998,"15min":5.1749999999999998},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.51,"5min":1.51,"15min":1.51},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.8159999999999998,"5min":2.8159999999999998,"15min":2.8159999999999998},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Jun 14 23:09:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":1.2869999999999999,"15min":1.3500000000000001},"min":{"1min":0.41599999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.2109999999999999,"5min":5.8559999999999999,"15min":7.7240000000000002},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.242,"15min":1.256},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":2.6619999999999999,"5min":7.9649999999999999,"15min":7.9649999999999999},"last":0.61899999999999999}]}]},{"osd":2,"up_from":133,"seq":571230650567,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":730712,"kb_used_data":327568,"kb_used_omap":6,"kb_used_meta":403129,"kb_avail":93014440,"statfs":{"total":95995035648,"available":95246786560,"internally_reserved":0,"allocated":335429632,"data_stored":333919390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6667,"internal_metadata":412804597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8440000000000001,"5min":2.3660000000000001,"15min":1.325},"min":{"1min":0.19700000000000001,"5min":0.18099999999999999,"15min":0.16700000000000001},"max":{"1min":7.492,"5min":55.851999999999997,"15min":55.851999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.879,"15min":1.3779999999999999},"min":{"1min":0.20699999999999999,"5min":0.19900000000000001,"15min":0.182},"max":{"1min":6.1760000000000002,"5min":56.171999999999997,"15min":56.171999999999997},"last":0.28399999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.712,"15min":1.3120000000000001},"min":{"1min":0.308,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.093,"5min":9.8140000000000001,"15min":21.920000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":2.161,"5min":1.883,"15min":1.3140000000000001},"min":{"1min":0.253,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":11.791,"5min":14.416,"15min":21.573},"last":0.55900000000000005}]},{"osd":3,"last update":"Fri Jun 14 23:09:39 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":1.3400000000000001,"15min":1.375},"min":{"1min":0.22800000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.4340000000000002,"5min":8.8889999999999993,"15min":21.66},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.4199999999999999,"15min":1.403},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":2.6549999999999998,"5min":10.297000000000001,"15min":22.341000000000001},"last":0.72499999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.3,"15min":1.4710000000000001},"min":{"1min":0.32800000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.4100000000000001,"5min":6.6349999999999998,"15min":31.928000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.9950000000000001,"5min":1.536,"15min":1.444},"min":{"1min":0.36499999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.0459999999999994,"5min":8.0459999999999994,"15min":31.954999999999998},"last":0.58299999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.9510000000000001,"5min":1.6619999999999999,"15min":1.327},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.28299999999999997},"max":{"1min":7.4390000000000001,"5min":9.6310000000000002,"15min":31.539999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.47,"5min":1.327,"15min":1.3169999999999999},"min":{"1min":0.40600000000000003,"5min":0.30499999999999999,"15min":0.29399999999999998},"max":{"1min":6.9630000000000001,"5min":8.8140000000000001,"15min":31.481999999999999},"last":1.47}]},{"osd":6,"last update":"Fri Jun 14 23:09:02 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":1.97,"15min":1.97},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":8.1189999999999998,"5min":8.1189999999999998,"15min":8.1189999999999998},"last":0.86199999999999999},{"interface":"front","average":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.1379999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":8.0700000000000003,"5min":8.0700000000000003,"15min":8.0700000000000003},"last":1.2769999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:09:02 2024","interfaces":[{"interface":"back","average":{"1min":2.137,"5min":1.732,"15min":1.3680000000000001},"min":{"1min":0.55000000000000004,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":8.0099999999999998,"5min":9.7539999999999996,"15min":10.614000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":2.1059999999999999,"5min":1.716,"15min":1.181},"min":{"1min":0.43099999999999999,"5min":0.36599999999999999,"15min":0.35199999999999998},"max":{"1min":7.5209999999999999,"5min":9.5850000000000009,"15min":9.5850000000000009},"last":0.69399999999999995}]}]},{"osd":1,"up_from":116,"seq":498216206584,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":877012,"kb_used_data":200780,"kb_used_omap":5,"kb_used_meta":676218,"kb_avail":92868140,"statfs":{"total":95995035648,"available":95096975360,"internally_reserved":0,"allocated":205598720,"data_stored":204091748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":692447512},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:09:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.639,"15min":1.3620000000000001},"min":{"1min":0.216,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":9.4109999999999996,"5min":9.4109999999999996,"15min":11.257999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.522,"15min":1.3109999999999999},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.186},"max":{"1min":6.7539999999999996,"5min":13.331,"15min":13.672000000000001},"last":0.33100000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:09:07 2024","interfaces":[{"interface":"back","average":{"1min":2.1400000000000001,"5min":1.879,"15min":1.5549999999999999},"min":{"1min":0.28699999999999998,"5min":0.20999999999999999,"15min":0.19500000000000001},"max":{"1min":9.3829999999999991,"5min":14.597,"15min":14.597},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.9930000000000001,"5min":1.6220000000000001,"15min":1.216},"min":{"1min":0.248,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":10.092000000000001,"5min":14.711,"15min":14.711},"last":2.1709999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:09:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.51,"15min":1.46},"min":{"1min":0.26600000000000001,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":3.3519999999999999,"5min":11.454000000000001,"15min":21.422000000000001},"last":2.0009999999999999},{"interface":"front","average":{"1min":1.036,"5min":1.6830000000000001,"15min":1.589},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.214},"max":{"1min":4.2859999999999996,"5min":10.584,"15min":21.016999999999999},"last":1.429}]},{"osd":4,"last update":"Fri Jun 14 23:09:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.21,"15min":1.2010000000000001},"min":{"1min":0.33100000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.2309999999999999,"5min":6.2939999999999996,"15min":7.673},"last":0.749},{"interface":"front","average":{"1min":2.1360000000000001,"5min":1.5620000000000001,"15min":1.4630000000000001},"min":{"1min":0.35499999999999998,"5min":0.33000000000000002,"15min":0.29199999999999998},"max":{"1min":9.9580000000000002,"5min":9.9580000000000002,"15min":9.9580000000000002},"last":1.272}]},{"osd":5,"last update":"Fri Jun 14 23:09:07 2024","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.591,"15min":1.391},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":10.513999999999999,"5min":19.132999999999999,"15min":19.132999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.456,"15min":1.2669999999999999},"min":{"1min":0.35399999999999998,"5min":0.29899999999999999,"15min":0.28599999999999998},"max":{"1min":5.3010000000000002,"5min":19.123000000000001,"15min":19.123000000000001},"last":0.86799999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:09:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.5510000000000002,"5min":6.5510000000000002,"15min":6.5510000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":9.9849999999999994,"5min":9.9849999999999994,"15min":9.9849999999999994},"last":1.3049999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:09:07 2024","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":1.5169999999999999,"15min":1.4550000000000001},"min":{"1min":0.34499999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":9.9280000000000008,"5min":9.9280000000000008,"15min":9.9280000000000008},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.8220000000000001,"5min":1.349,"15min":1.2370000000000001},"min":{"1min":0.35899999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":10.382999999999999,"5min":10.382999999999999,"15min":10.382999999999999},"last":2.0249999999999999}]}]},{"osd":0,"up_from":0,"seq":390842024239,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2152,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273612800,"data_stored":273612800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":401473536,"data_stored":401473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":360251392,"data_stored":360251392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":313131008,"data_stored":313131008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":160235520,"data_stored":160235520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":501219328,"data_stored":501219328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":633733120,"data_stored":633733120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:10:00.910 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-14T23:10:00.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-06-14T23:10:00.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:01.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:01.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:01.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:01.085 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:01.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:01.186 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:01.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:01.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:01.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:01.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:01.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:01.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:01.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:01.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 27120 27104 188.206 149.438 0.00291765 0.00530016 2024-06-14T23:10:01.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:01.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:01.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:01.641 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:10:01.641 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:10:01.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:01.703 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:01 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:10:01.703 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:01 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:10:01.703 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2075734675' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:10:01.703 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:01 smithi137 ceph-mon[90634]: osdmap e306: 8 total, 8 up, 7 in 2024-06-14T23:10:01.703 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:01 smithi137 ceph-mon[90634]: pgmap v1141: 119 pgs: 119 active+clean; 1.2 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-06-14T23:10:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:01.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:01.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:01.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:10:01.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:10:01.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2075734675' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:10:01.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[102754]: osdmap e306: 8 total, 8 up, 7 in 2024-06-14T23:10:01.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[102754]: pgmap v1141: 119 pgs: 119 active+clean; 1.2 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-06-14T23:10:01.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:10:01.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:10:01.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2075734675' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:10:01.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[97913]: osdmap e306: 8 total, 8 up, 7 in 2024-06-14T23:10:01.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:01 smithi086 ceph-mon[97913]: pgmap v1141: 119 pgs: 119 active+clean; 1.2 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-06-14T23:10:01.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:01.874 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:01.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:01.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:02.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:02.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:02.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:02.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:02.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:02.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:02.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:02.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:02.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:02.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:02.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-14T23:10:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:02.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:02.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:02.503 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 16 29590 29574 184.822 154.375 0.00603244 0.00540505 2024-06-14T23:10:02.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:02.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:02.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:02.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:02.700 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:02.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:02.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:02 smithi086 ceph-mon[97913]: 1.1a scrub starts 2024-06-14T23:10:02.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:02 smithi086 ceph-mon[97913]: 1.1a scrub ok 2024-06-14T23:10:02.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:02 smithi086 ceph-mon[97913]: 1.13 scrub starts 2024-06-14T23:10:02.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:02 smithi086 ceph-mon[97913]: 1.13 scrub ok 2024-06-14T23:10:02.739 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:02 smithi086 ceph-mon[102754]: 1.1a scrub starts 2024-06-14T23:10:02.739 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:02 smithi086 ceph-mon[102754]: 1.1a scrub ok 2024-06-14T23:10:02.739 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:02 smithi086 ceph-mon[102754]: 1.13 scrub starts 2024-06-14T23:10:02.739 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:02 smithi086 ceph-mon[102754]: 1.13 scrub ok 2024-06-14T23:10:02.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:02 smithi137 ceph-mon[90634]: 1.1a scrub starts 2024-06-14T23:10:02.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:02 smithi137 ceph-mon[90634]: 1.1a scrub ok 2024-06-14T23:10:02.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:02 smithi137 ceph-mon[90634]: 1.13 scrub starts 2024-06-14T23:10:02.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:02 smithi137 ceph-mon[90634]: 1.13 scrub ok 2024-06-14T23:10:02.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:02.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:02.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:02.853 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:10:02.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:02.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:02.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:03.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:03.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:03.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:03.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:03.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:03.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:03.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:03.504 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 31797 31781 180.559 137.938 0.00601414 0.00553308 2024-06-14T23:10:03.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:03.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:03.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:03.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:03.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:03.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:03.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:03 smithi137 ceph-mon[90634]: 1.1b scrub starts 2024-06-14T23:10:03.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:03 smithi137 ceph-mon[90634]: 1.1b scrub ok 2024-06-14T23:10:03.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:03 smithi137 ceph-mon[90634]: pgmap v1142: 119 pgs: 119 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-06-14T23:10:03.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:03.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:03.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:03.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:03 smithi086 ceph-mon[97913]: 1.1b scrub starts 2024-06-14T23:10:03.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:03 smithi086 ceph-mon[97913]: 1.1b scrub ok 2024-06-14T23:10:03.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:03 smithi086 ceph-mon[97913]: pgmap v1142: 119 pgs: 119 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-06-14T23:10:03.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:03 smithi086 ceph-mon[102754]: 1.1b scrub starts 2024-06-14T23:10:03.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:03 smithi086 ceph-mon[102754]: 1.1b scrub ok 2024-06-14T23:10:03.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:03 smithi086 ceph-mon[102754]: pgmap v1142: 119 pgs: 119 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-06-14T23:10:03.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:03.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:03.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:04.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:04.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:04.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:04.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:04.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:04.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:04.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:04.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:04.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:04.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:04.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:10:04.439 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:04.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:04.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 16 33553 33537 174.657 109.75 0.0114567 0.00571858 2024-06-14T23:10:04.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:04.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:04.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:04.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:04.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:04.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:04.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:04 smithi137 ceph-mon[90634]: 1.3 scrub starts 2024-06-14T23:10:04.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:04 smithi137 ceph-mon[90634]: 1.3 scrub ok 2024-06-14T23:10:04.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:04 smithi137 ceph-mon[90634]: 1.17 scrub starts 2024-06-14T23:10:04.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:04 smithi137 ceph-mon[90634]: 1.17 scrub ok 2024-06-14T23:10:04.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:04.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:04 smithi086 ceph-mon[102754]: 1.3 scrub starts 2024-06-14T23:10:04.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:04 smithi086 ceph-mon[102754]: 1.3 scrub ok 2024-06-14T23:10:04.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:04 smithi086 ceph-mon[102754]: 1.17 scrub starts 2024-06-14T23:10:04.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:04 smithi086 ceph-mon[102754]: 1.17 scrub ok 2024-06-14T23:10:04.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:04 smithi086 ceph-mon[97913]: 1.3 scrub starts 2024-06-14T23:10:04.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:04 smithi086 ceph-mon[97913]: 1.3 scrub ok 2024-06-14T23:10:04.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:04 smithi086 ceph-mon[97913]: 1.17 scrub starts 2024-06-14T23:10:04.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:04 smithi086 ceph-mon[97913]: 1.17 scrub ok 2024-06-14T23:10:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:04.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:04.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:04.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:04.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:04.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:05.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:05.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:05.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:05.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:05.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:05.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:05.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:05.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:05.340 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:10:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:10:04] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.43.0" 2024-06-14T23:10:05.357 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:05.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:05.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:05.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:05.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:05.504 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 35101 35085 168.664 96.75 0.0028038 0.00591503 2024-06-14T23:10:05.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:05.624 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:05.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:05.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:05.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:05 smithi137 ceph-mon[90634]: pgmap v1143: 119 pgs: 119 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 167 MiB/s wr, 2.66k op/s 2024-06-14T23:10:05.754 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:05.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:05.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:05 smithi086 ceph-mon[97913]: pgmap v1143: 119 pgs: 119 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 167 MiB/s wr, 2.66k op/s 2024-06-14T23:10:05.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:05 smithi086 ceph-mon[102754]: pgmap v1143: 119 pgs: 119 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 167 MiB/s wr, 2.66k op/s 2024-06-14T23:10:05.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:05.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:05.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:05.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:06.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:06.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:06.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:06.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:06.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:06.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:06.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:06.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:06.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:06.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:06.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:06.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:06.504 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 36495 36479 162.839 87.125 0.00224221 0.00605057 2024-06-14T23:10:06.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:06.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:06.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:06.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:06.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:06.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:06.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:06.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:06.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:06.934 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:06.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:07.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:07.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:07.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:07.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:07.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:07.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:07.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:07.327 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:07.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:07.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:07.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:07.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:07.504 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 38158 38142 158.911 103.938 0.00653356 0.00628843 2024-06-14T23:10:07.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:07.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:07.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:07.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:07.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:07.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:07.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:07.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:07.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:08.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:08.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:08.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:08.135 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:07 smithi086 ceph-mon[97913]: 1.c scrub starts 2024-06-14T23:10:08.135 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:07 smithi086 ceph-mon[97913]: 1.c scrub ok 2024-06-14T23:10:08.135 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:07 smithi086 ceph-mon[97913]: pgmap v1144: 119 pgs: 119 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 162 MiB/s wr, 2.58k op/s 2024-06-14T23:10:08.136 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:07 smithi086 ceph-mon[102754]: 1.c scrub starts 2024-06-14T23:10:08.136 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:07 smithi086 ceph-mon[102754]: 1.c scrub ok 2024-06-14T23:10:08.136 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:07 smithi086 ceph-mon[102754]: pgmap v1144: 119 pgs: 119 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 162 MiB/s wr, 2.58k op/s 2024-06-14T23:10:08.152 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:08.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:07 smithi137 ceph-mon[90634]: 1.c scrub starts 2024-06-14T23:10:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:07 smithi137 ceph-mon[90634]: 1.c scrub ok 2024-06-14T23:10:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:07 smithi137 ceph-mon[90634]: pgmap v1144: 119 pgs: 119 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 162 MiB/s wr, 2.58k op/s 2024-06-14T23:10:08.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:08.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:08.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:08.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:08.419 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:08.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:08.504 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 40005 39989 156.193 115.438 0.00918017 0.00639753 2024-06-14T23:10:08.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:08.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:08.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:08.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:08.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:08.697 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:08.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:08.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:08.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:08.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:08.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:08.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:08.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:08.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:09.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:09.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:09.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:09.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:09.210 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:09.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:09.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:09.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:10:09.262 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:10:09.301 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:10:09.301 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:10:09.329 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:10:09.329 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:10:09.345 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:10:09.345 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:10:09.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:09.361 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:10:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:10:09.390 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:10:09.390 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:10:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:09.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:09.428 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:10:09.428 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:10:09.459 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:10:09.459 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:10:09.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:09.504 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 16 41755 41739 153.439 109.375 0.00903845 0.00651225 2024-06-14T23:10:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:09.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:09.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:09.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:09.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:09.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:09.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:09.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:09.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:09.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:09.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:09.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:10.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:10.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:10.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:10.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:10.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:10.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:09 smithi137 ceph-mon[90634]: pgmap v1145: 119 pgs: 119 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-06-14T23:10:10.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1178120085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:10.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:09 smithi086 ceph-mon[97913]: pgmap v1145: 119 pgs: 119 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-06-14T23:10:10.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1178120085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:10.288 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:09 smithi086 ceph-mon[102754]: pgmap v1145: 119 pgs: 119 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-06-14T23:10:10.288 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1178120085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:10.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:10.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:10.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:10.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:10.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:10.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:10.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:10.505 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 43630 43614 151.416 117.188 0.00529667 0.00659962 2024-06-14T23:10:10.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:10.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:10.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:10.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:10.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:10.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:10.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:10.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:10.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:10.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:11.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:11.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:11.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:11.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:11.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:11.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:11.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:11.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:11.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:11.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:11.505 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 45381 45365 149.207 109.438 0.00902258 0.00669769 2024-06-14T23:10:11.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:11.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:11.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:11.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:11.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:11.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:11.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:11.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:11.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:11.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:11.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:12.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:12.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:12.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:12.210 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:11 smithi086 ceph-mon[102754]: pgmap v1146: 119 pgs: 119 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-06-14T23:10:12.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:11 smithi086 ceph-mon[97913]: pgmap v1146: 119 pgs: 119 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-06-14T23:10:12.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:11 smithi137 ceph-mon[90634]: pgmap v1146: 119 pgs: 119 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-06-14T23:10:12.255 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:12.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:12.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:12.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:12.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:12.421 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:10:12.421 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:10:12.421 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:10:12.421 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:10:12.421 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:10:12.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:12.505 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:10:12.504879+0000 min lat: 0.00120633 max lat: 0.275433 avg lat: 0.00679031 2024-06-14T23:10:12.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:10:12.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 47112 47096 147.154 108.188 0.00784502 0.00679031 2024-06-14T23:10:12.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:12.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:12.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:12.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:12.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:12.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:12.806 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:12.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:12.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:12.946 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:12.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:13.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:13.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:13.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:13.112 INFO:teuthology.orchestra.run.smithi086.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-06-14T23:10:13.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:13.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-14T23:10:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:13.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:13.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:13.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:13.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:13.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:13.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:13.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:13.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:13.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:13 smithi137 ceph-mon[90634]: pgmap v1147: 119 pgs: 119 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-06-14T23:10:13.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/820996415' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-06-14T23:10:13.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 15 48679 48664 144.813 98 0.0114827 0.00689998 2024-06-14T23:10:13.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:13.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:13.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:13.627 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:10:13.651 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:13.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:13.657 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:13.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:13.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:13 smithi086 ceph-mon[97913]: pgmap v1147: 119 pgs: 119 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-06-14T23:10:13.658 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/820996415' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-06-14T23:10:13.658 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:13.658 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:13.658 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:13 smithi086 ceph-mon[102754]: pgmap v1147: 119 pgs: 119 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-06-14T23:10:13.658 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/820996415' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-06-14T23:10:13.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:13.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:13.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:13.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:13.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:10:13.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:13.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:14.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:14.077 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:14.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:14.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:14.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:14.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:14.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:14.353 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:14.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:14.414 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:10:14.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:14.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:14.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:14.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 50336 50320 142.935 103.5 0.00372173 0.00699094 2024-06-14T23:10:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-14T23:10:14.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:14.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:14.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:14.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:14.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1906051405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:10:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:14.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:14.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:14.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1906051405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:10:14.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1906051405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:10:14.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:14.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:14.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:14.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:14.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:15.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:15.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:15.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:15.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:15.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:15.213 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:10:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:10:14] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.43.0" 2024-06-14T23:10:15.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:15.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:15.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:15.341 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:15.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:15.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:15.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:15.483 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:15 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:10:15.483 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1906051405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:10:15.483 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:15 smithi086 ceph-mon[97913]: osdmap e307: 8 total, 8 up, 7 in 2024-06-14T23:10:15.483 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:15 smithi086 ceph-mon[97913]: pgmap v1149: 119 pgs: 119 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-06-14T23:10:15.483 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:15 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:10:15.483 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1906051405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:10:15.484 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:15 smithi086 ceph-mon[102754]: osdmap e307: 8 total, 8 up, 7 in 2024-06-14T23:10:15.484 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:15 smithi086 ceph-mon[102754]: pgmap v1149: 119 pgs: 119 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-06-14T23:10:15.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 52151 52135 141.652 113.438 0.00731794 0.00705435 2024-06-14T23:10:15.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:15.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:15.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:15.615 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:15 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:10:15.616 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1906051405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:10:15.616 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:15 smithi137 ceph-mon[90634]: osdmap e307: 8 total, 8 up, 7 in 2024-06-14T23:10:15.616 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:15 smithi137 ceph-mon[90634]: pgmap v1149: 119 pgs: 119 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-06-14T23:10:15.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:15.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:15.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:15.775 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252314 10.9 10:960f94a9:::benchmark_data_smithi086_189745_object121346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.011224+0000", 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 354.75937801100002, 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0900273949999999, 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.776 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252314 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011224+0000", 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011224+0000", 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011223+0000", 2024-06-14T23:10:15.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:10:15.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011230+0000", 2024-06-14T23:10:15.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6009999999999997e-06 2024-06-14T23:10:15.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011246+0000", 2024-06-14T23:10:15.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6090999999999999e-05 2024-06-14T23:10:15.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011252+0000", 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9660000000000001e-06 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011316+0000", 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4170000000000004e-05 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011325+0000", 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1210000000000007e-06 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.041481+0000", 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0301560869999999 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.041565+0000", 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4305e-05 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.041641+0000", 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6340000000000004e-05 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.059215+0000", 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017573808 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101090+0000", 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041874407000000002 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101182+0000", 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2931999999999997e-05 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101222+0000", 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9650000000000002e-05 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101252+0000", 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9663999999999999e-05 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.783 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252317 10.9 10:91aeda83:::benchmark_data_smithi086_189745_object121349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.022551+0000", 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 354.74805100600003, 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.078809973, 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252317 2024-06-14T23:10:15.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022551+0000", 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022551+0000", 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022550+0000", 2024-06-14T23:10:15.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022556+0000", 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4269999999999999e-06 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022566+0000", 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0010000000000001e-05 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022570+0000", 2024-06-14T23:10:15.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2189999999999998e-06 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022606+0000", 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5809000000000002e-05 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022610+0000", 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8049999999999997e-06 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.045161+0000", 2024-06-14T23:10:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0225505130000001 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.045235+0000", 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4148999999999998e-05 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.045310+0000", 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4604000000000006e-05 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.059224+0000", 2024-06-14T23:10:15.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013914517 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101136+0000", 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041911325999999999 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101303+0000", 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016734900000000001 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101339+0000", 2024-06-14T23:10:15.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6380000000000001e-05 2024-06-14T23:10:15.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101361+0000", 2024-06-14T23:10:15.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2092000000000001e-05 2024-06-14T23:10:15.791 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.791 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.791 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.791 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252678 10.3 10:ceff32ef:::benchmark_data_smithi086_189745_object121710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.792 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.023839+0000", 2024-06-14T23:10:15.792 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.74676387, 2024-06-14T23:10:15.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98525532999999998, 2024-06-14T23:10:15.792 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.792 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.792 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252678 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023839+0000", 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023839+0000", 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023837+0000", 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023844+0000", 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.455e-06 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023858+0000", 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4619e-05 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023864+0000", 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3279999999999996e-06 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023971+0000", 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010751600000000001 2024-06-14T23:10:15.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023979+0000", 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4390000000000003e-06 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.997909+0000", 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97393003199999995 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.998001+0000", 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2132000000000005e-05 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.998077+0000", 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6552999999999997e-05 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002272+0000", 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041949149999999996 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.008946+0000", 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066735199999999996 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.008996+0000", 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0719e-05 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009049+0000", 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2899999999999998e-05 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009094+0000", 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4501e-05 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252708 10.3 10:c07e5468:::benchmark_data_smithi086_189745_object121740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.055163+0000", 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.71543954600003, 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95415895699999997, 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.800 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252708 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055163+0000", 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055163+0000", 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055162+0000", 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055167+0000", 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2140000000000004e-06 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055177+0000", 2024-06-14T23:10:15.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.013e-05 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055181+0000", 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8460000000000001e-06 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055222+0000", 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0878999999999999e-05 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055230+0000", 2024-06-14T23:10:15.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.276e-06 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002317+0000", 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94708661800000005 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002441+0000", 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124435 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002570+0000", 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012846799999999999 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009132+0000", 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065624999999999998 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009196+0000", 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3718000000000002e-05 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009236+0000", 2024-06-14T23:10:15.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0268000000000001e-05 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009286+0000", 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0078999999999999e-05 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009322+0000", 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5590999999999999e-05 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.806 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252709 10.3 10:c41e1665:::benchmark_data_smithi086_189745_object121741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.057035+0000", 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.71356730000002, 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95242822400000005, 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252709 2024-06-14T23:10:15.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057035+0000", 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057035+0000", 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057034+0000", 2024-06-14T23:10:15.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057039+0000", 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1690000000000002e-06 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057049+0000", 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0148e-05 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057054+0000", 2024-06-14T23:10:15.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7590000000000003e-06 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057093+0000", 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8566000000000003e-05 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057100+0000", 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9859999999999997e-06 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.005750+0000", 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94864968599999999 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.005829+0000", 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9124999999999999e-05 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.005919+0000", 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0216000000000005e-05 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009353+0000", 2024-06-14T23:10:15.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034340680000000002 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009357+0000", 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8970000000000001e-06 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009391+0000", 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4400000000000003e-05 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009429+0000", 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7137e-05 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009463+0000", 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4786000000000002e-05 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252721 10.3 10:cae392c3:::benchmark_data_smithi086_189745_object121753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.813 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.067872+0000", 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.702730428, 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95780526600000004, 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252721 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067872+0000", 2024-06-14T23:10:15.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067872+0000", 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067870+0000", 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067876+0000", 2024-06-14T23:10:15.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6359999999999999e-06 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067888+0000", 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1966e-05 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067894+0000", 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1739999999999997e-06 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067944+0000", 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0259000000000001e-05 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067951+0000", 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8650000000000003e-06 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009492+0000", 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94154064699999995 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009567+0000", 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5333999999999994e-05 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009654+0000", 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6735000000000002e-05 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017286+0000", 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076317240000000003 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017292+0000", 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1759999999999998e-06 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025571+0000", 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082794340000000004 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025636+0000", 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.444e-05 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025677+0000", 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1607999999999999e-05 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252748 10.3 10:c31065d0:::benchmark_data_smithi086_189745_object121780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.118605+0000", 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.651997016, 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90724995399999997, 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.820 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252748 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118605+0000", 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118605+0000", 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118604+0000", 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118611+0000", 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7259999999999996e-06 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118622+0000", 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1523e-05 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118628+0000", 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8119999999999996e-06 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118675+0000", 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6828e-05 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118685+0000", 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7990000000000005e-06 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.012981+0000", 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89429594199999995 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.013114+0000", 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013327200000000001 2024-06-14T23:10:15.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.013233+0000", 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119317 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017334+0000", 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041004850000000001 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025532+0000", 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081980309999999997 2024-06-14T23:10:15.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025763+0000", 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023137200000000001 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025816+0000", 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2525000000000003e-05 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025855+0000", 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9762999999999999e-05 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252755 10.3 10:c538d66a:::benchmark_data_smithi086_189745_object121787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.127097+0000", 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.64350513699998, 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89934203300000004, 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252755 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127097+0000", 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127097+0000", 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127096+0000", 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127102+0000", 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5610000000000004e-06 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127117+0000", 2024-06-14T23:10:15.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4346e-05 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127122+0000", 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6799999999999998e-06 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127178+0000", 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5414000000000002e-05 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127189+0000", 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1756000000000001e-05 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017353+0000", 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89016351900000001 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017457+0000", 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010399 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017571+0000", 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113579 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025689+0000", 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081183660000000001 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025731+0000", 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1641000000000003e-05 2024-06-14T23:10:15.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026328+0000", 2024-06-14T23:10:15.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000597077 2024-06-14T23:10:15.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026402+0000", 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4680999999999999e-05 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026439+0000", 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7027000000000003e-05 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252760 10.3 10:cf81e509:::benchmark_data_smithi086_189745_object121792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.63932397799999, 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89530984300000005, 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252760 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.835 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:10:15.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131281+0000", 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6860000000000002e-06 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131287+0000", 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1029999999999998e-06 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131291+0000", 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8959999999999996e-06 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131331+0000", 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0021999999999997e-05 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131336+0000", 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3900000000000001e-06 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.021730+0000", 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89039394699999996 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.021825+0000", 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5179000000000005e-05 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.021931+0000", 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010621899999999999 2024-06-14T23:10:15.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025902+0000", 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039701329999999998 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025913+0000", 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1089e-05 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026503+0000", 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00059051700000000004 2024-06-14T23:10:15.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026545+0000", 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1745000000000001e-05 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026588+0000", 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3331000000000002e-05 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252780 10.3 10:c1a81b18:::benchmark_data_smithi086_189745_object121812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.158993+0000", 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.611609714, 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.88218134500000001, 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252780 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.158993+0000", 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.158993+0000", 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.158991+0000", 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.158998+0000", 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7039999999999997e-06 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.159010+0000", 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2255999999999999e-05 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.159016+0000", 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5350000000000004e-06 2024-06-14T23:10:15.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.159069+0000", 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2539e-05 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.159078+0000", 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5060000000000001e-06 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026962+0000", 2024-06-14T23:10:15.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86788375399999995 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.027071+0000", 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010958100000000001 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.027186+0000", 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011420499999999999 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.040994+0000", 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013808572 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041051+0000", 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7226999999999999e-05 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041089+0000", 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7701999999999999e-05 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041135+0000", 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6351999999999999e-05 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041174+0000", 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8625999999999999e-05 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252800 10.3 10:c6176e12:::benchmark_data_smithi086_189745_object121832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.189322+0000", 2024-06-14T23:10:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.58128043300002, 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85191730099999996, 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252800 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189322+0000", 2024-06-14T23:10:15.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189322+0000", 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189321+0000", 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189326+0000", 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7350000000000001e-06 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189338+0000", 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1694e-05 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189343+0000", 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5850000000000001e-06 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189375+0000", 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2588e-05 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.189380+0000", 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9710000000000003e-06 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.031447+0000", 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84206642600000003 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.031551+0000", 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104193 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.031661+0000", 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109812 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041023+0000", 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093619159999999996 2024-06-14T23:10:15.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041082+0000", 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9357000000000001e-05 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041182+0000", 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100129 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041213+0000", 2024-06-14T23:10:15.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0380000000000001e-05 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.041239+0000", 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.675e-05 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252812 10.3 10:c0951fb7:::benchmark_data_smithi086_189745_object121844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.215115+0000", 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 352.55548722399999, 2024-06-14T23:10:15.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84339402699999999, 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252812 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215115+0000", 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215115+0000", 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215114+0000", 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215119+0000", 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7150000000000003e-06 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215131+0000", 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1573000000000001e-05 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215135+0000", 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2520000000000001e-06 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215169+0000", 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3906999999999999e-05 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.215174+0000", 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8210000000000001e-06 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.036052+0000", 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82087817500000004 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.036175+0000", 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012256600000000001 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.036304+0000", 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012888999999999999 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.044326+0000", 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0080227460000000007 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.049185+0000", 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048587730000000003 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.058419+0000", 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0092341750000000007 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.058471+0000", 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1332000000000003e-05 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.058509+0000", 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8512e-05 2024-06-14T23:10:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136541 11.5 11:ac4784f5:::benchmark_data_smithi086_226760_object9220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.932519+0000", 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 195.83808351499999, 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0219499169999999, 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136541 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932519+0000", 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932519+0000", 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932518+0000", 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932523+0000", 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.356e-06 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932535+0000", 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2167999999999999e-05 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932541+0000", 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7400000000000001e-06 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932575+0000", 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3946000000000003e-05 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932684+0000", 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010943299999999999 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932803+0000", 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118439 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.946273+0000", 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013470650000000001 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.946281+0000", 2024-06-14T23:10:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6260000000000003e-06 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.954404+0000", 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.008122773 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.954469+0000", 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4986000000000004e-05 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136539 11.8 11:1b96f922:::benchmark_data_smithi086_226760_object9218:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 195.82108639500001, 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.00353812, 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136539 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949518+0000", 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.83e-06 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949525+0000", 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4980000000000003e-06 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949529+0000", 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8340000000000001e-06 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949580+0000", 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1199000000000002e-05 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949590+0000", 2024-06-14T23:10:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9499999999999996e-06 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.952967+0000", 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.003376877 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953054+0000", 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7299000000000006e-05 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136542 11.8 11:118834dc:::benchmark_data_smithi086_226760_object9221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.949549+0000", 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 195.82105382399999, 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.013177585, 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136542 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949549+0000", 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949549+0000", 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949548+0000", 2024-06-14T23:10:15.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949551+0000", 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9459999999999998e-06 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949557+0000", 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6400000000000002e-06 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949560+0000", 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6849999999999999e-06 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949608+0000", 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7905000000000003e-05 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949613+0000", 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5349999999999998e-06 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953079+0000", 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0034659319999999 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953172+0000", 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3556999999999999e-05 2024-06-14T23:10:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962668+0000", 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094954789999999994 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962726+0000", 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8313999999999998e-05 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136550 11.a 11:5c3c9776:::benchmark_data_smithi086_226760_object9229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 195.82102701400001, 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0136263190000001, 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136550 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949577+0000", 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7889999999999999e-06 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949583+0000", 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3689999999999998e-06 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949586+0000", 2024-06-14T23:10:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3380000000000001e-06 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.950035+0000", 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044843299999999998 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.950042+0000", 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.922e-06 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.952960+0000", 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.002918226 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953096+0000", 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000136355 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953224+0000", 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000128019 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962228+0000", 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090035859999999992 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963063+0000", 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000835453 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963118+0000", 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5297000000000003e-05 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963202+0000", 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3250000000000004e-05 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136554 11.8 11:1bd3cba3:::benchmark_data_smithi086_226760_object9233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.957712+0000", 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 195.81289025999999, 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.004780947, 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136554 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957712+0000", 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957712+0000", 2024-06-14T23:10:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957710+0000", 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957717+0000", 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0480000000000001e-06 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957731+0000", 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3701e-05 2024-06-14T23:10:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957738+0000", 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4860000000000002e-06 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957791+0000", 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3304999999999999e-05 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957799+0000", 2024-06-14T23:10:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9039999999999992e-06 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953252+0000", 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99545359899999997 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953371+0000", 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118045 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953468+0000", 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7633000000000005e-05 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957490+0000", 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040219909999999999 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962312+0000", 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048222350000000002 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962399+0000", 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6630000000000002e-05 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962493+0000", 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4086999999999996e-05 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136561 11.a 11:5eddd569:::benchmark_data_smithi086_226760_object9240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.966732+0000", 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 195.80387026599999, 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0042532740000001, 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136561 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966732+0000", 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966732+0000", 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966731+0000", 2024-06-14T23:10:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966736+0000", 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0629999999999996e-06 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966747+0000", 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0335e-05 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966751+0000", 2024-06-14T23:10:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0840000000000002e-06 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966802+0000", 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0970999999999997e-05 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966812+0000", 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9860000000000006e-06 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957486+0000", 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99067426400000003 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957592+0000", 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105921 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957699+0000", 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106839 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962260+0000", 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045612719999999999 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.970899+0000", 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086394460000000003 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.970938+0000", 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9079000000000003e-05 2024-06-14T23:10:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.970985+0000", 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6937999999999997e-05 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211914 11.5 11:a4813f22:::benchmark_data_smithi086_226760_object84593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.083822+0000", 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 107.686779965, 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97223605199999996, 2024-06-14T23:10:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211914 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083822+0000", 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083822+0000", 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083821+0000", 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083827+0000", 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5289999999999999e-06 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083835+0000", 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4849999999999992e-06 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083840+0000", 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4460000000000003e-06 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083879+0000", 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8899000000000001e-05 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083885+0000", 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.93e-06 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.040652+0000", 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95676742400000003 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.040790+0000", 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013773400000000001 2024-06-14T23:10:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.041240+0000", 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00045052500000000001 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.047857+0000", 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066170179999999997 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.047945+0000", 2024-06-14T23:10:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7732999999999995e-05 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.055720+0000", 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077744800000000003 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.055823+0000", 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103556 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.056058+0000", 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023532300000000001 2024-06-14T23:10:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211934 11.5 11:a2c54b8c:::benchmark_data_smithi086_226760_object84613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.154477+0000", 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 107.61612531, 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90184410999999998, 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211934 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154477+0000", 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154477+0000", 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154476+0000", 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154483+0000", 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4680000000000003e-06 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154497+0000", 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4273999999999999e-05 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154504+0000", 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2550000000000003e-06 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154557+0000", 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3291999999999997e-05 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154568+0000", 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1347e-05 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044096+0000", 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.88952724999999999 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044425+0000", 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000329408 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.045198+0000", 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000773363 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048409+0000", 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003210638 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048463+0000", 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3637999999999997e-05 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.056114+0000", 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076511490000000003 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.056177+0000", 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3575e-05 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.056321+0000", 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014390200000000001 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:15.912 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:10:15.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:15.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:16.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:16.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:16.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:16.216 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:16.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:16.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:16.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:16.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:16.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:16.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:16.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:16.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 53922 53906 140.361 110.688 0.0110026 0.00711969 2024-06-14T23:10:16.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:16.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:16.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:16.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:16.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:16.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:16.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:16.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:16.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:16.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-14T23:10:17.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:17.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:17.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:17.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:17.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:17.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:10:17.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:17.334 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:17.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:17.429 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:10:17.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:17.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:17.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:17.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 55599 55583 138.939 104.812 0.00401947 0.00719193 2024-06-14T23:10:17.566 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:10:17.567 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:10:17.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:17.603 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:10:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:10:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:17.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:17.637 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:10:17.637 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:10:17.673 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:10:17.673 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:10:17.708 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:10:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:10:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:17.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:17.742 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:10:17.742 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:10:17.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:17.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:17.783 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:10:17.783 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:10:17.816 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:10:17.816 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:10:17.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:17.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:17.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:17.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:17.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:18.034 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:18.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:18.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:17 smithi086 ceph-mon[97913]: pgmap v1150: 119 pgs: 119 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-06-14T23:10:18.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3545982500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:17 smithi086 ceph-mon[102754]: pgmap v1150: 119 pgs: 119 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-06-14T23:10:18.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3545982500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:18.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:18.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:18.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:18.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:17 smithi137 ceph-mon[90634]: pgmap v1150: 119 pgs: 119 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-06-14T23:10:18.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3545982500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:18.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:18.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:18.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:18.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:18.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:18.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:18.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 57221 57205 137.494 101.375 0.0091258 0.00726789 2024-06-14T23:10:18.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:18.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:18.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:18.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:18.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:18.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:18.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:18.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:18.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:18.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:19.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:19.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:19.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:19.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:19.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:19.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:19.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:19.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:19.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:19.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:19.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:19.456 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:10:19.456 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:10:19.506 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 58907 58891 136.304 105.375 0.00752712 0.00733078 2024-06-14T23:10:19.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:19.566 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 6, 7, 2] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-06-14T23:10:19.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:10:19.566 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [1, 3, 4, 5, 6, 7, 2] 2024-06-14T23:10:19.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd out 1 2024-06-14T23:10:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:19.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:19.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:19.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:19.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:19.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:19.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:19.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:19.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:20.004 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:20.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:20.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:20.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:20.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:20.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:19 smithi137 ceph-mon[90634]: pgmap v1151: 119 pgs: 119 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-06-14T23:10:20.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:20.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:20.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:20.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:19 smithi086 ceph-mon[97913]: pgmap v1151: 119 pgs: 119 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-06-14T23:10:20.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:19 smithi086 ceph-mon[102754]: pgmap v1151: 119 pgs: 119 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-06-14T23:10:20.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-14T23:10:20.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:20.433 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:20.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:20.507 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 60505 60489 135.003 99.875 0.00936956 0.0074026 2024-06-14T23:10:20.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:20.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:20.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:20.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:20.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:20.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:20.779 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:10:20.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:20.910 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:20.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:20.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:20.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:21.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:21.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:21.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:21.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:10:21.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:21.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:21.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:21.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:21.337 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:21.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:21.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:21.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:21.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:21.507 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 61955 61939 133.471 90.625 0.00608368 0.00748742 2024-06-14T23:10:21.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:21.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:21.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:21.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:21.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:21.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:21.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:21.898 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:21.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:21.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:10:22.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:22.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:22.106 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:21 smithi137 ceph-mon[90634]: pgmap v1152: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-06-14T23:10:22.106 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1159231800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:10:22.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:22.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:22.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:22.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:22.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:22.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:22.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:22.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:21 smithi086 ceph-mon[97913]: pgmap v1152: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-06-14T23:10:22.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1159231800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:10:22.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:21 smithi086 ceph-mon[102754]: pgmap v1152: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-06-14T23:10:22.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1159231800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:10:22.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:22.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:22.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:22.507 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 63422 63406 132.078 91.6875 0.00935325 0.00756596 2024-06-14T23:10:22.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:22.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:22.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:22.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:22.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:22.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:22.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:22.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:22.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:22.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:23.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:23.023 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:23.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:23.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:23.157 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:23.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:23.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:23.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:23.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:23.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:23.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:22 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:10:23.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1159231800' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:10:23.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:22 smithi086 ceph-mon[102754]: osdmap e308: 8 total, 8 up, 7 in 2024-06-14T23:10:23.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:22 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:10:23.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1159231800' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:10:23.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:22 smithi086 ceph-mon[97913]: osdmap e308: 8 total, 8 up, 7 in 2024-06-14T23:10:23.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:23.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:23.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:23.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:22 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:10:23.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1159231800' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:10:23.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:22 smithi137 ceph-mon[90634]: osdmap e308: 8 total, 8 up, 7 in 2024-06-14T23:10:23.507 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 64924 64908 130.846 93.875 0.00633682 0.00763798 2024-06-14T23:10:23.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:23.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:23.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:23.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:23.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:23.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:23.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:23.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:23.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:23.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:23.977 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:23.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:24.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:24.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:24 smithi086 ceph-mon[102754]: pgmap v1154: 119 pgs: 119 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-14T23:10:24.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:24 smithi086 ceph-mon[97913]: pgmap v1154: 119 pgs: 119 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-14T23:10:24.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:24.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:24.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:24.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:24.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:24.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:24.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:24.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:24.387 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:24 smithi137 ceph-mon[90634]: pgmap v1154: 119 pgs: 119 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-14T23:10:24.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:24.507 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 66388 66372 129.616 91.5 0.00990763 0.00770944 2024-06-14T23:10:24.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:24.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:24.544 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:10:24.544 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:10:24.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:10:24.322+0000 7fe0161db640 -1 mon.a@0(leader).osd e308 definitely_dead 0 2024-06-14T23:10:24.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:24.665 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:24.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:24.725 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:24.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:24.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:24.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:24.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:24.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:25.025 INFO:teuthology.orchestra.run.smithi086.stderr:marked out osd.1. 2024-06-14T23:10:25.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:25.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:25.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:25.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:25.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:25.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:25.209 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1962924376' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-06-14T23:10:25.209 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:25 smithi137 ceph-mon[90634]: Client client.admin marked osd.1 out, while it was still marked up 2024-06-14T23:10:25.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-14T23:10:25.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:25.320 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1962924376' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-06-14T23:10:25.321 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:25 smithi086 ceph-mon[97913]: Client client.admin marked osd.1 out, while it was still marked up 2024-06-14T23:10:25.321 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:10:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:10:24] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.43.0" 2024-06-14T23:10:25.321 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1962924376' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-06-14T23:10:25.321 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:25 smithi086 ceph-mon[102754]: Client client.admin marked osd.1 out, while it was still marked up 2024-06-14T23:10:25.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:25.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:25.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:25.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:25.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:25.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:25.507 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 16 67225 67209 127.273 52.3125 0.00240726 0.00774443 2024-06-14T23:10:25.526 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:10:25.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:25.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:25.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:25.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:25.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:25.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:25.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:25.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:25.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:25.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:25.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:25.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:26.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:26.073 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:26.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:26.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:26.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:26.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:26 smithi137 ceph-mon[90634]: pgmap v1155: 119 pgs: 119 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-06-14T23:10:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1962924376' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-06-14T23:10:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:26 smithi137 ceph-mon[90634]: osdmap e309: 8 total, 8 up, 6 in 2024-06-14T23:10:26.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:26.317 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:26 smithi086 ceph-mon[97913]: pgmap v1155: 119 pgs: 119 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-06-14T23:10:26.317 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1962924376' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-06-14T23:10:26.317 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:26 smithi086 ceph-mon[97913]: osdmap e309: 8 total, 8 up, 6 in 2024-06-14T23:10:26.317 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:26 smithi086 ceph-mon[102754]: pgmap v1155: 119 pgs: 119 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-06-14T23:10:26.317 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1962924376' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-06-14T23:10:26.318 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:26 smithi086 ceph-mon[102754]: osdmap e309: 8 total, 8 up, 6 in 2024-06-14T23:10:26.321 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:26.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:26.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:26.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:26.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:26.507 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 67225 67209 123.53 0 - 0.00774443 2024-06-14T23:10:26.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:26.568 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:26.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:26.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:10:26.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:26.715 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:10:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:10:26.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:26.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:26.732 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:10:26.733 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:10:26.750 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:10:26.750 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:10:26.766 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:10:26.767 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:10:26.783 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:10:26.783 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:10:26.799 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:10:26.799 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:10:26.815 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:10:26.815 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:10:26.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:26.832 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:10:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:10:26.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:26.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:26.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:26.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:26.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:27.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:27.104 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:27.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:27.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:27 smithi137 ceph-mon[90634]: osdmap e310: 8 total, 8 up, 6 in 2024-06-14T23:10:27.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/695097249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:27.142 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:27 smithi137 ceph-mon[90634]: osdmap e311: 8 total, 8 up, 6 in 2024-06-14T23:10:27.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:27.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:27.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:27.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:27 smithi086 ceph-mon[102754]: osdmap e310: 8 total, 8 up, 6 in 2024-06-14T23:10:27.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/695097249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:27.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:27 smithi086 ceph-mon[102754]: osdmap e311: 8 total, 8 up, 6 in 2024-06-14T23:10:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:27 smithi086 ceph-mon[97913]: osdmap e310: 8 total, 8 up, 6 in 2024-06-14T23:10:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/695097249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:27 smithi086 ceph-mon[97913]: osdmap e311: 8 total, 8 up, 6 in 2024-06-14T23:10:27.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:27.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:27.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:27.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:27.507 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 67488 67472 120.47 8.21875 0.00438085 0.00829532 2024-06-14T23:10:27.525 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:27.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:27.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:27.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:27.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:27.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:27.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:27.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:27.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:27.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:27.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:28.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:28.055 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:28.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:28.099 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:10:28.099 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:10:28.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:10:28.099 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:10:28.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.099 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:252208 10.8 e224/223 10:14ea0a56:::benchmark_data_smithi086_189745_object121240:head v 224'15600, mlcod=222'15593)", 2024-06-14T23:10:28.099 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.013331+0000", 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 367.08082665500001, 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.107265625, 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.013331+0000", 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.013331+0000", 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.013331+0000", 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.013335+0000", 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7550000000000002e-06 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.013343+0000", 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7970000000000001e-06 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.013347+0000", 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7270000000000001e-06 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.018157+0000", 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048102170000000003 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.018178+0000", 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1151000000000001e-05 2024-06-14T23:10:28.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.120552+0000", 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.102373533 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.120597+0000", 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4909e-05 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.103 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:252213 10.8 e224/223 10:166b668c:::benchmark_data_smithi086_189745_object121245:head v 224'15601, mlcod=222'15593)", 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.015412+0000", 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 367.07874596699997, 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10521865699999999, 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.015412+0000", 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.015412+0000", 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.015411+0000", 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.015431+0000", 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0356999999999998e-05 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.015442+0000", 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0203999999999999e-05 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.015447+0000", 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2939999999999996e-06 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.020874+0000", 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005426686 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.020886+0000", 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.222e-05 2024-06-14T23:10:28.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.120602+0000", 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.099716473999999999 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.120630+0000", 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8022e-05 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:252230 10.8 e224/223 10:13e69e69:::benchmark_data_smithi086_189745_object121262:head v 224'15602, mlcod=224'15594)", 2024-06-14T23:10:28.107 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.017684+0000", 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 367.07647383599999, 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10654833499999999, 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.017684+0000", 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.017684+0000", 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.017683+0000", 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.017688+0000", 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8180000000000003e-06 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.017695+0000", 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2479999999999999e-06 2024-06-14T23:10:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.017699+0000", 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9419999999999997e-06 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022974+0000", 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052745600000000002 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022989+0000", 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5248e-05 2024-06-14T23:10:28.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.124203+0000", 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.101213939 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.124232+0000", 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9074999999999999e-05 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:252234 10.8 e224/223 10:1b0daa4f:::benchmark_data_smithi086_189745_object121266:head v 224'15603, mlcod=224'15594)", 2024-06-14T23:10:28.111 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.019964+0000", 2024-06-14T23:10:28.112 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 367.07419368799998, 2024-06-14T23:10:28.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10428638599999999, 2024-06-14T23:10:28.112 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.112 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.112 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.019964+0000", 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.019964+0000", 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.019963+0000", 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.019970+0000", 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2849999999999997e-06 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.019982+0000", 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.182e-05 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.019988+0000", 2024-06-14T23:10:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.161e-06 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.025518+0000", 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055300469999999997 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.025531+0000", 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2576e-05 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.124234+0000", 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.098703581999999998 2024-06-14T23:10:28.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.124250+0000", 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5933000000000001e-05 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252316 10.c 10:3fc471b4:::benchmark_data_smithi086_189745_object121348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.021866+0000", 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 367.07229185, 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.100991316, 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.116 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252316 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.021866+0000", 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.021866+0000", 2024-06-14T23:10:28.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.021864+0000", 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.021871+0000", 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8639999999999998e-06 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.021881+0000", 2024-06-14T23:10:28.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0134e-05 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.021885+0000", 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8720000000000004e-06 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.021934+0000", 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8368999999999997e-05 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022010+0000", 2024-06-14T23:10:28.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6780999999999996e-05 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022087+0000", 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6639999999999998e-05 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.025316+0000", 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032285299999999999 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026162+0000", 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00084607599999999997 2024-06-14T23:10:28.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.122777+0000", 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.096615369000000006 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.122822+0000", 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4951999999999999e-05 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.122857+0000", 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5179000000000003e-05 2024-06-14T23:10:28.121 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:252255 10.8 e224/223 10:1398ced5:::benchmark_data_smithi086_189745_object121287:head v 224'15604, mlcod=224'15597)", 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.023309+0000", 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 367.07084899900002, 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10422709199999999, 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.023309+0000", 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.023309+0000", 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.023308+0000", 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.023312+0000", 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7570000000000002e-06 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.023318+0000", 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5979999999999996e-06 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.023321+0000", 2024-06-14T23:10:28.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5149999999999998e-06 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.116998+0000", 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.093676869999999995 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.117024+0000", 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6262e-05 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.127501+0000", 2024-06-14T23:10:28.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010476997 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.127536+0000", 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4421999999999997e-05 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:252256 10.8 e224/223 10:1f556f68:::benchmark_data_smithi086_189745_object121288:head v 224'15605, mlcod=224'15597)", 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.026401+0000", 2024-06-14T23:10:28.126 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 367.06775625, 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10447026, 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026401+0000", 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026401+0000", 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026401+0000", 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026407+0000", 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7880000000000003e-06 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026415+0000", 2024-06-14T23:10:28.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8349999999999993e-06 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.026420+0000", 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7169999999999999e-06 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.120667+0000", 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.094246609999999995 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.120689+0000", 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1877000000000001e-05 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.130837+0000", 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010148502 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.130872+0000", 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4570999999999998e-05 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.45821.0:252257 10.8 e224/223 10:158927dc:::benchmark_data_smithi086_189745_object121289:head v 224'15606, mlcod=224'15598)", 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.029835+0000", 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 367.06432232399999, 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.104297502, 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.029835+0000", 2024-06-14T23:10:28.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.029835+0000", 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.029835+0000", 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.029840+0000", 2024-06-14T23:10:28.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.933e-06 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.117797+0000", 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.087957015 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.117805+0000", 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4079999999999998e-06 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.124276+0000", 2024-06-14T23:10:28.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064706069999999997 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.124292+0000", 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6030000000000001e-05 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.134100+0000", 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098078409999999994 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.134133+0000", 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3139000000000002e-05 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136543 11.4 11:20fdfdc0:::benchmark_data_smithi086_226760_object9222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.932504+0000", 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.16165353299999, 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0301282709999999, 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:10:28.135 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136543 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932504+0000", 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932504+0000", 2024-06-14T23:10:28.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932502+0000", 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932509+0000", 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9290000000000001e-06 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932524+0000", 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4260000000000001e-05 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932531+0000", 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0319999999999998e-06 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932570+0000", 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9038999999999997e-05 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932685+0000", 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115599 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932804+0000", 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118542 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958388+0000", 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025584134000000001 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958395+0000", 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.559e-06 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962564+0000", 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0041688070000001 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962632+0000", 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8134999999999997e-05 2024-06-14T23:10:28.140 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136544 11.4 11:2b4506cd:::benchmark_data_smithi086_226760_object9223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.932542+0000", 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.16161522499999, 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0454412900000001, 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.141 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136544 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932542+0000", 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932542+0000", 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932542+0000", 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932546+0000", 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5669999999999999e-06 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932552+0000", 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4899999999999997e-06 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932556+0000", 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2259999999999999e-06 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.936405+0000", 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038487030000000002 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.936523+0000", 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117587 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.936636+0000", 2024-06-14T23:10:28.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011364100000000001 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958423+0000", 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021786934000000001 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958429+0000", 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5520000000000004e-06 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962848+0000", 2024-06-14T23:10:28.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.004419715 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962996+0000", 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014767099999999999 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.977890+0000", 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014894220999999999 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.977984+0000", 2024-06-14T23:10:28.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3461000000000005e-05 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136549 11.3 11:c6ade1fb:::benchmark_data_smithi086_226760_object9228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.945488+0000", 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.148669778, 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.012765999, 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.147 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136549 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945488+0000", 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945488+0000", 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945486+0000", 2024-06-14T23:10:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945494+0000", 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5989999999999996e-06 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945506+0000", 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1997999999999999e-05 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945512+0000", 2024-06-14T23:10:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9959999999999999e-06 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945896+0000", 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038445299999999999 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945903+0000", 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6089999999999997e-06 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.949574+0000", 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0036716459999999 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.951889+0000", 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023141390000000002 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1", 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.951987+0000", 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8564e-05 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.958120+0000", 2024-06-14T23:10:28.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061325629999999997 2024-06-14T23:10:28.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.958132+0000", 2024-06-14T23:10:28.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2276000000000001e-05 2024-06-14T23:10:28.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.958175+0000", 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2803999999999999e-05 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.958254+0000", 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9128999999999994e-05 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.153 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136555 11.4 11:26a47b0e:::benchmark_data_smithi086_226760_object9234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.958695+0000", 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.135462878, 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0183652350000001, 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136555 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958695+0000", 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958695+0000", 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958694+0000", 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958698+0000", 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1710000000000004e-06 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958708+0000", 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0266e-05 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958714+0000", 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6110000000000001e-06 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958763+0000", 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9339000000000003e-05 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958770+0000", 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9199999999999998e-06 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963038+0000", 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.004267749 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963138+0000", 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100512 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963255+0000", 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011699199999999999 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.969271+0000", 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060157680000000003 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.976977+0000", 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077061819999999998 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.977010+0000", 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2836999999999997e-05 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.977060+0000", 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0037000000000002e-05 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136560 11.4 11:227da07d:::benchmark_data_smithi086_226760_object9239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.966742+0000", 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.12741520899999, 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0114153640000001, 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136560 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966742+0000", 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966742+0000", 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966740+0000", 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966752+0000", 2024-06-14T23:10:28.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1289e-05 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966766+0000", 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4632e-05 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966773+0000", 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2979999999999997e-06 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966824+0000", 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1702000000000001e-05 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966834+0000", 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3659999999999998e-06 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.966441+0000", 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99960689800000002 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.966541+0000", 2024-06-14T23:10:28.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100348 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.966655+0000", 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113701 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.969390+0000", 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027356390000000002 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.978031+0000", 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086405820000000008 2024-06-14T23:10:28.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.978078+0000", 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6854999999999998e-05 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.978158+0000", 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9998000000000002e-05 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211922 11.4 11:2352494a:::benchmark_data_smithi086_226760_object84601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:10:28.165 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.120211+0000", 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.973946801, 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.924341317, 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211922 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120211+0000", 2024-06-14T23:10:28.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120211+0000", 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120209+0000", 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120217+0000", 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0220000000000004e-06 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120237+0000", 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9423999999999999e-05 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120245+0000", 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4309999999999994e-06 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120307+0000", 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1558999999999998e-05 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120315+0000", 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5199999999999997e-06 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.034034+0000", 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91371895000000003 2024-06-14T23:10:28.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.034193+0000", 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015935299999999999 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.034585+0000", 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039136599999999998 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.038389+0000", 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038037409999999998 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.038424+0000", 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5942999999999998e-05 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044266+0000", 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058418389999999997 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044320+0000", 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3764999999999998e-05 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044552+0000", 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000232072 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211928 11.4 11:21e85ab3:::benchmark_data_smithi086_226760_object84607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.139112+0000", 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.95504524899999, 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90937327199999995, 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211928 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139112+0000", 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139112+0000", 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139111+0000", 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139118+0000", 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1829999999999997e-06 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139132+0000", 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3835999999999999e-05 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139138+0000", 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9959999999999999e-06 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139190+0000", 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1583999999999999e-05 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139201+0000", 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1466e-05 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.036511+0000", 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89731032300000002 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.036651+0000", 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000139321 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.036980+0000", 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032952300000000002 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.038457+0000", 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014771299999999999 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.038565+0000", 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010767100000000001 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048205+0000", 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009639613 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048262+0000", 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7645999999999997e-05 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048486+0000", 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022344300000000001 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:218464 11.4 11:2641b213:::benchmark_data_smithi086_226760_object91143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:35.022580+0000", 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 113.071577409, 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.117434257, 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.179 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218464 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.022580+0000", 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.022580+0000", 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.022579+0000", 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.022584+0000", 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.66e-06 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.022599+0000", 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5092000000000001e-05 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.022606+0000", 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6479999999999997e-06 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027501+0000", 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004895379 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027626+0000", 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125067 2024-06-14T23:10:28.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.029039+0000", 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001412188 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.036642+0000", 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076036919999999996 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.041814+0000", 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051716840000000002 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.139589+0000", 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.097775179000000004 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.139713+0000", 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123569 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.140014+0000", 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030178200000000001 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.184 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:218465 11.4 11:25deed0a:::benchmark_data_smithi086_226760_object91144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:35.026266+0000", 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 113.067892112, 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.115000381, 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218465 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.026266+0000", 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.026266+0000", 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.026264+0000", 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.026273+0000", 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6670000000000002e-06 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.026290+0000", 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7510999999999999e-05 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.026297+0000", 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0219999999999999e-06 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.030682+0000", 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043853349999999998 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.030793+0000", 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011089799999999999 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.031100+0000", 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000307156 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.039813+0000", 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0087124149999999994 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.041856+0000", 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002042936 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.140727+0000", 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.098871237000000001 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.140897+0000", 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000169717 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.141266+0000", 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036914100000000002 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:218467 11.4 11:227a5100:::benchmark_data_smithi086_226760_object91146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:35.027042+0000", 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 113.06711528, 2024-06-14T23:10:28.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.11490993100000001, 2024-06-14T23:10:28.191 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.191 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.191 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.191 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.191 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.191 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218467 2024-06-14T23:10:28.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.191 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:28.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:28 smithi137 ceph-mon[90634]: pgmap v1158: 119 pgs: 21 peering, 3 remapped+peering, 1 unknown, 94 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-14T23:10:28.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:28 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-14T23:10:28.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:28.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:28.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:28 smithi137 ceph-mon[90634]: osdmap e312: 8 total, 8 up, 6 in 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027042+0000", 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027042+0000", 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027042+0000", 2024-06-14T23:10:28.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027046+0000", 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4789999999999998e-06 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027055+0000", 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0259999999999996e-06 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027060+0000", 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.51e-06 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.034546+0000", 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0074864420000000003 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.034651+0000", 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104588 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:10:28.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.034986+0000", 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033550799999999999 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.039861+0000", 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048747560000000001 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.041895+0000", 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002033862 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.141736+0000", 2024-06-14T23:10:28.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.099841368 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.141811+0000", 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4632e-05 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.141952+0000", 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014160999999999999 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:218468 11.14 11:2e3fc304:::benchmark_data_smithi086_226760_object91147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:35.027064+0000", 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 113.067093352, 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.113659132, 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218468 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027064+0000", 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027064+0000", 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027064+0000", 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027066+0000", 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2579999999999998e-06 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027071+0000", 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8640000000000004e-06 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.027074+0000", 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1750000000000001e-06 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.032939+0000", 2024-06-14T23:10:28.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058647730000000002 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.033062+0000", 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012288300000000001 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.033352+0000", 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028977499999999998 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.039842+0000", 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064899049999999998 2024-06-14T23:10:28.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.041845+0000", 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020030349999999998 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.140257+0000", 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.098412586999999996 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.140400+0000", 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014291299999999999 2024-06-14T23:10:28.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.140723+0000", 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032328100000000002 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:218477 11.4 11:25f1922e:::benchmark_data_smithi086_226760_object91156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:35.035333+0000", 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 113.058824562, 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.106819604, 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218477 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.035333+0000", 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.035333+0000", 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.035332+0000", 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.035338+0000", 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.545e-06 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.035352+0000", 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4379000000000001e-05 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.035359+0000", 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4350000000000002e-06 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.036706+0000", 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001347015 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.036935+0000", 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000229037 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.037596+0000", 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00066111100000000003 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.039925+0000", 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023293070000000001 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.044371+0000", 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044457389999999998 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.141992+0000", 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.097620587999999994 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.142035+0000", 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3625000000000003e-05 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:35.142153+0000", 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011736299999999999 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.208 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.209 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:28.209 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:28.209 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:10:28.240 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:28.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:28.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[102754]: pgmap v1158: 119 pgs: 21 peering, 3 remapped+peering, 1 unknown, 94 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-14T23:10:28.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-14T23:10:28.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:28.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:28.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[102754]: osdmap e312: 8 total, 8 up, 6 in 2024-06-14T23:10:28.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[97913]: pgmap v1158: 119 pgs: 21 peering, 3 remapped+peering, 1 unknown, 94 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-14T23:10:28.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-14T23:10:28.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:28.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:28.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:28 smithi086 ceph-mon[97913]: osdmap e312: 8 total, 8 up, 6 in 2024-06-14T23:10:28.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:28.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:28.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:28.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:28.507 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 68456 68440 118.804 60.5 0.00796251 0.00841019 2024-06-14T23:10:28.525 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:28.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:28.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:28.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:28.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:28.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:28.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:28.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:28.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:28.934 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:28.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:29.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:29.062 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:29.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:29.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:29.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:29.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:29.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:29.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:29.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T23:10:29.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:29.368 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:29.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:29.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:29.508 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 69422 69406 117.225 60.375 0.0394486 0.00852368 2024-06-14T23:10:29.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:29.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:29.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:29.658 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:29.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:29.724 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:10:29.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:29.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:29.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:29.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:29.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:29.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:30.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:30.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:30.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:30.194 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:30.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:30.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:30 smithi137 ceph-mon[90634]: pgmap v1161: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 30 peering, 3 remapped+peering, 1 unknown, 83 active+clean; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 5.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 8341/203028 objects degraded (4.108%); 82 KiB/s, 1 objects/s recovering 2024-06-14T23:10:30.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:30 smithi137 ceph-mon[90634]: osdmap e313: 8 total, 8 up, 6 in 2024-06-14T23:10:30.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:30 smithi137 ceph-mon[90634]: Health check failed: Degraded data redundancy: 8341/203028 objects degraded (4.108%), 2 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:30.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:30.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:30 smithi086 ceph-mon[97913]: pgmap v1161: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 30 peering, 3 remapped+peering, 1 unknown, 83 active+clean; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 5.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 8341/203028 objects degraded (4.108%); 82 KiB/s, 1 objects/s recovering 2024-06-14T23:10:30.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:30 smithi086 ceph-mon[97913]: osdmap e313: 8 total, 8 up, 6 in 2024-06-14T23:10:30.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:30 smithi086 ceph-mon[97913]: Health check failed: Degraded data redundancy: 8341/203028 objects degraded (4.108%), 2 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:30.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:30 smithi086 ceph-mon[102754]: pgmap v1161: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 30 peering, 3 remapped+peering, 1 unknown, 83 active+clean; 4.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 5.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 8341/203028 objects degraded (4.108%); 82 KiB/s, 1 objects/s recovering 2024-06-14T23:10:30.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:30 smithi086 ceph-mon[102754]: osdmap e313: 8 total, 8 up, 6 in 2024-06-14T23:10:30.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:30 smithi086 ceph-mon[102754]: Health check failed: Degraded data redundancy: 8341/203028 objects degraded (4.108%), 2 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:30.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:30.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:30.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:30.481 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:30.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:30.508 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 70283 70267 115.556 53.8125 0.0318187 0.0086475 2024-06-14T23:10:30.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:30.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:30.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:30.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:30.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:30.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:30.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:30.896 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:30.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:30.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:10:30.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:31.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:31.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:31.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:31.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:31.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:31.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:31.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:31.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:31.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:31.508 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 16 71274 71258 114.181 61.9375 0.0246148 0.00874778 2024-06-14T23:10:31.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:31.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:31.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:31.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:31.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:31.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:10:31.867 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:10:31.867 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:10:31.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.867 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:71585 12.6 12:6eab0c81:::benchmark_data_smithi086_261492_object71584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.851236+0000", 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.012239432, 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012271921, 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 71585 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851236+0000", 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851236+0000", 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851235+0000", 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:31.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851295+0000", 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9865999999999999e-05 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851306+0000", 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1079e-05 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851311+0000", 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8200000000000002e-06 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.859537+0000", 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082251579999999998 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.859664+0000", 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012768400000000001 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.859789+0000", 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124499 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:71589 12.6 12:61e56957:::benchmark_data_smithi086_261492_object71588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.851691+0000", 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.011785082000000001, 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011891751000000001, 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:31.872 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 71589 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851691+0000", 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851691+0000", 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851690+0000", 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851778+0000", 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7868999999999995e-05 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851790+0000", 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2721000000000001e-05 2024-06-14T23:10:31.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.851796+0000", 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7810000000000002e-06 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862631+0000", 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010834463000000001 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862802+0000", 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000171304 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.863003+0000", 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020105000000000001 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.46469.0:71582 12.4 e313/310 ondisk, result = 0)", 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.858436+0000", 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.005039567, 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052073060000000001, 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858436+0000", 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858436+0000", 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858435+0000", 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858441+0000", 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8300000000000001e-06 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858454+0000", 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2724999999999999e-05 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858459+0000", 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7090000000000001e-06 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858516+0000", 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7590000000000003e-05 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858524+0000", 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5599999999999996e-06 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:71593 12.f 12:f14a0eb9:::benchmark_data_smithi086_261492_object71592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-06-14T23:10:31.879 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.858812+0000", 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0046633899999999999, 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048851620000000002, 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 71593 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858812+0000", 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858812+0000", 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858811+0000", 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858902+0000", 2024-06-14T23:10:31.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1263999999999999e-05 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858912+0000", 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0096e-05 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858922+0000", 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0288e-05 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.859018+0000", 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6111000000000002e-05 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.859118+0000", 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9381000000000003e-05 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.859238+0000", 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120394 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861996+0000", 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027577830000000002 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:71596 12.8 e313/299 12:1fb56f22:::benchmark_data_smithi086_261492_object71595:head v 313'4585, mlcod=313'4583)", 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.861755+0000", 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.001720566, 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020104200000000002, 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:10:31.884 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861755+0000", 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861755+0000", 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861754+0000", 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:31.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861796+0000", 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2061999999999998e-05 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861807+0000", 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0618999999999999e-05 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861812+0000", 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1510000000000003e-06 2024-06-14T23:10:31.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862025+0000", 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000213119 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862041+0000", 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6147e-05 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:71597 12.9 e313/295 12:96225c1a:::benchmark_data_smithi086_261492_object71596:head v 313'4529, mlcod=313'4527)", 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.862012+0000", 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0014632499999999999, 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001804604, 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862012+0000", 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862012+0000", 2024-06-14T23:10:31.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862012+0000", 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862046+0000", 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4038000000000002e-05 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862056+0000", 2024-06-14T23:10:31.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0129000000000001e-05 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862061+0000", 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9019999999999998e-06 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862100+0000", 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9591000000000001e-05 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.862117+0000", 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6628000000000001e-05 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:71576 12.5 12:a215d63a:::benchmark_data_smithi086_261492_object71575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.840218+0000", 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.023258006000000001, 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023654661, 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:31.891 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 71576 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.840218+0000", 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.840218+0000", 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.840217+0000", 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.840248+0000", 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0899999999999999e-05 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.840256+0000", 2024-06-14T23:10:31.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6170000000000007e-06 2024-06-14T23:10:31.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.840261+0000", 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.639e-06 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.843912+0000", 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036514809999999998 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.844051+0000", 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013828899999999999 2024-06-14T23:10:31.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.844196+0000", 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000144948 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.860891+0000", 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016695323000000001 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:31.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.860954+0000", 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2853000000000003e-05 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:71580 12.5 12:a20bde48:::benchmark_data_smithi086_261492_object71579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.843575+0000", 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.019900538999999998, 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020381409999999999, 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:31.897 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 71580 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.843575+0000", 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.843575+0000", 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.843573+0000", 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999976 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.843610+0000", 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6792999999999998e-05 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.843624+0000", 2024-06-14T23:10:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3895e-05 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.843638+0000", 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4825e-05 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846894+0000", 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032552900000000001 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.847041+0000", 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014738500000000001 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.847238+0000", 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019650599999999999 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.860925+0000", 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013687563999999999 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.860987+0000", 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1643999999999995e-05 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:71581 12.5 12:ab223c94:::benchmark_data_smithi086_261492_object71580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.846752+0000", 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.016723802999999999, 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017280172999999999, 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.902 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 71581 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846752+0000", 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846752+0000", 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846750+0000", 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846790+0000", 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0000000000000003e-05 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846806+0000", 2024-06-14T23:10:31.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5512e-05 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846812+0000", 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6340000000000003e-06 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861009+0000", 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014196677 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861136+0000", 2024-06-14T23:10:31.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012704799999999999 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.861491+0000", 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000355402 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:71582 12.4 12:229251e3:::benchmark_data_smithi086_261492_object71581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:10:31.846823+0000", 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.01665268, 2024-06-14T23:10:31.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017273549999999999, 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 71582 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846823+0000", 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846823+0000", 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846822+0000", 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846851+0000", 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8527e-05 2024-06-14T23:10:31.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846858+0000", 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8449999999999997e-06 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846862+0000", 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7890000000000001e-06 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.846925+0000", 2024-06-14T23:10:31.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3269999999999996e-05 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.847084+0000", 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015916699999999999 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.847260+0000", 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000176417 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.856901+0000", 2024-06-14T23:10:31.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096409479999999999 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.856911+0000", 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2979999999999997e-06 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858530+0000", 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016191529999999999 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:10:31.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:10:31.858609+0000", 2024-06-14T23:10:31.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9399000000000003e-05 2024-06-14T23:10:31.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.912 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:10:31.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:10:31.912 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T23:10:31.912 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 10 2024-06-14T23:10:31.912 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:10:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:31.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:31.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:10:32.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:32.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:32.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:32.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:32.185 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:32.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:32.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:32.321 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:32.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:32.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:32 smithi086 ceph-mon[97913]: pgmap v1163: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 peering, 3 remapped+peering, 104 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 4.3 KiB/s rd, 50 MiB/s wr, 809 op/s; 8341/205140 objects degraded (4.066%); 4174/205140 objects misplaced (2.035%); 1.2 MiB/s, 19 objects/s recovering 2024-06-14T23:10:32.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:32 smithi086 ceph-mon[102754]: pgmap v1163: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 peering, 3 remapped+peering, 104 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 4.3 KiB/s rd, 50 MiB/s wr, 809 op/s; 8341/205140 objects degraded (4.066%); 4174/205140 objects misplaced (2.035%); 1.2 MiB/s, 19 objects/s recovering 2024-06-14T23:10:32.394 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 2] out_osds: [0, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-06-14T23:10:32.394 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:10:32.394 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 7, 6] 2024-06-14T23:10:32.394 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-14T23:10:32.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.2 2024-06-14T23:10:32.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:32.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:32.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:32.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:32 smithi137 ceph-mon[90634]: pgmap v1163: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 peering, 3 remapped+peering, 104 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 4.3 KiB/s rd, 50 MiB/s wr, 809 op/s; 8341/205140 objects degraded (4.066%); 4174/205140 objects misplaced (2.035%); 1.2 MiB/s, 19 objects/s recovering 2024-06-14T23:10:32.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:10:32.507031+0000 min lat: 0.00120633 max lat: 2.27207 avg lat: 0.00886825 2024-06-14T23:10:32.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:10:32.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 72150 72134 112.695 54.75 0.0156949 0.00886825 2024-06-14T23:10:32.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:32.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:32.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:32.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:32.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:32.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:32.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:32.836 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:32 smithi086 systemd[1]: Stopping Ceph osd.2 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:10:32.870 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:32.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:32.972 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:33.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:33.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:33.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:33.044 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:33.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:33.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:33.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:33.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:33.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:33.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:33.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:33.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:33.508 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 73130 73114 111.44 61.25 0.0364512 0.00896722 2024-06-14T23:10:33.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:33.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:33.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:33.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:33.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:33.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:33.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:33.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:33.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:33.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:33.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:34.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:34.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:34.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:34.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:34.251 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:34.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:34.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:34 smithi086 ceph-mon[102754]: pgmap v1164: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 9 peering, 104 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 3.5 KiB/s rd, 36 MiB/s wr, 582 op/s; 20810/208797 objects degraded (9.967%); 4174/208797 objects misplaced (1.999%); 972 KiB/s, 15 objects/s recovering 2024-06-14T23:10:34.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:34 smithi086 ceph-mon[102754]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:10:34.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:34 smithi086 ceph-mon[97913]: pgmap v1164: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 9 peering, 104 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 3.5 KiB/s rd, 36 MiB/s wr, 582 op/s; 20810/208797 objects degraded (9.967%); 4174/208797 objects misplaced (1.999%); 972 KiB/s, 15 objects/s recovering 2024-06-14T23:10:34.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:34 smithi086 ceph-mon[97913]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:10:34.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:34.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:34.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:34.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:34 smithi137 ceph-mon[90634]: pgmap v1164: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 9 peering, 104 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 3.5 KiB/s rd, 36 MiB/s wr, 582 op/s; 20810/208797 objects degraded (9.967%); 4174/208797 objects misplaced (1.999%); 972 KiB/s, 15 objects/s recovering 2024-06-14T23:10:34.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:34 smithi137 ceph-mon[90634]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:10:34.508 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 16 74022 74006 110.114 55.75 0.00947386 0.00907533 2024-06-14T23:10:34.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:34.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:34.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:34.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:34.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:34.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:34.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:34.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:34.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:34.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:34.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:35.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:35.084 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:35.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:35.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:35.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:35.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:35.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:35.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:35.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3363111418' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:10:35.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:35 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:10:35.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:10:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:10:34] "GET /metrics HTTP/1.1" 200 36598 "" "Prometheus/2.43.0" 2024-06-14T23:10:35.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3363111418' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:10:35.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:35 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:10:35.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:35.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:35.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:35.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3363111418' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:10:35.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:35 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:10:35.500 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:35.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:35.508 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 74954 74938 108.908 58.25 0.0132539 0.00917606 2024-06-14T23:10:35.572 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:10:35.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:35.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:35.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:35.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:35.764 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:35.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:35.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:35.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:35.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:36.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:36.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:36.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:36.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:36.180 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:36.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:36.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:36.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:36.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:36.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:36.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[97913]: pgmap v1165: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 2.7 KiB/s rd, 44 MiB/s wr, 710 op/s; 20810/215157 objects degraded (9.672%); 4174/215157 objects misplaced (1.940%); 738 KiB/s, 11 objects/s recovering 2024-06-14T23:10:36.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 20810/215157 objects degraded (9.672%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[97913]: osdmap e314: 8 total, 8 up, 6 in 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[102754]: pgmap v1165: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 2.7 KiB/s rd, 44 MiB/s wr, 710 op/s; 20810/215157 objects degraded (9.672%); 4174/215157 objects misplaced (1.940%); 738 KiB/s, 11 objects/s recovering 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 20810/215157 objects degraded (9.672%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:10:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-mon[102754]: osdmap e314: 8 total, 8 up, 6 in 2024-06-14T23:10:36.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:36.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:36.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:36.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:36 smithi137 ceph-mon[90634]: pgmap v1165: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 2.7 KiB/s rd, 44 MiB/s wr, 710 op/s; 20810/215157 objects degraded (9.672%); 4174/215157 objects misplaced (1.940%); 738 KiB/s, 11 objects/s recovering 2024-06-14T23:10:36.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:36 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 20810/215157 objects degraded (9.672%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:36.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:36 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:10:36.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:36 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:10:36.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:36 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:10:36.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:36 smithi137 ceph-mon[90634]: osdmap e314: 8 total, 8 up, 6 in 2024-06-14T23:10:36.508 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 16 75866 75850 107.728 57 0.00660362 0.00927625 2024-06-14T23:10:36.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:36.581 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:36.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:36.670 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[132386]: 2024-06-14T23:10:36.549+0000 7fb4b988d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:10:36.670 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[132386]: 2024-06-14T23:10:36.549+0000 7fb4b988d640 -1 osd.2 314 *** Got signal Terminated *** 2024-06-14T23:10:36.670 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:36 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[132386]: 2024-06-14T23:10:36.549+0000 7fb4b988d640 -1 osd.2 314 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:10:36.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:36.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:36.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:36.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:36.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:36.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:36.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-14T23:10:36.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:36.987 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:36.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:37.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:37.111 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:10:37.111 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:10:37.128 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:37.128 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:10:37.129 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:10:37.129 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-14T23:10:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:37.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:37.145 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:10:37.146 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:10:37.162 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:10:37.162 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:10:37.178 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:10:37.179 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:10:37.195 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:10:37.195 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:10:37.211 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:10:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:10:37.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:37 smithi137 ceph-mon[90634]: osd.2 marked itself down and dead 2024-06-14T23:10:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:37 smithi137 ceph-mon[90634]: pgmap v1167: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 52 MiB/s wr, 827 op/s; 20810/222900 objects degraded (9.336%); 4025/222900 objects misplaced (1.806%); 1.8 MiB/s, 29 objects/s recovering 2024-06-14T23:10:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:37.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:37.279 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:10:37.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:37 smithi086 ceph-mon[97913]: osd.2 marked itself down and dead 2024-06-14T23:10:37.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:37 smithi086 ceph-mon[97913]: pgmap v1167: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 52 MiB/s wr, 827 op/s; 20810/222900 objects degraded (9.336%); 4025/222900 objects misplaced (1.806%); 1.8 MiB/s, 29 objects/s recovering 2024-06-14T23:10:37.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:37 smithi086 ceph-mon[102754]: osd.2 marked itself down and dead 2024-06-14T23:10:37.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:37 smithi086 ceph-mon[102754]: pgmap v1167: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 52 MiB/s wr, 827 op/s; 20810/222900 objects degraded (9.336%); 4025/222900 objects misplaced (1.806%); 1.8 MiB/s, 29 objects/s recovering 2024-06-14T23:10:37.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:37.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:37.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:37.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:37.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:37.508 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 76380 76364 106.048 32.125 0.0470004 0.00933651 2024-06-14T23:10:37.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:37.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:37.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:37.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:37.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:37.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:37.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:37.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:37.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:37.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:37.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:37.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:37.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:10:38.007 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:10:38.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:38.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:38.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:38.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:38.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:38.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:38.336 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:38 smithi086 podman[266349]: 2024-06-14 23:10:38.060620804 +0000 UTC m=+4.691273647 container died 866667701cf9b5515eb213854504f9204c5a578c5d786ef9c6bb3686f7c771be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=) 2024-06-14T23:10:38.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:38 smithi086 ceph-mon[102754]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T23:10:38.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:38 smithi086 ceph-mon[102754]: osdmap e315: 8 total, 7 up, 6 in 2024-06-14T23:10:38.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:38 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1177044377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:38.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:38 smithi086 ceph-mon[102754]: 1.16 scrub starts 2024-06-14T23:10:38.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:38 smithi086 ceph-mon[97913]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T23:10:38.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:38 smithi086 ceph-mon[97913]: osdmap e315: 8 total, 7 up, 6 in 2024-06-14T23:10:38.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:38 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1177044377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:38.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:38 smithi086 ceph-mon[97913]: 1.16 scrub starts 2024-06-14T23:10:38.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:38.390 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:38.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:38.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:10:38.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:38.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:38 smithi137 ceph-mon[90634]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T23:10:38.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:38 smithi137 ceph-mon[90634]: osdmap e315: 8 total, 7 up, 6 in 2024-06-14T23:10:38.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:38 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1177044377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:38.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:38 smithi137 ceph-mon[90634]: 1.16 scrub starts 2024-06-14T23:10:38.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 16 76432 76416 103.813 3.25 0.00304565 0.00950112 2024-06-14T23:10:38.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:38.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:38.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:38.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:38.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:38.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:38.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:38.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:38.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:38 smithi086 podman[266349]: 2024-06-14 23:10:38.510081581 +0000 UTC m=+5.140734421 container cleanup 866667701cf9b5515eb213854504f9204c5a578c5d786ef9c6bb3686f7c771be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:10:38.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:38 smithi086 bash[266349]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2 2024-06-14T23:10:38.829 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:38.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:38.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:38.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:38.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:38.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:39.010 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:39.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:39.071 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:38 smithi086 podman[266809]: 2024-06-14 23:10:38.778256166 +0000 UTC m=+1.043473183 container remove 866667701cf9b5515eb213854504f9204c5a578c5d786ef9c6bb3686f7c771be (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T23:10:39.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:39.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:39.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:39.146 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:39 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:10:39.146 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:39 smithi137 ceph-mon[90634]: osdmap e316: 8 total, 7 up, 6 in 2024-06-14T23:10:39.146 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:39 smithi137 ceph-mon[90634]: pgmap v1170: 119 pgs: 2 remapped+peering, 26 active+undersized, 1 stale+active+recovering+undersized+remapped, 9 stale+active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+undersized+degraded, 73 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 12 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 22068/227559 objects degraded (9.698%); 4025/227559 objects misplaced (1.769%); 1.6 MiB/s, 24 objects/s recovering 2024-06-14T23:10:39.161 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:10:39.161 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:10:39.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:39.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:39.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:39.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:39 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:10:39.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:39 smithi086 ceph-mon[97913]: osdmap e316: 8 total, 7 up, 6 in 2024-06-14T23:10:39.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:39 smithi086 ceph-mon[97913]: pgmap v1170: 119 pgs: 2 remapped+peering, 26 active+undersized, 1 stale+active+recovering+undersized+remapped, 9 stale+active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+undersized+degraded, 73 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 12 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 22068/227559 objects degraded (9.698%); 4025/227559 objects misplaced (1.769%); 1.6 MiB/s, 24 objects/s recovering 2024-06-14T23:10:39.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:39 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:10:39.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:39 smithi086 ceph-mon[102754]: osdmap e316: 8 total, 7 up, 6 in 2024-06-14T23:10:39.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:39 smithi086 ceph-mon[102754]: pgmap v1170: 119 pgs: 2 remapped+peering, 26 active+undersized, 1 stale+active+recovering+undersized+remapped, 9 stale+active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+undersized+degraded, 73 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 12 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 22068/227559 objects degraded (9.698%); 4025/227559 objects misplaced (1.769%); 1.6 MiB/s, 24 objects/s recovering 2024-06-14T23:10:39.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:39.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:39.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:39.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:39.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 76453 76437 101.632 1.3125 0.0092436 0.0097185 2024-06-14T23:10:39.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:39.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:39.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:39 smithi086 podman[266882]: 2024-06-14 23:10:39.332110941 +0000 UTC m=+0.799820976 container died 1625d2847a7daa6bd374d2b49612308e420b2e85ece5a691492ad9a42ae1bcfa (image=quay.ceph.io/ceph-ci/ceph:quincy, name=distracted_hertz, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:10:39.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:39.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:39.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:39.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:39.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:39.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:39.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:39.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:40.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:10:40.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:40.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:40.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:40.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:40.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:40.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:40.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:40.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-14T23:10:40.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:40 smithi086 ceph-mon[97913]: osdmap e317: 8 total, 7 up, 6 in 2024-06-14T23:10:40.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:40 smithi086 ceph-mon[97913]: 1.16 scrub starts 2024-06-14T23:10:40.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:40 smithi086 ceph-mon[102754]: osdmap e317: 8 total, 7 up, 6 in 2024-06-14T23:10:40.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:40 smithi086 ceph-mon[102754]: 1.16 scrub starts 2024-06-14T23:10:40.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:40.346 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:40 smithi137 ceph-mon[90634]: osdmap e317: 8 total, 7 up, 6 in 2024-06-14T23:10:40.346 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:40 smithi137 ceph-mon[90634]: 1.16 scrub starts 2024-06-14T23:10:40.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:40.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:40.452 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:40.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:40.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 77389 77373 100.734 58.5 0.00914196 0.00992086 2024-06-14T23:10:40.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:40.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:40.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:40.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:40.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:40.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:40.788 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:10:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:40.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:40.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:40.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:40.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:40.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:41.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:41.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:41.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:41.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:41.188 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:41 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:10:41.189 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:41 smithi137 ceph-mon[90634]: pgmap v1172: 119 pgs: 2 remapped+peering, 35 active+undersized, 3 stale+active+clean, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded, 69 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 12 KiB/s rd, 47 MiB/s wr, 764 op/s; 31855/228450 objects degraded (13.944%); 3973/228450 objects misplaced (1.739%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:10:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:41.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:41.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:41.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:41.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:41.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:41.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:41.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:41.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:41.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 78301 78285 99.8409 57 0.0172257 0.010008 2024-06-14T23:10:41.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:41.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:41 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:10:41.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:41 smithi086 ceph-mon[97913]: pgmap v1172: 119 pgs: 2 remapped+peering, 35 active+undersized, 3 stale+active+clean, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded, 69 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 12 KiB/s rd, 47 MiB/s wr, 764 op/s; 31855/228450 objects degraded (13.944%); 3973/228450 objects misplaced (1.739%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:10:41.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:41 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:10:41.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:41 smithi086 ceph-mon[102754]: pgmap v1172: 119 pgs: 2 remapped+peering, 35 active+undersized, 3 stale+active+clean, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded, 69 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 12 KiB/s rd, 47 MiB/s wr, 764 op/s; 31855/228450 objects degraded (13.944%); 3973/228450 objects misplaced (1.739%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:10:41.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:41.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:41.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:41.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:41.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:41.866 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:41.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:41.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:41.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:41.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:42.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:42.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:42.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:42.178 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:42 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 31855/228450 objects degraded (13.944%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:42.179 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:42 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 31855/228450 objects degraded (13.944%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:42.202 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:42 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 31855/228450 objects degraded (13.944%), 10 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:42.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:42.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:42.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:42.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:42.408 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:42.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:42.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:42.510 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 79152 79136 98.9078 53.1875 0.011834 0.0101049 2024-06-14T23:10:42.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:42.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:42.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:42.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:42.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:42.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:42.806 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:42.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:42.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:42.934 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:42.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:43.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:43.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:43.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:43.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:43.181 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:10:43.182 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:10:43.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:43.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:43.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:43.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:43.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:43.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:43.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:43.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:43.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:43.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 79980 79964 97.9831 51.75 0.0108289 0.0102005 2024-06-14T23:10:43.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:43.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:43.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:43.684 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:43.685 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:43.685 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/487191375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:10:43.685 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:43 smithi086 ceph-mon[97913]: pgmap v1173: 119 pgs: 1 active+recovery_wait+degraded+remapped, 2 remapped+peering, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 12 KiB/s rd, 31 MiB/s wr, 511 op/s; 47007/231945 objects degraded (20.266%); 8229/231945 objects misplaced (3.548%); 3.1 MiB/s, 49 objects/s recovering 2024-06-14T23:10:43.685 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:43.685 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:43.685 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/487191375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:10:43.685 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:43 smithi086 ceph-mon[102754]: pgmap v1173: 119 pgs: 1 active+recovery_wait+degraded+remapped, 2 remapped+peering, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 12 KiB/s rd, 31 MiB/s wr, 511 op/s; 47007/231945 objects degraded (20.266%); 8229/231945 objects misplaced (3.548%); 3.1 MiB/s, 49 objects/s recovering 2024-06-14T23:10:43.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:43.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:43.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:43.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:43.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:43.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:43.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/487191375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:10:43.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:43 smithi137 ceph-mon[90634]: pgmap v1173: 119 pgs: 1 active+recovery_wait+degraded+remapped, 2 remapped+peering, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 4.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 12 KiB/s rd, 31 MiB/s wr, 511 op/s; 47007/231945 objects degraded (20.266%); 8229/231945 objects misplaced (3.548%); 3.1 MiB/s, 49 objects/s recovering 2024-06-14T23:10:43.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:43.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:43.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:43.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:43.980 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:43.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:44.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-14T23:10:44.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:44.129 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:44.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:44.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:44.261 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:44.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:44.336 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:44 smithi086 podman[267584]: 2024-06-14 23:10:44.120696774 +0000 UTC m=+0.561466049 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T23:10:44.337 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:44 smithi086 podman[267584]: 2024-06-14 23:10:44.288945743 +0000 UTC m=+0.729715016 container create b4cf93786782b987dcad86cfd9d1818e0ce7d405968740a62f021f058eaaf0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3) 2024-06-14T23:10:44.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:44.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:44.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:44.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:44.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 16 80911 80895 97.2178 58.1875 0.01437 0.0102803 2024-06-14T23:10:44.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:44.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:44.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:44.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:44.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:44.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:44 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:10:44.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:44 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:10:44.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/487191375' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:10:44.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:44 smithi137 ceph-mon[90634]: osdmap e318: 8 total, 7 up, 6 in 2024-06-14T23:10:44.710 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:44 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:10:44.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:10:44.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:10:44.718 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/487191375' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:10:44.719 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[102754]: osdmap e318: 8 total, 7 up, 6 in 2024-06-14T23:10:44.719 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:10:44.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:10:44.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:10:44.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/487191375' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:10:44.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[97913]: osdmap e318: 8 total, 7 up, 6 in 2024-06-14T23:10:44.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:10:44.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:44.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:44.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:44.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:44.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:44.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:44.983 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:10:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:10:44] "GET /metrics HTTP/1.1" 200 36643 "" "Prometheus/2.43.0" 2024-06-14T23:10:44.984 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:44 smithi086 podman[267584]: 2024-06-14 23:10:44.832463944 +0000 UTC m=+1.273233221 container init b4cf93786782b987dcad86cfd9d1818e0ce7d405968740a62f021f058eaaf0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-06-14T23:10:44.984 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:44 smithi086 podman[267584]: 2024-06-14 23:10:44.843398336 +0000 UTC m=+1.284167605 container start b4cf93786782b987dcad86cfd9d1818e0ce7d405968740a62f021f058eaaf0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, io.buildah.version=1.35.3) 2024-06-14T23:10:44.984 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:44 smithi086 podman[267584]: 2024-06-14 23:10:44.982126986 +0000 UTC m=+1.422896268 container attach b4cf93786782b987dcad86cfd9d1818e0ce7d405968740a62f021f058eaaf0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, ceph=True) 2024-06-14T23:10:45.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:45.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:45.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:45.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:45.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:45.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:45.321 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:10:45.322 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:10:45.322 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:10:45.322 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:10:45.322 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-14T23:10:45.345 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:10:45.345 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:10:45.376 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:10:45.376 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:10:45.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:45.402 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:10:45.403 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:10:45.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:45.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:45.463 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:10:45.464 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:10:45.487 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:10:45.487 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:10:45.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 16 81811 81795 96.4447 56.25 0.0139752 0.0103629 2024-06-14T23:10:45.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:45.544 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:10:45.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:45.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:45.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:45.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:45 smithi086 ceph-mon[97913]: pgmap v1175: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 10 KiB/s rd, 41 MiB/s wr, 663 op/s; 56988/236382 objects degraded (24.108%); 17019/236382 objects misplaced (7.200%); 2.7 MiB/s, 43 objects/s recovering 2024-06-14T23:10:45.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3461173769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:45.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:45 smithi086 ceph-mon[102754]: pgmap v1175: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 10 KiB/s rd, 41 MiB/s wr, 663 op/s; 56988/236382 objects degraded (24.108%); 17019/236382 objects misplaced (7.200%); 2.7 MiB/s, 43 objects/s recovering 2024-06-14T23:10:45.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3461173769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:45.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:45.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:45.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:45.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:45 smithi137 ceph-mon[90634]: pgmap v1175: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 10 KiB/s rd, 41 MiB/s wr, 663 op/s; 56988/236382 objects degraded (24.108%); 17019/236382 objects misplaced (7.200%); 2.7 MiB/s, 43 objects/s recovering 2024-06-14T23:10:45.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3461173769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:10:45.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:45.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:45.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:45.949 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:45.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:46.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:46.076 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:46.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:46.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:46.210 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:46.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:46.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:46.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:46.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:46.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:46.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:46.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:46.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 82715 82699 95.7049 56.5 0.019236 0.0104427 2024-06-14T23:10:46.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:46.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:46.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:46.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:46.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:10:46.788 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:10:46.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:46.910 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:46.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:46.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:47.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:47.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:47.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:47.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:47.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:47.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:47.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:47.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:47.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:47.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:47 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 58413/243849 objects degraded (23.955%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:47.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:47 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 58413/243849 objects degraded (23.955%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:47.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:47.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:47.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:47.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:47 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 58413/243849 objects degraded (23.955%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:47.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 83620 83604 94.9932 56.5625 0.0310593 0.0105206 2024-06-14T23:10:47.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:47.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:47.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:47.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:47.734 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:47.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:47.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:47.870 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:47.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:47.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:48.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:48.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:48.039 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:48 smithi137 ceph-mon[90634]: pgmap v1176: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 42 MiB/s wr, 678 op/s; 58413/243849 objects degraded (23.955%); 17258/243849 objects misplaced (7.077%); 4.4 MiB/s, 69 objects/s recovering 2024-06-14T23:10:48.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:48.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:48.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:48.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:48.286 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:48.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:48.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:48 smithi086 ceph-mon[97913]: pgmap v1176: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 42 MiB/s wr, 678 op/s; 58413/243849 objects degraded (23.955%); 17258/243849 objects misplaced (7.077%); 4.4 MiB/s, 69 objects/s recovering 2024-06-14T23:10:48.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:48 smithi086 ceph-mon[102754]: pgmap v1176: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 42 MiB/s wr, 678 op/s; 58413/243849 objects degraded (23.955%); 17258/243849 objects misplaced (7.077%); 4.4 MiB/s, 69 objects/s recovering 2024-06-14T23:10:48.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:48.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:48.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:48.509 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 84583 84567 94.3716 60.1875 0.00714594 0.0105903 2024-06-14T23:10:48.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:48.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:48.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:48.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:48.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:48.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:48.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:10:48.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:48.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:48.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:48.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:48.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:48.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:49.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:49.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:49.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:49.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:49.256 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:49.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:49.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:49.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:49.394 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:49.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:49.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:49.510 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 85473 85457 93.6915 55.625 0.0190597 0.0106671 2024-06-14T23:10:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:49.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:49.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:49.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:49.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:49.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:49.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:49.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:49.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:49.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:49.952 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:49.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:50.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:50.085 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:50.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:50.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:50.118 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:50 smithi137 ceph-mon[90634]: pgmap v1177: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 47 MiB/s wr, 754 op/s; 58986/249876 objects degraded (23.606%); 17800/249876 objects misplaced (7.124%); 3.5 MiB/s, 56 objects/s recovering 2024-06-14T23:10:50.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:50.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:50.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:50.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:50.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:50 smithi086 ceph-mon[97913]: pgmap v1177: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 47 MiB/s wr, 754 op/s; 58986/249876 objects degraded (23.606%); 17800/249876 objects misplaced (7.124%); 3.5 MiB/s, 56 objects/s recovering 2024-06-14T23:10:50.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:50 smithi086 ceph-mon[102754]: pgmap v1177: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 47 MiB/s wr, 754 op/s; 58986/249876 objects degraded (23.606%); 17800/249876 objects misplaced (7.124%); 3.5 MiB/s, 56 objects/s recovering 2024-06-14T23:10:50.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:50.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:50.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:50.497 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:50.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:50.510 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 16 86413 86397 93.089 58.75 0.0308669 0.0107368 2024-06-14T23:10:50.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:50.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:50.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:50.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:50.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:50.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:50.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:50.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:50.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:50.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:51.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:51.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:51.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:51.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:51.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:51.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:51.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:51.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:51.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:51.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:51.510 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 87307 87291 92.4582 55.875 0.011582 0.0108106 2024-06-14T23:10:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:51.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:51.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:51.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:51.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:51.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:51.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:51.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:51.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:51 smithi137 ceph-mon[90634]: pgmap v1178: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 48 MiB/s wr, 773 op/s; 59578/251652 objects degraded (23.675%); 17800/251652 objects misplaced (7.073%); 4.7 MiB/s, 75 objects/s recovering 2024-06-14T23:10:51.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:51.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:51.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:51.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:51.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:51.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:51.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:51 smithi086 ceph-mon[97913]: pgmap v1178: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 48 MiB/s wr, 773 op/s; 59578/251652 objects degraded (23.675%); 17800/251652 objects misplaced (7.073%); 4.7 MiB/s, 75 objects/s recovering 2024-06-14T23:10:51.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:51.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:51.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:51 smithi086 ceph-mon[102754]: pgmap v1178: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 48 MiB/s wr, 773 op/s; 59578/251652 objects degraded (23.675%); 17800/251652 objects misplaced (7.073%); 4.7 MiB/s, 75 objects/s recovering 2024-06-14T23:10:51.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:51.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:51.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:52.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:52.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:10:52.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:52.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:52.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:52.213 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:52.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:52.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:52.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:52.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:52.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:52.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:52.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:52.510 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:10:52.509181+0000 min lat: 0.00120633 max lat: 2.44501 avg lat: 0.0108766 2024-06-14T23:10:52.510 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:10:52.510 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 88248 88232 91.8974 58.8125 0.0154594 0.0108766 2024-06-14T23:10:52.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:52.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:52.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:52.685 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:52.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:52 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:10:52.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:52.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:52.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:52.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:52 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:10:52.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:52 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:10:52.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:52.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:52.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:53.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:53.054 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:53.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:53.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:53.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:53.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:53.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:53.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:53.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:53.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:53.466 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:53.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:53.510 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 89178 89162 91.3436 58.125 0.0188718 0.0109421 2024-06-14T23:10:53.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:53.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:53 smithi086 ceph-mon[97913]: pgmap v1179: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 53 MiB/s wr, 851 op/s; 60407/257487 objects degraded (23.460%); 18094/257487 objects misplaced (7.027%); 3.4 MiB/s, 54 objects/s recovering 2024-06-14T23:10:53.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 59578/251652 objects degraded (23.675%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:53.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:53 smithi086 ceph-mon[102754]: pgmap v1179: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 53 MiB/s wr, 851 op/s; 60407/257487 objects degraded (23.460%); 18094/257487 objects misplaced (7.027%); 3.4 MiB/s, 54 objects/s recovering 2024-06-14T23:10:53.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 59578/251652 objects degraded (23.675%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:53.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:53.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:53.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:53.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:53.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:53.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:53.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:53 smithi137 ceph-mon[90634]: pgmap v1179: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 53 MiB/s wr, 851 op/s; 60407/257487 objects degraded (23.460%); 18094/257487 objects misplaced (7.027%); 3.4 MiB/s, 54 objects/s recovering 2024-06-14T23:10:53.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 59578/251652 objects degraded (23.675%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:53.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:53.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:53.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:53.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:53.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:54.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:54.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:54.045 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:53 smithi086 conmon[267790]: conmon b4cf93786782b987dcad : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-b4cf93786782b987dcad86cfd9d1818e0ce7d405968740a62f021f058eaaf0f6.scope/memory.events 2024-06-14T23:10:54.045 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:10:53 smithi086 podman[267584]: 2024-06-14 23:10:53.695190647 +0000 UTC m=+10.135959921 container died b4cf93786782b987dcad86cfd9d1818e0ce7d405968740a62f021f058eaaf0f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-14T23:10:54.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:54.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:54.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:54.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:54.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:54.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:54.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:54.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:54.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:54.510 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 90140 90124 90.8401 60.125 0.0172427 0.0110031 2024-06-14T23:10:54.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:54.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:54.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:54.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:54.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:54.694 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:54.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:54.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:54.833 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:54.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:54.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:54.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:54.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:55.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:55.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:55.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:55.169 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:10:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:10:54] "GET /metrics HTTP/1.1" 200 36643 "" "Prometheus/2.43.0" 2024-06-14T23:10:55.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.2.service' 2024-06-14T23:10:55.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:55.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:55.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:55.267 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:10:55.267 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-14T23:10:55.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:55.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:55.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:55.409 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:10:55.410 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:10:55.410 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-06-14T23:10:55.410 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:10:55.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:55.512 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 91104 91088 90.3522 60.25 0.0152834 0.0110622 2024-06-14T23:10:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:55.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:55.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:55.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:55.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:55.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:55.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:55.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:55.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:55.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:56.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:56.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:56.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:56.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:55 smithi086 ceph-mon[97913]: pgmap v1180: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 49 MiB/s wr, 790 op/s; 61020/263748 objects degraded (23.136%); 18690/263748 objects misplaced (7.086%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T23:10:56.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:55 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:10:56.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:55 smithi086 ceph-mon[102754]: pgmap v1180: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 49 MiB/s wr, 790 op/s; 61020/263748 objects degraded (23.136%); 18690/263748 objects misplaced (7.086%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T23:10:56.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:55 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:10:56.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-14T23:10:56.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:56.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:56.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:56.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:55 smithi137 ceph-mon[90634]: pgmap v1180: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 49 MiB/s wr, 790 op/s; 61020/263748 objects degraded (23.136%); 18690/263748 objects misplaced (7.086%); 2.9 MiB/s, 46 objects/s recovering 2024-06-14T23:10:56.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:55 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:10:56.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:56.331 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:56.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:56.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:56.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:56.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:56.512 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 92099 92083 89.9115 62.1875 0.0392113 0.0111166 2024-06-14T23:10:56.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:56.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:56.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:56.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:56.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:56.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:56.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:56.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:56.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:56.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:56.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:57.044 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:10:57.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:57.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:57.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:57.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:10:57.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:57.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:10:57.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:56 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:10:57.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:56 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:10:57.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:57.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:57.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:57.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:56 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:10:57.512 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 93019 93003 89.4128 57.5 0.0106962 0.011179 2024-06-14T23:10:57.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:57.548 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:57.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:57.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:57.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:57.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:57.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:57.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:57.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:10:57.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:57.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:57.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:58.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:58.100 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:58.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:58.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:58.238 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:58.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:58.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:58.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:57 smithi086 ceph-mon[102754]: pgmap v1181: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 61 MiB/s wr, 981 op/s; 62477/271731 objects degraded (22.992%); 18803/271731 objects misplaced (6.920%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T23:10:58.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:58.265 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:58.265 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:57 smithi086 ceph-mon[97913]: pgmap v1181: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 61 MiB/s wr, 981 op/s; 62477/271731 objects degraded (22.992%); 18803/271731 objects misplaced (6.920%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T23:10:58.265 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:58.265 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:58.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:58.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:58.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:58.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:10:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:57 smithi137 ceph-mon[90634]: pgmap v1181: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 61 MiB/s wr, 981 op/s; 62477/271731 objects degraded (22.992%); 18803/271731 objects misplaced (6.920%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T23:10:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:10:58.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:10:58.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:58.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:58.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:10:58.512 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 93951 93935 88.9406 58.25 0.044892 0.0112378 2024-06-14T23:10:58.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:58.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:58.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:58.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:58.772 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:58.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:58.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:58.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:10:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:58.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:58.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:10:59.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:10:59.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:59.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:10:59.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:59.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:59.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:59.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:59 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:10:59.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:10:59 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 62477/271731 objects degraded (22.992%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:59.268 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:59 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:10:59.268 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:10:59 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 62477/271731 objects degraded (22.992%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:59.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:59.296 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:10:59.296 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:10:59.297 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:10:59.297 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:10:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:10:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:10:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-14T23:10:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:10:59.297 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:10:59.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:59.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:59.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:59.335 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:10:59.335 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:10:59.371 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:10:59.371 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:10:59.396 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:10:59.396 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:10:59.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:59.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:59.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:59 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:10:59.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:10:59 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 62477/271731 objects degraded (22.992%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:10:59.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 94883 94867 88.4825 58.25 0.0172634 0.0112962 2024-06-14T23:10:59.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:10:59.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:59.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:10:59.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:59.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:59.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:59.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:10:59.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:10:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:59.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:10:59.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:10:59.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:10:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:10:59.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:00.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:00.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:00.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:00.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:00.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:00.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:00.268 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 2] out_osds: [0, 1] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 7, 6] 2024-06-14T23:11:00.268 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-06-14T23:11:00.268 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-06-14T23:11:00.268 DEBUG:teuthology.orchestra.run.smithi086:osd.2> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.2.service 2024-06-14T23:11:00.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl start ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.2 2024-06-14T23:11:00.323 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:00 smithi086 ceph-mon[97913]: pgmap v1182: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 59 MiB/s wr, 945 op/s; 63085/277908 objects degraded (22.700%); 19374/277908 objects misplaced (6.971%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:00.323 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1945380874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:00.326 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:00 smithi086 ceph-mon[102754]: pgmap v1182: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 59 MiB/s wr, 945 op/s; 63085/277908 objects degraded (22.700%); 19374/277908 objects misplaced (6.971%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:00.326 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1945380874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:00.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:00.392 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:00.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:00.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:00.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:00 smithi137 ceph-mon[90634]: pgmap v1182: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 59 MiB/s wr, 945 op/s; 63085/277908 objects degraded (22.700%); 19374/277908 objects misplaced (6.971%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:00.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1945380874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:00.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 95836 95820 88.0571 59.5625 0.0368373 0.0113513 2024-06-14T23:11:00.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:00.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:00.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:00 smithi086 systemd[1]: Starting Ceph osd.2 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:11:00.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:00.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:00.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:00.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:00.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:00.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:00.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:00.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:01.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:01.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:01.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:01 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:11:01.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:01 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:11:01.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:01.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:01.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:01.234 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:11:01.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:11:01.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:01.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:01.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:01.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:01.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:01.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:01.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:01 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:11:01.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 96802 96786 87.6559 60.375 0.022797 0.0114029 2024-06-14T23:11:01.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:01.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:01.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:01.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:01.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:01.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:01.869 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:01.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:01.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:02.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:02.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:02.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:02.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:02.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:02.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:02.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:02.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:02.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:02 smithi086 ceph-mon[97913]: pgmap v1183: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 52 MiB/s wr, 830 op/s; 63711/279786 objects degraded (22.771%); 19189/279786 objects misplaced (6.858%); 3.5 MiB/s, 55 objects/s recovering 2024-06-14T23:11:02.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:02 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:11:02.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:02 smithi086 ceph-mon[102754]: pgmap v1183: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 52 MiB/s wr, 830 op/s; 63711/279786 objects degraded (22.771%); 19189/279786 objects misplaced (6.858%); 3.5 MiB/s, 55 objects/s recovering 2024-06-14T23:11:02.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:02 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:11:02.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:02.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:02.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:02.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:02 smithi137 ceph-mon[90634]: pgmap v1183: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 52 MiB/s wr, 830 op/s; 63711/279786 objects degraded (22.771%); 19189/279786 objects misplaced (6.858%); 3.5 MiB/s, 55 objects/s recovering 2024-06-14T23:11:02.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:02 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:11:02.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:02.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 97741 97725 87.242 58.6875 0.0108037 0.0114555 2024-06-14T23:11:02.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:02.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:02.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:02.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:02.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:02.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:02.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:02.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:02.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:02.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:02.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:03.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:03.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:03.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:03.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:03.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:03.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:03.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:11:03.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:03.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:03.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:03.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:03.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:03.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:03.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 98709 98693 86.8653 60.5 0.0124802 0.011506 2024-06-14T23:11:03.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:03.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:03.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:03.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:03.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:03.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:03.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:03.890 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:03 smithi086 podman[269864]: 2024-06-14 23:11:03.615904078 +0000 UTC m=+0.866891142 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T23:11:03.891 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:03 smithi086 podman[269864]: 2024-06-14 23:11:03.889080716 +0000 UTC m=+1.140067767 container create d836bc0abe60303ba87019ebdaac6135d56ec80b2e0b8d2514773cc5c6c5c9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T23:11:03.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:03.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:04.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:04.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:04.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:04.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:04.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:04.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:04.301 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:04 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:11:04.301 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:04 smithi086 ceph-mon[102754]: pgmap v1184: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 60 MiB/s wr, 953 op/s; 64596/285975 objects degraded (22.588%); 19486/285975 objects misplaced (6.814%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T23:11:04.301 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:04 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 64596/285975 objects degraded (22.588%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:04.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:04 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:11:04.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:04 smithi086 ceph-mon[97913]: pgmap v1184: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 60 MiB/s wr, 953 op/s; 64596/285975 objects degraded (22.588%); 19486/285975 objects misplaced (6.814%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T23:11:04.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:04 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 64596/285975 objects degraded (22.588%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:04.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:04.337 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:04.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:04.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:04.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:04.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:04 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:11:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:04 smithi137 ceph-mon[90634]: pgmap v1184: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 60 MiB/s wr, 953 op/s; 64596/285975 objects degraded (22.588%); 19486/285975 objects misplaced (6.814%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T23:11:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:04 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 64596/285975 objects degraded (22.588%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:04.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 99641 99625 86.4678 58.25 0.00806607 0.0115595 2024-06-14T23:11:04.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:04.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:04 smithi086 podman[269864]: 2024-06-14 23:11:04.298972376 +0000 UTC m=+1.549959428 container init d836bc0abe60303ba87019ebdaac6135d56ec80b2e0b8d2514773cc5c6c5c9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True) 2024-06-14T23:11:04.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:04 smithi086 podman[269864]: 2024-06-14 23:11:04.308738389 +0000 UTC m=+1.559725440 container start d836bc0abe60303ba87019ebdaac6135d56ec80b2e0b8d2514773cc5c6c5c9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.35.3) 2024-06-14T23:11:04.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:04.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:04.698 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:04.732 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:04.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:04.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:04.873 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:04.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:04.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:05.013 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:05.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:05.042 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:11:05.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:05.152 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3814109034' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:11:05.153 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:05 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:11:05.153 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3814109034' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:11:05.153 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:05 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:11:05.153 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:11:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:11:04] "GET /metrics HTTP/1.1" 200 36636 "" "Prometheus/2.43.0" 2024-06-14T23:11:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:05.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:05.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:05.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:05.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:05.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:05.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:05.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:05.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3814109034' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:11:05.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:05 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:11:05.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 100561 100545 86.071 57.5 0.0308504 0.0116134 2024-06-14T23:11:05.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:05.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:05.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:05.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:05 smithi086 podman[269864]: 2024-06-14 23:11:05.151115862 +0000 UTC m=+2.402102928 container attach d836bc0abe60303ba87019ebdaac6135d56ec80b2e0b8d2514773cc5c6c5c9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-14T23:11:05.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:05.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:05.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:05.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:05.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:05.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:05.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:05.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:05.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:06.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:06.123 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:06.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:06.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:06.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:06.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:06.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:11:06.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[97913]: pgmap v1185: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 60 MiB/s wr, 963 op/s; 65177/292185 objects degraded (22.307%); 20100/292185 objects misplaced (6.879%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T23:11:06.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:11:06.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:11:06.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[97913]: osdmap e319: 8 total, 7 up, 6 in 2024-06-14T23:11:06.289 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:11:06.289 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[102754]: pgmap v1185: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 60 MiB/s wr, 963 op/s; 65177/292185 objects degraded (22.307%); 20100/292185 objects misplaced (6.879%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T23:11:06.289 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:11:06.289 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:11:06.289 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-mon[102754]: osdmap e319: 8 total, 7 up, 6 in 2024-06-14T23:11:06.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:06.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:06.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:06.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:06.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:06 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:11:06.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:06 smithi137 ceph-mon[90634]: pgmap v1185: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 5.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 60 MiB/s wr, 963 op/s; 65177/292185 objects degraded (22.307%); 20100/292185 objects misplaced (6.879%); 2.0 MiB/s, 31 objects/s recovering 2024-06-14T23:11:06.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:06 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:11:06.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:06 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:11:06.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:06 smithi137 ceph-mon[90634]: osdmap e319: 8 total, 7 up, 6 in 2024-06-14T23:11:06.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 16 101511 101495 85.7101 59.375 0.0223449 0.0116621 2024-06-14T23:11:06.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:06.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:06.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:06.661 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:06.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:06.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:06.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:06.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:06.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:06.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:06.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:06.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[270099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-14T23:11:06.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 bash[269864]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-14T23:11:06.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[270099]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-14T23:11:06.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 bash[269864]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-14T23:11:06.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[270099]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-14T23:11:06.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 bash[269864]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-14T23:11:06.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[270099]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-14T23:11:06.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 bash[269864]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-14T23:11:06.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[270099]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-14T23:11:06.936 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 bash[269864]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-14T23:11:06.936 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[270099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-14T23:11:06.936 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 bash[269864]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-14T23:11:07.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:07.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:07.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:07.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:07.191 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate[270099]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-14T23:11:07.191 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 bash[269864]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-14T23:11:07.191 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:06 smithi086 conmon[270099]: conmon d836bc0abe60303ba870 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-d836bc0abe60303ba87019ebdaac6135d56ec80b2e0b8d2514773cc5c6c5c9d3.scope/memory.events 2024-06-14T23:11:07.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:07.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:07.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:07.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:07.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:07.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:07.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:07.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:07.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 102436 102420 85.3381 57.8125 0.00283879 0.0117084 2024-06-14T23:11:07.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:07.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:07.645 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:11:07.645 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:11:07.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:07.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:07.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:07.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:07.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:07.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:07.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:08.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:08.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:08.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:08.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:08.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:08.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:08.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:08.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:08.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:08 smithi086 ceph-mon[102754]: pgmap v1187: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 949 op/s; 66666/300219 objects degraded (22.206%); 20246/300219 objects misplaced (6.744%); 2.4 MiB/s, 37 objects/s recovering 2024-06-14T23:11:08.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:08 smithi086 ceph-mon[97913]: pgmap v1187: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 949 op/s; 66666/300219 objects degraded (22.206%); 20246/300219 objects misplaced (6.744%); 2.4 MiB/s, 37 objects/s recovering 2024-06-14T23:11:08.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:08.426 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:08.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:08.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:08 smithi137 ceph-mon[90634]: pgmap v1187: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 949 op/s; 66666/300219 objects degraded (22.206%); 20246/300219 objects misplaced (6.744%); 2.4 MiB/s, 37 objects/s recovering 2024-06-14T23:11:08.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-14T23:11:08.513 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 103375 103359 84.9874 58.6875 0.0190716 0.0117606 2024-06-14T23:11:08.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:08.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:08.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:08.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:08.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:08.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:08.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:08.836 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:08 smithi086 podman[269864]: 2024-06-14 23:11:08.503260413 +0000 UTC m=+5.754247460 container died d836bc0abe60303ba87019ebdaac6135d56ec80b2e0b8d2514773cc5c6c5c9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-06-14T23:11:08.852 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:08.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:08.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:08.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:08.981 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:08.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:09.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:09.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:09.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:09.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:09.179 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:09 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 66666/300219 objects degraded (22.206%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:09.179 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:09 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 66666/300219 objects degraded (22.206%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:09.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:11:09.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:09.258 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:09.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:09.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:09.398 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:09.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:09.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:09 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 66666/300219 objects degraded (22.206%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:09.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:09.514 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 16 104324 104308 84.6539 59.3125 0.0145857 0.0118076 2024-06-14T23:11:09.523 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:09 smithi086 podman[270462]: 2024-06-14 23:11:09.195094953 +0000 UTC m=+2.213912503 container remove d836bc0abe60303ba87019ebdaac6135d56ec80b2e0b8d2514773cc5c6c5c9d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:11:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:09.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:09.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:09.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:09.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:09.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:09.709 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:09.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:09.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:09.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:09.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:09.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:09.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:10.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:10 smithi086 ceph-mon[97913]: pgmap v1188: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 950 op/s; 67232/306426 objects degraded (21.941%); 20842/306426 objects misplaced (6.802%); 2.4 MiB/s, 37 objects/s recovering 2024-06-14T23:11:10.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:10 smithi086 ceph-mon[102754]: pgmap v1188: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 950 op/s; 67232/306426 objects degraded (21.941%); 20842/306426 objects misplaced (6.802%); 2.4 MiB/s, 37 objects/s recovering 2024-06-14T23:11:10.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:10.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:10.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:10.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:10.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:10.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:10.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:10.396 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:10 smithi137 ceph-mon[90634]: pgmap v1188: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 950 op/s; 67232/306426 objects degraded (21.941%); 20842/306426 objects misplaced (6.802%); 2.4 MiB/s, 37 objects/s recovering 2024-06-14T23:11:10.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:10.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:10.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:10.514 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 105260 105244 84.3185 58.5 0.0185128 0.0118538 2024-06-14T23:11:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:10.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:10.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:10.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:10.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:10.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:10.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:10.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:10.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:10.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:10.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:10.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:10.996 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:10 smithi086 podman[270864]: 2024-06-14 23:11:10.611476965 +0000 UTC m=+0.656699505 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T23:11:10.996 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:10 smithi086 podman[270864]: 2024-06-14 23:11:10.77468716 +0000 UTC m=+0.819909707 container create 8d3db12678a64641df0dd43eeccbf68efc095493c8138d732bf37d3ad7857178 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T23:11:11.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:11.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:11.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:11.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:11.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:11.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:11.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:11.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:11.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:11.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:11.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:11.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:11.514 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 106119 106103 83.9307 53.6875 0.00720555 0.0119096 2024-06-14T23:11:11.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:11.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:11.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:11.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:11.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:12.037 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:12.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:12.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:12.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:12.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:12.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:12.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:12.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:12.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:12.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:12.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:12 smithi086 ceph-mon[97913]: pgmap v1189: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 943 op/s; 67792/308106 objects degraded (22.003%); 20639/308106 objects misplaced (6.699%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:12.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:12 smithi086 ceph-mon[102754]: pgmap v1189: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 943 op/s; 67792/308106 objects degraded (22.003%); 20639/308106 objects misplaced (6.699%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:12.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:12.472 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:12.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:12.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:12 smithi137 ceph-mon[90634]: pgmap v1189: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 943 op/s; 67792/308106 objects degraded (22.003%); 20639/308106 objects misplaced (6.699%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:12.514 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:11:12.513347+0000 min lat: 0.00120633 max lat: 2.44501 avg lat: 0.0119641 2024-06-14T23:11:12.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:11:12.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 106970 106954 83.546 53.1875 0.023319 0.0119641 2024-06-14T23:11:12.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:12.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:12.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:12.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:12.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:12.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:12.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:12.857 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:12.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:12.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:12.991 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:12.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:13.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:13.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:13.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:13.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:13.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:13.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:13.264 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:13 smithi086 podman[270864]: 2024-06-14 23:11:13.100675963 +0000 UTC m=+3.145898528 container init 8d3db12678a64641df0dd43eeccbf68efc095493c8138d732bf37d3ad7857178 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-14T23:11:13.264 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:13 smithi086 podman[270864]: 2024-06-14 23:11:13.112360991 +0000 UTC m=+3.157583533 container start 8d3db12678a64641df0dd43eeccbf68efc095493c8138d732bf37d3ad7857178 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240528) 2024-06-14T23:11:13.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:13.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:13.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:13.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:13.514 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 107953 107937 83.273 61.4375 0.00954228 0.012003 2024-06-14T23:11:13.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:13.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:13.587 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-osd[271306]: -- 172.21.15.86:0/366560747 <== mon.0 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55de554b91e0 con 0x55de55457c00 2024-06-14T23:11:13.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:13.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:13.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-mon[97913]: pgmap v1190: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 58 MiB/s wr, 934 op/s; 68615/314034 objects degraded (21.850%); 20930/314034 objects misplaced (6.665%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:13.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 67792/308106 objects degraded (22.003%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:13.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:13.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:13.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-mon[102754]: pgmap v1190: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 58 MiB/s wr, 934 op/s; 68615/314034 objects degraded (21.850%); 20930/314034 objects misplaced (6.665%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:13.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 67792/308106 objects degraded (22.003%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:13.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:13.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:13.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:13.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:13 smithi137 ceph-mon[90634]: pgmap v1190: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 58 MiB/s wr, 934 op/s; 68615/314034 objects degraded (21.850%); 20930/314034 objects misplaced (6.665%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 67792/308106 objects degraded (22.003%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:13.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:13.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:13.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:13.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:13.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:13.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:14.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:14.042 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[270968]: 2024-06-14T23:11:13.793+0000 7f28f40f3640 -1 Falling back to public interface 2024-06-14T23:11:14.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:14.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:14.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:14.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:14.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:14.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:14.337 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:14.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:14.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:14.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:14.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:14.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:14.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 108917 108901 82.9922 60.25 0.00622732 0.0120432 2024-06-14T23:11:14.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:14.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:14.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:14.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:14.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:14.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:14.779 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:11:14.780 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:11:14.808 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:11:14.808 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:11:14.809 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-06-14T23:11:14.831 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:11:14.831 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:11:14.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:14.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T23:11:14.858 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:11:14.858 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:11:14.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:11:14.888 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:11:14.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:14.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:14.935 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:11:14.936 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:11:14.966 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:11:14.966 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:11:14.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:15.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:15.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:15.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1992065843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:15.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1992065843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:15.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:11:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:11:14] "GET /metrics HTTP/1.1" 200 36637 "" "Prometheus/2.43.0" 2024-06-14T23:11:15.087 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:14 smithi086 bash[270864]: 8d3db12678a64641df0dd43eeccbf68efc095493c8138d732bf37d3ad7857178 2024-06-14T23:11:15.087 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:14 smithi086 systemd[1]: Started Ceph osd.2 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T23:11:15.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:15.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:15.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:15.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1992065843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:15.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:15.308 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:15.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:15.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:15.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:15.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:15.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 16 109880 109864 82.7174 60.1875 0.009713 0.0120844 2024-06-14T23:11:15.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:15.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:15.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:15.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:15.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:15.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:15.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:15.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:15.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:15.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:15.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:15.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:16.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:15 smithi086 ceph-mon[97913]: pgmap v1191: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 58 MiB/s wr, 929 op/s; 69236/320091 objects degraded (21.630%); 21486/320091 objects misplaced (6.712%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:16.030 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:15 smithi086 ceph-mon[102754]: pgmap v1191: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 58 MiB/s wr, 929 op/s; 69236/320091 objects degraded (21.630%); 21486/320091 objects misplaced (6.712%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:11:16.062 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:11:16.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:16.116 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:16.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:16.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:16.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:15 smithi137 ceph-mon[90634]: pgmap v1191: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 58 MiB/s wr, 929 op/s; 69236/320091 objects degraded (21.630%); 21486/320091 objects misplaced (6.712%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:11:16.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:16.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:16.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:16.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:16.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:16.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:16.517 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 110838 110822 82.4454 59.875 0.0119456 0.0121237 2024-06-14T23:11:16.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:16.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:16.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:16.673 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:16.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:16.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:16.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:16.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:16.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:16.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:16.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:17.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:17.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:17.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:17.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:17.221 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252098 10.8 10:194b0bc1:::benchmark_data_smithi086_189745_object121130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:19.961385+0000", 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.25549277900001, 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.045269832, 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.222 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252098 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.961385+0000", 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.961385+0000", 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.961384+0000", 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.961393+0000", 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7820000000000005e-06 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.961405+0000", 2024-06-14T23:11:17.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2038999999999999e-05 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.961411+0000", 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6670000000000001e-06 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.961463+0000", 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1858999999999999e-05 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.961473+0000", 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0132999999999999e-05 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.999195+0000", 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0377220650000001 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.999333+0000", 2024-06-14T23:11:17.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013792000000000001 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.999504+0000", 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017045299999999999 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.003027+0000", 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003523923 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.006517+0000", 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003489626 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.006584+0000", 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7096999999999998e-05 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.006616+0000", 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1696000000000003e-05 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.006655+0000", 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9042999999999999e-05 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252108 10.8 10:10b914c8:::benchmark_data_smithi086_189745_object121140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:19.966252+0000", 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.25062585199998, 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0510758, 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.229 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252108 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.966252+0000", 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.966252+0000", 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.966251+0000", 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.966257+0000", 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5509999999999996e-06 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.966271+0000", 2024-06-14T23:11:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3799e-05 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.966278+0000", 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.798e-06 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.966326+0000", 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8041000000000003e-05 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:19.966342+0000", 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5710999999999999e-05 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.003063+0000", 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0367219059999999 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.003143+0000", 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9031999999999998e-05 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.003205+0000", 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2182999999999997e-05 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.008816+0000", 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056109369999999999 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.011009+0000", 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002193806 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017280+0000", 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0062703890000000003 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017307+0000", 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6673e-05 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017328+0000", 2024-06-14T23:11:17.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1338000000000001e-05 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252140 10.8 10:15294e77:::benchmark_data_smithi086_189745_object121172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.001312+0000", 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.21556557999997, 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0212052460000001, 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.237 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252140 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.001312+0000", 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.001312+0000", 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.001311+0000", 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.001318+0000", 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.635e-06 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.001332+0000", 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4233000000000001e-05 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.001339+0000", 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0169999999999999e-06 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.001384+0000", 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5649999999999998e-05 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.001394+0000", 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6779999999999995e-06 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.005006+0000", 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.00361229 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.005064+0000", 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7309000000000003e-05 2024-06-14T23:11:17.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.005125+0000", 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1795e-05 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.011015+0000", 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058895320000000003 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.013066+0000", 2024-06-14T23:11:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020508990000000001 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022435+0000", 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0093694169999999997 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022481+0000", 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5244000000000003e-05 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022518+0000", 2024-06-14T23:11:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6996000000000003e-05 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252165 10.8 10:1e490178:::benchmark_data_smithi086_189745_object121197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.030275+0000", 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.18660333000003, 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99242177399999998, 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252165 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.030275+0000", 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.030275+0000", 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.030274+0000", 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.030279+0000", 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9899999999999997e-06 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.030289+0000", 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0455000000000001e-05 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.030305+0000", 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5620999999999998e-05 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.030339+0000", 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4372000000000002e-05 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.030349+0000", 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0138e-05 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.006689+0000", 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97633952099999999 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.006757+0000", 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8034999999999995e-05 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.007010+0000", 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025322000000000002 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.013071+0000", 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060610029999999997 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017219+0000", 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041477739999999999 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022616+0000", 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053970010000000002 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022658+0000", 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2341e-05 2024-06-14T23:11:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022696+0000", 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8362000000000003e-05 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252170 10.8 10:16e69ad3:::benchmark_data_smithi086_189745_object121202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.032571+0000", 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.18430696199999, 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99030390899999998, 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252170 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.032571+0000", 2024-06-14T23:11:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.032571+0000", 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.032570+0000", 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.032575+0000", 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4249999999999997e-06 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.032585+0000", 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9150000000000008e-06 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.032591+0000", 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3110000000000002e-06 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.032632+0000", 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0491000000000002e-05 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.032640+0000", 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7780000000000007e-06 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.008838+0000", 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97619878299999996 2024-06-14T23:11:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.008961+0000", 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000122404 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.009093+0000", 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000131855 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017184+0000", 2024-06-14T23:11:17.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080911049999999995 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017244+0000", 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0495999999999998e-05 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022793+0000", 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055483659999999999 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022839+0000", 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6037e-05 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022875+0000", 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6157e-05 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252180 10.8 10:108b5542:::benchmark_data_smithi086_189745_object121212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.048028+0000", 2024-06-14T23:11:17.261 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.16884968199997, 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98148007699999995, 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252180 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.048028+0000", 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.048028+0000", 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.048027+0000", 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.048032+0000", 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9470000000000002e-06 2024-06-14T23:11:17.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.048041+0000", 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8720000000000008e-06 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.048045+0000", 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6090000000000003e-06 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.048098+0000", 2024-06-14T23:11:17.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2234999999999997e-05 2024-06-14T23:11:17.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.048107+0000", 2024-06-14T23:11:17.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4800000000000007e-06 2024-06-14T23:11:17.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.011033+0000", 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96292582500000001 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.011137+0000", 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000104536 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.011250+0000", 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011297 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017250+0000", 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059998669999999999 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017358+0000", 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000107452 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.029430+0000", 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012071791 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.029474+0000", 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4218000000000003e-05 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.029508+0000", 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4513e-05 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.268 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252208 10.8 10:14ea0a56:::benchmark_data_smithi086_189745_object121240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.077704+0000", 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.13917371299999, 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0431709410000001, 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252208 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.077704+0000", 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.077704+0000", 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.077703+0000", 2024-06-14T23:11:17.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.077709+0000", 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5600000000000001e-06 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.077720+0000", 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1167e-05 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.077725+0000", 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9409999999999998e-06 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.077768+0000", 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3467999999999998e-05 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.077778+0000", 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0156000000000001e-05 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.013090+0000", 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93531119100000004 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.013198+0000", 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108164 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.013318+0000", 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119874 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017256+0000", 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003938179 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022566+0000", 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053106660000000003 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.120791+0000", 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.098224846000000005 2024-06-14T23:11:17.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.120840+0000", 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8871e-05 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.120875+0000", 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4975999999999999e-05 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252213 10.8 10:166b668c:::benchmark_data_smithi086_189745_object121245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.085045+0000", 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.13183269400002, 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.035977981, 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.276 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252213 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.085045+0000", 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.085045+0000", 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.085044+0000", 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.085050+0000", 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7789999999999998e-06 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.085065+0000", 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.429e-05 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.085072+0000", 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3699999999999997e-06 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.085112+0000", 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9509999999999999e-05 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.085120+0000", 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9100000000000005e-06 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.015230+0000", 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93011032299999996 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.015293+0000", 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3238000000000007e-05 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.015379+0000", 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.577e-05 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022401+0000", 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070221090000000003 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022747+0000", 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00034599799999999998 2024-06-14T23:11:17.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.120930+0000", 2024-06-14T23:11:17.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.098183193000000002 2024-06-14T23:11:17.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:17.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:17.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:17.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.120991+0000", 2024-06-14T23:11:17.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0368000000000002e-05 2024-06-14T23:11:17.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.121023+0000", 2024-06-14T23:11:17.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2636999999999999e-05 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252230 10.8 10:13e69e69:::benchmark_data_smithi086_189745_object121262:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.107144+0000", 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.10973347599997, 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.018020841, 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.285 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252230 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.107144+0000", 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.107144+0000", 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.107143+0000", 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.107150+0000", 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2180000000000002e-06 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.107163+0000", 2024-06-14T23:11:17.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2648e-05 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.107170+0000", 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0620000000000003e-06 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.107218+0000", 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8238000000000001e-05 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.107228+0000", 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5899999999999997e-06 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017427+0000", 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.91019922399999997 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017539+0000", 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112281 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.017649+0000", 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110411 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022576+0000", 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049266539999999999 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022931+0000", 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035502100000000001 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.125080+0000", 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10214875399999999 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.125128+0000", 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8238000000000001e-05 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.125165+0000", 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7144999999999998e-05 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.291 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252234 10.8 10:1b0daa4f:::benchmark_data_smithi086_189745_object121266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.109980+0000", 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.10689793500001, 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.01527418, 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252234 2024-06-14T23:11:17.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.109980+0000", 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.109980+0000", 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.109979+0000", 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.109985+0000", 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1380000000000004e-06 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.109998+0000", 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3118999999999999e-05 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.110004+0000", 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4819999999999999e-06 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.110048+0000", 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3572999999999998e-05 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.110058+0000", 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4939999999999992e-06 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.019681+0000", 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.90962306000000004 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.019795+0000", 2024-06-14T23:11:17.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011429000000000001 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.019924+0000", 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129068 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022755+0000", 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00283139 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.022979+0000", 2024-06-14T23:11:17.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000223728 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.125196+0000", 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.102216952 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.125227+0000", 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0741999999999998e-05 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.125254+0000", 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7413e-05 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252255 10.8 10:1398ced5:::benchmark_data_smithi086_189745_object121287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.140522+0000", 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.07635625799998, 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.987351015, 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252255 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.140522+0000", 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.140522+0000", 2024-06-14T23:11:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.140521+0000", 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.140524+0000", 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.552e-06 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.140532+0000", 2024-06-14T23:11:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4749999999999996e-06 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.140535+0000", 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7100000000000001e-06 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.140580+0000", 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4376999999999999e-05 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.140587+0000", 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2250000000000003e-06 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.023004+0000", 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.88241721500000003 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.023112+0000", 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000107697 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.023243+0000", 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013079399999999999 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.029389+0000", 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061465039999999997 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.029518+0000", 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012908 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.127785+0000", 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.098266890999999995 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.127831+0000", 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5838999999999999e-05 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.127873+0000", 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1591999999999998e-05 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252256 10.8 10:1f556f68:::benchmark_data_smithi086_189745_object121288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.141171+0000", 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.07570669699999, 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99007454699999997, 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252256 2024-06-14T23:11:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141171+0000", 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141171+0000", 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141170+0000", 2024-06-14T23:11:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141176+0000", 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4210000000000003e-06 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141188+0000", 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2262000000000001e-05 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141194+0000", 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6340000000000003e-06 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141231+0000", 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6210000000000001e-05 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141238+0000", 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1919999999999999e-06 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.026056+0000", 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.88481860899999998 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.026197+0000", 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014062999999999999 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.026298+0000", 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101011 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.029555+0000", 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0032566740000000002 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.032351+0000", 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002795695 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.131079+0000", 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.098728754000000002 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.131127+0000", 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7608999999999998e-05 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.131246+0000", 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118888 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252257 10.8 10:158927dc:::benchmark_data_smithi086_189745_object121289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.141249+0000", 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.07562892800001, 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.996371648, 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252257 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141249+0000", 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141249+0000", 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141248+0000", 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141253+0000", 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3850000000000001e-06 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141267+0000", 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4154999999999999e-05 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141276+0000", 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6240000000000008e-06 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141309+0000", 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3331000000000003e-05 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.141316+0000", 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3940000000000001e-06 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.029581+0000", 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.88826563800000002 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.029679+0000", 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7862999999999995e-05 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.029790+0000", 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110296 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.034600+0000", 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048100189999999996 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.034608+0000", 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.405e-06 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.137478+0000", 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.102870008 2024-06-14T23:11:17.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.137582+0000", 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103498 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.137621+0000", 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9101999999999999e-05 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.319 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252271 10.8 10:1bc37808:::benchmark_data_smithi086_189745_object121303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.168496+0000", 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.048382355, 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96924561200000003, 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252271 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.168496+0000", 2024-06-14T23:11:17.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.168496+0000", 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.168495+0000", 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.168500+0000", 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.271e-06 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.168511+0000", 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1308000000000001e-05 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.168518+0000", 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.311e-06 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.168564+0000", 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5760999999999998e-05 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.168573+0000", 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6770000000000001e-06 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.032373+0000", 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.86380061100000005 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.032469+0000", 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5437000000000001e-05 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.032576+0000", 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010719 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.034656+0000", 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020794210000000001 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.038988+0000", 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043321369999999998 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.137672+0000", 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.098684157999999994 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.137709+0000", 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7181000000000002e-05 2024-06-14T23:11:17.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.137741+0000", 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2157999999999999e-05 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252283 10.8 10:1d455687:::benchmark_data_smithi086_189745_object121315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:20.195857+0000", 2024-06-14T23:11:17.326 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.021020432, 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.94198149499999995, 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252283 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.195857+0000", 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.195857+0000", 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.195856+0000", 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.195862+0000", 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9100000000000002e-06 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.195876+0000", 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4226999999999999e-05 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.195883+0000", 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1400000000000002e-06 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.195943+0000", 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9642000000000003e-05 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:20.195950+0000", 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1840000000000002e-06 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.034692+0000", 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.83874132999999995 2024-06-14T23:11:17.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.034791+0000", 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9128000000000004e-05 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,7", 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.034908+0000", 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116808 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.038995+0000", 2024-06-14T23:11:17.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0040878870000000001 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.039024+0000", 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8345999999999999e-05 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.137775+0000", 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.098751412999999996 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.137809+0000", 2024-06-14T23:11:17.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3828000000000002e-05 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:21.137839+0000", 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9941e-05 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252576 10.5 10:a39274f2:::benchmark_data_smithi086_189745_object121608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:22.102440+0000", 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 415.11443831000003, 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.90060157200000002, 2024-06-14T23:11:17.333 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252576 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.102440+0000", 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.102440+0000", 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.102439+0000", 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.102444+0000", 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9190000000000004e-06 2024-06-14T23:11:17.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.102460+0000", 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6062999999999999e-05 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.102468+0000", 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.605e-06 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.102830+0000", 2024-06-14T23:11:17.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036210399999999997 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.102838+0000", 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9650000000000008e-06 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.993183+0000", 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.89034523700000001 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.993279+0000", 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5879999999999997e-05 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.993380+0000", 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100277 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.997483+0000", 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041033479999999997 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.000213+0000", 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027299970000000001 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.002968+0000", 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027553019999999998 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.002999+0000", 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0448999999999999e-05 2024-06-14T23:11:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.003041+0000", 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2505e-05 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252592 10.5 10:a288115e:::benchmark_data_smithi086_189745_object121624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:22.118469+0000", 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 415.09840878799997, 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.88753291000000001, 2024-06-14T23:11:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252592 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.118469+0000", 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.118469+0000", 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.118468+0000", 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.118473+0000", 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5229999999999997e-06 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.118482+0000", 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4860000000000003e-06 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.118488+0000", 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8250000000000001e-06 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.118687+0000", 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019881099999999999 2024-06-14T23:11:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.118695+0000", 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4330000000000004e-06 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.997521+0000", 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.87882636400000003 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.997609+0000", 2024-06-14T23:11:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7484000000000005e-05 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.997682+0000", 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3114999999999994e-05 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.005740+0000", 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080578100000000003 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.005831+0000", 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1377000000000003e-05 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.005904+0000", 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2475e-05 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.005961+0000", 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7173999999999998e-05 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.006002+0000", 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1124000000000001e-05 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252598 10.5 10:add4900e:::benchmark_data_smithi086_189745_object121630:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:22.125259+0000", 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 415.09161888900002, 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.88095028099999995, 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252598 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.125259+0000", 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.125259+0000", 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.125257+0000", 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.125264+0000", 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.125279+0000", 2024-06-14T23:11:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5780000000000001e-05 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.125288+0000", 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2360000000000004e-06 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.125335+0000", 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7403e-05 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.125345+0000", 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.046e-05 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.000255+0000", 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.87490910799999999 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.000374+0000", 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011937199999999999 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:11:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.000504+0000", 2024-06-14T23:11:17.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129741 2024-06-14T23:11:17.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.006067+0000", 2024-06-14T23:11:17.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00556319 2024-06-14T23:11:17.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.006078+0000", 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0971999999999999e-05 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.006135+0000", 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7021000000000002e-05 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.006177+0000", 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2095000000000003e-05 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.006209+0000", 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2329999999999997e-05 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252621 10.5 10:a4a82ac2:::benchmark_data_smithi086_189745_object121653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:11:17.354 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:22.143117+0000", 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 415.07376098100002, 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.87447841199999998, 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252621 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.143117+0000", 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.143117+0000", 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.143116+0000", 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.143122+0000", 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4200000000000004e-06 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.143135+0000", 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2789000000000001e-05 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.143142+0000", 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3860000000000001e-06 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.143182+0000", 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9969000000000003e-05 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.143191+0000", 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1300000000000007e-06 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.003076+0000", 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.859884333 2024-06-14T23:11:17.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.003166+0000", 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0296000000000004e-05 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.003251+0000", 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5292999999999998e-05 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.010164+0000", 2024-06-14T23:11:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069131720000000004 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.010197+0000", 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3022999999999999e-05 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.017508+0000", 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0073103899999999999 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.017554+0000", 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5822999999999998e-05 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.017595+0000", 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1658e-05 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.45821.0:252633 10.5 10:a9fc19cd:::benchmark_data_smithi086_189745_object121665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:04:22.155206+0000", 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 415.061672004, 2024-06-14T23:11:17.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.86257089799999997, 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.45821", 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 252633 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.155206+0000", 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.155206+0000", 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.155205+0000", 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.155209+0000", 2024-06-14T23:11:17.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3810000000000001e-06 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.155220+0000", 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0623e-05 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.155226+0000", 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4809999999999997e-06 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.155284+0000", 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8669999999999999e-05 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:22.155293+0000", 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5339999999999999e-06 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.006244+0000", 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.850950816 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.006358+0000", 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114373 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.006469+0000", 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111352 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.017656+0000", 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011186644000000001 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.017701+0000", 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4703000000000002e-05 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.017709+0000", 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4800000000000001e-06 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.017746+0000", 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7259999999999999e-05 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:04:23.017777+0000", 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0450000000000002e-05 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:11:17.368 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:11:17.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:17.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:17.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:17.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 111820 111804 82.1974 61.375 0.0124761 0.0121601 2024-06-14T23:11:17.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:17.567 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:17.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:17.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:17.700 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:17.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:17.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:17.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:17.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:17.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:17.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:17.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:18.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:11:18.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:18.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:18.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:18.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:18.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:18.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:18.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:17 smithi086 ceph-mon[97913]: pgmap v1192: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 62 MiB/s wr, 999 op/s; 70684/327885 objects degraded (21.558%); 21556/327885 objects misplaced (6.574%); 3.1 MiB/s, 50 objects/s recovering 2024-06-14T23:11:18.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:18 smithi086 ceph-mon[102754]: pgmap v1192: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 62 MiB/s wr, 999 op/s; 70684/327885 objects degraded (21.558%); 21556/327885 objects misplaced (6.574%); 3.1 MiB/s, 50 objects/s recovering 2024-06-14T23:11:18.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:18.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:18.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:18.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:17 smithi137 ceph-mon[90634]: pgmap v1192: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.7 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 62 MiB/s wr, 999 op/s; 70684/327885 objects degraded (21.558%); 21556/327885 objects misplaced (6.574%); 3.1 MiB/s, 50 objects/s recovering 2024-06-14T23:11:18.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:18.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 112800 112784 81.9538 61.25 0.00596609 0.0121967 2024-06-14T23:11:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:18.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:18.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:18.661 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:18.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:18.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:18.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:18.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:18.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-14T23:11:18.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:18.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:18.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:19.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:19.036 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:19 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 70684/327885 objects degraded (21.558%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:19.037 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:19 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 70684/327885 objects degraded (21.558%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:19.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:19.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:19.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:19.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:19.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:19.284 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:19.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:19.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:19.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:19.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:19.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:19.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:19.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:19 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 70684/327885 objects degraded (21.558%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:19.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 113737 113721 81.6849 58.5625 0.0214302 0.0122371 2024-06-14T23:11:19.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:19.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:19.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:19.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:19.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:19.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:19.858 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:19.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:19.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:19.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:20.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:20.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:20.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:20.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:20.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:20.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:20.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:20.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:20 smithi086 ceph-mon[102754]: pgmap v1193: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 59 MiB/s wr, 948 op/s; 71325/334368 objects degraded (21.331%); 22136/334368 objects misplaced (6.620%); 2.1 MiB/s, 33 objects/s recovering 2024-06-14T23:11:20.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:20.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:20 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:20.192 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:20 smithi086 ceph-mon[97913]: pgmap v1193: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 59 MiB/s wr, 948 op/s; 71325/334368 objects degraded (21.331%); 22136/334368 objects misplaced (6.620%); 2.1 MiB/s, 33 objects/s recovering 2024-06-14T23:11:20.192 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:20.192 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:20 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:20.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:20.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:20.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:20.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:20.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:20.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:20.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:20 smithi137 ceph-mon[90634]: pgmap v1193: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 59 MiB/s wr, 948 op/s; 71325/334368 objects degraded (21.331%); 22136/334368 objects misplaced (6.620%); 2.1 MiB/s, 33 objects/s recovering 2024-06-14T23:11:20.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:20.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:20 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:20.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 114663 114647 81.4143 57.875 0.0133294 0.0122779 2024-06-14T23:11:20.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:20.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:20.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:20.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:20.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:20.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:20.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:20.857 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:20.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:20.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:20.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:20.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:21.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:21.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:21.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:21.137 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:21.140 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:21.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:11:21.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:11:21.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:11:21.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:21.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2010678893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:21.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:21.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:11:21.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:11:21.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:21.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2010678893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:21.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:21.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:21.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:21.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:21.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:21.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:11:21.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:11:21.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:21.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2010678893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:21 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:21.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:21.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 115578 115562 81.1421 57.1875 0.0068866 0.0123182 2024-06-14T23:11:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:21.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:21.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:21.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:21.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:21.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:21.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:21.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:21.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:21.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:21.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:21.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:21.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:21.980 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[270968]: 2024-06-14T23:11:21.713+0000 7f28f40f3640 -1 osd.2 314 log_to_monitors true 2024-06-14T23:11:22.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:22.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:22.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:22.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:22.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:22.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:22.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[97913]: pgmap v1194: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 52 MiB/s wr, 824 op/s; 71325/334368 objects degraded (21.331%); 22136/334368 objects misplaced (6.620%); 2.3 MiB/s, 36 objects/s recovering 2024-06-14T23:11:22.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[97913]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-14T23:11:22.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:11:22.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:11:22.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[97913]: osdmap e320: 8 total, 7 up, 6 in 2024-06-14T23:11:22.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[97913]: from='osd.2 [v2:172.21.15.86:6818/3308282667,v1:172.21.15.86:6819/3308282667]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T23:11:22.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[97913]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T23:11:22.250 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[102754]: pgmap v1194: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 52 MiB/s wr, 824 op/s; 71325/334368 objects degraded (21.331%); 22136/334368 objects misplaced (6.620%); 2.3 MiB/s, 36 objects/s recovering 2024-06-14T23:11:22.250 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[102754]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-14T23:11:22.251 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:11:22.251 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:11:22.251 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[102754]: osdmap e320: 8 total, 7 up, 6 in 2024-06-14T23:11:22.251 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[102754]: from='osd.2 [v2:172.21.15.86:6818/3308282667,v1:172.21.15.86:6819/3308282667]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T23:11:22.251 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-mon[102754]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T23:11:22.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:22.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:22.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:22.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:22.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:22 smithi137 ceph-mon[90634]: pgmap v1194: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 52 MiB/s wr, 824 op/s; 71325/334368 objects degraded (21.331%); 22136/334368 objects misplaced (6.620%); 2.3 MiB/s, 36 objects/s recovering 2024-06-14T23:11:22.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:22 smithi137 ceph-mon[90634]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-14T23:11:22.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:22 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:11:22.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:22 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:11:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:22 smithi137 ceph-mon[90634]: osdmap e320: 8 total, 7 up, 6 in 2024-06-14T23:11:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:22 smithi137 ceph-mon[90634]: from='osd.2 [v2:172.21.15.86:6818/3308282667,v1:172.21.15.86:6819/3308282667]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T23:11:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:22 smithi137 ceph-mon[90634]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-14T23:11:22.497 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:22.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:22.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 90 1 116502 116501 80.8925 58.6875 0.0235469 0.0123572 2024-06-14T23:11:22.580 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:11:22 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[270968]: 2024-06-14T23:11:22.263+0000 7f28e7e95640 -1 osd.2 314 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T23:11:22.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:22.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:22.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:22.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:22.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:22.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:22.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:22.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:22.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:22.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:22.991 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:22.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:22.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:23.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:23.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:23.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:23.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:23.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:23.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[102754]: pgmap v1196: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 7.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 72892/342549 objects degraded (21.279%); 22209/342549 objects misplaced (6.483%); 2.7 MiB/s, 43 objects/s recovering 2024-06-14T23:11:23.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[102754]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-14T23:11:23.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[102754]: osdmap e321: 8 total, 7 up, 6 in 2024-06-14T23:11:23.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[102754]: from='osd.2 [v2:172.21.15.86:6818/3308282667,v1:172.21.15.86:6819/3308282667]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T23:11:23.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[102754]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T23:11:23.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[97913]: pgmap v1196: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 7.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 72892/342549 objects degraded (21.279%); 22209/342549 objects misplaced (6.483%); 2.7 MiB/s, 43 objects/s recovering 2024-06-14T23:11:23.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[97913]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-14T23:11:23.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[97913]: osdmap e321: 8 total, 7 up, 6 in 2024-06-14T23:11:23.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[97913]: from='osd.2 [v2:172.21.15.86:6818/3308282667,v1:172.21.15.86:6819/3308282667]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T23:11:23.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:23 smithi086 ceph-mon[97913]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T23:11:23.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:23.361 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:23.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:23.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:23.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:23.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:23.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0123 2024-06-14T23:11:23.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 116502 2024-06-14T23:11:23.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T23:11:23.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T23:11:23.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 80.8932 2024-06-14T23:11:23.515 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 46.2113 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 248.062 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 0 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1294 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 739.39 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 3969 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 0 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0123573 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.0354378 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 2.44501 2024-06-14T23:11:23.516 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00120633 2024-06-14T23:11:23.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:23.608 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:23.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:23.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:23.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:23.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:23.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:23 smithi137 ceph-mon[90634]: pgmap v1196: 119 pgs: 3 active+recovery_wait+degraded+remapped, 38 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 66 active+clean; 7.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 72892/342549 objects degraded (21.279%); 22209/342549 objects misplaced (6.483%); 2.7 MiB/s, 43 objects/s recovering 2024-06-14T23:11:23.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:23 smithi137 ceph-mon[90634]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-14T23:11:23.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:23 smithi137 ceph-mon[90634]: osdmap e321: 8 total, 7 up, 6 in 2024-06-14T23:11:23.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:23 smithi137 ceph-mon[90634]: from='osd.2 [v2:172.21.15.86:6818/3308282667,v1:172.21.15.86:6819/3308282667]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T23:11:23.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:23 smithi137 ceph-mon[90634]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-06-14T23:11:23.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:23.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:23.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:23.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:23.979 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:23.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:24.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:24.103 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:24.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:24.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:24.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:24.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:24.308 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[97913]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[97913]: osd.2 [v2:172.21.15.86:6818/3308282667,v1:172.21.15.86:6819/3308282667] boot 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[97913]: osdmap e322: 8 total, 8 up, 6 in 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 72892/342549 objects degraded (21.279%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[97913]: osdmap e323: 8 total, 8 up, 6 in 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[102754]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[102754]: osd.2 [v2:172.21.15.86:6818/3308282667,v1:172.21.15.86:6819/3308282667] boot 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[102754]: osdmap e322: 8 total, 8 up, 6 in 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T23:11:24.309 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 72892/342549 objects degraded (21.279%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:24.310 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-mon[102754]: osdmap e323: 8 total, 8 up, 6 in 2024-06-14T23:11:24.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:24.349 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:24.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:24.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:24.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:24.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:24.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:24.628 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:24.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:24.684 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T23:11:24.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:24 smithi137 ceph-mon[90634]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T23:11:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:24 smithi137 ceph-mon[90634]: osd.2 [v2:172.21.15.86:6818/3308282667,v1:172.21.15.86:6819/3308282667] boot 2024-06-14T23:11:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:24 smithi137 ceph-mon[90634]: osdmap e322: 8 total, 8 up, 6 in 2024-06-14T23:11:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:24 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-14T23:11:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:24 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 72892/342549 objects degraded (21.279%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:24 smithi137 ceph-mon[90634]: osdmap e323: 8 total, 8 up, 6 in 2024-06-14T23:11:24.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:24.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:24.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:24.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:24.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:24.950 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:11:24.950 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:11:24.950 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-06-14T23:11:24.950 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:11:25.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:25.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:25.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:25.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:25.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:25.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:25.239 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:11:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:11:24] "GET /metrics HTTP/1.1" 200 36637 "" "Prometheus/2.43.0" 2024-06-14T23:11:25.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:25.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:25.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:25.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:25.432 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:25.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:25.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:25.570 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:25.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:25.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:25 smithi086 ceph-mon[97913]: pgmap v1200: 119 pgs: 2 remapped+peering, 10 peering, 1 active+recovery_wait+degraded+remapped, 30 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 66 active+clean; 7.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 69 MiB/s wr, 1.10k op/s; 56499/347607 objects degraded (16.254%); 9364/347607 objects misplaced (2.694%); 3.1 MiB/s, 49 objects/s recovering 2024-06-14T23:11:25.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:25 smithi086 ceph-mon[97913]: osdmap e324: 8 total, 8 up, 6 in 2024-06-14T23:11:25.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:25 smithi086 ceph-mon[102754]: pgmap v1200: 119 pgs: 2 remapped+peering, 10 peering, 1 active+recovery_wait+degraded+remapped, 30 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 66 active+clean; 7.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 69 MiB/s wr, 1.10k op/s; 56499/347607 objects degraded (16.254%); 9364/347607 objects misplaced (2.694%); 3.1 MiB/s, 49 objects/s recovering 2024-06-14T23:11:25.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:25 smithi086 ceph-mon[102754]: osdmap e324: 8 total, 8 up, 6 in 2024-06-14T23:11:25.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:25.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 3 1 2024-06-14T23:11:25.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:25.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:25.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:25 smithi137 ceph-mon[90634]: pgmap v1200: 119 pgs: 2 remapped+peering, 10 peering, 1 active+recovery_wait+degraded+remapped, 30 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 66 active+clean; 7.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 69 MiB/s wr, 1.10k op/s; 56499/347607 objects degraded (16.254%); 9364/347607 objects misplaced (2.694%); 3.1 MiB/s, 49 objects/s recovering 2024-06-14T23:11:25.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:25 smithi137 ceph-mon[90634]: osdmap e324: 8 total, 8 up, 6 in 2024-06-14T23:11:25.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:25.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:25.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:25.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:25.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:26.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:26.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:26.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:26.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:26.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:26.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:26.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:26.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:26.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:26.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:26.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:26.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:26.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:26.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:26.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:26.762 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:11:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:11:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:26.782 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:11:26.783 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:11:26.792 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:26.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:26.810 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:11:26.810 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-06-14T23:11:26.835 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:11:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:11:26.856 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:11:26.856 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:11:26.883 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:11:26.883 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:11:26.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:26.902 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:11:26.902 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:11:26.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:26.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:26.926 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:11:26.927 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:11:27.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:27.055 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:27.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:27.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:27.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:27.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:27.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:27.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:27.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:27.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:27.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:27.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:27.551 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:27.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:27.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:27.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:27.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:27.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:27.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:27 smithi137 ceph-mon[90634]: pgmap v1202: 119 pgs: 2 remapped+peering, 10 peering, 1 active+recovery_wait+degraded+remapped, 27 active+undersized, 6 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 69 active+clean; 7.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 32 MiB/s wr, 537 op/s; 40178/348414 objects degraded (11.532%); 6551/348414 objects misplaced (1.880%); 3.2 MiB/s, 51 objects/s recovering 2024-06-14T23:11:27.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3715771147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:27.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:27.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:27 smithi086 ceph-mon[102754]: pgmap v1202: 119 pgs: 2 remapped+peering, 10 peering, 1 active+recovery_wait+degraded+remapped, 27 active+undersized, 6 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 69 active+clean; 7.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 32 MiB/s wr, 537 op/s; 40178/348414 objects degraded (11.532%); 6551/348414 objects misplaced (1.880%); 3.2 MiB/s, 51 objects/s recovering 2024-06-14T23:11:27.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3715771147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:27.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:27 smithi086 ceph-mon[97913]: pgmap v1202: 119 pgs: 2 remapped+peering, 10 peering, 1 active+recovery_wait+degraded+remapped, 27 active+undersized, 6 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 69 active+clean; 7.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 32 MiB/s wr, 537 op/s; 40178/348414 objects degraded (11.532%); 6551/348414 objects misplaced (1.880%); 3.2 MiB/s, 51 objects/s recovering 2024-06-14T23:11:27.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3715771147' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:27.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:27.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:27.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:27.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:27.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:28.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:28.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:28.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:28.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:11:28.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:28.262 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:28.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:28.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:28.407 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:28.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:28.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:28.552 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:28.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:28.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:28.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:28.738 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:28.738 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:28.738 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 40178/348414 objects degraded (11.532%), 11 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:28.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:28.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:28.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 40178/348414 objects degraded (11.532%), 11 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:28.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:28.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:28.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 40178/348414 objects degraded (11.532%), 11 pgs degraded (PG_DEGRADED) 2024-06-14T23:11:28.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:28.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:28.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:28.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:28.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:28.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:29.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:29.075 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:29.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:29.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:29.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:29.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:29.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:29.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:29.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:29.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:29.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:29.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:29.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:29.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:29.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:29.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:29.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:29.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:29.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:29.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:29 smithi137 ceph-mon[90634]: pgmap v1203: 119 pgs: 2 remapped+peering, 10 peering, 10 active+undersized, 8 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 86 active+clean; 7.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 2.0 KiB/s rd, 24 MiB/s wr, 466 op/s; 37094/348264 objects degraded (10.651%); 4.2 MiB/s, 66 objects/s recovering 2024-06-14T23:11:29.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:29 smithi086 ceph-mon[97913]: pgmap v1203: 119 pgs: 2 remapped+peering, 10 peering, 10 active+undersized, 8 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 86 active+clean; 7.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 2.0 KiB/s rd, 24 MiB/s wr, 466 op/s; 37094/348264 objects degraded (10.651%); 4.2 MiB/s, 66 objects/s recovering 2024-06-14T23:11:29.804 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:29 smithi086 ceph-mon[102754]: pgmap v1203: 119 pgs: 2 remapped+peering, 10 peering, 10 active+undersized, 8 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 86 active+clean; 7.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 2.0 KiB/s rd, 24 MiB/s wr, 466 op/s; 37094/348264 objects degraded (10.651%); 4.2 MiB/s, 66 objects/s recovering 2024-06-14T23:11:29.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:29.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:29.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:29.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:30.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:30.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:30.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:30.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:30.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:30.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:30.270 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:30.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:30.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:30.399 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:30.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:30.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:30.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:30.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:30.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:30.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:30.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:30.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:30.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:30.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:30.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:30.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:30.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:30.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:31.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:31.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:31.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:31.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:31.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:31.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:31.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:31.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:31.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:31.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:31.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:31.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:31.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:31.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:31.594 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:31.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:31.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:31.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:31.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:31.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:31 smithi137 ceph-mon[90634]: pgmap v1204: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 7.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.9 KiB/s rd, 19 MiB/s wr, 501 op/s; 41129/345411 objects degraded (11.907%); 6503/345411 objects misplaced (1.883%); 8.2 MiB/s, 131 objects/s recovering 2024-06-14T23:11:31.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:31 smithi086 ceph-mon[97913]: pgmap v1204: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 7.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.9 KiB/s rd, 19 MiB/s wr, 501 op/s; 41129/345411 objects degraded (11.907%); 6503/345411 objects misplaced (1.883%); 8.2 MiB/s, 131 objects/s recovering 2024-06-14T23:11:31.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:31 smithi086 ceph-mon[102754]: pgmap v1204: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 7.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.9 KiB/s rd, 19 MiB/s wr, 501 op/s; 41129/345411 objects degraded (11.907%); 6503/345411 objects misplaced (1.883%); 8.2 MiB/s, 131 objects/s recovering 2024-06-14T23:11:31.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:31.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:31.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:31.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:32.008 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:32.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:32.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:32.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:32.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:32.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:32.261 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:32.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:32.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:32.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:32.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:32.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:32.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:32.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:32.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:32.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:32.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:32.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:32.807 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:32.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:32.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:32.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:32.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:33.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:33.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:33.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:33.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:33.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:33.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:33.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:33.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:33.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:33.473 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:33.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:33.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:33.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:33.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:33.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:33.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:33.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:33.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:33.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:33 smithi137 ceph-mon[90634]: pgmap v1205: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.5 KiB/s rd, 9.5 MiB/s wr, 494 op/s; 40945/340524 objects degraded (12.024%); 6503/340524 objects misplaced (1.910%); 7.8 MiB/s, 124 objects/s recovering 2024-06-14T23:11:33.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 40945/340524 objects degraded (12.024%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:33.763 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:33 smithi086 ceph-mon[102754]: pgmap v1205: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.5 KiB/s rd, 9.5 MiB/s wr, 494 op/s; 40945/340524 objects degraded (12.024%); 6503/340524 objects misplaced (1.910%); 7.8 MiB/s, 124 objects/s recovering 2024-06-14T23:11:33.763 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 40945/340524 objects degraded (12.024%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:33.763 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:33 smithi086 ceph-mon[97913]: pgmap v1205: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.5 KiB/s rd, 9.5 MiB/s wr, 494 op/s; 40945/340524 objects degraded (12.024%); 6503/340524 objects misplaced (1.910%); 7.8 MiB/s, 124 objects/s recovering 2024-06-14T23:11:33.763 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 40945/340524 objects degraded (12.024%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:33.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:33.865 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:33.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:33.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:33.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:33.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:34.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:34.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:34.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:34.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:34.243 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:11:34.243 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:11:34.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:34.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:34.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:34.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:34.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:34.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:34.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:11:34.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:34.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:34.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:34.673 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:34.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:34.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:34.809 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:34.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:34.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:34.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:34.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:35.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:35.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:35.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:35.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:11:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:11:34] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.43.0" 2024-06-14T23:11:35.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:35.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:35.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:35.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:35.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:35.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:35.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:35.473 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:35.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:35.575 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:35.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:35.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:35.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:35.706 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:35 smithi086 ceph-mon[97913]: pgmap v1206: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.3 KiB/s rd, 4.1 MiB/s wr, 494 op/s; 40342/336759 objects degraded (11.979%); 6333/336759 objects misplaced (1.881%); 12 MiB/s, 187 objects/s recovering 2024-06-14T23:11:35.706 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1527575474' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:11:35.706 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:35 smithi086 ceph-mon[102754]: pgmap v1206: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.3 KiB/s rd, 4.1 MiB/s wr, 494 op/s; 40342/336759 objects degraded (11.979%); 6333/336759 objects misplaced (1.881%); 12 MiB/s, 187 objects/s recovering 2024-06-14T23:11:35.706 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1527575474' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:11:35.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:35.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:35 smithi137 ceph-mon[90634]: pgmap v1206: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.3 KiB/s rd, 4.1 MiB/s wr, 494 op/s; 40342/336759 objects degraded (11.979%); 6333/336759 objects misplaced (1.881%); 12 MiB/s, 187 objects/s recovering 2024-06-14T23:11:35.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1527575474' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:11:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:35.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:35.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-14T23:11:35.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:35.863 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.3 to 1 (10000) 2024-06-14T23:11:35.878 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:35.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:35.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:36.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:36.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:36.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:36.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:36.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:36.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:36.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:36.276 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:36.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:36.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:36.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:36.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:36.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:36.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:36.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:36.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 4 1 2024-06-14T23:11:36.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:36.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:36.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:36.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1527575474' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:11:36.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:36 smithi137 ceph-mon[90634]: osdmap e325: 8 total, 8 up, 6 in 2024-06-14T23:11:36.752 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1527575474' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:11:36.752 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:36 smithi086 ceph-mon[97913]: osdmap e325: 8 total, 8 up, 6 in 2024-06-14T23:11:36.753 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1527575474' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:11:36.753 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:36 smithi086 ceph-mon[102754]: osdmap e325: 8 total, 8 up, 6 in 2024-06-14T23:11:36.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:36.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:36.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:36.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:36.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:36.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:37.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:37.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:37.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:37.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:37.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:37.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:37.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:37.330 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:37.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:37.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:37.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:37.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:37.464 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:37.466 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:37.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:37.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:37.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:37.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:37.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:37.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:37.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:37 smithi137 ceph-mon[90634]: pgmap v1208: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.3 KiB/s rd, 1.8 MiB/s wr, 521 op/s; 40342/334569 objects degraded (12.058%); 6333/334569 objects misplaced (1.893%); 10 MiB/s, 166 objects/s recovering 2024-06-14T23:11:37.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:37.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:37 smithi086 ceph-mon[97913]: pgmap v1208: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.3 KiB/s rd, 1.8 MiB/s wr, 521 op/s; 40342/334569 objects degraded (12.058%); 6333/334569 objects misplaced (1.893%); 10 MiB/s, 166 objects/s recovering 2024-06-14T23:11:37.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:37 smithi086 ceph-mon[102754]: pgmap v1208: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.3 KiB/s rd, 1.8 MiB/s wr, 521 op/s; 40342/334569 objects degraded (12.058%); 6333/334569 objects misplaced (1.893%); 10 MiB/s, 166 objects/s recovering 2024-06-14T23:11:37.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:37.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:37.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:37.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:37.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:38.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:38.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:38.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:38.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:38.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:38.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:38.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:38.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:38.470 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:38.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:38.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:38.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:38.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:38.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:38.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:38.722 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 40342/334569 objects degraded (12.058%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:38.723 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 40342/334569 objects degraded (12.058%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:38.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 40342/334569 objects degraded (12.058%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:38.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:38.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:38.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:38.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:38.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:38.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:39.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:39.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:39.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:39.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:39.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:39.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:39.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:39.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:39.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:39.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:39.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:39.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:39.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:39.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:39.569 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:39.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:39.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:39.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:39.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:39.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:39.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:39 smithi137 ceph-mon[90634]: pgmap v1209: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 102 B/s wr, 549 op/s; 40153/331770 objects degraded (12.103%); 6333/331770 objects misplaced (1.909%); 10 MiB/s, 166 objects/s recovering 2024-06-14T23:11:39.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:39 smithi086 ceph-mon[97913]: pgmap v1209: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 102 B/s wr, 549 op/s; 40153/331770 objects degraded (12.103%); 6333/331770 objects misplaced (1.909%); 10 MiB/s, 166 objects/s recovering 2024-06-14T23:11:39.749 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:39 smithi086 ceph-mon[102754]: pgmap v1209: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 102 B/s rd, 102 B/s wr, 549 op/s; 40153/331770 objects degraded (12.103%); 6333/331770 objects misplaced (1.909%); 10 MiB/s, 166 objects/s recovering 2024-06-14T23:11:39.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:39.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:39.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:39.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:40.010 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:40.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:40.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:40.139 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:40.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:40.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:40.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:40.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:40.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:40.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:40.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:40.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:40.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:40.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:40.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:40.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:40.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:40.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:40.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:40.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:40.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:40.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:40.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:41.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:41.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:41.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:41.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:41.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:41.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:41.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:41.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:41.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:41.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:41.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:41.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:41.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:41.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:41.496 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:11:41.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:41.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:41.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:41.665 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:41 smithi086 ceph-mon[97913]: pgmap v1210: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 588 op/s; 39485/327744 objects degraded (12.048%); 6149/327744 objects misplaced (1.876%); 12 MiB/s, 199 objects/s recovering 2024-06-14T23:11:41.666 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1625619698' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:11:41.666 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:41 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:11:41.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:41 smithi086 ceph-mon[102754]: pgmap v1210: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 588 op/s; 39485/327744 objects degraded (12.048%); 6149/327744 objects misplaced (1.876%); 12 MiB/s, 199 objects/s recovering 2024-06-14T23:11:41.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1625619698' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:11:41.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:41 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:11:41.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:41.723 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:41.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:41.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:41 smithi137 ceph-mon[90634]: pgmap v1210: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 588 op/s; 39485/327744 objects degraded (12.048%); 6149/327744 objects misplaced (1.876%); 12 MiB/s, 199 objects/s recovering 2024-06-14T23:11:41.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1625619698' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:11:41.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:41 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:11:41.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:41.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:41.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:41.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:41.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:41.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:42.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:42.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:42.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:42.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:42.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:42.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:42.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:42.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:42.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:42.392 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:42.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:42.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:42.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:42.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:42.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:42.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:42.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:42.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:42 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:11:42.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:42 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:11:42.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:42 smithi137 ceph-mon[90634]: osdmap e326: 8 total, 8 up, 6 in 2024-06-14T23:11:42.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:42.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:42.822 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:42.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:42.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:42 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:11:42.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:42 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:11:42.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:42 smithi086 ceph-mon[97913]: osdmap e326: 8 total, 8 up, 6 in 2024-06-14T23:11:42.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:42.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:42 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:11:42.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:42 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:11:42.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:42 smithi086 ceph-mon[102754]: osdmap e326: 8 total, 8 up, 6 in 2024-06-14T23:11:42.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:42.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:42.953 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:42.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:42.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:43.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:43.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:43.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:43.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:43.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:43.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:43.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:43.342 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:43.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:43.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:43.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:43.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:43.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:43.608 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:43.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:43.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:43.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:43.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:43.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:43 smithi086 ceph-mon[97913]: pgmap v1212: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 590 op/s; 39305/322593 objects degraded (12.184%); 6149/322593 objects misplaced (1.906%); 9.5 MiB/s, 152 objects/s recovering 2024-06-14T23:11:43.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 39305/322593 objects degraded (12.184%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:43.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:43 smithi086 ceph-mon[102754]: pgmap v1212: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 590 op/s; 39305/322593 objects degraded (12.184%); 6149/322593 objects misplaced (1.906%); 9.5 MiB/s, 152 objects/s recovering 2024-06-14T23:11:43.796 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 39305/322593 objects degraded (12.184%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:43.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:43.873 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:43.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:43.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:43.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:43 smithi137 ceph-mon[90634]: pgmap v1212: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 590 op/s; 39305/322593 objects degraded (12.184%); 6149/322593 objects misplaced (1.906%); 9.5 MiB/s, 152 objects/s recovering 2024-06-14T23:11:43.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 39305/322593 objects degraded (12.184%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:44.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:44.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:44.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:44.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:44.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:44.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:44.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:44.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:44.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:44.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:11:44.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:44.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:44.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:44.548 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:44.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:44.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:44.679 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:44.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:44.696 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2516596884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:44.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2516596884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:44.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:44.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:44.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:44.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:44.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:44.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:44.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:44.953 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:11:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:11:44] "GET /metrics HTTP/1.1" 200 36720 "" "Prometheus/2.43.0" 2024-06-14T23:11:44.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2516596884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:45.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:45.065 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:45.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:45.146 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:11:45.146 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:11:45.146 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:11:45.146 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:11:45.146 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:11:45.147 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:11:45.147 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:11:45.147 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:11:45.147 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:11:45.147 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:11:45.147 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:11:45.147 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:11:45.147 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:11:45.148 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:11:45.148 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:11:45.148 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:11:45.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:45.195 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:45.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:45.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:45.330 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:45.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:45.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:45.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:45.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:45.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:45.591 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:45.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:45.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:45.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:45.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:45.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:45.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:45 smithi086 ceph-mon[97913]: pgmap v1213: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 597 op/s; 38558/318795 objects degraded (12.095%); 5970/318795 objects misplaced (1.873%); 15 MiB/s, 244 objects/s recovering 2024-06-14T23:11:45.801 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:45 smithi086 ceph-mon[102754]: pgmap v1213: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 597 op/s; 38558/318795 objects degraded (12.095%); 5970/318795 objects misplaced (1.873%); 15 MiB/s, 244 objects/s recovering 2024-06-14T23:11:45.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:45.865 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:45.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:45.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:45.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:45.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:45 smithi137 ceph-mon[90634]: pgmap v1213: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 597 op/s; 38558/318795 objects degraded (12.095%); 5970/318795 objects misplaced (1.873%); 15 MiB/s, 244 objects/s recovering 2024-06-14T23:11:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:45.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:45.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:46.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:46.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:46.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:46.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:46.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:46.258 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:46.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:46.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:46.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:46.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:46.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:46.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:46.512 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.4 to 1 (10000) 2024-06-14T23:11:46.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:46.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:46.599 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:11:46.599 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:11:46.599 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:11:46.599 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:11:46.599 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:11:46.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:46.673 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:46.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:46.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:46.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:46.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:46.836 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3570417944' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:11:46.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3570417944' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:11:46.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:46.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:46.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:46.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3570417944' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:11:47.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:47.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:47.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:47.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:47.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:47.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:47.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:47.334 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:47.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:47.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:47.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:47.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:47.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:47.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:47.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 5 1 2024-06-14T23:11:47.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:47.742 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:47.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:47.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:47 smithi086 ceph-mon[97913]: pgmap v1214: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 38558/316380 objects degraded (12.187%); 5970/316380 objects misplaced (1.887%); 13 MiB/s, 214 objects/s recovering 2024-06-14T23:11:47.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3570417944' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:11:47.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:47 smithi086 ceph-mon[97913]: osdmap e327: 8 total, 8 up, 6 in 2024-06-14T23:11:47.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:47 smithi086 ceph-mon[102754]: pgmap v1214: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 38558/316380 objects degraded (12.187%); 5970/316380 objects misplaced (1.887%); 13 MiB/s, 214 objects/s recovering 2024-06-14T23:11:47.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3570417944' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:11:47.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:47 smithi086 ceph-mon[102754]: osdmap e327: 8 total, 8 up, 6 in 2024-06-14T23:11:47.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:47.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:11:47.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:47.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:47.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:47.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:47 smithi137 ceph-mon[90634]: pgmap v1214: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 38558/316380 objects degraded (12.187%); 5970/316380 objects misplaced (1.887%); 13 MiB/s, 214 objects/s recovering 2024-06-14T23:11:47.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3570417944' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:11:47.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:47 smithi137 ceph-mon[90634]: osdmap e327: 8 total, 8 up, 6 in 2024-06-14T23:11:48.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:48.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:48.041 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:48.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-14T23:11:48.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:48.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:48.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:48.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:48.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:48.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:48.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:48.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:48.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:48.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:48.588 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:48.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:48.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:48.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:48.722 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:48.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:48.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:48.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 38558/316380 objects degraded (12.187%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:48.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 38558/316380 objects degraded (12.187%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:48.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:48.896 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:48.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:48.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:48.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:48.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 38558/316380 objects degraded (12.187%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:49.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:49.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:49.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:49.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:49.253 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:49.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:49.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:49.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:49.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:49.482 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:49.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:49.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:49.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:49.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:49.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:49.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:49 smithi086 ceph-mon[97913]: pgmap v1216: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 590 op/s; 38383/313575 objects degraded (12.240%); 5970/313575 objects misplaced (1.904%); 10 MiB/s, 160 objects/s recovering 2024-06-14T23:11:49.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:49 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:11:49.733 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:49 smithi086 ceph-mon[102754]: pgmap v1216: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 590 op/s; 38383/313575 objects degraded (12.240%); 5970/313575 objects misplaced (1.904%); 10 MiB/s, 160 objects/s recovering 2024-06-14T23:11:49.733 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:49 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:11:49.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:49.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:49.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:49.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:49.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:49.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:49.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:49 smithi137 ceph-mon[90634]: pgmap v1216: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 590 op/s; 38383/313575 objects degraded (12.240%); 5970/313575 objects misplaced (1.904%); 10 MiB/s, 160 objects/s recovering 2024-06-14T23:11:49.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:49 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:11:50.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:50.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:50.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:50.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:50.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:50.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:50.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:50.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:50.274 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:50.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:50.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:50.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:50.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:50.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:50.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:50.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:50.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:50.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:50.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:50.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:50.700 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:50.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:50.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:50.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:50.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:50 smithi086 ceph-mon[97913]: osdmap e328: 8 total, 8 up, 6 in 2024-06-14T23:11:50.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:50 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:11:50.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:50 smithi086 ceph-mon[102754]: osdmap e328: 8 total, 8 up, 6 in 2024-06-14T23:11:50.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:50 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:11:50.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:50.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:50.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:50.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:50 smithi137 ceph-mon[90634]: osdmap e328: 8 total, 8 up, 6 in 2024-06-14T23:11:50.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:50 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:11:51.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:51.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:51.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:51.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:51.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:51.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:51.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:51.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:51.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:51.354 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:51.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:51.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:51.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:51.556 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:11:51.556 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:51 smithi086 ceph-mon[97913]: pgmap v1218: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 545 op/s; 37382/309498 objects degraded (12.078%); 5763/309498 objects misplaced (1.862%); 18 MiB/s, 288 objects/s recovering 2024-06-14T23:11:51.556 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:51 smithi086 ceph-mon[97913]: osdmap e329: 8 total, 8 up, 6 in 2024-06-14T23:11:51.556 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1745159050' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:51.557 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:51.557 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:51 smithi086 ceph-mon[102754]: pgmap v1218: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 545 op/s; 37382/309498 objects degraded (12.078%); 5763/309498 objects misplaced (1.862%); 18 MiB/s, 288 objects/s recovering 2024-06-14T23:11:51.557 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:51 smithi086 ceph-mon[102754]: osdmap e329: 8 total, 8 up, 6 in 2024-06-14T23:11:51.557 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1745159050' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:51.557 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:51.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:51.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:51.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:51.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:51.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:51.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:51.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:51.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:51.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:51.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:51.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:51.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:51.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:51 smithi137 ceph-mon[90634]: pgmap v1218: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 545 op/s; 37382/309498 objects degraded (12.078%); 5763/309498 objects misplaced (1.862%); 18 MiB/s, 288 objects/s recovering 2024-06-14T23:11:51.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:51 smithi137 ceph-mon[90634]: osdmap e329: 8 total, 8 up, 6 in 2024-06-14T23:11:51.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1745159050' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:51.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:11:52.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:52.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:52.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:52.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:52.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:52.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:52.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:52.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:52.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:52.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:52.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:52.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:52.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:52.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:52.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:52.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:52.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:52.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:52 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:11:52.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:52 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:11:52.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:52 smithi086 ceph-mon[102754]: osdmap e330: 8 total, 8 up, 6 in 2024-06-14T23:11:52.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:52 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:11:52.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:52 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:11:52.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:52 smithi086 ceph-mon[97913]: osdmap e330: 8 total, 8 up, 6 in 2024-06-14T23:11:52.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:52.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:52.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:52.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:52 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:11:52.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:52 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:11:52.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:52 smithi137 ceph-mon[90634]: osdmap e330: 8 total, 8 up, 6 in 2024-06-14T23:11:53.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:53.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:53.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:53.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:53.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:53.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:53.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:53.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:53.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:53.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:53.451 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:53.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:53.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:53.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:53.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:53.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:53.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:53.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:53.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:53.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:53 smithi086 ceph-mon[97913]: pgmap v1221: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 684 op/s; 37208/304596 objects degraded (12.216%); 5763/304596 objects misplaced (1.892%); 17 MiB/s, 271 objects/s recovering 2024-06-14T23:11:53.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 37208/304596 objects degraded (12.216%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:53.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:53 smithi086 ceph-mon[102754]: pgmap v1221: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 684 op/s; 37208/304596 objects degraded (12.216%); 5763/304596 objects misplaced (1.892%); 17 MiB/s, 271 objects/s recovering 2024-06-14T23:11:53.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 37208/304596 objects degraded (12.216%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:53.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:53.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:53.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:53.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:53.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:53.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:53 smithi137 ceph-mon[90634]: pgmap v1221: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 684 op/s; 37208/304596 objects degraded (12.216%); 5763/304596 objects misplaced (1.892%); 17 MiB/s, 271 objects/s recovering 2024-06-14T23:11:53.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 37208/304596 objects degraded (12.216%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:54.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:54.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:54.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:54.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:54.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:54.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:54.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:54.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:54.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:54.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:54.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:54.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:54.566 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.5 to 1 (10000) 2024-06-14T23:11:54.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:54.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:54.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:54.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:54.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:54.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:54.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:54.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:54 smithi086 ceph-mon[97913]: 1.1c deep-scrub starts 2024-06-14T23:11:54.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1690439867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:11:54.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:54 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:11:54.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:54 smithi086 ceph-mon[102754]: 1.1c deep-scrub starts 2024-06-14T23:11:54.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1690439867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:11:54.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:54 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:11:54.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:54.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:54.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:54.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:54.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:11:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:54 smithi137 ceph-mon[90634]: 1.1c deep-scrub starts 2024-06-14T23:11:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:54 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1690439867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:11:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:54 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:11:55.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:55.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:55.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:55.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:55.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:55.166 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:55.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:55.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:55.300 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:55.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:55.336 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:11:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:11:54] "GET /metrics HTTP/1.1" 200 36720 "" "Prometheus/2.43.0" 2024-06-14T23:11:55.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:55.433 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:55.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:55.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:55.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:55.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:55.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:55.712 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:55 smithi086 ceph-mon[97913]: pgmap v1222: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 713 op/s; 36955/300729 objects degraded (12.288%); 5556/300729 objects misplaced (1.848%); 19 MiB/s, 306 objects/s recovering 2024-06-14T23:11:55.712 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:55 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:11:55.712 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:55 smithi086 ceph-mon[97913]: osdmap e331: 8 total, 8 up, 6 in 2024-06-14T23:11:55.713 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:55 smithi086 ceph-mon[102754]: pgmap v1222: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 713 op/s; 36955/300729 objects degraded (12.288%); 5556/300729 objects misplaced (1.848%); 19 MiB/s, 306 objects/s recovering 2024-06-14T23:11:55.713 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:55 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:11:55.713 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:55 smithi086 ceph-mon[102754]: osdmap e331: 8 total, 8 up, 6 in 2024-06-14T23:11:55.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:55.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:55.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:55.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:55.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:55.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:55.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:55 smithi137 ceph-mon[90634]: pgmap v1222: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 713 op/s; 36955/300729 objects degraded (12.288%); 5556/300729 objects misplaced (1.848%); 19 MiB/s, 306 objects/s recovering 2024-06-14T23:11:55.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:55 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:11:55.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:55 smithi137 ceph-mon[90634]: osdmap e331: 8 total, 8 up, 6 in 2024-06-14T23:11:56.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:56.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:56.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:56.092 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:11:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:11:56.117 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:11:56.117 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:11:56.140 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:11:56.140 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:11:56.165 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:11:56.165 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:11:56.189 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:11:56.189 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:11:56.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:56.211 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:11:56.211 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:11:56.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:56.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:56.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:11:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:11:56.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:11:56.256 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:11:56.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:56.365 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:56.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:56.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 6 1 2024-06-14T23:11:56.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:56.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:56.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:56.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:56.632 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:56.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:56.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:56.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:56.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:56.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1969631460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:56.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1969631460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:56.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:56.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:56.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:56.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:56.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1969631460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:11:56.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:11:57.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:57.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:57.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:11:57.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:57.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:57.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:57.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:57.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:57.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:57.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:57.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:57.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:57.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:57.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:57.614 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:57 smithi086 ceph-mon[102754]: pgmap v1224: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 620 op/s; 36955/298329 objects degraded (12.387%); 5556/298329 objects misplaced (1.862%); 6.6 MiB/s, 105 objects/s recovering 2024-06-14T23:11:57.614 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:57.614 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:57.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:57 smithi086 ceph-mon[97913]: pgmap v1224: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 620 op/s; 36955/298329 objects degraded (12.387%); 5556/298329 objects misplaced (1.862%); 6.6 MiB/s, 105 objects/s recovering 2024-06-14T23:11:57.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:57.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:57.653 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:57.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:57.677 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:11:57.678 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:11:57.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:57.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:57.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:57.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:57.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:57.934 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:11:57.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:57.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:57.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:57.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:57.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:57 smithi137 ceph-mon[90634]: pgmap v1224: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 620 op/s; 36955/298329 objects degraded (12.387%); 5556/298329 objects misplaced (1.862%); 6.6 MiB/s, 105 objects/s recovering 2024-06-14T23:11:57.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:11:57.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:11:58.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:58.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:58.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:58.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:11:58.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:58.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:11:58.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:58.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:58.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:58.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:58.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:11:58.507 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:58.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:11:58.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:58.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:58.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:58.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:11:58.774 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:58.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:58.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:11:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:58 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:11:58.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:58 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 36955/298329 objects degraded (12.387%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:58.791 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:58 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:11:58.791 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:58 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 36955/298329 objects degraded (12.387%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:58.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:58.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:58.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:58.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:58 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:11:58.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:58 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 36955/298329 objects degraded (12.387%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:11:59.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:59.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:59.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:59.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:59.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:59.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:59.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:59.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:59.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:59.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:59.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:11:59.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:11:59.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:59.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:11:59.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:11:59.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:11:59 smithi086 ceph-mon[97913]: pgmap v1225: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 525 op/s; 36739/295176 objects degraded (12.446%); 5556/295176 objects misplaced (1.882%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:11:59.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:11:59 smithi086 ceph-mon[102754]: pgmap v1225: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 525 op/s; 36739/295176 objects degraded (12.446%); 5556/295176 objects misplaced (1.882%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:11:59.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:59.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:59.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:11:59.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:11:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:11:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:59.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:11:59.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-14T23:11:59.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:59.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:59.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:11:59.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:11:59.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:11:59.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:11:59 smithi137 ceph-mon[90634]: pgmap v1225: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 6.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 525 op/s; 36739/295176 objects degraded (12.446%); 5556/295176 objects misplaced (1.882%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:11:59.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:11:59.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:00.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:00.137 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:00.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:00.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:00.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:00.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:00.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:00.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:00.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:00.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:00.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:00.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:00.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:00.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:00.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:00.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:00.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:00.726 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:00.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:00.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:00.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:00.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:00 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:00.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:00 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:00.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:00.925 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:00.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:00.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:00 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:01.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:01.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:01.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:01.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:01.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:01.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:01.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:01.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:01.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:01.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:01.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:01.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:01.582 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:01.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:01.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:01.706 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:01 smithi086 ceph-mon[97913]: pgmap v1226: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 628 op/s; 36549/290910 objects degraded (12.564%); 5322/290910 objects misplaced (1.829%); 9.2 MiB/s, 146 objects/s recovering 2024-06-14T23:12:01.706 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:01 smithi086 ceph-mon[102754]: pgmap v1226: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 628 op/s; 36549/290910 objects degraded (12.564%); 5322/290910 objects misplaced (1.829%); 9.2 MiB/s, 146 objects/s recovering 2024-06-14T23:12:01.712 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:01.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:01.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:01.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:01.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:01.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:01.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:01.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:01.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:01 smithi137 ceph-mon[90634]: pgmap v1226: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 628 op/s; 36549/290910 objects degraded (12.564%); 5322/290910 objects misplaced (1.829%); 9.2 MiB/s, 146 objects/s recovering 2024-06-14T23:12:02.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:02.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:02.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:02.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:02.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:02.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:02.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:02.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:02.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:02.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:02.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:02.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:02.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:02.607 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:12:02.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:02.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:02.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:02.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:02.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:02.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:02.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1157507643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:12:02.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:02 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:12:02.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:02 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:02.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1157507643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:12:02.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:02 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:12:02.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:02 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:02.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:02.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:02.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:02.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1157507643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:12:02.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:02 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:12:02.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:02 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:03.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:03.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:03.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:03.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:03.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:03.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:03.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:03.308 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:03.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:03.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:03.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:03.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:03.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:03.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:03.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:03.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:03.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:03.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:03.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:03.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:03.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:03.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:03.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:03.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:03.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:03 smithi137 ceph-mon[90634]: pgmap v1227: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 646 op/s; 36328/285189 objects degraded (12.738%); 5322/285189 objects misplaced (1.866%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:12:03.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:03 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:03.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:03 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:12:03.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:03 smithi137 ceph-mon[90634]: osdmap e332: 8 total, 8 up, 6 in 2024-06-14T23:12:03.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 36328/285189 objects degraded (12.738%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:04.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[97913]: pgmap v1227: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 646 op/s; 36328/285189 objects degraded (12.738%); 5322/285189 objects misplaced (1.866%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:12:04.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:04.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:12:04.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[97913]: osdmap e332: 8 total, 8 up, 6 in 2024-06-14T23:12:04.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 36328/285189 objects degraded (12.738%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:04.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[102754]: pgmap v1227: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 646 op/s; 36328/285189 objects degraded (12.738%); 5322/285189 objects misplaced (1.866%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:12:04.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:04.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:12:04.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[102754]: osdmap e332: 8 total, 8 up, 6 in 2024-06-14T23:12:04.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 36328/285189 objects degraded (12.738%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:04.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:04.114 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:04.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:04.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:04.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:04.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:04.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:04.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:04.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:04.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:04.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:04.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.6 to 1 (10000) 2024-06-14T23:12:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:04.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:04.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:04.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:04.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:04.872 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:04 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2345595566' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:12:04.872 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:04 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2345595566' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:12:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:04.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:04.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:04 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2345595566' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:12:04.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:05.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:05.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:05.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:05.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:05.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:05.172 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:12:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:12:04] "GET /metrics HTTP/1.1" 200 36718 "" "Prometheus/2.43.0" 2024-06-14T23:12:05.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:05.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:05.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:05.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:05.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:05.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:05.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:05.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 7 1 2024-06-14T23:12:05.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:05.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:05.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:05.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:05.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:05.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:05.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:05.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:05.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:05 smithi086 ceph-mon[97913]: pgmap v1229: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 685 op/s; 36116/280827 objects degraded (12.861%); 5108/280827 objects misplaced (1.819%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:12:05.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2345595566' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:12:05.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:05 smithi086 ceph-mon[97913]: osdmap e333: 8 total, 8 up, 6 in 2024-06-14T23:12:05.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:05 smithi086 ceph-mon[102754]: pgmap v1229: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 685 op/s; 36116/280827 objects degraded (12.861%); 5108/280827 objects misplaced (1.819%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:12:05.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2345595566' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:12:05.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:05 smithi086 ceph-mon[102754]: osdmap e333: 8 total, 8 up, 6 in 2024-06-14T23:12:05.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:05.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:05.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:05.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:05 smithi137 ceph-mon[90634]: pgmap v1229: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 685 op/s; 36116/280827 objects degraded (12.861%); 5108/280827 objects misplaced (1.819%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:12:05.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2345595566' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:12:05.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:05 smithi137 ceph-mon[90634]: osdmap e333: 8 total, 8 up, 6 in 2024-06-14T23:12:06.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:06.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:06.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:06.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:06.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:06.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:06.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:06.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:06.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:06.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:06.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:06.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:06.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:06.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:06.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:06.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:06.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:06.742 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:06.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:06.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:12:06.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:06.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:06.915 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:06.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:07.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:07.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:07.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:07.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:07.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:07.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:07.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:07.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:07.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:07.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:07.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:07.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:07.569 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:07.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:07.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:07.702 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:07.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:07.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:07.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:07.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:07.899 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:07 smithi086 ceph-mon[97913]: pgmap v1231: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 708 op/s; 36116/278172 objects degraded (12.983%); 5108/278172 objects misplaced (1.836%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:12:07.900 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:07 smithi086 ceph-mon[102754]: pgmap v1231: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 708 op/s; 36116/278172 objects degraded (12.983%); 5108/278172 objects misplaced (1.836%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:12:07.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:07.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:07.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:07 smithi137 ceph-mon[90634]: pgmap v1231: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 708 op/s; 36116/278172 objects degraded (12.983%); 5108/278172 objects misplaced (1.836%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:12:08.039 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:12:08.039 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:12:08.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:08.100 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:08.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:08.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:08.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:08.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:08.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:08.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:08.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:08.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:08.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:08.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:08.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:08.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:08.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:08.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:08.677 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:08.678 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:12:08.678 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:12:08.691 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:08.714 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:12:08.714 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:12:08.744 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:12:08.744 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:12:08.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:08.768 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:12:08.768 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:12:08.768 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 36116/278172 objects degraded (12.983%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:08.769 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 36116/278172 objects degraded (12.983%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:08.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:08.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:08.794 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:12:08.794 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:12:08.820 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:12:08.820 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:12:08.845 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:12:08.845 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:12:08.870 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:12:08.870 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:12:08.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:08.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:08.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:08.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:08.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 36116/278172 objects degraded (12.983%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:09.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:09.051 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:09.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:09.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:09.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:09.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:09.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:09.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:09.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:09.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-14T23:12:09.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:09.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:09.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:09.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:09.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:09.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:09.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:09.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:09.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:09.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:09.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:09.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:09.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:09.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:09.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:09.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:09.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:09.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:09 smithi137 ceph-mon[90634]: pgmap v1232: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 665 op/s; 35902/274926 objects degraded (13.059%); 5108/274926 objects misplaced (1.858%); 6.7 MiB/s, 107 objects/s recovering 2024-06-14T23:12:09.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/646818233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:10.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:09 smithi086 ceph-mon[97913]: pgmap v1232: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 665 op/s; 35902/274926 objects degraded (13.059%); 5108/274926 objects misplaced (1.858%); 6.7 MiB/s, 107 objects/s recovering 2024-06-14T23:12:10.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/646818233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:10.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:09 smithi086 ceph-mon[102754]: pgmap v1232: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 665 op/s; 35902/274926 objects degraded (13.059%); 5108/274926 objects misplaced (1.858%); 6.7 MiB/s, 107 objects/s recovering 2024-06-14T23:12:10.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/646818233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:10.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:10.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:10.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:10.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:10.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:10.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:10.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:10.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:10.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:10.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:10.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:10.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:10.646 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:10.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:10.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:10.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:10.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:10.911 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:10.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:11.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:11.050 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:11.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:11.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:11.180 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:11.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:11.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:11.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:11.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:11.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:11.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:11.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:11.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:11.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:11.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:11.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:11.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:11.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:11.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:11.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:11 smithi086 ceph-mon[97913]: pgmap v1233: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 619 op/s; 35674/270324 objects degraded (13.197%); 4900/270324 objects misplaced (1.813%); 8.4 MiB/s, 134 objects/s recovering 2024-06-14T23:12:11.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:11 smithi086 ceph-mon[102754]: pgmap v1233: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 619 op/s; 35674/270324 objects degraded (13.197%); 4900/270324 objects misplaced (1.813%); 8.4 MiB/s, 134 objects/s recovering 2024-06-14T23:12:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:11.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:11.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:11.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:11.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:11 smithi137 ceph-mon[90634]: pgmap v1233: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 619 op/s; 35674/270324 objects degraded (13.197%); 4900/270324 objects misplaced (1.813%); 8.4 MiB/s, 134 objects/s recovering 2024-06-14T23:12:12.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:12.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:12.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:12.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:12.236 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:12.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:12.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:12.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:12.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:12.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:12.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:12.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:12.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:12.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:12.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:12.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:12.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:12.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:12.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:12.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:12.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:12.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:13.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:13.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:13.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:13.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:13.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:13.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:13.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:13.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:13.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:13.452 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:13.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:13.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:13.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:13.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:13 smithi086 ceph-mon[102754]: pgmap v1234: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 710 op/s; 35472/264633 objects degraded (13.404%); 4900/264633 objects misplaced (1.852%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:12:13.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:13.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:13.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 35472/264633 objects degraded (13.404%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:13.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:13 smithi086 ceph-mon[97913]: pgmap v1234: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 710 op/s; 35472/264633 objects degraded (13.404%); 4900/264633 objects misplaced (1.852%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:12:13.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:13.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:13.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 35472/264633 objects degraded (13.404%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:13.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:13.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:13.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:13.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:13.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:13 smithi137 ceph-mon[90634]: pgmap v1234: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 710 op/s; 35472/264633 objects degraded (13.404%); 4900/264633 objects misplaced (1.852%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:12:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:13.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:13.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 35472/264633 objects degraded (13.404%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:13.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:13.861 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:13.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:13.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:13.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:13.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:13.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:14.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:14.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:14.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:14.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:14.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:14.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:14.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:14.395 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:14.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:14.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:14.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:14.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:14.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:14.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:14.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:14.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:14.785 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:14.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:14.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:14.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:14.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:15.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:12:15.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:15.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:15.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:15.124 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:12:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:12:14] "GET /metrics HTTP/1.1" 200 36717 "" "Prometheus/2.43.0" 2024-06-14T23:12:15.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:15.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:15.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:15.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:15.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:15.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:15.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:15.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:15.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:15.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:15.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:15.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:15 smithi086 ceph-mon[97913]: pgmap v1235: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 689 op/s; 35269/260139 objects degraded (13.558%); 4674/260139 objects misplaced (1.797%); 8.0 MiB/s, 128 objects/s recovering 2024-06-14T23:12:15.700 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:15 smithi086 ceph-mon[102754]: pgmap v1235: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 689 op/s; 35269/260139 objects degraded (13.558%); 4674/260139 objects misplaced (1.797%); 8.0 MiB/s, 128 objects/s recovering 2024-06-14T23:12:15.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:15.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:15 smithi137 ceph-mon[90634]: pgmap v1235: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 689 op/s; 35269/260139 objects degraded (13.558%); 4674/260139 objects misplaced (1.797%); 8.0 MiB/s, 128 objects/s recovering 2024-06-14T23:12:15.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:15.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:15.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:15.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:15.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:15.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:16.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:16.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:16.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:16.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:16.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:16.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:16.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:16.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:16.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:16.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:16.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:16.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:16.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:16.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:16.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:16.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:16.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:16.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:16.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:16.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:16.965 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:16.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:17.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:17.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:17.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:17.239 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:17.240 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:17.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:17.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:17.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:17.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:17.416 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.7 to 1 (10000) 2024-06-14T23:12:17.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:17.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:17.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:17.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:17.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:17.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:17.705 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:12:17.705 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:12:17.705 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:12:17.706 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:12:17.706 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:12:17.706 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:17 smithi086 ceph-mon[97913]: pgmap v1236: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 671 op/s; 35269/257394 objects degraded (13.702%); 4674/257394 objects misplaced (1.816%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:12:17.706 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4181856537' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:12:17.706 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:17 smithi086 ceph-mon[102754]: pgmap v1236: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 671 op/s; 35269/257394 objects degraded (13.702%); 4674/257394 objects misplaced (1.816%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:12:17.706 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4181856537' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:12:17.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:17 smithi137 ceph-mon[90634]: pgmap v1236: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 671 op/s; 35269/257394 objects degraded (13.702%); 4674/257394 objects misplaced (1.816%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:12:17.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4181856537' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:12:17.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:17.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:17.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:17.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:17.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:18.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:18.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:18.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:18.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:18.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:18.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:18.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:18.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:18.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:18.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:18.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:18.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:18.422 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:12:18.447 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:18.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:18.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:18.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:18.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:18.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4181856537' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:12:18.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[97913]: osdmap e334: 8 total, 8 up, 6 in 2024-06-14T23:12:18.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/148911631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:12:18.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:12:18.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 35269/257394 objects degraded (13.702%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:18.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4181856537' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:12:18.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[102754]: osdmap e334: 8 total, 8 up, 6 in 2024-06-14T23:12:18.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/148911631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:12:18.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:12:18.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 35269/257394 objects degraded (13.702%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:18.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:18.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:18.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:18.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4181856537' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:12:18.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:18 smithi137 ceph-mon[90634]: osdmap e334: 8 total, 8 up, 6 in 2024-06-14T23:12:18.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/148911631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:12:18.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:18 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:12:18.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 35269/257394 objects degraded (13.702%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:18.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:18.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:18.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:18.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:18.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:18.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:19.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:19.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:19.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:19.237 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:19.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:19.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:19.377 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:19.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:19.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:19.505 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:19.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:19.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:19 smithi086 ceph-mon[97913]: pgmap v1238: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 694 op/s; 35051/254091 objects degraded (13.795%); 4674/254091 objects misplaced (1.839%); 8.0 MiB/s, 128 objects/s recovering 2024-06-14T23:12:19.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:19 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:19.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:19 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:12:19.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:19 smithi086 ceph-mon[97913]: osdmap e335: 8 total, 8 up, 6 in 2024-06-14T23:12:19.563 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:19 smithi086 ceph-mon[102754]: pgmap v1238: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 694 op/s; 35051/254091 objects degraded (13.795%); 4674/254091 objects misplaced (1.839%); 8.0 MiB/s, 128 objects/s recovering 2024-06-14T23:12:19.563 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:19 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:19.563 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:19 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:12:19.563 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:19 smithi086 ceph-mon[102754]: osdmap e335: 8 total, 8 up, 6 in 2024-06-14T23:12:19.607 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:19.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:19.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:19.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:19.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:19 smithi137 ceph-mon[90634]: pgmap v1238: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 694 op/s; 35051/254091 objects degraded (13.795%); 4674/254091 objects misplaced (1.839%); 8.0 MiB/s, 128 objects/s recovering 2024-06-14T23:12:19.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:19 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:19.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:19 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:12:19.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:19 smithi137 ceph-mon[90634]: osdmap e335: 8 total, 8 up, 6 in 2024-06-14T23:12:19.772 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:19.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:19.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:19.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:19.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:19.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-14T23:12:20.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:20.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:20.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:20.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:20.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:20.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:20.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:20.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:20.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:20.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 2 1 2024-06-14T23:12:20.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:20.488 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:20.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:20.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:20.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:20.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:20.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:20.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:20.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:20.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:20.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:20.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:21.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:21.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:21.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:21.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:21.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:21.188 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:21.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:21.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:21.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:12:21.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:21.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:21.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:21.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:21.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:21.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:21.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:21.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:21.715 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:21 smithi137 ceph-mon[90634]: pgmap v1240: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 638 op/s; 34820/249294 objects degraded (13.967%); 4453/249294 objects misplaced (1.786%); 8.6 MiB/s, 137 objects/s recovering 2024-06-14T23:12:21.716 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:12:21.716 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.716 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.716 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.716 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:21 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:21.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:21.797 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[102754]: pgmap v1240: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 638 op/s; 34820/249294 objects degraded (13.967%); 4453/249294 objects misplaced (1.786%); 8.6 MiB/s, 137 objects/s recovering 2024-06-14T23:12:21.797 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:12:21.797 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.797 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.797 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.798 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[97913]: pgmap v1240: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 638 op/s; 34820/249294 objects degraded (13.967%); 4453/249294 objects misplaced (1.786%); 8.6 MiB/s, 137 objects/s recovering 2024-06-14T23:12:21.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:12:21.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:21 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:21.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:21.869 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:21.925 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:21.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:22.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:22.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:22.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:22.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:22.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:22.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:22.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:22.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:22.362 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:22.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:22.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:22.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:22.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:22.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:22.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:22.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:22.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:22.770 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:22.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:22.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:22.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:22.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:23.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:23.010 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:12:23.010 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:12:23.038 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:12:23.039 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:12:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:23.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:23.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:23.060 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:12:23.060 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:12:23.085 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:12:23.085 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:12:23.118 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:12:23.119 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:12:23.143 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:12:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:12:23.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:23.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:23.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:23.184 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:12:23.184 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:12:23.225 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:12:23.225 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:12:23.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:23.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:23.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:23.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:23.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:23.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:23.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:23.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:23.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:23.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:23.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:23 smithi086 ceph-mon[97913]: pgmap v1241: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 699 op/s; 34598/243354 objects degraded (14.217%); 4453/243354 objects misplaced (1.830%); 7.0 MiB/s, 111 objects/s recovering 2024-06-14T23:12:23.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/168412440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:23.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 34598/243354 objects degraded (14.217%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:23.696 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:23 smithi086 ceph-mon[102754]: pgmap v1241: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 699 op/s; 34598/243354 objects degraded (14.217%); 4453/243354 objects misplaced (1.830%); 7.0 MiB/s, 111 objects/s recovering 2024-06-14T23:12:23.696 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/168412440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:23.697 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 34598/243354 objects degraded (14.217%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:23.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:23.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:23 smithi137 ceph-mon[90634]: pgmap v1241: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 5.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 699 op/s; 34598/243354 objects degraded (14.217%); 4453/243354 objects misplaced (1.830%); 7.0 MiB/s, 111 objects/s recovering 2024-06-14T23:12:23.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/168412440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:23.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 34598/243354 objects degraded (14.217%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:23.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:23.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:23.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:23.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:23.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:23.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:24.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:24.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:24.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:24.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:24.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:24.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:24.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:24.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:24.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:24.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:24.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:24.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:24.584 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:24.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:24.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:24.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:24.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:24.807 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:24.808 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:24.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:24.864 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:24.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:24.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:25.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:25.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:25.086 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:12:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:12:24] "GET /metrics HTTP/1.1" 200 36717 "" "Prometheus/2.43.0" 2024-06-14T23:12:25.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:25.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:25.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:25.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:25.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:25.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:25.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:25.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:25.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:25.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:25 smithi137 ceph-mon[90634]: pgmap v1242: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 773 op/s; 34380/238824 objects degraded (14.396%); 4260/238824 objects misplaced (1.784%); 10 MiB/s, 162 objects/s recovering 2024-06-14T23:12:25.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:25.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:25.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:25.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:25 smithi086 ceph-mon[97913]: pgmap v1242: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 773 op/s; 34380/238824 objects degraded (14.396%); 4260/238824 objects misplaced (1.784%); 10 MiB/s, 162 objects/s recovering 2024-06-14T23:12:25.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:25 smithi086 ceph-mon[102754]: pgmap v1242: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 773 op/s; 34380/238824 objects degraded (14.396%); 4260/238824 objects misplaced (1.784%); 10 MiB/s, 162 objects/s recovering 2024-06-14T23:12:25.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:25.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:25.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:25.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:25.838 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:25.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:25.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:25.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:25.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:26.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:26.106 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:26.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:26.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:26.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:26.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:26.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:26.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:26.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:26.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:26.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:26.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:26.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:26.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:26.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:26.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:26.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:26.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:26.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:26.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:26.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:27.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:27.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:27.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:27.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:27.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:27.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:27.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:27.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:12:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:27.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:27.434 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:27.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:27.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:27.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:27.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:27.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:27.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:27.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:27 smithi086 ceph-mon[97913]: pgmap v1243: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 672 op/s; 34380/236250 objects degraded (14.552%); 4260/236250 objects misplaced (1.803%); 7.7 MiB/s, 122 objects/s recovering 2024-06-14T23:12:27.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:27.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:27.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:27.676 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:27 smithi086 ceph-mon[102754]: pgmap v1243: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 672 op/s; 34380/236250 objects degraded (14.552%); 4260/236250 objects misplaced (1.803%); 7.7 MiB/s, 122 objects/s recovering 2024-06-14T23:12:27.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:27.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:27.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:27.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:27.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:27.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:27.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:27.879 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:27.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:27.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:27.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:27 smithi137 ceph-mon[90634]: pgmap v1243: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 672 op/s; 34380/236250 objects degraded (14.552%); 4260/236250 objects misplaced (1.803%); 7.7 MiB/s, 122 objects/s recovering 2024-06-14T23:12:27.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:27.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:27.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:28.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:28.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:28.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:28.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:28.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:28.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:28.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:28.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:28.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:28.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:28.430 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:28.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:28.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:28.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:28.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:28.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:28.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:28.758 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:28.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:28.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 34380/236250 objects degraded (14.552%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:28.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 34380/236250 objects degraded (14.552%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:28.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:28.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:28.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:28.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:28.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 34380/236250 objects degraded (14.552%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:29.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:29.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:29.159 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:29.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:29.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:29.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:29.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:29.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:29.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:29.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:29.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:29.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:29.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:29.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:29.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:29.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:29.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:29.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:29.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:29 smithi086 ceph-mon[97913]: pgmap v1244: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 706 op/s; 34176/232875 objects degraded (14.676%); 4260/232875 objects misplaced (1.829%); 8.1 MiB/s, 128 objects/s recovering 2024-06-14T23:12:29.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:29 smithi086 ceph-mon[102754]: pgmap v1244: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 706 op/s; 34176/232875 objects degraded (14.676%); 4260/232875 objects misplaced (1.829%); 8.1 MiB/s, 128 objects/s recovering 2024-06-14T23:12:29.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:29.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:29.917 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:12:29.917 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:12:29.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:29.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:29 smithi137 ceph-mon[90634]: pgmap v1244: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.7 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 706 op/s; 34176/232875 objects degraded (14.676%); 4260/232875 objects misplaced (1.829%); 8.1 MiB/s, 128 objects/s recovering 2024-06-14T23:12:30.008 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:30.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:30.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:30.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:30.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:30.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:30.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:30.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:30.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:30.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:30.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:30.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:30.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:30.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:30.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:30.648 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:12:30.651 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.2 to 1 (10000) 2024-06-14T23:12:30.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:30.665 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4018585720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:12:30.666 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:30 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:12:30.666 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3395327333' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:12:30.666 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:30 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:12:30.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4018585720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:12:30.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:30 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:12:30.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3395327333' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:12:30.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:30 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:12:30.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:30.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:30.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:30.816 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:30.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:30.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:30.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:30.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:30.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:30.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:30.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4018585720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:12:30.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:30 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:12:30.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3395327333' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:12:30.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:30 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:12:31.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:31.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:31.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:31.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:31.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-14T23:12:31.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:31.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:31.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:31.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:31.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:31.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:31.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:31.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:31.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:31.628 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:31.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:31.638 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:12:31.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:31.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:31.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:31.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:31.803 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[102754]: pgmap v1245: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 649 op/s; 33943/228255 objects degraded (14.871%); 4039/228255 objects misplaced (1.770%); 9.2 MiB/s, 147 objects/s recovering 2024-06-14T23:12:31.803 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:31.803 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:12:31.803 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:12:31.804 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[102754]: osdmap e336: 8 total, 8 up, 6 in 2024-06-14T23:12:31.804 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:31.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[97913]: pgmap v1245: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 649 op/s; 33943/228255 objects degraded (14.871%); 4039/228255 objects misplaced (1.770%); 9.2 MiB/s, 147 objects/s recovering 2024-06-14T23:12:31.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:31.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:12:31.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:12:31.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[97913]: osdmap e336: 8 total, 8 up, 6 in 2024-06-14T23:12:31.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:31 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:31.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:31.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:31.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:31.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:31 smithi137 ceph-mon[90634]: pgmap v1245: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 649 op/s; 33943/228255 objects degraded (14.871%); 4039/228255 objects misplaced (1.770%); 9.2 MiB/s, 147 objects/s recovering 2024-06-14T23:12:31.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:31 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:31.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:31 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:12:31.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:31 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:12:31.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:31 smithi137 ceph-mon[90634]: osdmap e336: 8 total, 8 up, 6 in 2024-06-14T23:12:31.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:31 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:32.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:32.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:32.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:32.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:32.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:32.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:32.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:32.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:32.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:32.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:32.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:32.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:32.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:32.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:32.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:32.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:32.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:32.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:32.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:32.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:32.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:32 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:32.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:32 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:32.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:32.955 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:32 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:32.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:32.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:33.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:33.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:33.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:33.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:33.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:33.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:33.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:33.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:33.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:33.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:33.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:33.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:33.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:33.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:33.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:33.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:33.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:33.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:33 smithi137 ceph-mon[90634]: pgmap v1247: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 704 op/s; 33710/222213 objects degraded (15.170%); 4039/222213 objects misplaced (1.818%); 8.1 MiB/s, 130 objects/s recovering 2024-06-14T23:12:33.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 33710/222213 objects degraded (15.170%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:33.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:33.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:33.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:33.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:33.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:33.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:34.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:34.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:34.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:33 smithi086 ceph-mon[97913]: pgmap v1247: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 704 op/s; 33710/222213 objects degraded (15.170%); 4039/222213 objects misplaced (1.818%); 8.1 MiB/s, 130 objects/s recovering 2024-06-14T23:12:34.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 33710/222213 objects degraded (15.170%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:34.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:33 smithi086 ceph-mon[102754]: pgmap v1247: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 704 op/s; 33710/222213 objects degraded (15.170%); 4039/222213 objects misplaced (1.818%); 8.1 MiB/s, 130 objects/s recovering 2024-06-14T23:12:34.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 33710/222213 objects degraded (15.170%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:34.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:34.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:34.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:34.203 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-14T23:12:34.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:12:34.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:34.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:12:34.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:34.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:34.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:34.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:34.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:34.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:34.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:34.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:34.636 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:12:34.637 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:12:34.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:34.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:34 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:34.756 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:34 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:34.756 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:34 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:34.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:34.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:34.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:34.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:34.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:34.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:35.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:35.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:35.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:12:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:12:34] "GET /metrics HTTP/1.1" 200 36719 "" "Prometheus/2.43.0" 2024-06-14T23:12:35.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:35.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:35.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:35.297 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:35.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:35.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:35.432 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:35.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:35.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:35.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:35.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:35.647 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:35.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:35.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:35.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:35.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:35 smithi086 ceph-mon[97913]: pgmap v1248: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 33466/217365 objects degraded (15.396%); 3809/217365 objects misplaced (1.752%); 8.5 MiB/s, 136 objects/s recovering 2024-06-14T23:12:35.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:35 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:35.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:35 smithi086 ceph-mon[102754]: pgmap v1248: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 33466/217365 objects degraded (15.396%); 3809/217365 objects misplaced (1.752%); 8.5 MiB/s, 136 objects/s recovering 2024-06-14T23:12:35.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:35 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:35.849 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:35.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:35.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:35.991 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:35.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:35.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:35 smithi137 ceph-mon[90634]: pgmap v1248: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 33466/217365 objects degraded (15.396%); 3809/217365 objects misplaced (1.752%); 8.5 MiB/s, 136 objects/s recovering 2024-06-14T23:12:35.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:35 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:36.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-14T23:12:36.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:36.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:36.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:36.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:36.297 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:12:36.297 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:12:36.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:36.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:36.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:36.330 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:12:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:12:36.355 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:12:36.356 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:12:36.382 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:12:36.382 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:12:36.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:36.412 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:12:36.412 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:12:36.437 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:12:36.437 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:12:36.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:36.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:36.467 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:12:36.467 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:12:36.494 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:12:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:12:36.518 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:12:36.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:36.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:36.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:36.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:36.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:36.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:36.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1711316059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:36.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1711316059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:36.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:36.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:36.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:36.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:36.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:36.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1711316059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:37.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:37.127 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:37.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:37.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:37.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:37.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:37.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:37.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:37.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:37.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:37.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:37.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:37.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:37.717 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:37.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:37.802 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:37.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:37.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:37.932 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:37.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:37.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:37.947 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:37 smithi137 ceph-mon[90634]: pgmap v1249: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 716 op/s; 33466/214761 objects degraded (15.583%); 3809/214761 objects misplaced (1.774%); 8.5 MiB/s, 136 objects/s recovering 2024-06-14T23:12:38.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:38.053 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:37 smithi086 ceph-mon[97913]: pgmap v1249: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 716 op/s; 33466/214761 objects degraded (15.583%); 3809/214761 objects misplaced (1.774%); 8.5 MiB/s, 136 objects/s recovering 2024-06-14T23:12:38.054 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:37 smithi086 ceph-mon[102754]: pgmap v1249: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.4 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 716 op/s; 33466/214761 objects degraded (15.583%); 3809/214761 objects misplaced (1.774%); 8.5 MiB/s, 136 objects/s recovering 2024-06-14T23:12:38.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:38.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:38.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:38.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:38.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:38.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:38.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:38.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:38.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:38.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:38.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:38.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:38.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:38.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:38.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:38.686 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:12:38.687 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:12:38.687 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:12:38.687 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:12:38.687 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:12:38.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:38.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:38.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:38.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:38.873 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:38.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:38.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:38.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 33466/214761 objects degraded (15.583%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:38.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:38 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:39.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:39.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 33466/214761 objects degraded (15.583%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:39.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:38 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:39.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 33466/214761 objects degraded (15.583%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:39.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:38 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:39.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:39.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:39.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:39.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:39.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:39.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:39.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:12:39.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:39.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:39.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:39.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-14T23:12:39.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:39.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:39.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:39.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:39.721 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:39.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:39.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:39.869 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:39.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:39.917 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:12:39.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:39.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:39 smithi137 ceph-mon[90634]: pgmap v1250: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 33236/211398 objects degraded (15.722%); 3809/211398 objects misplaced (1.802%); 8.7 MiB/s, 139 objects/s recovering 2024-06-14T23:12:39.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:40.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:39 smithi086 ceph-mon[97913]: pgmap v1250: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 33236/211398 objects degraded (15.722%); 3809/211398 objects misplaced (1.802%); 8.7 MiB/s, 139 objects/s recovering 2024-06-14T23:12:40.082 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:39 smithi086 ceph-mon[102754]: pgmap v1250: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 33236/211398 objects degraded (15.722%); 3809/211398 objects misplaced (1.802%); 8.7 MiB/s, 139 objects/s recovering 2024-06-14T23:12:40.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:40.131 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:40.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:40.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:40.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:40.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:40.395 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:40.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:40.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:40.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:40.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:40.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:40.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:40.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:40.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:40.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:40.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:40.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:40.944 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:40.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:41.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:41.080 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:41.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:41.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:41.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:41.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:41.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:41.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:41.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:41.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:41.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:41.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:41.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:41.608 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:41.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:41.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:41.747 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:41.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:41.819 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:12:41.820 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:12:41.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:41.870 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:41 smithi137 ceph-mon[90634]: pgmap v1251: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 32999/206790 objects degraded (15.958%); 3595/206790 objects misplaced (1.738%); 8.7 MiB/s, 138 objects/s recovering 2024-06-14T23:12:41.871 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:41 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:41.893 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:41.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:41.925 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:12:41.925 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:12:41.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:12:41.925 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:12:41.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.925 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252334 10.e 10:703c40e1:::benchmark_data_smithi086_189745_object121366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:12:41.925 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.132222+0000", 2024-06-14T23:12:41.925 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 500.78749480499999, 2024-06-14T23:12:41.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15311908299999999, 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252334 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132222+0000", 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132222+0000", 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132220+0000", 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132225+0000", 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5709999999999999e-06 2024-06-14T23:12:41.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132237+0000", 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2354000000000001e-05 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132243+0000", 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4859999999999997e-06 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132349+0000", 2024-06-14T23:12:41.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105629 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132454+0000", 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105302 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.132573+0000", 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119226 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.142887+0000", 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010313708 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.142945+0000", 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8133000000000001e-05 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.142955+0000", 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0077e-05 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285197+0000", 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.142242058 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285240+0000", 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3432e-05 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285341+0000", 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010046499999999999 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252339 10.e 10:70dcdd8f:::benchmark_data_smithi086_189745_object121371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.137784+0000", 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 500.781932649, 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14782557299999999, 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252339 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137784+0000", 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137784+0000", 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137782+0000", 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:12:41.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137789+0000", 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8140000000000003e-06 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137802+0000", 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2449999999999999e-05 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137807+0000", 2024-06-14T23:12:41.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6620000000000002e-06 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139832+0000", 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020249109999999999 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139964+0000", 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000132223 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:12:41.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.140134+0000", 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000169362 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143030+0000", 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028967730000000001 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143035+0000", 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5199999999999999e-06 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.148876+0000", 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005840914 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285457+0000", 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13658071899999999 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285498+0000", 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1788e-05 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285610+0000", 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011121299999999999 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252341 10.e 10:75c51104:::benchmark_data_smithi086_189745_object121373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.137972+0000", 2024-06-14T23:12:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 500.78174492199997, 2024-06-14T23:12:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14802895199999999, 2024-06-14T23:12:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252341 2024-06-14T23:12:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137972+0000", 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137972+0000", 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137971+0000", 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137975+0000", 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3629999999999998e-06 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137982+0000", 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7340000000000008e-06 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.137986+0000", 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1819999999999999e-06 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143052+0000", 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050652889999999997 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143170+0000", 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118755 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.143294+0000", 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012369700000000001 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.148940+0000", 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056457319999999997 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.148970+0000", 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0195000000000001e-05 2024-06-14T23:12:41.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149013+0000", 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3353999999999999e-05 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285649+0000", 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.136635389 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.285769+0000", 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120219 2024-06-14T23:12:41.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.286001+0000", 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000231642 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252342 10.e 10:7c0afdb9:::benchmark_data_smithi086_189745_object121374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.139687+0000", 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 500.78002919900001, 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14649146099999999, 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.945 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252342 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139687+0000", 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139687+0000", 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139686+0000", 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139692+0000", 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6049999999999997e-06 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139708+0000", 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6384000000000001e-05 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.139714+0000", 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6910000000000002e-06 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.146284+0000", 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065695550000000004 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.146430+0000", 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014615099999999999 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.146550+0000", 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120195 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149006+0000", 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002455975 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149044+0000", 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8155000000000003e-05 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.151425+0000", 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023813290000000002 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.286039+0000", 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.134613646 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.286081+0000", 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1538000000000001e-05 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.286179+0000", 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8363999999999995e-05 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252346 10.e 10:7351a8c3:::benchmark_data_smithi086_189745_object121378:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:12:41.951 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.145483+0000", 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 500.77423344699997, 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.142599638, 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252346 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145483+0000", 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145483+0000", 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145482+0000", 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145487+0000", 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6800000000000001e-06 2024-06-14T23:12:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145498+0000", 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1216000000000001e-05 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.145504+0000", 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5180000000000001e-06 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149077+0000", 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035729210000000002 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149176+0000", 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9083e-05 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5,6", 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.149278+0000", 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101383 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.151657+0000", 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023791870000000001 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.151664+0000", 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1459999999999999e-06 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.152401+0000", 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073650199999999999 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.287931+0000", 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13553063700000001 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.287982+0000", 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0846000000000001e-05 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.288083+0000", 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100682 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252602 10.e 10:73840a31:::benchmark_data_smithi086_189745_object121634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.127817+0000", 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 499.79189919599997, 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89931418799999996, 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252602 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127817+0000", 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127817+0000", 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127816+0000", 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127822+0000", 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9739999999999999e-06 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127834+0000", 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1814000000000001e-05 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127839+0000", 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4689999999999996e-06 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127961+0000", 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121651 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.127968+0000", 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6889999999999998e-06 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.992502+0000", 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86453456799999995 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.992591+0000", 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9153000000000005e-05 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.992668+0000", 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6419000000000001e-05 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024151+0000", 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031483186000000003 2024-06-14T23:12:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024178+0000", 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7098e-05 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027048+0000", 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028697240000000001 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027091+0000", 2024-06-14T23:12:41.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3090000000000002e-05 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027132+0000", 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0772999999999997e-05 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252608 10.e 10:774df1e1:::benchmark_data_smithi086_189745_object121640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:12:41.964 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.133959+0000", 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 499.78575747299999, 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89326967700000004, 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252608 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:41 smithi086 ceph-mon[97913]: pgmap v1251: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 32999/206790 objects degraded (15.958%); 3595/206790 objects misplaced (1.738%); 8.7 MiB/s, 138 objects/s recovering 2024-06-14T23:12:41.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:41 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:41.967 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:41 smithi086 ceph-mon[102754]: pgmap v1251: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 715 op/s; 32999/206790 objects degraded (15.958%); 3595/206790 objects misplaced (1.738%); 8.7 MiB/s, 138 objects/s recovering 2024-06-14T23:12:41.967 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:41 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:41.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133959+0000", 2024-06-14T23:12:41.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133959+0000", 2024-06-14T23:12:41.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133958+0000", 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133964+0000", 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5610000000000004e-06 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133988+0000", 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3663999999999999e-05 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.133995+0000", 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8249999999999999e-06 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.134043+0000", 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8720000000000001e-05 2024-06-14T23:12:41.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.134051+0000", 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8499999999999994e-06 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.995475+0000", 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86142369100000005 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.995601+0000", 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125754 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.995731+0000", 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000130681 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024203+0000", 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028471988 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024228+0000", 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.48e-05 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027140+0000", 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002911832 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027193+0000", 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3399000000000002e-05 2024-06-14T23:12:41.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027229+0000", 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5485999999999999e-05 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252623 10.e 10:716ab276:::benchmark_data_smithi086_189745_object121655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.148259+0000", 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 499.77145722500001, 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87907613699999998, 2024-06-14T23:12:41.973 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252623 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148259+0000", 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148259+0000", 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148258+0000", 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148263+0000", 2024-06-14T23:12:41.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6250000000000004e-06 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148273+0000", 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5950000000000005e-06 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148278+0000", 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1440000000000002e-06 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148323+0000", 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4616000000000001e-05 2024-06-14T23:12:41.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.148336+0000", 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3771000000000001e-05 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.998870+0000", 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85053394100000002 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.999017+0000", 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014683900000000001 2024-06-14T23:12:41.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.999155+0000", 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013765199999999999 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024253+0000", 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025097715999999999 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024278+0000", 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5605e-05 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027270+0000", 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029918240000000001 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027306+0000", 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5534999999999998e-05 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027336+0000", 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9986999999999999e-05 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252629 10.e 10:71d7498e:::benchmark_data_smithi086_189745_object121661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.154557+0000", 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 499.76515986499999, 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87284944099999995, 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.980 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252629 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.154557+0000", 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.154557+0000", 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.154540+0000", 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999833 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158057+0000", 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035170869999999999 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158076+0000", 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8581999999999999e-05 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158083+0000", 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0350000000000004e-06 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158731+0000", 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064791399999999998 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.158738+0000", 2024-06-14T23:12:41.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1169999999999998e-06 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.002862+0000", 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84412396499999998 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.003212+0000", 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035084500000000002 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:12:41.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.003371+0000", 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015807500000000001 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024303+0000", 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020932504000000001 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024327+0000", 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4170999999999999e-05 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027347+0000", 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030197610000000001 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027377+0000", 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9986999999999999e-05 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027406+0000", 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9277000000000001e-05 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.986 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252647 10.e 10:72729fba:::benchmark_data_smithi086_189745_object121679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.172478+0000", 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 499.74723874099999, 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.854998225, 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252647 2024-06-14T23:12:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172478+0000", 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172478+0000", 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172476+0000", 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172482+0000", 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2740000000000004e-06 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172498+0000", 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6018000000000001e-05 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172504+0000", 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0970000000000002e-06 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172546+0000", 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2064000000000002e-05 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.172554+0000", 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4070000000000004e-06 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.009106+0000", 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83655226299999996 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.009214+0000", 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107619 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.009335+0000", 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121106 2024-06-14T23:12:41.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024353+0000", 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015018337999999999 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024378+0000", 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4732e-05 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027411+0000", 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030333130000000002 2024-06-14T23:12:41.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027443+0000", 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1362000000000002e-05 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027476+0000", 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3546e-05 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.993 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252651 10.e 10:711dfc34:::benchmark_data_smithi086_189745_object121683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.178987+0000", 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 499.74072939600001, 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84855638899999997, 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252651 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.178987+0000", 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.178987+0000", 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.178986+0000", 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.178994+0000", 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2700000000000004e-06 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.179007+0000", 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3383e-05 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.179013+0000", 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7339999999999996e-06 2024-06-14T23:12:41.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.179048+0000", 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5388000000000001e-05 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.179054+0000", 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6319999999999996e-06 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.012456+0000", 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83340202200000002 2024-06-14T23:12:41.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.012590+0000", 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133652 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.012733+0000", 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000142991 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.024409+0000", 2024-06-14T23:12:41.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011676116 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027040+0000", 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026314519999999998 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027482+0000", 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044158400000000002 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:41.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027513+0000", 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1535999999999998e-05 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027544+0000", 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0182999999999999e-05 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252662 10.e 10:75f70bd0:::benchmark_data_smithi086_189745_object121694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:12:42.000 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.201859+0000", 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 499.71785749999998, 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82584559400000002, 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252662 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201859+0000", 2024-06-14T23:12:42.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201859+0000", 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201858+0000", 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:42.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201863+0000", 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5469999999999998e-06 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201873+0000", 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0339e-05 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201877+0000", 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0430000000000001e-06 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201927+0000", 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0423999999999997e-05 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.201936+0000", 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3990000000000004e-06 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.015444+0000", 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.81350812299999997 2024-06-14T23:12:42.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.015563+0000", 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0001189 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.015691+0000", 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012809400000000001 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027265+0000", 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011573669999999999 2024-06-14T23:12:42.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027549+0000", 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000284787 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027646+0000", 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6413e-05 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027677+0000", 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0862999999999999e-05 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027705+0000", 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8107e-05 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252665 10.e 10:7d9cb9fd:::benchmark_data_smithi086_189745_object121697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-06-14T23:12:42.007 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:22.214428+0000", 2024-06-14T23:12:42.008 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 499.70528829099999, 2024-06-14T23:12:42.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.81702328300000004, 2024-06-14T23:12:42.008 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:42.008 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:42.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:42.009 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:42.009 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252665 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214428+0000", 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214428+0000", 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214427+0000", 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214432+0000", 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7999999999999998e-06 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214442+0000", 2024-06-14T23:12:42.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0379e-05 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214446+0000", 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0810000000000004e-06 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214494+0000", 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7839000000000001e-05 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.214500+0000", 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9889999999999997e-06 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.019682+0000", 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80518220100000004 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.019778+0000", 2024-06-14T23:12:42.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5921000000000005e-05 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.019885+0000", 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010651500000000001 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027554+0000", 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076687140000000001 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.027731+0000", 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000176863 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.031365+0000", 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036342330000000002 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.031413+0000", 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8579999999999999e-05 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.031452+0000", 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8331000000000003e-05 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36237.0:135097 11.d e261/237 11:b57b3051:::benchmark_data_smithi086_226760_object7776:head v 261'8402, mlcod=261'8400)", 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.307313+0000", 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.61240344499998, 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14073733699999999, 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:42.016 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307313+0000", 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307313+0000", 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307312+0000", 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:12:42.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307320+0000", 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9000000000000006e-06 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307333+0000", 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3521e-05 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307339+0000", 2024-06-14T23:12:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4439999999999999e-06 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307392+0000", 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3584000000000001e-05 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.307411+0000", 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8717999999999999e-05 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447999+0000", 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14058846 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448051+0000", 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1109000000000001e-05 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:135101 11.e 11:7b4cbd7b:::benchmark_data_smithi086_226760_object7780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.308293+0000", 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.611423785, 2024-06-14T23:12:42.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15223837100000001, 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135101 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308293+0000", 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308293+0000", 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308291+0000", 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308299+0000", 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6960000000000005e-06 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308312+0000", 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3261999999999999e-05 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308318+0000", 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1879999999999999e-06 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308365+0000", 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6332000000000003e-05 2024-06-14T23:12:42.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308470+0000", 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105128 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448019+0000", 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13954949 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.453774+0000", 2024-06-14T23:12:42.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005755151 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.453816+0000", 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1721999999999998e-05 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460440+0000", 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066234040000000003 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460497+0000", 2024-06-14T23:12:42.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7669e-05 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460531+0000", 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4076999999999999e-05 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36237.0:135100 11.c e261/237 11:33d4c343:::benchmark_data_smithi086_226760_object7779:head v 261'8533, mlcod=261'8531)", 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.308431+0000", 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.61128535500001, 2024-06-14T23:12:42.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.139588079, 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308431+0000", 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308431+0000", 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:42.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308431+0000", 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308439+0000", 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8979999999999996e-06 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308448+0000", 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4350000000000003e-06 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308452+0000", 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9600000000000002e-06 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308488+0000", 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5855000000000001e-05 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.308505+0000", 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.749e-05 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447983+0000", 2024-06-14T23:12:42.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.139477667 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448019+0000", 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6445000000000001e-05 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36237.0:135104 11.d e261/237 11:bcf46504:::benchmark_data_smithi086_226760_object7783:head v 261'8403, mlcod=261'8400)", 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.316932+0000", 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.60278510199998, 2024-06-14T23:12:42.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14248889100000001, 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.316932+0000", 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.316932+0000", 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:42.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.316931+0000", 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.316937+0000", 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6900000000000003e-06 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447970+0000", 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13103320500000001 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:42.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447984+0000", 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3740000000000001e-05 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448082+0000", 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7369999999999998e-05 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.448099+0000", 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7337999999999999e-05 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.459388+0000", 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011289344999999999 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.459420+0000", 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2211e-05 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:135114 11.e 11:7774b920:::benchmark_data_smithi086_226760_object7793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-06-14T23:12:42.034 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:58.324411+0000", 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.59530578300001, 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.136254772, 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135114 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.324411+0000", 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.324411+0000", 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.324409+0000", 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.324418+0000", 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7600000000000008e-06 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447972+0000", 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12355432099999999 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.447994+0000", 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2172e-05 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.451216+0000", 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032218099999999999 2024-06-14T23:12:42.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.451315+0000", 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8837000000000003e-05 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.451394+0000", 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240999999999996e-05 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460361+0000", 2024-06-14T23:12:42.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0089670789999999993 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460563+0000", 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020152599999999999 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460568+0000", 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1529999999999996e-06 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460635+0000", 2024-06-14T23:12:42.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6655000000000005e-05 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:58.460666+0000", 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1031000000000002e-05 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:144797 12.2 12:42e3d442:::benchmark_data_smithi086_261492_object28292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:12:09.873259+0000", 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 32.046458139999999, 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.071350900999999994, 2024-06-14T23:12:42.040 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 144797 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.873259+0000", 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:42.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.873259+0000", 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.873257+0000", 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.873266+0000", 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1230000000000006e-06 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.873281+0000", 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5197e-05 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.873288+0000", 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4659999999999996e-06 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.876788+0000", 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035000790000000001 2024-06-14T23:12:42.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.876906+0000", 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117997 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.877039+0000", 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013306 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.882252+0000", 2024-06-14T23:12:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052130309999999999 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.882265+0000", 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3056e-05 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.944411+0000", 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.062146169000000001 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.944480+0000", 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9332000000000006e-05 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:09.944609+0000", 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012913700000000001 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:146923 12.0 12:0fca8f43:::benchmark_data_smithi086_261492_object30418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-06-14T23:12:42.046 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:12:12.890085+0000", 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 29.029631187, 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.073105508999999999, 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 146923 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.890085+0000", 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.890085+0000", 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.890084+0000", 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.890090+0000", 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.319e-06 2024-06-14T23:12:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.890102+0000", 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1891e-05 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.890108+0000", 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1099999999999999e-06 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.890152+0000", 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3312000000000001e-05 2024-06-14T23:12:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.890274+0000", 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122199 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.890396+0000", 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012256499999999999 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.893533+0000", 2024-06-14T23:12:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003136425 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.894102+0000", 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00056878600000000005 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.960909+0000", 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.066807508000000002 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.960972+0000", 2024-06-14T23:12:42.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3250999999999995e-05 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:12.963191+0000", 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002218603 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:12:42.052 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:12:42.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:42.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:42.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:42.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:42.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:42.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:42.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:42.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:42.457 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:42.491 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:42.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:42.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:42.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:42.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:42.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:42.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:42.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:42.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:42.901 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:42.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:42.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:42 smithi086 ceph-mon[97913]: from='client.46661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:12:42.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:42.950 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:42 smithi086 ceph-mon[102754]: from='client.46661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:12:42.950 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:42.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:42 smithi137 ceph-mon[90634]: from='client.46661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:12:42.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:43.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:43.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:43.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:43.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:43.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:43.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:43.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:43.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:43.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:43.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:43.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1251,"stamp":"2024-06-14T23:12:40.257906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4516519406,"num_objects":68930,"num_object_clones":0,"num_object_copies":206790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32999,"num_objects_misplaced":3595,"num_objects_unfound":0,"num_objects_dirty":68930,"num_whiteouts":0,"num_read":56,"num_read_kb":104,"num_write":164188,"num_write_kb":7456431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14238,"num_bytes_recovered":932156630,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83430,"ondisk_log_size":83430,"up":357,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":16014312,"kb_used_data":11444920,"kb_used_omap":27,"kb_used_meta":4569124,"kb_avail":546456600,"statfs":{"total":575970213888,"available":559571558400,"internally_reserved":0,"allocated":11719598080,"data_stored":11709261315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":4678783430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-468910080,"num_objects":-7155,"num_object_clones":0,"num_object_copies":-21465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-944,"num_objects_misplaced":-444,"num_objects_unfound":0,"num_objects_dirty":-7155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":90963968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003389"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":715,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423100+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:12:18.423100+0000","last_peered":"2024-06-14T23:12:18.423100+0000","last_clean":"2024-06-14T23:12:18.423100+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:12:18.423100+0000","last_undegraded":"2024-06-14T23:12:18.423100+0000","last_fullsized":"2024-06-14T23:12:18.423100+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":153,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424524+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:12:18.424524+0000","last_peered":"2024-06-14T23:12:18.424524+0000","last_clean":"2024-06-14T23:12:18.424524+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:12:18.424524+0000","last_undegraded":"2024-06-14T23:12:18.424524+0000","last_fullsized":"2024-06-14T23:12:18.424524+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":323,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.426192+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:12:18.426192+0000","last_peered":"2024-06-14T23:12:18.426192+0000","last_clean":"2024-06-14T23:12:18.426192+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:12:18.426192+0000","last_undegraded":"2024-06-14T23:12:18.426192+0000","last_fullsized":"2024-06-14T23:12:18.426192+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":228,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.413836+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:12:18.413836+0000","last_peered":"2024-06-14T23:12:18.413836+0000","last_clean":"2024-06-14T23:12:18.413836+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:12:18.413836+0000","last_undegraded":"2024-06-14T23:12:18.413836+0000","last_fullsized":"2024-06-14T23:12:18.413836+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":118,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.414501+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:12:18.414501+0000","last_peered":"2024-06-14T23:12:18.414501+0000","last_clean":"2024-06-14T23:12:18.414501+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:12:18.414501+0000","last_undegraded":"2024-06-14T23:12:18.414501+0000","last_fullsized":"2024-06-14T23:12:18.414501+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":722,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:35.376950+0000","last_change":"2024-06-14T23:12:35.376950+0000","last_active":"2024-06-14T23:12:35.376950+0000","last_peered":"2024-06-14T23:12:35.376950+0000","last_clean":"2024-06-14T23:12:35.376950+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:12:35.376950+0000","last_undegraded":"2024-06-14T23:12:35.376950+0000","last_fullsized":"2024-06-14T23:12:35.376950+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:09:56.196559+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:09:56.196559+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":299,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424012+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:12:18.424012+0000","last_peered":"2024-06-14T23:12:18.424012+0000","last_clean":"2024-06-14T23:12:18.424012+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:12:18.424012+0000","last_undegraded":"2024-06-14T23:12:18.424012+0000","last_fullsized":"2024-06-14T23:12:18.424012+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":340,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423963+0000","last_change":"2024-06-14T23:11:23.413009+0000","last_active":"2024-06-14T23:12:18.423963+0000","last_peered":"2024-06-14T23:12:18.423963+0000","last_clean":"2024-06-14T23:12:18.423963+0000","last_became_active":"2024-06-14T23:11:23.412491+0000","last_became_peered":"2024-06-14T23:11:23.412491+0000","last_unstale":"2024-06-14T23:12:18.423963+0000","last_undegraded":"2024-06-14T23:12:18.423963+0000","last_fullsized":"2024-06-14T23:12:18.423963+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:57.263547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":486,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423351+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:12:18.423351+0000","last_peered":"2024-06-14T23:12:18.423351+0000","last_clean":"2024-06-14T23:12:18.423351+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:12:18.423351+0000","last_undegraded":"2024-06-14T23:12:18.423351+0000","last_fullsized":"2024-06-14T23:12:18.423351+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":313,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425419+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:12:18.425419+0000","last_peered":"2024-06-14T23:12:18.425419+0000","last_clean":"2024-06-14T23:12:18.425419+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:12:18.425419+0000","last_undegraded":"2024-06-14T23:12:18.425419+0000","last_fullsized":"2024-06-14T23:12:18.425419+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":340,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425942+0000","last_change":"2024-06-14T23:11:23.427771+0000","last_active":"2024-06-14T23:12:18.425942+0000","last_peered":"2024-06-14T23:12:18.425942+0000","last_clean":"2024-06-14T23:12:18.425942+0000","last_became_active":"2024-06-14T23:11:23.427481+0000","last_became_peered":"2024-06-14T23:11:23.427481+0000","last_unstale":"2024-06-14T23:12:18.425942+0000","last_undegraded":"2024-06-14T23:12:18.425942+0000","last_fullsized":"2024-06-14T23:12:18.425942+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:01:04.162256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":129,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425391+0000","last_change":"2024-06-14T23:10:26.053647+0000","last_active":"2024-06-14T23:12:18.425391+0000","last_peered":"2024-06-14T23:12:18.425391+0000","last_clean":"2024-06-14T23:12:18.425391+0000","last_became_active":"2024-06-14T23:10:26.053318+0000","last_became_peered":"2024-06-14T23:10:26.053318+0000","last_unstale":"2024-06-14T23:12:18.425391+0000","last_undegraded":"2024-06-14T23:12:18.425391+0000","last_fullsized":"2024-06-14T23:12:18.425391+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:32.647107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":128,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.414142+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:12:18.414142+0000","last_peered":"2024-06-14T23:12:18.414142+0000","last_clean":"2024-06-14T23:12:18.414142+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:12:18.414142+0000","last_undegraded":"2024-06-14T23:12:18.414142+0000","last_fullsized":"2024-06-14T23:12:18.414142+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":639,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.427330+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:12:18.427330+0000","last_peered":"2024-06-14T23:12:18.427330+0000","last_clean":"2024-06-14T23:12:18.427330+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:12:18.427330+0000","last_undegraded":"2024-06-14T23:12:18.427330+0000","last_fullsized":"2024-06-14T23:12:18.427330+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":362,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424022+0000","last_change":"2024-06-14T23:11:23.413241+0000","last_active":"2024-06-14T23:12:18.424022+0000","last_peered":"2024-06-14T23:12:18.424022+0000","last_clean":"2024-06-14T23:12:18.424022+0000","last_became_active":"2024-06-14T23:11:23.412936+0000","last_became_peered":"2024-06-14T23:11:23.412936+0000","last_unstale":"2024-06-14T23:12:18.424022+0000","last_undegraded":"2024-06-14T23:12:18.424022+0000","last_fullsized":"2024-06-14T23:12:18.424022+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:36.581716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":148,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424713+0000","last_change":"2024-06-14T23:10:26.038549+0000","last_active":"2024-06-14T23:12:18.424713+0000","last_peered":"2024-06-14T23:12:18.424713+0000","last_clean":"2024-06-14T23:12:18.424713+0000","last_became_active":"2024-06-14T23:10:26.038260+0000","last_became_peered":"2024-06-14T23:10:26.038260+0000","last_unstale":"2024-06-14T23:12:18.424713+0000","last_undegraded":"2024-06-14T23:12:18.424713+0000","last_fullsized":"2024-06-14T23:12:18.424713+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:44.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":742,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.420747+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:12:18.420747+0000","last_peered":"2024-06-14T23:12:18.420747+0000","last_clean":"2024-06-14T23:12:18.420747+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:12:18.420747+0000","last_undegraded":"2024-06-14T23:12:18.420747+0000","last_fullsized":"2024-06-14T23:12:18.420747+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":132,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424036+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:12:18.424036+0000","last_peered":"2024-06-14T23:12:18.424036+0000","last_clean":"2024-06-14T23:12:18.424036+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:12:18.424036+0000","last_undegraded":"2024-06-14T23:12:18.424036+0000","last_fullsized":"2024-06-14T23:12:18.424036+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":195,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:20.371048+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:12:20.371048+0000","last_peered":"2024-06-14T23:12:20.371048+0000","last_clean":"2024-06-14T23:12:20.371048+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:12:20.371048+0000","last_undegraded":"2024-06-14T23:12:20.371048+0000","last_fullsized":"2024-06-14T23:12:20.371048+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":690,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421942+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:12:18.421942+0000","last_peered":"2024-06-14T23:12:18.421942+0000","last_clean":"2024-06-14T23:12:18.421942+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:12:18.421942+0000","last_undegraded":"2024-06-14T23:12:18.421942+0000","last_fullsized":"2024-06-14T23:12:18.421942+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":56,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.414290+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:12:18.414290+0000","last_peered":"2024-06-14T23:12:18.414290+0000","last_clean":"2024-06-14T23:12:18.414290+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:12:18.414290+0000","last_undegraded":"2024-06-14T23:12:18.414290+0000","last_fullsized":"2024-06-14T23:12:18.414290+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":287,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.430285+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:12:18.430285+0000","last_peered":"2024-06-14T23:12:18.430285+0000","last_clean":"2024-06-14T23:12:18.430285+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:12:18.430285+0000","last_undegraded":"2024-06-14T23:12:18.430285+0000","last_fullsized":"2024-06-14T23:12:18.430285+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":642,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423331+0000","last_change":"2024-06-14T23:10:29.062659+0000","last_active":"2024-06-14T23:12:18.423331+0000","last_peered":"2024-06-14T23:12:18.423331+0000","last_clean":"2024-06-14T23:12:18.423331+0000","last_became_active":"2024-06-14T23:10:29.062256+0000","last_became_peered":"2024-06-14T23:10:29.062256+0000","last_unstale":"2024-06-14T23:12:18.423331+0000","last_undegraded":"2024-06-14T23:12:18.423331+0000","last_fullsized":"2024-06-14T23:12:18.423331+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:38.814157+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":318,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.422130+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:12:18.422130+0000","last_peered":"2024-06-14T23:12:18.422130+0000","last_clean":"2024-06-14T23:12:18.422130+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:12:18.422130+0000","last_undegraded":"2024-06-14T23:12:18.422130+0000","last_fullsized":"2024-06-14T23:12:18.422130+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":183,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424515+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:12:18.424515+0000","last_peered":"2024-06-14T23:12:18.424515+0000","last_clean":"2024-06-14T23:12:18.424515+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:12:18.424515+0000","last_undegraded":"2024-06-14T23:12:18.424515+0000","last_fullsized":"2024-06-14T23:12:18.424515+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":642,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424852+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:12:18.424852+0000","last_peered":"2024-06-14T23:12:18.424852+0000","last_clean":"2024-06-14T23:12:18.424852+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:12:18.424852+0000","last_undegraded":"2024-06-14T23:12:18.424852+0000","last_fullsized":"2024-06-14T23:12:18.424852+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":201,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423802+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:12:18.423802+0000","last_peered":"2024-06-14T23:12:18.423802+0000","last_clean":"2024-06-14T23:12:18.423802+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:12:18.423802+0000","last_undegraded":"2024-06-14T23:12:18.423802+0000","last_fullsized":"2024-06-14T23:12:18.423802+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":307,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424333+0000","last_change":"2024-06-14T23:10:26.053008+0000","last_active":"2024-06-14T23:12:18.424333+0000","last_peered":"2024-06-14T23:12:18.424333+0000","last_clean":"2024-06-14T23:12:18.424333+0000","last_became_active":"2024-06-14T23:10:26.052701+0000","last_became_peered":"2024-06-14T23:10:26.052701+0000","last_unstale":"2024-06-14T23:12:18.424333+0000","last_undegraded":"2024-06-14T23:12:18.424333+0000","last_fullsized":"2024-06-14T23:12:18.424333+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:15:03.929062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":666,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.414064+0000","last_change":"2024-06-14T23:11:23.411926+0000","last_active":"2024-06-14T23:12:18.414064+0000","last_peered":"2024-06-14T23:12:18.414064+0000","last_clean":"2024-06-14T23:12:18.414064+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:12:18.414064+0000","last_undegraded":"2024-06-14T23:12:18.414064+0000","last_fullsized":"2024-06-14T23:12:18.414064+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":219,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423343+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:12:18.423343+0000","last_peered":"2024-06-14T23:12:18.423343+0000","last_clean":"2024-06-14T23:12:18.423343+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:12:18.423343+0000","last_undegraded":"2024-06-14T23:12:18.423343+0000","last_fullsized":"2024-06-14T23:12:18.423343+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":349,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.426738+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:12:18.426738+0000","last_peered":"2024-06-14T23:12:18.426738+0000","last_clean":"2024-06-14T23:12:18.426738+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:12:18.426738+0000","last_undegraded":"2024-06-14T23:12:18.426738+0000","last_fullsized":"2024-06-14T23:12:18.426738+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":512,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424304+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:12:18.424304+0000","last_peered":"2024-06-14T23:12:18.424304+0000","last_clean":"2024-06-14T23:12:18.424304+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:12:18.424304+0000","last_undegraded":"2024-06-14T23:12:18.424304+0000","last_fullsized":"2024-06-14T23:12:18.424304+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":319,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424447+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:12:18.424447+0000","last_peered":"2024-06-14T23:12:18.424447+0000","last_clean":"2024-06-14T23:12:18.424447+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:12:18.424447+0000","last_undegraded":"2024-06-14T23:12:18.424447+0000","last_fullsized":"2024-06-14T23:12:18.424447+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":323,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.426001+0000","last_change":"2024-06-14T23:10:26.047451+0000","last_active":"2024-06-14T23:12:18.426001+0000","last_peered":"2024-06-14T23:12:18.426001+0000","last_clean":"2024-06-14T23:12:18.426001+0000","last_became_active":"2024-06-14T23:10:26.046781+0000","last_became_peered":"2024-06-14T23:10:26.046781+0000","last_unstale":"2024-06-14T23:12:18.426001+0000","last_undegraded":"2024-06-14T23:12:18.426001+0000","last_fullsized":"2024-06-14T23:12:18.426001+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:07:15.892864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":551,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425626+0000","last_change":"2024-06-14T23:10:39.253251+0000","last_active":"2024-06-14T23:12:18.425626+0000","last_peered":"2024-06-14T23:12:18.425626+0000","last_clean":"2024-06-14T23:12:18.425626+0000","last_became_active":"2024-06-14T23:10:29.061386+0000","last_became_peered":"2024-06-14T23:10:29.061386+0000","last_unstale":"2024-06-14T23:12:18.425626+0000","last_undegraded":"2024-06-14T23:12:18.425626+0000","last_fullsized":"2024-06-14T23:12:18.425626+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.735431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.735431+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"336'10188","reported_seq":10249,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:35.322788+0000","last_change":"2024-06-14T23:11:24.652983+0000","last_active":"2024-06-14T23:12:35.322788+0000","last_peered":"2024-06-14T23:12:35.322788+0000","last_clean":"2024-06-14T23:10:37.054302+0000","last_became_active":"2024-06-14T23:11:24.646468+0000","last_became_peered":"2024-06-14T23:11:24.646468+0000","last_unstale":"2024-06-14T23:12:35.322788+0000","last_undegraded":"2024-06-14T23:11:24.406142+0000","last_fullsized":"2024-06-14T23:11:24.380989+0000","mapping_epoch":323,"log_start":"318'5805","ondisk_log_start":"318'5805","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:33:08.769394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287834112,"num_objects":4392,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10188,"num_write_kb":466560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1924}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":296,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423402+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:12:18.423402+0000","last_peered":"2024-06-14T23:12:18.423402+0000","last_clean":"2024-06-14T23:12:18.423402+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:12:18.423402+0000","last_undegraded":"2024-06-14T23:12:18.423402+0000","last_fullsized":"2024-06-14T23:12:18.423402+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":308,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.430100+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:12:18.430100+0000","last_peered":"2024-06-14T23:12:18.430100+0000","last_clean":"2024-06-14T23:12:18.430100+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:12:18.430100+0000","last_undegraded":"2024-06-14T23:12:18.430100+0000","last_fullsized":"2024-06-14T23:12:18.430100+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":94,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425489+0000","last_change":"2024-06-14T23:10:26.058423+0000","last_active":"2024-06-14T23:12:18.425489+0000","last_peered":"2024-06-14T23:12:18.425489+0000","last_clean":"2024-06-14T23:12:18.425489+0000","last_became_active":"2024-06-14T23:10:26.050244+0000","last_became_peered":"2024-06-14T23:10:26.050244+0000","last_unstale":"2024-06-14T23:12:18.425489+0000","last_undegraded":"2024-06-14T23:12:18.425489+0000","last_fullsized":"2024-06-14T23:12:18.425489+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:52.060325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":660,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421816+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:12:18.421816+0000","last_peered":"2024-06-14T23:12:18.421816+0000","last_clean":"2024-06-14T23:12:18.421816+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:12:18.421816+0000","last_undegraded":"2024-06-14T23:12:18.421816+0000","last_fullsized":"2024-06-14T23:12:18.421816+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"336'10316","reported_seq":10389,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:35.367141+0000","last_change":"2024-06-14T23:11:24.639438+0000","last_active":"2024-06-14T23:12:35.367141+0000","last_peered":"2024-06-14T23:12:35.367141+0000","last_clean":"2024-06-14T23:10:37.032734+0000","last_became_active":"2024-06-14T23:11:24.634884+0000","last_became_peered":"2024-06-14T23:11:24.634884+0000","last_unstale":"2024-06-14T23:12:35.367141+0000","last_undegraded":"2024-06-14T23:11:24.403675+0000","last_fullsized":"2024-06-14T23:11:24.380974+0000","mapping_epoch":323,"log_start":"318'6000","ondisk_log_start":"318'6000","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:48:19.382266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288751616,"num_objects":4406,"num_object_clones":0,"num_object_copies":13218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":471104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":249,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423408+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:12:18.423408+0000","last_peered":"2024-06-14T23:12:18.423408+0000","last_clean":"2024-06-14T23:12:18.423408+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:12:18.423408+0000","last_undegraded":"2024-06-14T23:12:18.423408+0000","last_fullsized":"2024-06-14T23:12:18.423408+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":299,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425770+0000","last_change":"2024-06-14T23:10:26.051450+0000","last_active":"2024-06-14T23:12:18.425770+0000","last_peered":"2024-06-14T23:12:18.425770+0000","last_clean":"2024-06-14T23:12:18.425770+0000","last_became_active":"2024-06-14T23:10:26.050830+0000","last_became_peered":"2024-06-14T23:10:26.050830+0000","last_unstale":"2024-06-14T23:12:18.425770+0000","last_undegraded":"2024-06-14T23:12:18.425770+0000","last_fullsized":"2024-06-14T23:12:18.425770+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:16:16.175014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":50,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424209+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:12:18.424209+0000","last_peered":"2024-06-14T23:12:18.424209+0000","last_clean":"2024-06-14T23:12:18.424209+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:12:18.424209+0000","last_undegraded":"2024-06-14T23:12:18.424209+0000","last_fullsized":"2024-06-14T23:12:18.424209+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":750,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424783+0000","last_change":"2024-06-14T23:11:23.401974+0000","last_active":"2024-06-14T23:12:18.424783+0000","last_peered":"2024-06-14T23:12:18.424783+0000","last_clean":"2024-06-14T23:12:18.424783+0000","last_became_active":"2024-06-14T23:11:23.401625+0000","last_became_peered":"2024-06-14T23:11:23.401625+0000","last_unstale":"2024-06-14T23:12:18.424783+0000","last_undegraded":"2024-06-14T23:12:18.424783+0000","last_fullsized":"2024-06-14T23:12:18.424783+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.214300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.214300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:05:54.552362+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"336'10171","reported_seq":10233,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:35.391627+0000","last_change":"2024-06-14T23:11:24.651371+0000","last_active":"2024-06-14T23:12:35.391627+0000","last_peered":"2024-06-14T23:12:35.391627+0000","last_clean":"2024-06-14T23:10:37.020885+0000","last_became_active":"2024-06-14T23:11:24.646284+0000","last_became_peered":"2024-06-14T23:11:24.646284+0000","last_unstale":"2024-06-14T23:12:35.391627+0000","last_undegraded":"2024-06-14T23:11:24.405009+0000","last_fullsized":"2024-06-14T23:11:24.380990+0000","mapping_epoch":323,"log_start":"318'5842","ondisk_log_start":"318'5842","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:26:58.760076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293797888,"num_objects":4483,"num_object_clones":0,"num_object_copies":13449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10171,"num_write_kb":468928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1927}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":147,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.422123+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:12:18.422123+0000","last_peered":"2024-06-14T23:12:18.422123+0000","last_clean":"2024-06-14T23:12:18.422123+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:12:18.422123+0000","last_undegraded":"2024-06-14T23:12:18.422123+0000","last_fullsized":"2024-06-14T23:12:18.422123+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":321,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425009+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:12:18.425009+0000","last_peered":"2024-06-14T23:12:18.425009+0000","last_clean":"2024-06-14T23:12:18.425009+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:12:18.425009+0000","last_undegraded":"2024-06-14T23:12:18.425009+0000","last_fullsized":"2024-06-14T23:12:18.425009+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":272,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.414148+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:12:18.414148+0000","last_peered":"2024-06-14T23:12:18.414148+0000","last_clean":"2024-06-14T23:12:18.414148+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:12:18.414148+0000","last_undegraded":"2024-06-14T23:12:18.414148+0000","last_fullsized":"2024-06-14T23:12:18.414148+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":635,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.432790+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:12:18.432790+0000","last_peered":"2024-06-14T23:12:18.432790+0000","last_clean":"2024-06-14T23:12:18.432790+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:12:18.432790+0000","last_undegraded":"2024-06-14T23:12:18.432790+0000","last_fullsized":"2024-06-14T23:12:18.432790+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"336'10272","reported_seq":14856,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:35.356577+0000","last_change":"2024-06-14T23:11:24.647584+0000","last_active":"2024-06-14T23:12:35.356577+0000","last_peered":"2024-06-14T23:12:35.356577+0000","last_clean":"2024-06-14T23:10:25.009797+0000","last_became_active":"2024-06-14T23:11:24.638765+0000","last_became_peered":"2024-06-14T23:11:24.638765+0000","last_unstale":"2024-06-14T23:12:35.356577+0000","last_undegraded":"2024-06-14T23:11:24.535343+0000","last_fullsized":"2024-06-14T23:11:24.381001+0000","mapping_epoch":323,"log_start":"318'5972","ondisk_log_start":"318'5972","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:37.543254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294912000,"num_objects":4500,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10272,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2251,"num_bytes_recovered":147521536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[3,2],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,2,3","objects":1374},{"shards":"1,3,4","objects":2529},{"shards":"2,3","objects":-633}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":112,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421430+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:12:18.421430+0000","last_peered":"2024-06-14T23:12:18.421430+0000","last_clean":"2024-06-14T23:12:18.421430+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:12:18.421430+0000","last_undegraded":"2024-06-14T23:12:18.421430+0000","last_fullsized":"2024-06-14T23:12:18.421430+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":56,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.413973+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:12:18.413973+0000","last_peered":"2024-06-14T23:12:18.413973+0000","last_clean":"2024-06-14T23:12:18.413973+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:12:18.413973+0000","last_undegraded":"2024-06-14T23:12:18.413973+0000","last_fullsized":"2024-06-14T23:12:18.413973+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":351,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424090+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:12:18.424090+0000","last_peered":"2024-06-14T23:12:18.424090+0000","last_clean":"2024-06-14T23:12:18.424090+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:12:18.424090+0000","last_undegraded":"2024-06-14T23:12:18.424090+0000","last_fullsized":"2024-06-14T23:12:18.424090+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":615,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.430162+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:12:18.430162+0000","last_peered":"2024-06-14T23:12:18.430162+0000","last_clean":"2024-06-14T23:12:18.430162+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:12:18.430162+0000","last_undegraded":"2024-06-14T23:12:18.430162+0000","last_fullsized":"2024-06-14T23:12:18.430162+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"336'10423","reported_seq":15661,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:38.207301+0000","last_change":"2024-06-14T23:11:50.568053+0000","last_active":"2024-06-14T23:12:38.207301+0000","last_peered":"2024-06-14T23:12:38.207301+0000","last_clean":"2024-06-14T23:12:38.207301+0000","last_became_active":"2024-06-14T23:11:50.555786+0000","last_became_peered":"2024-06-14T23:11:50.555786+0000","last_unstale":"2024-06-14T23:12:38.207301+0000","last_undegraded":"2024-06-14T23:12:38.207301+0000","last_fullsized":"2024-06-14T23:12:38.207301+0000","mapping_epoch":328,"log_start":"317'5000","ondisk_log_start":"317'5000","created":295,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5423,"log_dups_size":0,"ondisk_log_size":5423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:36.382663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285147136,"num_objects":4351,"num_object_clones":0,"num_object_copies":13053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10423,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2575,"num_bytes_recovered":168755200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":287,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.413710+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:12:18.413710+0000","last_peered":"2024-06-14T23:12:18.413710+0000","last_clean":"2024-06-14T23:12:18.413710+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:12:18.413710+0000","last_undegraded":"2024-06-14T23:12:18.413710+0000","last_fullsized":"2024-06-14T23:12:18.413710+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":334,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.426113+0000","last_change":"2024-06-14T23:10:26.049033+0000","last_active":"2024-06-14T23:12:18.426113+0000","last_peered":"2024-06-14T23:12:18.426113+0000","last_clean":"2024-06-14T23:12:18.426113+0000","last_became_active":"2024-06-14T23:10:26.047775+0000","last_became_peered":"2024-06-14T23:10:26.047775+0000","last_unstale":"2024-06-14T23:12:18.426113+0000","last_undegraded":"2024-06-14T23:12:18.426113+0000","last_fullsized":"2024-06-14T23:12:18.426113+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:15:52.051858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":328,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421655+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:12:18.421655+0000","last_peered":"2024-06-14T23:12:18.421655+0000","last_clean":"2024-06-14T23:12:18.421655+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:12:18.421655+0000","last_undegraded":"2024-06-14T23:12:18.421655+0000","last_fullsized":"2024-06-14T23:12:18.421655+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":729,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421118+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:12:18.421118+0000","last_peered":"2024-06-14T23:12:18.421118+0000","last_clean":"2024-06-14T23:12:18.421118+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:12:18.421118+0000","last_undegraded":"2024-06-14T23:12:18.421118+0000","last_fullsized":"2024-06-14T23:12:18.421118+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"336'10368","reported_seq":10438,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:36.647944+0000","last_change":"2024-06-14T23:11:24.656131+0000","last_active":"2024-06-14T23:12:36.647944+0000","last_peered":"2024-06-14T23:12:36.647944+0000","last_clean":"2024-06-14T23:10:37.043860+0000","last_became_active":"2024-06-14T23:11:24.647995+0000","last_became_peered":"2024-06-14T23:11:24.647995+0000","last_unstale":"2024-06-14T23:12:36.647944+0000","last_undegraded":"2024-06-14T23:11:24.403317+0000","last_fullsized":"2024-06-14T23:11:24.380506+0000","mapping_epoch":323,"log_start":"318'5400","ondisk_log_start":"318'5400","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4968,"log_dups_size":0,"ondisk_log_size":4968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:32:10.827694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283639808,"num_objects":4328,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":470272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1827}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":296,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424414+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:12:18.424414+0000","last_peered":"2024-06-14T23:12:18.424414+0000","last_clean":"2024-06-14T23:12:18.424414+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:12:18.424414+0000","last_undegraded":"2024-06-14T23:12:18.424414+0000","last_fullsized":"2024-06-14T23:12:18.424414+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":332,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.413700+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:12:18.413700+0000","last_peered":"2024-06-14T23:12:18.413700+0000","last_clean":"2024-06-14T23:12:18.413700+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:12:18.413700+0000","last_undegraded":"2024-06-14T23:12:18.413700+0000","last_fullsized":"2024-06-14T23:12:18.413700+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":137,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.414437+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:12:18.414437+0000","last_peered":"2024-06-14T23:12:18.414437+0000","last_clean":"2024-06-14T23:12:18.414437+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:12:18.414437+0000","last_undegraded":"2024-06-14T23:12:18.414437+0000","last_fullsized":"2024-06-14T23:12:18.414437+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":623,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423785+0000","last_change":"2024-06-14T23:11:23.413224+0000","last_active":"2024-06-14T23:12:18.423785+0000","last_peered":"2024-06-14T23:12:18.423785+0000","last_clean":"2024-06-14T23:12:18.423785+0000","last_became_active":"2024-06-14T23:11:23.412680+0000","last_became_peered":"2024-06-14T23:11:23.412680+0000","last_unstale":"2024-06-14T23:12:18.423785+0000","last_undegraded":"2024-06-14T23:12:18.423785+0000","last_fullsized":"2024-06-14T23:12:18.423785+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:48:10.967651+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.e","version":"336'10249","reported_seq":10308,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:38.353413+0000","last_change":"2024-06-14T23:09:41.994118+0000","last_active":"2024-06-14T23:12:38.353413+0000","last_peered":"2024-06-14T23:12:38.353413+0000","last_clean":"2024-06-14T23:12:38.353413+0000","last_became_active":"2024-06-14T23:09:41.993702+0000","last_became_peered":"2024-06-14T23:09:41.993702+0000","last_unstale":"2024-06-14T23:12:38.353413+0000","last_undegraded":"2024-06-14T23:12:38.353413+0000","last_fullsized":"2024-06-14T23:12:38.353413+0000","mapping_epoch":299,"log_start":"317'4900","ondisk_log_start":"317'4900","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5349,"log_dups_size":0,"ondisk_log_size":5349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:25.994252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280690688,"num_objects":4283,"num_object_clones":0,"num_object_copies":12849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10249,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":305,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.413852+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:12:18.413852+0000","last_peered":"2024-06-14T23:12:18.413852+0000","last_clean":"2024-06-14T23:12:18.413852+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:12:18.413852+0000","last_undegraded":"2024-06-14T23:12:18.413852+0000","last_fullsized":"2024-06-14T23:12:18.413852+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":184,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423248+0000","last_change":"2024-06-14T23:09:41.991652+0000","last_active":"2024-06-14T23:12:18.423248+0000","last_peered":"2024-06-14T23:12:18.423248+0000","last_clean":"2024-06-14T23:12:18.423248+0000","last_became_active":"2024-06-14T23:09:41.991317+0000","last_became_peered":"2024-06-14T23:09:41.991317+0000","last_unstale":"2024-06-14T23:12:18.423248+0000","last_undegraded":"2024-06-14T23:12:18.423248+0000","last_fullsized":"2024-06-14T23:12:18.423248+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:16:18.209329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":301,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:25.385319+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:12:25.385319+0000","last_peered":"2024-06-14T23:12:25.385319+0000","last_clean":"2024-06-14T23:12:25.385319+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:12:25.385319+0000","last_undegraded":"2024-06-14T23:12:25.385319+0000","last_fullsized":"2024-06-14T23:12:25.385319+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":543,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423832+0000","last_change":"2024-06-14T23:10:29.063981+0000","last_active":"2024-06-14T23:12:18.423832+0000","last_peered":"2024-06-14T23:12:18.423832+0000","last_clean":"2024-06-14T23:12:18.423832+0000","last_became_active":"2024-06-14T23:10:29.063546+0000","last_became_peered":"2024-06-14T23:10:29.063546+0000","last_unstale":"2024-06-14T23:12:18.423832+0000","last_undegraded":"2024-06-14T23:12:18.423832+0000","last_fullsized":"2024-06-14T23:12:18.423832+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:26:23.411322+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"336'10351","reported_seq":16569,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:38.714291+0000","last_change":"2024-06-14T23:11:24.664901+0000","last_active":"2024-06-14T23:12:38.714291+0000","last_peered":"2024-06-14T23:12:38.714291+0000","last_clean":"2024-06-14T23:10:24.989118+0000","last_became_active":"2024-06-14T23:11:24.647313+0000","last_became_peered":"2024-06-14T23:11:24.647313+0000","last_unstale":"2024-06-14T23:12:38.714291+0000","last_undegraded":"2024-06-14T23:11:24.555289+0000","last_fullsized":"2024-06-14T23:11:24.381668+0000","mapping_epoch":323,"log_start":"318'5500","ondisk_log_start":"318'5500","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:48:12.784805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276693016,"num_objects":4223,"num_object_clones":0,"num_object_copies":12669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10351,"num_write_kb":466305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":201064448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,3],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,2,7","objects":1119},{"shards":"1,3,7","objects":2484},{"shards":"3,7","objects":620}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":251,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.436461+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:12:18.436461+0000","last_peered":"2024-06-14T23:12:18.436461+0000","last_clean":"2024-06-14T23:12:18.436461+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:12:18.436461+0000","last_undegraded":"2024-06-14T23:12:18.436461+0000","last_fullsized":"2024-06-14T23:12:18.436461+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":257,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421590+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:12:18.421590+0000","last_peered":"2024-06-14T23:12:18.421590+0000","last_clean":"2024-06-14T23:12:18.421590+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:12:18.421590+0000","last_undegraded":"2024-06-14T23:12:18.421590+0000","last_fullsized":"2024-06-14T23:12:18.421590+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":338,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.422183+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:12:18.422183+0000","last_peered":"2024-06-14T23:12:18.422183+0000","last_clean":"2024-06-14T23:12:18.422183+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:12:18.422183+0000","last_undegraded":"2024-06-14T23:12:18.422183+0000","last_fullsized":"2024-06-14T23:12:18.422183+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":615,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.414235+0000","last_change":"2024-06-14T23:11:23.406149+0000","last_active":"2024-06-14T23:12:18.414235+0000","last_peered":"2024-06-14T23:12:18.414235+0000","last_clean":"2024-06-14T23:12:18.414235+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:12:18.414235+0000","last_undegraded":"2024-06-14T23:12:18.414235+0000","last_fullsized":"2024-06-14T23:12:18.414235+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.751667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.751667+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.7","version":"336'10306","reported_seq":16499,"reported_epoch":336,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-14T23:12:38.699192+0000","last_change":"2024-06-14T23:11:24.669886+0000","last_active":"2024-06-14T23:12:38.699192+0000","last_peered":"2024-06-14T23:12:38.699192+0000","last_clean":"2024-06-14T23:10:25.005348+0000","last_became_active":"2024-06-14T23:11:24.634913+0000","last_became_peered":"2024-06-14T23:11:24.634913+0000","last_unstale":"2024-06-14T23:12:38.699192+0000","last_undegraded":"2024-06-14T23:12:38.699192+0000","last_fullsized":"2024-06-14T23:11:24.386154+0000","mapping_epoch":323,"log_start":"318'5400","ondisk_log_start":"318'5400","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:26:38.637559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274989056,"num_objects":4196,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3595,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10306,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3056,"num_bytes_recovered":200278016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,4],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,2,7","objects":1101},{"shards":"1,4,7","objects":2494},{"shards":"4,7","objects":601}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":133,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.422598+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:12:18.422598+0000","last_peered":"2024-06-14T23:12:18.422598+0000","last_clean":"2024-06-14T23:12:18.422598+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:12:18.422598+0000","last_undegraded":"2024-06-14T23:12:18.422598+0000","last_fullsized":"2024-06-14T23:12:18.422598+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":305,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423539+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:12:18.423539+0000","last_peered":"2024-06-14T23:12:18.423539+0000","last_clean":"2024-06-14T23:12:18.423539+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:12:18.423539+0000","last_undegraded":"2024-06-14T23:12:18.423539+0000","last_fullsized":"2024-06-14T23:12:18.423539+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":727,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421297+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:12:18.421297+0000","last_peered":"2024-06-14T23:12:18.421297+0000","last_clean":"2024-06-14T23:12:18.421297+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:12:18.421297+0000","last_undegraded":"2024-06-14T23:12:18.421297+0000","last_fullsized":"2024-06-14T23:12:18.421297+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"336'10079","reported_seq":10597,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:36.643825+0000","last_change":"2024-06-14T23:11:24.664127+0000","last_active":"2024-06-14T23:12:36.643825+0000","last_peered":"2024-06-14T23:12:36.643825+0000","last_clean":"2024-06-14T23:10:25.013628+0000","last_became_active":"2024-06-14T23:10:27.203384+0000","last_became_peered":"2024-06-14T23:10:27.203384+0000","last_unstale":"2024-06-14T23:12:36.643825+0000","last_undegraded":"2024-06-14T23:11:24.661184+0000","last_fullsized":"2024-06-14T23:10:27.026246+0000","mapping_epoch":310,"log_start":"318'5100","ondisk_log_start":"318'5100","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4979,"log_dups_size":0,"ondisk_log_size":4979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:40.932486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274792448,"num_objects":4193,"num_object_clones":0,"num_object_copies":12579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10079,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":14548992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"1,6,7","objects":1210},{"shards":"6,7","objects":262}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":323,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425699+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:12:18.425699+0000","last_peered":"2024-06-14T23:12:18.425699+0000","last_clean":"2024-06-14T23:12:18.425699+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:12:18.425699+0000","last_undegraded":"2024-06-14T23:12:18.425699+0000","last_fullsized":"2024-06-14T23:12:18.425699+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":334,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423644+0000","last_change":"2024-06-14T23:11:23.427812+0000","last_active":"2024-06-14T23:12:18.423644+0000","last_peered":"2024-06-14T23:12:18.423644+0000","last_clean":"2024-06-14T23:12:18.423644+0000","last_became_active":"2024-06-14T23:11:23.427563+0000","last_became_peered":"2024-06-14T23:11:23.427563+0000","last_unstale":"2024-06-14T23:12:18.423644+0000","last_undegraded":"2024-06-14T23:12:18.423644+0000","last_fullsized":"2024-06-14T23:12:18.423644+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.941006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":695,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.422059+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:12:18.422059+0000","last_peered":"2024-06-14T23:12:18.422059+0000","last_clean":"2024-06-14T23:12:18.422059+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:12:18.422059+0000","last_undegraded":"2024-06-14T23:12:18.422059+0000","last_fullsized":"2024-06-14T23:12:18.422059+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"336'10099","reported_seq":16036,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:36.667323+0000","last_change":"2024-06-14T23:11:24.656651+0000","last_active":"2024-06-14T23:12:36.667323+0000","last_peered":"2024-06-14T23:12:36.667323+0000","last_clean":"2024-06-14T23:10:25.016384+0000","last_became_active":"2024-06-14T23:11:24.643623+0000","last_became_peered":"2024-06-14T23:11:24.643623+0000","last_unstale":"2024-06-14T23:12:36.667323+0000","last_undegraded":"2024-06-14T23:11:24.519360+0000","last_fullsized":"2024-06-14T23:11:24.380111+0000","mapping_epoch":323,"log_start":"318'5100","ondisk_log_start":"318'5100","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4999,"log_dups_size":0,"ondisk_log_size":4999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:18:43.920069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278069248,"num_objects":4243,"num_object_clones":0,"num_object_copies":12729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3680,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10099,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2929,"num_bytes_recovered":191954944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,3],"avail_no_missing":["6"],"object_location_counts":[{"shards":"1,2,6","objects":1219},{"shards":"1,3,6","objects":2461},{"shards":"3,6","objects":563}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":353,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424966+0000","last_change":"2024-06-14T23:11:23.416264+0000","last_active":"2024-06-14T23:12:18.424966+0000","last_peered":"2024-06-14T23:12:18.424966+0000","last_clean":"2024-06-14T23:12:18.424966+0000","last_became_active":"2024-06-14T23:11:23.415946+0000","last_became_peered":"2024-06-14T23:11:23.415946+0000","last_unstale":"2024-06-14T23:12:18.424966+0000","last_undegraded":"2024-06-14T23:12:18.424966+0000","last_fullsized":"2024-06-14T23:12:18.424966+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:22.668569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":204,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423935+0000","last_change":"2024-06-14T23:10:26.045363+0000","last_active":"2024-06-14T23:12:18.423935+0000","last_peered":"2024-06-14T23:12:18.423935+0000","last_clean":"2024-06-14T23:12:18.423935+0000","last_became_active":"2024-06-14T23:10:26.043844+0000","last_became_peered":"2024-06-14T23:10:26.043844+0000","last_unstale":"2024-06-14T23:12:18.423935+0000","last_undegraded":"2024-06-14T23:12:18.423935+0000","last_fullsized":"2024-06-14T23:12:18.423935+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:11:27.891184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":718,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.420598+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:12:18.420598+0000","last_peered":"2024-06-14T23:12:18.420598+0000","last_clean":"2024-06-14T23:12:18.420598+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:12:18.420598+0000","last_undegraded":"2024-06-14T23:12:18.420598+0000","last_fullsized":"2024-06-14T23:12:18.420598+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"336'10272","reported_seq":10563,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:36.642099+0000","last_change":"2024-06-14T23:10:39.484566+0000","last_active":"2024-06-14T23:12:36.642099+0000","last_peered":"2024-06-14T23:12:36.642099+0000","last_clean":"2024-06-14T23:10:24.998059+0000","last_became_active":"2024-06-14T23:10:27.239025+0000","last_became_peered":"2024-06-14T23:10:27.239025+0000","last_unstale":"2024-06-14T23:12:36.642099+0000","last_undegraded":"2024-06-14T23:10:27.071395+0000","last_fullsized":"2024-06-14T23:10:27.025672+0000","mapping_epoch":310,"log_start":"318'5300","ondisk_log_start":"318'5300","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:52:40.136978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280625152,"num_objects":4282,"num_object_clones":0,"num_object_copies":12846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10272,"num_write_kb":465728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":7602176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":1172},{"shards":"3,6","objects":229}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":313,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424361+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:12:18.424361+0000","last_peered":"2024-06-14T23:12:18.424361+0000","last_clean":"2024-06-14T23:12:18.424361+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:12:18.424361+0000","last_undegraded":"2024-06-14T23:12:18.424361+0000","last_fullsized":"2024-06-14T23:12:18.424361+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":134,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425192+0000","last_change":"2024-06-14T23:10:26.056252+0000","last_active":"2024-06-14T23:12:18.425192+0000","last_peered":"2024-06-14T23:12:18.425192+0000","last_clean":"2024-06-14T23:12:18.425192+0000","last_became_active":"2024-06-14T23:10:26.046629+0000","last_became_peered":"2024-06-14T23:10:26.046629+0000","last_unstale":"2024-06-14T23:12:18.425192+0000","last_undegraded":"2024-06-14T23:12:18.425192+0000","last_fullsized":"2024-06-14T23:12:18.425192+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:51:19.277386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":606,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.414271+0000","last_change":"2024-06-14T23:11:23.428346+0000","last_active":"2024-06-14T23:12:18.414271+0000","last_peered":"2024-06-14T23:12:18.414271+0000","last_clean":"2024-06-14T23:12:18.414271+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:12:18.414271+0000","last_undegraded":"2024-06-14T23:12:18.414271+0000","last_fullsized":"2024-06-14T23:12:18.414271+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.656210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.656210+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"336'10387","reported_seq":10446,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:38.663254+0000","last_change":"2024-06-14T23:09:41.992020+0000","last_active":"2024-06-14T23:12:38.663254+0000","last_peered":"2024-06-14T23:12:38.663254+0000","last_clean":"2024-06-14T23:12:38.663254+0000","last_became_active":"2024-06-14T23:09:41.991722+0000","last_became_peered":"2024-06-14T23:09:41.991722+0000","last_unstale":"2024-06-14T23:12:38.663254+0000","last_undegraded":"2024-06-14T23:12:38.663254+0000","last_fullsized":"2024-06-14T23:12:38.663254+0000","mapping_epoch":299,"log_start":"318'5500","ondisk_log_start":"318'5500","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:35:54.751263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275972096,"num_objects":4211,"num_object_clones":0,"num_object_copies":12633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10387,"num_write_kb":467136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":133,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425113+0000","last_change":"2024-06-14T23:10:26.047111+0000","last_active":"2024-06-14T23:12:18.425113+0000","last_peered":"2024-06-14T23:12:18.425113+0000","last_clean":"2024-06-14T23:12:18.425113+0000","last_became_active":"2024-06-14T23:10:26.045540+0000","last_became_peered":"2024-06-14T23:10:26.045540+0000","last_unstale":"2024-06-14T23:12:18.425113+0000","last_undegraded":"2024-06-14T23:12:18.425113+0000","last_fullsized":"2024-06-14T23:12:18.425113+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:04.131400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":314,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425500+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:12:18.425500+0000","last_peered":"2024-06-14T23:12:18.425500+0000","last_clean":"2024-06-14T23:12:18.425500+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:12:18.425500+0000","last_undegraded":"2024-06-14T23:12:18.425500+0000","last_fullsized":"2024-06-14T23:12:18.425500+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":574,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.425149+0000","last_change":"2024-06-14T23:09:17.382466+0000","last_active":"2024-06-14T23:12:18.425149+0000","last_peered":"2024-06-14T23:12:18.425149+0000","last_clean":"2024-06-14T23:12:18.425149+0000","last_became_active":"2024-06-14T23:09:12.030205+0000","last_became_peered":"2024-06-14T23:09:12.030205+0000","last_unstale":"2024-06-14T23:12:18.425149+0000","last_undegraded":"2024-06-14T23:12:18.425149+0000","last_fullsized":"2024-06-14T23:12:18.425149+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:09:17.382389+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:33.121472+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"336'10055","reported_seq":10104,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:38.374682+0000","last_change":"2024-06-14T23:09:29.408859+0000","last_active":"2024-06-14T23:12:38.374682+0000","last_peered":"2024-06-14T23:12:38.374682+0000","last_clean":"2024-06-14T23:12:38.374682+0000","last_became_active":"2024-06-14T23:09:29.408548+0000","last_became_peered":"2024-06-14T23:09:29.408548+0000","last_unstale":"2024-06-14T23:12:38.374682+0000","last_undegraded":"2024-06-14T23:12:38.374682+0000","last_fullsized":"2024-06-14T23:12:38.374682+0000","mapping_epoch":295,"log_start":"317'4700","ondisk_log_start":"317'4700","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5355,"log_dups_size":0,"ondisk_log_size":5355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:19.810173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275709952,"num_objects":4207,"num_object_clones":0,"num_object_copies":12621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":456384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":287,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.426621+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:12:18.426621+0000","last_peered":"2024-06-14T23:12:18.426621+0000","last_clean":"2024-06-14T23:12:18.426621+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:12:18.426621+0000","last_undegraded":"2024-06-14T23:12:18.426621+0000","last_fullsized":"2024-06-14T23:12:18.426621+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":231,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423808+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:12:18.423808+0000","last_peered":"2024-06-14T23:12:18.423808+0000","last_clean":"2024-06-14T23:12:18.423808+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:12:18.423808+0000","last_undegraded":"2024-06-14T23:12:18.423808+0000","last_fullsized":"2024-06-14T23:12:18.423808+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":615,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.430226+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:12:18.430226+0000","last_peered":"2024-06-14T23:12:18.430226+0000","last_clean":"2024-06-14T23:12:18.430226+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:12:18.430226+0000","last_undegraded":"2024-06-14T23:12:18.430226+0000","last_fullsized":"2024-06-14T23:12:18.430226+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"336'10217","reported_seq":10266,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:36.661959+0000","last_change":"2024-06-14T23:09:29.403408+0000","last_active":"2024-06-14T23:12:36.661959+0000","last_peered":"2024-06-14T23:12:36.661959+0000","last_clean":"2024-06-14T23:12:36.661959+0000","last_became_active":"2024-06-14T23:09:29.403000+0000","last_became_peered":"2024-06-14T23:09:29.403000+0000","last_unstale":"2024-06-14T23:12:36.661959+0000","last_undegraded":"2024-06-14T23:12:36.661959+0000","last_fullsized":"2024-06-14T23:12:36.661959+0000","mapping_epoch":295,"log_start":"318'5200","ondisk_log_start":"318'5200","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5017,"log_dups_size":0,"ondisk_log_size":5017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:16.838865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282001408,"num_objects":4303,"num_object_clones":0,"num_object_copies":12909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10217,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":323,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423578+0000","last_change":"2024-06-14T23:10:26.040201+0000","last_active":"2024-06-14T23:12:18.423578+0000","last_peered":"2024-06-14T23:12:18.423578+0000","last_clean":"2024-06-14T23:12:18.423578+0000","last_became_active":"2024-06-14T23:10:26.039865+0000","last_became_peered":"2024-06-14T23:10:26.039865+0000","last_unstale":"2024-06-14T23:12:18.423578+0000","last_undegraded":"2024-06-14T23:12:18.423578+0000","last_fullsized":"2024-06-14T23:12:18.423578+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:34.621202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":301,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.419013+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:12:18.419013+0000","last_peered":"2024-06-14T23:12:18.419013+0000","last_clean":"2024-06-14T23:12:18.419013+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:12:18.419013+0000","last_undegraded":"2024-06-14T23:12:18.419013+0000","last_fullsized":"2024-06-14T23:12:18.419013+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":743,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421892+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:12:18.421892+0000","last_peered":"2024-06-14T23:12:18.421892+0000","last_clean":"2024-06-14T23:12:18.421892+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:12:18.421892+0000","last_undegraded":"2024-06-14T23:12:18.421892+0000","last_fullsized":"2024-06-14T23:12:18.421892+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"336'10348","reported_seq":10414,"reported_epoch":336,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:38.290177+0000","last_change":"2024-06-14T23:11:24.632384+0000","last_active":"2024-06-14T23:12:38.290177+0000","last_peered":"2024-06-14T23:12:38.290177+0000","last_clean":"2024-06-14T23:10:37.053680+0000","last_became_active":"2024-06-14T23:11:24.623684+0000","last_became_peered":"2024-06-14T23:11:24.623684+0000","last_unstale":"2024-06-14T23:12:38.290177+0000","last_undegraded":"2024-06-14T23:11:24.400366+0000","last_fullsized":"2024-06-14T23:11:24.379058+0000","mapping_epoch":323,"log_start":"318'5000","ondisk_log_start":"318'5000","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5348,"log_dups_size":0,"ondisk_log_size":5348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:34:53.331507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282066944,"num_objects":4304,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10348,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":1750}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":296,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.420819+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:12:18.420819+0000","last_peered":"2024-06-14T23:12:18.420819+0000","last_clean":"2024-06-14T23:12:18.420819+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:12:18.420819+0000","last_undegraded":"2024-06-14T23:12:18.420819+0000","last_fullsized":"2024-06-14T23:12:18.420819+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":314,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.474481+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:12:18.474481+0000","last_peered":"2024-06-14T23:12:18.474481+0000","last_clean":"2024-06-14T23:12:18.474481+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:12:18.474481+0000","last_undegraded":"2024-06-14T23:12:18.474481+0000","last_fullsized":"2024-06-14T23:12:18.474481+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":769,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.424645+0000","last_change":"2024-06-14T23:11:23.418239+0000","last_active":"2024-06-14T23:12:18.424645+0000","last_peered":"2024-06-14T23:12:18.424645+0000","last_clean":"2024-06-14T23:12:18.424645+0000","last_became_active":"2024-06-14T23:11:23.417873+0000","last_became_peered":"2024-06-14T23:11:23.417873+0000","last_unstale":"2024-06-14T23:12:18.424645+0000","last_undegraded":"2024-06-14T23:12:18.424645+0000","last_fullsized":"2024-06-14T23:12:18.424645+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:04:50.740175+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":322,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.426584+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:12:18.426584+0000","last_peered":"2024-06-14T23:12:18.426584+0000","last_clean":"2024-06-14T23:12:18.426584+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:12:18.426584+0000","last_undegraded":"2024-06-14T23:12:18.426584+0000","last_fullsized":"2024-06-14T23:12:18.426584+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":318,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.422485+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:12:18.422485+0000","last_peered":"2024-06-14T23:12:18.422485+0000","last_clean":"2024-06-14T23:12:18.422485+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:12:18.422485+0000","last_undegraded":"2024-06-14T23:12:18.422485+0000","last_fullsized":"2024-06-14T23:12:18.422485+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":691,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421366+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:12:18.421366+0000","last_peered":"2024-06-14T23:12:18.421366+0000","last_clean":"2024-06-14T23:12:18.421366+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:12:18.421366+0000","last_undegraded":"2024-06-14T23:12:18.421366+0000","last_fullsized":"2024-06-14T23:12:18.421366+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":220,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423327+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:12:18.423327+0000","last_peered":"2024-06-14T23:12:18.423327+0000","last_clean":"2024-06-14T23:12:18.423327+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:12:18.423327+0000","last_undegraded":"2024-06-14T23:12:18.423327+0000","last_fullsized":"2024-06-14T23:12:18.423327+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":199,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423422+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:12:18.423422+0000","last_peered":"2024-06-14T23:12:18.423422+0000","last_clean":"2024-06-14T23:12:18.423422+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:12:18.423422+0000","last_undegraded":"2024-06-14T23:12:18.423422+0000","last_fullsized":"2024-06-14T23:12:18.423422+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":657,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423017+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:12:18.423017+0000","last_peered":"2024-06-14T23:12:18.423017+0000","last_clean":"2024-06-14T23:12:18.423017+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:12:18.423017+0000","last_undegraded":"2024-06-14T23:12:18.423017+0000","last_fullsized":"2024-06-14T23:12:18.423017+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":37,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.413964+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:12:18.413964+0000","last_peered":"2024-06-14T23:12:18.413964+0000","last_clean":"2024-06-14T23:12:18.413964+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:12:18.413964+0000","last_undegraded":"2024-06-14T23:12:18.413964+0000","last_fullsized":"2024-06-14T23:12:18.413964+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":296,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.423923+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:12:18.423923+0000","last_peered":"2024-06-14T23:12:18.423923+0000","last_clean":"2024-06-14T23:12:18.423923+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:12:18.423923+0000","last_undegraded":"2024-06-14T23:12:18.423923+0000","last_fullsized":"2024-06-14T23:12:18.423923+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":655,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.420514+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:12:18.420514+0000","last_peered":"2024-06-14T23:12:18.420514+0000","last_clean":"2024-06-14T23:12:18.420514+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:12:18.420514+0000","last_undegraded":"2024-06-14T23:12:18.420514+0000","last_fullsized":"2024-06-14T23:12:18.420514+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":348,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.416011+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:12:18.416011+0000","last_peered":"2024-06-14T23:12:18.416011+0000","last_clean":"2024-06-14T23:12:18.416011+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:12:18.416011+0000","last_undegraded":"2024-06-14T23:12:18.416011+0000","last_fullsized":"2024-06-14T23:12:18.416011+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":37,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.414250+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:12:18.414250+0000","last_peered":"2024-06-14T23:12:18.414250+0000","last_clean":"2024-06-14T23:12:18.414250+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:12:18.414250+0000","last_undegraded":"2024-06-14T23:12:18.414250+0000","last_fullsized":"2024-06-14T23:12:18.414250+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":768,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-06-14T23:12:18.421035+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:12:18.421035+0000","last_peered":"2024-06-14T23:12:18.421035+0000","last_clean":"2024-06-14T23:12:18.421035+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:12:18.421035+0000","last_undegraded":"2024-06-14T23:12:18.421035+0000","last_fullsized":"2024-06-14T23:12:18.421035+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":4515692568,"num_objects":68905,"num_object_clones":0,"num_object_copies":206715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32999,"num_objects_misplaced":3595,"num_objects_unfound":0,"num_objects_dirty":68905,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164101,"num_write_kb":7456129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14218,"num_bytes_recovered":931790848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13925363712,"data_stored":13925351496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78382,"ondisk_log_size":78382,"up":48,"acting":37,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":821296,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":103,"num_write":26,"num_write_kb":273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":365100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234463,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3581956,"kb_used_data":2382332,"kb_used_omap":1,"kb_used_meta":1199614,"kb_avail":90163196,"statfs":{"total":95995035648,"available":92327112704,"internally_reserved":0,"allocated":2439507968,"data_stored":2437768746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1228405080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":0.95599999999999996,"15min":0.99099999999999999},"min":{"1min":0.24199999999999999,"5min":0.218,"15min":0.19800000000000001},"max":{"1min":5.7229999999999999,"5min":5.7229999999999999,"15min":6.9539999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.169,"15min":1.1950000000000001},"min":{"1min":0.29599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":5.5350000000000001,"5min":5.5350000000000001,"15min":6.9550000000000001},"last":0.47099999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:12:11 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4410000000000001,"15min":1.3180000000000001},"min":{"1min":0.38700000000000001,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":3.9460000000000002,"5min":12.010999999999999,"15min":12.010999999999999},"last":0.499},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4219999999999999,"15min":1.345},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.22900000000000001},"max":{"1min":3.738,"5min":13.531000000000001,"15min":13.531000000000001},"last":0.68600000000000005}]},{"osd":2,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.1970000000000001,"5min":5.1970000000000001,"15min":5.1970000000000001},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":1.7010000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5609999999999999,"15min":1.4179999999999999},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":5.2270000000000003,"5min":20.645,"15min":20.645},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.625,"15min":1.494},"min":{"1min":0.36499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.3639999999999999,"5min":22.042999999999999,"15min":22.042999999999999},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.127,"15min":1.077},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.3319999999999999,"5min":5.3319999999999999,"15min":11.342000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.208,"15min":1.1459999999999999},"min":{"1min":0.32600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3769999999999998,"5min":8.5899999999999999,"15min":16.879000000000001},"last":1.7290000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.026,"15min":1.109},"min":{"1min":0.39100000000000001,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":5.6849999999999996,"5min":5.6849999999999996,"15min":6.9000000000000004},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.407,"5min":1.222,"15min":1.1499999999999999},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.3949999999999996,"5min":5.3949999999999996,"15min":6.8529999999999998},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:12:00 2024","interfaces":[{"interface":"back","average":{"1min":2.7090000000000001,"5min":1.554,"15min":1.393},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":9.8949999999999996,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":2.8809999999999998},{"interface":"front","average":{"1min":1.431,"5min":1.3660000000000001,"15min":1.3759999999999999},"min":{"1min":0.57099999999999995,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":3.8879999999999999,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.57999999999999996}]}]},{"osd":6,"up_from":272,"seq":1168231104569,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2910256,"kb_used_data":2513128,"kb_used_omap":7,"kb_used_meta":397048,"kb_avail":90834896,"statfs":{"total":95995035648,"available":93014933504,"internally_reserved":0,"allocated":2573443072,"data_stored":2571723863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":406577594},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.2450000000000001,"15min":1.6659999999999999},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.1190000000000002,"5min":6.6440000000000001,"15min":6.6440000000000001},"last":0.498},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.3089999999999999,"15min":1.7190000000000001},"min":{"1min":0.48599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.484,"5min":6.6790000000000003,"15min":6.6790000000000003},"last":0.58299999999999996}]},{"osd":1,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.5900000000000001,"15min":2.1800000000000002},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.4660000000000002,"5min":8.173,"15min":8.173},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.506,"15min":2.0859999999999999},"min":{"1min":0.36799999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.5209999999999999,"5min":10.380000000000001,"15min":10.380000000000001},"last":0.55100000000000005}]},{"osd":2,"last update":"Fri Jun 14 23:12:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":3.4489999999999998,"5min":3.4489999999999998,"15min":3.4489999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":0.64600000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.4279999999999999,"15min":1.827},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.5779999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.363,"5min":1.7190000000000001,"15min":2.0219999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.6239999999999997,"5min":8.4740000000000002,"15min":8.4740000000000002},"last":0.59699999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.3520000000000001,"15min":1.4259999999999999},"min":{"1min":0.28999999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":6.8920000000000003,"5min":11.010999999999999,"15min":11.010999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.1599999999999999,"15min":1.266},"min":{"1min":0.39400000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.605,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.139,"15min":1.371},"min":{"1min":0.34000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":3.5510000000000002,"5min":7.7469999999999999,"15min":7.7469999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.472,"5min":1.48,"15min":1.6020000000000001},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.5810000000000004,"5min":7.8680000000000003,"15min":7.8680000000000003},"last":0.61899999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.6930000000000001,"15min":1.7410000000000001},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.5419999999999998,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.429,"15min":1.5389999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.9219999999999997,"5min":7.4619999999999997,"15min":7.4619999999999997},"last":0.70099999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757738,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1734892,"kb_used_data":559460,"kb_used_omap":1,"kb_used_meta":1175422,"kb_avail":92010260,"statfs":{"total":95995035648,"available":94218506240,"internally_reserved":0,"allocated":572887040,"data_stored":571167737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1203632472},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:11:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.93200000000000005,"15min":1.347},"min":{"1min":0.251,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.0270000000000001,"5min":6.1950000000000003,"15min":14.94},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.628,"5min":1.105,"15min":1.3700000000000001},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.219},"max":{"1min":2.3380000000000001,"5min":8.3520000000000003,"15min":9.3819999999999997},"last":0.36799999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.91900000000000004,"15min":0.99099999999999999},"min":{"1min":0.28999999999999998,"5min":0.245,"15min":0.23999999999999999},"max":{"1min":1.7509999999999999,"5min":6.9480000000000004,"15min":7.4370000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.96599999999999997,"15min":1.23},"min":{"1min":0.27400000000000002,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":3.0819999999999999,"5min":5.1920000000000002,"15min":9.5039999999999996},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.0579999999999998,"5min":3.0579999999999998,"15min":3.0579999999999998},"last":0.312},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.125,"5min":3.125,"15min":3.125},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:11:41 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.3009999999999999,"15min":1.6419999999999999},"min":{"1min":0.26900000000000002,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":3.0670000000000002,"5min":8.4350000000000005,"15min":45.744},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.234,"15min":1.669},"min":{"1min":0.32100000000000001,"5min":0.318,"15min":0.28899999999999998},"max":{"1min":3.1030000000000002,"5min":7.4290000000000003,"15min":45.838000000000001},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.71599999999999997,"15min":0.89600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.1719999999999999,"5min":3.012,"15min":14.352},"last":1.2689999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.80500000000000005,"15min":1.0489999999999999},"min":{"1min":0.34100000000000003,"5min":0.28799999999999998,"15min":0.253},"max":{"1min":1.9950000000000001,"5min":8.4039999999999999,"15min":9.5980000000000008},"last":1.139}]},{"osd":6,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.079,"15min":1.389},"min":{"1min":0.29599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.905,"5min":9.4710000000000001,"15min":9.4710000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.3600000000000001,"15min":1.746},"min":{"1min":0.379,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.1630000000000003,"5min":10.632,"15min":10.632},"last":0.68100000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.256,"15min":1.246},"min":{"1min":0.39100000000000001,"5min":0.29099999999999998,"15min":0.27900000000000003},"max":{"1min":26.981999999999999,"5min":26.981999999999999,"15min":26.981999999999999},"last":1.246},{"interface":"front","average":{"1min":2.0670000000000002,"5min":1.333,"15min":1.222},"min":{"1min":0.42499999999999999,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":23.326000000000001,"5min":23.326000000000001,"15min":23.326000000000001},"last":1.3779999999999999}]}]},{"osd":4,"up_from":30,"seq":128849019374,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2391220,"kb_used_data":1500524,"kb_used_omap":1,"kb_used_meta":890686,"kb_avail":91353932,"statfs":{"total":95995035648,"available":93546426368,"internally_reserved":0,"allocated":1536536576,"data_stored":1534812835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912062808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.89000000000000001,"15min":1.135},"min":{"1min":0.251,"5min":0.217,"15min":0.17100000000000001},"max":{"1min":2.7490000000000001,"5min":6.875,"15min":10.567},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.80100000000000005,"15min":1.1100000000000001},"min":{"1min":0.25800000000000001,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":1.7270000000000001,"5min":6.1379999999999999,"15min":8.3390000000000004},"last":0.82299999999999995}]},{"osd":1,"last update":"Fri Jun 14 23:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.383,"15min":1.5},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.23100000000000001},"max":{"1min":4.476,"5min":42.656999999999996,"15min":56.917000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.343,"15min":1.5609999999999999},"min":{"1min":0.30399999999999999,"5min":0.249,"15min":0.249},"max":{"1min":2.7280000000000002,"5min":42.192999999999998,"15min":56.789999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Fri Jun 14 23:12:23 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.72499999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.6559999999999999,"15min":1.3859999999999999},"min":{"1min":0.30599999999999999,"5min":0.25,"15min":0.25},"max":{"1min":1.768,"5min":44.389000000000003,"15min":44.389000000000003},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.845,"15min":1.5309999999999999},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.27500000000000002},"max":{"1min":2.7040000000000002,"5min":46.101999999999997,"15min":46.101999999999997},"last":0.629}]},{"osd":5,"last update":"Fri Jun 14 23:12:36 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.85799999999999998,"15min":1.5529999999999999},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.4159999999999999,"5min":4.04,"15min":139.13200000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":1.613},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.2360000000000002,"5min":6.9290000000000003,"15min":138.86000000000001},"last":0.68899999999999995}]},{"osd":6,"last update":"Fri Jun 14 23:12:06 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.117,"15min":1.105},"min":{"1min":0.36599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.46,"5min":6.9989999999999997,"15min":6.9989999999999997},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.383,"5min":1.2230000000000001,"15min":1.5069999999999999},"min":{"1min":0.32200000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.1070000000000002,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.75600000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:12:17 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.1859999999999999,"15min":1.087},"min":{"1min":0.33800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.0439999999999996,"5min":6.5910000000000002,"15min":8.9489999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.583,"5min":1.2210000000000001,"15min":1.1659999999999999},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.4219999999999997,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":0.89600000000000002}]}]},{"osd":3,"up_from":138,"seq":592705487072,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3938984,"kb_used_data":3145568,"kb_used_omap":4,"kb_used_meta":793275,"kb_avail":89806168,"statfs":{"total":95995035648,"available":91961516032,"internally_reserved":0,"allocated":3221061632,"data_stored":3219345656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":812314078},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":1.54,"15min":1.3979999999999999},"min":{"1min":0.16900000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":25.190000000000001,"15min":25.190000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":1.6359999999999999,"15min":1.365},"min":{"1min":0.29799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":4.6369999999999996,"5min":25.652000000000001,"15min":25.652000000000001},"last":0.309}]},{"osd":1,"last update":"Fri Jun 14 23:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.718,"15min":1.51},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":4.6840000000000002,"5min":25.754000000000001,"15min":25.754000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":2.0609999999999999,"15min":1.665},"min":{"1min":0.33900000000000002,"5min":0.251,"15min":0.25},"max":{"1min":1.429,"5min":27.992999999999999,"15min":27.992999999999999},"last":0.36399999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.7549999999999999,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.972,"5min":2.972,"15min":2.972},"last":0.374}]},{"osd":4,"last update":"Fri Jun 14 23:11:36 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":2.6819999999999999,"15min":1.7729999999999999},"min":{"1min":0.28299999999999997,"5min":0.28100000000000003,"15min":0.27900000000000003},"max":{"1min":5.4710000000000001,"5min":73.263000000000005,"15min":73.263000000000005},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.46,"5min":2.4590000000000001,"15min":1.6779999999999999},"min":{"1min":0.29499999999999998,"5min":0.27600000000000002,"15min":0.27300000000000002},"max":{"1min":3.8319999999999999,"5min":73.5,"15min":73.5},"last":0.44400000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.8129999999999999,"15min":1.8089999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":2.7999999999999998,"5min":25.521000000000001,"15min":81.438999999999993},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.7030000000000001,"15min":1.8120000000000001},"min":{"1min":0.376,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.5990000000000002,"5min":25.84,"15min":81.626999999999995},"last":1.893}]},{"osd":6,"last update":"Fri Jun 14 23:12:00 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.73,"15min":1.7170000000000001},"min":{"1min":0.39800000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":7.1379999999999999,"5min":25.869,"15min":25.869},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.5289999999999999,"15min":1.516},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.4340000000000002,"5min":25.143000000000001,"15min":25.143000000000001},"last":0.57199999999999995}]},{"osd":7,"last update":"Fri Jun 14 23:11:36 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.6120000000000001,"15min":1.498},"min":{"1min":0.35799999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.4960000000000004,"5min":25.120000000000001,"15min":25.120000000000001},"last":0.497},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.704,"15min":1.421},"min":{"1min":0.44400000000000001,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":4.7709999999999999,"5min":25.472000000000001,"15min":25.472000000000001},"last":0.46500000000000002}]}]},{"osd":2,"up_from":322,"seq":1382979469328,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1457004,"kb_used_data":1343908,"kb_used_omap":10,"kb_used_meta":113077,"kb_avail":92288148,"statfs":{"total":95995035648,"available":94503063552,"internally_reserved":0,"allocated":1376161792,"data_stored":1374442478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":115791398},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.313},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.30099999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.42899999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.251}]},{"osd":4,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.39000000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.544,"5min":2.544,"15min":2.544},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.377}]},{"osd":6,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.6950000000000003,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":0.82999999999999996}]},{"osd":7,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":3.008,"5min":3.008,"15min":3.008},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.548,"5min":4.548,"15min":4.548},"last":0.40100000000000002}]}]},{"osd":1,"up_from":0,"seq":498216206616,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024271,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2335309824,"data_stored":2335309824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1352404992,"data_stored":1352400920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3193049088,"data_stored":3193045016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1522925568,"data_stored":1522925568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":564002816,"data_stored":564002816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2544238592,"data_stored":2544238592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2413432832,"data_stored":2413428760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:12:43.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:12:43.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:43.418 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:43.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:43.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:43.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:43.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:43.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:43.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:43.704 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:12:43.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:43.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:43.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:43.858 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:43.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:43.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:43.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:43 smithi137 ceph-mon[90634]: pgmap v1252: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 724 op/s; 32760/200355 objects degraded (16.351%); 3595/200355 objects misplaced (1.794%); 8.8 MiB/s, 140 objects/s recovering 2024-06-14T23:12:43.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2475841998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:12:43.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 32760/200355 objects degraded (16.351%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:43.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:43.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:44.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:43 smithi086 ceph-mon[102754]: pgmap v1252: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 724 op/s; 32760/200355 objects degraded (16.351%); 3595/200355 objects misplaced (1.794%); 8.8 MiB/s, 140 objects/s recovering 2024-06-14T23:12:44.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2475841998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:12:44.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 32760/200355 objects degraded (16.351%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:44.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:43 smithi086 ceph-mon[97913]: pgmap v1252: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 724 op/s; 32760/200355 objects degraded (16.351%); 3595/200355 objects misplaced (1.794%); 8.8 MiB/s, 140 objects/s recovering 2024-06-14T23:12:44.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2475841998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:12:44.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 32760/200355 objects degraded (16.351%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:44.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:44.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:44.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:44.255 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:44.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:44.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:44.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:44.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:44.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:44.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:44.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:44.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:44.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:44.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:44.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:44.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:44.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:44.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:44.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:44.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:44.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:44 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:12:44.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2475841998' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:12:44.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:44 smithi137 ceph-mon[90634]: osdmap e337: 8 total, 8 up, 6 in 2024-06-14T23:12:45.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:45.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:45.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:45.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:44 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:12:45.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2475841998' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:12:45.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:44 smithi086 ceph-mon[97913]: osdmap e337: 8 total, 8 up, 6 in 2024-06-14T23:12:45.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:12:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:12:44] "GET /metrics HTTP/1.1" 200 36720 "" "Prometheus/2.43.0" 2024-06-14T23:12:45.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:44 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:12:45.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2475841998' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:12:45.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:44 smithi086 ceph-mon[102754]: osdmap e337: 8 total, 8 up, 6 in 2024-06-14T23:12:45.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:45.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:45.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:45.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:45.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:45.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:45.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:45.460 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:45.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:45.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:45.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:45.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:45.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:45.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:45.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:45.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:45.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:45.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:45.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:45.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:45.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:45 smithi137 ceph-mon[90634]: pgmap v1254: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 728 op/s; 32549/195492 objects degraded (16.650%); 3386/195492 objects misplaced (1.732%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:12:45.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:45 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:46.014 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:46.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:46.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:45 smithi086 ceph-mon[97913]: pgmap v1254: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 728 op/s; 32549/195492 objects degraded (16.650%); 3386/195492 objects misplaced (1.732%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:12:46.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:45 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:46.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:45 smithi086 ceph-mon[102754]: pgmap v1254: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 4.0 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 728 op/s; 32549/195492 objects degraded (16.650%); 3386/195492 objects misplaced (1.732%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:12:46.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:45 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:46.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:46.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:46.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:46.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:46.280 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:46.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:46.287 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:46.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:46.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:46.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:46.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:46.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:46.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:46.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:46.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:46.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:46.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:46.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:46.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:46.961 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:46.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:47.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:47.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:47.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:47.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:47.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:47.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:47.327 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:47.370 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:47.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:47.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:47.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:47.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:47.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:47.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:47.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:47.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:47.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:12:47.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:47.773 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:47.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:47.854 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:12:47.854 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:12:47.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:47.878 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:12:47.878 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:12:47.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:47.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:47.913 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:12:47.914 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:12:47.937 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:12:47.938 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:12:47.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:47 smithi086 ceph-mon[97913]: pgmap v1255: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 738 op/s; 32549/192591 objects degraded (16.901%); 3386/192591 objects misplaced (1.758%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:12:47.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:47 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:47.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:47 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:47.938 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:47 smithi086 ceph-mon[102754]: pgmap v1255: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 738 op/s; 32549/192591 objects degraded (16.901%); 3386/192591 objects misplaced (1.758%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:12:47.938 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:47 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:47.939 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:47 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:47.963 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:12:47.964 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:12:47.992 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:12:47.993 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:12:47.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:47 smithi137 ceph-mon[90634]: pgmap v1255: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 738 op/s; 32549/192591 objects degraded (16.901%); 3386/192591 objects misplaced (1.758%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:12:47.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:47 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:47.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:47 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:48.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:48.030 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:12:48.031 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:12:48.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:48.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:48.071 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:12:48.071 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:12:48.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:48.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:48.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:48.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:48.334 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:48.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:48.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T23:12:48.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:48.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:48.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:48.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:48.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:48.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:48.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:48.739 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:48.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:48.790 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:12:48.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:48.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:48.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:48.985 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:48.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:48 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:12:48.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:48 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:12:48.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:48 smithi137 ceph-mon[90634]: pgmap v1256: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 780 op/s; 32317/189246 objects degraded (17.077%); 3386/189246 objects misplaced (1.789%); 8.9 MiB/s, 141 objects/s recovering 2024-06-14T23:12:48.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:48 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:48.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3056230380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:48.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 32549/192591 objects degraded (16.901%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:48.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:48 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:49.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:49.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:49.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:12:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:12:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[97913]: pgmap v1256: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 780 op/s; 32317/189246 objects degraded (17.077%); 3386/189246 objects misplaced (1.789%); 8.9 MiB/s, 141 objects/s recovering 2024-06-14T23:12:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3056230380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 32549/192591 objects degraded (16.901%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:49.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:12:49.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:12:49.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[102754]: pgmap v1256: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 780 op/s; 32317/189246 objects degraded (17.077%); 3386/189246 objects misplaced (1.789%); 8.9 MiB/s, 141 objects/s recovering 2024-06-14T23:12:49.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:49.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3056230380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:49.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 32549/192591 objects degraded (16.901%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:49.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:48 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:49.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:49.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:49.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:49.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:49.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:49.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:49.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:49.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:49.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:49.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:49.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:49.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:49.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:49.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:49.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:49.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:49.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:49.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:49.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:49.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:49.961 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:49 smithi137 ceph-mon[90634]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-06-14T23:12:49.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:49.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:49.974 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:50.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:50.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:49 smithi086 ceph-mon[97913]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-06-14T23:12:50.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:49 smithi086 ceph-mon[102754]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-06-14T23:12:50.089 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:50.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:50.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:50.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:50.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:50.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:50.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:50.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:50.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:50.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:50.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:50.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:50.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:50.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:50.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:50.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:50.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:50.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:50.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:50.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:50.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:50.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:50 smithi137 ceph-mon[90634]: pgmap v1257: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 787 op/s; 32086/184455 objects degraded (17.395%); 3147/184455 objects misplaced (1.706%); 9.0 MiB/s, 143 objects/s recovering 2024-06-14T23:12:51.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:51.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:51.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:51.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:50 smithi086 ceph-mon[97913]: pgmap v1257: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 787 op/s; 32086/184455 objects degraded (17.395%); 3147/184455 objects misplaced (1.706%); 9.0 MiB/s, 143 objects/s recovering 2024-06-14T23:12:51.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:50 smithi086 ceph-mon[102754]: pgmap v1257: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.8 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 787 op/s; 32086/184455 objects degraded (17.395%); 3147/184455 objects misplaced (1.706%); 9.0 MiB/s, 143 objects/s recovering 2024-06-14T23:12:51.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:51.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:51.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:51.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:51.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:51.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:51.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:12:51.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:51.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:51.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:51.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:51.587 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:51.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:51.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:51.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:51.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:51.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:51.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:51.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:51.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:51.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:51.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:51.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:51.998 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:12:51.998 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:12:52.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:52.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:52.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:52.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:52.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:52.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:52.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:52.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:52.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:52.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:52.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:52.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:52.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:52.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:52.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:52.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:52.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-14T23:12:52.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:52.802 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:52.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:52.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:52.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:52.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:52.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:52 smithi137 ceph-mon[90634]: pgmap v1258: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 788 op/s; 31862/177990 objects degraded (17.901%); 3147/177990 objects misplaced (1.768%); 8.9 MiB/s, 142 objects/s recovering 2024-06-14T23:12:52.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:52 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:53.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:53.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:52 smithi086 ceph-mon[97913]: pgmap v1258: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 788 op/s; 31862/177990 objects degraded (17.901%); 3147/177990 objects misplaced (1.768%); 8.9 MiB/s, 142 objects/s recovering 2024-06-14T23:12:53.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:52 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:53.074 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:52 smithi086 ceph-mon[102754]: pgmap v1258: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 788 op/s; 31862/177990 objects degraded (17.901%); 3147/177990 objects misplaced (1.768%); 8.9 MiB/s, 142 objects/s recovering 2024-06-14T23:12:53.074 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:52 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:53.100 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:53.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:53.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:53.196 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:12:53.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:53.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:53.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:53.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:53.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:53.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:53.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:53.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:53.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:53.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:53.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:53.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:53.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:53.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:53.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:53.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:53.944 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:12:53.946 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:12:53.963 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 31862/177990 objects degraded (17.901%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:53.963 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:53 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:12:53.963 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 31862/177990 objects degraded (17.901%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:53.964 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:53 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:12:53.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 31862/177990 objects degraded (17.901%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:53.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:53 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:12:54.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:54.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:54.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:54.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:54.203 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:54.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:54.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:54.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:54.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:54.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:54.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:54.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:54.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:54.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:54.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:54.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:54.742 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:54.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:54.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:54.767 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:12:54.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:54.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:54.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:54.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:54 smithi086 ceph-mon[97913]: pgmap v1259: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 31862/175128 objects degraded (18.194%); 3147/175128 objects misplaced (1.797%); 8.4 MiB/s, 134 objects/s recovering 2024-06-14T23:12:54.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:54 smithi086 ceph-mon[97913]: from='client.46679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:12:54.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3756820248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:12:54.985 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:12:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:12:54] "GET /metrics HTTP/1.1" 200 36720 "" "Prometheus/2.43.0" 2024-06-14T23:12:54.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:54 smithi086 ceph-mon[102754]: pgmap v1259: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 31862/175128 objects degraded (18.194%); 3147/175128 objects misplaced (1.797%); 8.4 MiB/s, 134 objects/s recovering 2024-06-14T23:12:54.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:54 smithi086 ceph-mon[102754]: from='client.46679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:12:54.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3756820248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:12:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:54 smithi137 ceph-mon[90634]: pgmap v1259: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.6 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 745 op/s; 31862/175128 objects degraded (18.194%); 3147/175128 objects misplaced (1.797%); 8.4 MiB/s, 134 objects/s recovering 2024-06-14T23:12:54.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:54 smithi137 ceph-mon[90634]: from='client.46679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:12:54.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:54 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3756820248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:12:54.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:55.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:55.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:55.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:55.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:55.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:55.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:55.292 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:55.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:55.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:55.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:55.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:55.450 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1259,"stamp":"2024-06-14T23:12:53.719290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3824852462,"num_objects":58376,"num_object_clones":0,"num_object_copies":175128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31862,"num_objects_misplaced":3147,"num_objects_unfound":0,"num_objects_dirty":58376,"num_whiteouts":0,"num_read":56,"num_read_kb":104,"num_write":174742,"num_write_kb":7456431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15823,"num_bytes_recovered":1036031190,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83184,"ondisk_log_size":83184,"up":357,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":14297480,"kb_used_data":9659800,"kb_used_omap":27,"kb_used_meta":4637220,"kb_avail":548173432,"statfs":{"total":575970213888,"available":561329594368,"internally_reserved":0,"allocated":9891635200,"data_stored":9881251527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":4748513734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,0,0,0,0,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-489684992,"num_objects":-7472,"num_object_clones":0,"num_object_copies":-22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-898,"num_objects_misplaced":-448,"num_objects_unfound":0,"num_objects_dirty":-7472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":88211456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.019308"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":716,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.648358+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:12:30.648358+0000","last_peered":"2024-06-14T23:12:30.648358+0000","last_clean":"2024-06-14T23:12:30.648358+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:12:30.648358+0000","last_undegraded":"2024-06-14T23:12:30.648358+0000","last_fullsized":"2024-06-14T23:12:30.648358+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":154,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.669569+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:12:30.669569+0000","last_peered":"2024-06-14T23:12:30.669569+0000","last_clean":"2024-06-14T23:12:30.669569+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:12:30.669569+0000","last_undegraded":"2024-06-14T23:12:30.669569+0000","last_fullsized":"2024-06-14T23:12:30.669569+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":324,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.642643+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:12:30.642643+0000","last_peered":"2024-06-14T23:12:30.642643+0000","last_clean":"2024-06-14T23:12:30.642643+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:12:30.642643+0000","last_undegraded":"2024-06-14T23:12:30.642643+0000","last_fullsized":"2024-06-14T23:12:30.642643+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":229,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.644669+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:12:30.644669+0000","last_peered":"2024-06-14T23:12:30.644669+0000","last_clean":"2024-06-14T23:12:30.644669+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:12:30.644669+0000","last_undegraded":"2024-06-14T23:12:30.644669+0000","last_fullsized":"2024-06-14T23:12:30.644669+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":119,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645969+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:12:30.645969+0000","last_peered":"2024-06-14T23:12:30.645969+0000","last_clean":"2024-06-14T23:12:30.645969+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:12:30.645969+0000","last_undegraded":"2024-06-14T23:12:30.645969+0000","last_fullsized":"2024-06-14T23:12:30.645969+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":738,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:52.383327+0000","last_change":"2024-06-14T23:12:52.383327+0000","last_active":"2024-06-14T23:12:52.383327+0000","last_peered":"2024-06-14T23:12:52.383327+0000","last_clean":"2024-06-14T23:12:52.383327+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:12:52.383327+0000","last_undegraded":"2024-06-14T23:12:52.383327+0000","last_fullsized":"2024-06-14T23:12:52.383327+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:09:56.196559+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:09:56.196559+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":300,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.669093+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:12:30.669093+0000","last_peered":"2024-06-14T23:12:30.669093+0000","last_clean":"2024-06-14T23:12:30.669093+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:12:30.669093+0000","last_undegraded":"2024-06-14T23:12:30.669093+0000","last_fullsized":"2024-06-14T23:12:30.669093+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":341,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645763+0000","last_change":"2024-06-14T23:11:23.413009+0000","last_active":"2024-06-14T23:12:30.645763+0000","last_peered":"2024-06-14T23:12:30.645763+0000","last_clean":"2024-06-14T23:12:30.645763+0000","last_became_active":"2024-06-14T23:11:23.412491+0000","last_became_peered":"2024-06-14T23:11:23.412491+0000","last_unstale":"2024-06-14T23:12:30.645763+0000","last_undegraded":"2024-06-14T23:12:30.645763+0000","last_fullsized":"2024-06-14T23:12:30.645763+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:57.263547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":487,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.649294+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:12:30.649294+0000","last_peered":"2024-06-14T23:12:30.649294+0000","last_clean":"2024-06-14T23:12:30.649294+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:12:30.649294+0000","last_undegraded":"2024-06-14T23:12:30.649294+0000","last_fullsized":"2024-06-14T23:12:30.649294+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":314,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.668931+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:12:30.668931+0000","last_peered":"2024-06-14T23:12:30.668931+0000","last_clean":"2024-06-14T23:12:30.668931+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:12:30.668931+0000","last_undegraded":"2024-06-14T23:12:30.668931+0000","last_fullsized":"2024-06-14T23:12:30.668931+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":341,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650324+0000","last_change":"2024-06-14T23:11:23.427771+0000","last_active":"2024-06-14T23:12:30.650324+0000","last_peered":"2024-06-14T23:12:30.650324+0000","last_clean":"2024-06-14T23:12:30.650324+0000","last_became_active":"2024-06-14T23:11:23.427481+0000","last_became_peered":"2024-06-14T23:11:23.427481+0000","last_unstale":"2024-06-14T23:12:30.650324+0000","last_undegraded":"2024-06-14T23:12:30.650324+0000","last_fullsized":"2024-06-14T23:12:30.650324+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:01:04.162256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":130,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650198+0000","last_change":"2024-06-14T23:10:26.053647+0000","last_active":"2024-06-14T23:12:30.650198+0000","last_peered":"2024-06-14T23:12:30.650198+0000","last_clean":"2024-06-14T23:12:30.650198+0000","last_became_active":"2024-06-14T23:10:26.053318+0000","last_became_peered":"2024-06-14T23:10:26.053318+0000","last_unstale":"2024-06-14T23:12:30.650198+0000","last_undegraded":"2024-06-14T23:12:30.650198+0000","last_fullsized":"2024-06-14T23:12:30.650198+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:32.647107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":129,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645103+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:12:30.645103+0000","last_peered":"2024-06-14T23:12:30.645103+0000","last_clean":"2024-06-14T23:12:30.645103+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:12:30.645103+0000","last_undegraded":"2024-06-14T23:12:30.645103+0000","last_fullsized":"2024-06-14T23:12:30.645103+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":640,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.642383+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:12:30.642383+0000","last_peered":"2024-06-14T23:12:30.642383+0000","last_clean":"2024-06-14T23:12:30.642383+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:12:30.642383+0000","last_undegraded":"2024-06-14T23:12:30.642383+0000","last_fullsized":"2024-06-14T23:12:30.642383+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":363,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645858+0000","last_change":"2024-06-14T23:11:23.413241+0000","last_active":"2024-06-14T23:12:30.645858+0000","last_peered":"2024-06-14T23:12:30.645858+0000","last_clean":"2024-06-14T23:12:30.645858+0000","last_became_active":"2024-06-14T23:11:23.412936+0000","last_became_peered":"2024-06-14T23:11:23.412936+0000","last_unstale":"2024-06-14T23:12:30.645858+0000","last_undegraded":"2024-06-14T23:12:30.645858+0000","last_fullsized":"2024-06-14T23:12:30.645858+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:36.581716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":149,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650425+0000","last_change":"2024-06-14T23:10:26.038549+0000","last_active":"2024-06-14T23:12:30.650425+0000","last_peered":"2024-06-14T23:12:30.650425+0000","last_clean":"2024-06-14T23:12:30.650425+0000","last_became_active":"2024-06-14T23:10:26.038260+0000","last_became_peered":"2024-06-14T23:10:26.038260+0000","last_unstale":"2024-06-14T23:12:30.650425+0000","last_undegraded":"2024-06-14T23:12:30.650425+0000","last_fullsized":"2024-06-14T23:12:30.650425+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:44.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":743,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.666817+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:12:30.666817+0000","last_peered":"2024-06-14T23:12:30.666817+0000","last_clean":"2024-06-14T23:12:30.666817+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:12:30.666817+0000","last_undegraded":"2024-06-14T23:12:30.666817+0000","last_fullsized":"2024-06-14T23:12:30.666817+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":134,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.718446+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:12:43.718446+0000","last_peered":"2024-06-14T23:12:43.718446+0000","last_clean":"2024-06-14T23:12:43.718446+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:12:43.718446+0000","last_undegraded":"2024-06-14T23:12:43.718446+0000","last_fullsized":"2024-06-14T23:12:43.718446+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":196,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.649220+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:12:30.649220+0000","last_peered":"2024-06-14T23:12:30.649220+0000","last_clean":"2024-06-14T23:12:30.649220+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:12:30.649220+0000","last_undegraded":"2024-06-14T23:12:30.649220+0000","last_fullsized":"2024-06-14T23:12:30.649220+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":691,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.668020+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:12:30.668020+0000","last_peered":"2024-06-14T23:12:30.668020+0000","last_clean":"2024-06-14T23:12:30.668020+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:12:30.668020+0000","last_undegraded":"2024-06-14T23:12:30.668020+0000","last_fullsized":"2024-06-14T23:12:30.668020+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":57,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.644707+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:12:30.644707+0000","last_peered":"2024-06-14T23:12:30.644707+0000","last_clean":"2024-06-14T23:12:30.644707+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:12:30.644707+0000","last_undegraded":"2024-06-14T23:12:30.644707+0000","last_fullsized":"2024-06-14T23:12:30.644707+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":288,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.647938+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:12:30.647938+0000","last_peered":"2024-06-14T23:12:30.647938+0000","last_clean":"2024-06-14T23:12:30.647938+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:12:30.647938+0000","last_undegraded":"2024-06-14T23:12:30.647938+0000","last_fullsized":"2024-06-14T23:12:30.647938+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":643,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.643359+0000","last_change":"2024-06-14T23:10:29.062659+0000","last_active":"2024-06-14T23:12:30.643359+0000","last_peered":"2024-06-14T23:12:30.643359+0000","last_clean":"2024-06-14T23:12:30.643359+0000","last_became_active":"2024-06-14T23:10:29.062256+0000","last_became_peered":"2024-06-14T23:10:29.062256+0000","last_unstale":"2024-06-14T23:12:30.643359+0000","last_undegraded":"2024-06-14T23:12:30.643359+0000","last_fullsized":"2024-06-14T23:12:30.643359+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:38.814157+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":320,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716782+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:12:43.716782+0000","last_peered":"2024-06-14T23:12:43.716782+0000","last_clean":"2024-06-14T23:12:43.716782+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:12:43.716782+0000","last_undegraded":"2024-06-14T23:12:43.716782+0000","last_fullsized":"2024-06-14T23:12:43.716782+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":184,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.669649+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:12:30.669649+0000","last_peered":"2024-06-14T23:12:30.669649+0000","last_clean":"2024-06-14T23:12:30.669649+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:12:30.669649+0000","last_undegraded":"2024-06-14T23:12:30.669649+0000","last_fullsized":"2024-06-14T23:12:30.669649+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":643,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.657998+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:12:30.657998+0000","last_peered":"2024-06-14T23:12:30.657998+0000","last_clean":"2024-06-14T23:12:30.657998+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:12:30.657998+0000","last_undegraded":"2024-06-14T23:12:30.657998+0000","last_fullsized":"2024-06-14T23:12:30.657998+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":202,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.647407+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:12:30.647407+0000","last_peered":"2024-06-14T23:12:30.647407+0000","last_clean":"2024-06-14T23:12:30.647407+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:12:30.647407+0000","last_undegraded":"2024-06-14T23:12:30.647407+0000","last_fullsized":"2024-06-14T23:12:30.647407+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":308,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.668939+0000","last_change":"2024-06-14T23:10:26.053008+0000","last_active":"2024-06-14T23:12:30.668939+0000","last_peered":"2024-06-14T23:12:30.668939+0000","last_clean":"2024-06-14T23:12:30.668939+0000","last_became_active":"2024-06-14T23:10:26.052701+0000","last_became_peered":"2024-06-14T23:10:26.052701+0000","last_unstale":"2024-06-14T23:12:30.668939+0000","last_undegraded":"2024-06-14T23:12:30.668939+0000","last_fullsized":"2024-06-14T23:12:30.668939+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:15:03.929062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":667,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645018+0000","last_change":"2024-06-14T23:11:23.411926+0000","last_active":"2024-06-14T23:12:30.645018+0000","last_peered":"2024-06-14T23:12:30.645018+0000","last_clean":"2024-06-14T23:12:30.645018+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:12:30.645018+0000","last_undegraded":"2024-06-14T23:12:30.645018+0000","last_fullsized":"2024-06-14T23:12:30.645018+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":221,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.721468+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:12:43.721468+0000","last_peered":"2024-06-14T23:12:43.721468+0000","last_clean":"2024-06-14T23:12:43.721468+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:12:43.721468+0000","last_undegraded":"2024-06-14T23:12:43.721468+0000","last_fullsized":"2024-06-14T23:12:43.721468+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":350,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.642067+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:12:30.642067+0000","last_peered":"2024-06-14T23:12:30.642067+0000","last_clean":"2024-06-14T23:12:30.642067+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:12:30.642067+0000","last_undegraded":"2024-06-14T23:12:30.642067+0000","last_fullsized":"2024-06-14T23:12:30.642067+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":513,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.669270+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:12:30.669270+0000","last_peered":"2024-06-14T23:12:30.669270+0000","last_clean":"2024-06-14T23:12:30.669270+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:12:30.669270+0000","last_undegraded":"2024-06-14T23:12:30.669270+0000","last_fullsized":"2024-06-14T23:12:30.669270+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":320,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650372+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:12:30.650372+0000","last_peered":"2024-06-14T23:12:30.650372+0000","last_clean":"2024-06-14T23:12:30.650372+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:12:30.650372+0000","last_undegraded":"2024-06-14T23:12:30.650372+0000","last_fullsized":"2024-06-14T23:12:30.650372+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":324,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650388+0000","last_change":"2024-06-14T23:10:26.047451+0000","last_active":"2024-06-14T23:12:30.650388+0000","last_peered":"2024-06-14T23:12:30.650388+0000","last_clean":"2024-06-14T23:12:30.650388+0000","last_became_active":"2024-06-14T23:10:26.046781+0000","last_became_peered":"2024-06-14T23:10:26.046781+0000","last_unstale":"2024-06-14T23:12:30.650388+0000","last_undegraded":"2024-06-14T23:12:30.650388+0000","last_fullsized":"2024-06-14T23:12:30.650388+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:07:15.892864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":552,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.652067+0000","last_change":"2024-06-14T23:10:39.253251+0000","last_active":"2024-06-14T23:12:30.652067+0000","last_peered":"2024-06-14T23:12:30.652067+0000","last_clean":"2024-06-14T23:12:30.652067+0000","last_became_active":"2024-06-14T23:10:29.061386+0000","last_became_peered":"2024-06-14T23:10:29.061386+0000","last_unstale":"2024-06-14T23:12:30.652067+0000","last_undegraded":"2024-06-14T23:12:30.652067+0000","last_fullsized":"2024-06-14T23:12:30.652067+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.735431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.735431+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"337'10909","reported_seq":10971,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:50.386252+0000","last_change":"2024-06-14T23:11:24.652983+0000","last_active":"2024-06-14T23:12:50.386252+0000","last_peered":"2024-06-14T23:12:50.386252+0000","last_clean":"2024-06-14T23:10:37.054302+0000","last_became_active":"2024-06-14T23:11:24.646468+0000","last_became_peered":"2024-06-14T23:11:24.646468+0000","last_unstale":"2024-06-14T23:12:50.386252+0000","last_undegraded":"2024-06-14T23:11:24.406142+0000","last_fullsized":"2024-06-14T23:11:24.380989+0000","mapping_epoch":323,"log_start":"319'6605","ondisk_log_start":"319'6605","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:33:08.769394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240582656,"num_objects":3671,"num_object_clones":0,"num_object_copies":11013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10909,"num_write_kb":466560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1203}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":297,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.641516+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:12:30.641516+0000","last_peered":"2024-06-14T23:12:30.641516+0000","last_clean":"2024-06-14T23:12:30.641516+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:12:30.641516+0000","last_undegraded":"2024-06-14T23:12:30.641516+0000","last_fullsized":"2024-06-14T23:12:30.641516+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":309,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.647727+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:12:30.647727+0000","last_peered":"2024-06-14T23:12:30.647727+0000","last_clean":"2024-06-14T23:12:30.647727+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:12:30.647727+0000","last_undegraded":"2024-06-14T23:12:30.647727+0000","last_fullsized":"2024-06-14T23:12:30.647727+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":95,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.651944+0000","last_change":"2024-06-14T23:10:26.058423+0000","last_active":"2024-06-14T23:12:30.651944+0000","last_peered":"2024-06-14T23:12:30.651944+0000","last_clean":"2024-06-14T23:12:30.651944+0000","last_became_active":"2024-06-14T23:10:26.050244+0000","last_became_peered":"2024-06-14T23:10:26.050244+0000","last_unstale":"2024-06-14T23:12:30.651944+0000","last_undegraded":"2024-06-14T23:12:30.651944+0000","last_fullsized":"2024-06-14T23:12:30.651944+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:52.060325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":662,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.717613+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:12:43.717613+0000","last_peered":"2024-06-14T23:12:43.717613+0000","last_clean":"2024-06-14T23:12:43.717613+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:12:43.717613+0000","last_undegraded":"2024-06-14T23:12:43.717613+0000","last_fullsized":"2024-06-14T23:12:43.717613+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"337'11044","reported_seq":11118,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:50.338686+0000","last_change":"2024-06-14T23:11:24.639438+0000","last_active":"2024-06-14T23:12:50.338686+0000","last_peered":"2024-06-14T23:12:50.338686+0000","last_clean":"2024-06-14T23:10:37.032734+0000","last_became_active":"2024-06-14T23:11:24.634884+0000","last_became_peered":"2024-06-14T23:11:24.634884+0000","last_unstale":"2024-06-14T23:12:50.338686+0000","last_undegraded":"2024-06-14T23:11:24.403675+0000","last_fullsized":"2024-06-14T23:11:24.380974+0000","mapping_epoch":323,"log_start":"319'6700","ondisk_log_start":"319'6700","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4344,"log_dups_size":0,"ondisk_log_size":4344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:48:19.382266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241041408,"num_objects":3678,"num_object_clones":0,"num_object_copies":11034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11044,"num_write_kb":471104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":250,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650773+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:12:30.650773+0000","last_peered":"2024-06-14T23:12:30.650773+0000","last_clean":"2024-06-14T23:12:30.650773+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:12:30.650773+0000","last_undegraded":"2024-06-14T23:12:30.650773+0000","last_fullsized":"2024-06-14T23:12:30.650773+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":300,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.652136+0000","last_change":"2024-06-14T23:10:26.051450+0000","last_active":"2024-06-14T23:12:30.652136+0000","last_peered":"2024-06-14T23:12:30.652136+0000","last_clean":"2024-06-14T23:12:30.652136+0000","last_became_active":"2024-06-14T23:10:26.050830+0000","last_became_peered":"2024-06-14T23:10:26.050830+0000","last_unstale":"2024-06-14T23:12:30.652136+0000","last_undegraded":"2024-06-14T23:12:30.652136+0000","last_fullsized":"2024-06-14T23:12:30.652136+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:16:16.175014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":51,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.669196+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:12:30.669196+0000","last_peered":"2024-06-14T23:12:30.669196+0000","last_clean":"2024-06-14T23:12:30.669196+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:12:30.669196+0000","last_undegraded":"2024-06-14T23:12:30.669196+0000","last_fullsized":"2024-06-14T23:12:30.669196+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":751,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650513+0000","last_change":"2024-06-14T23:11:23.401974+0000","last_active":"2024-06-14T23:12:30.650513+0000","last_peered":"2024-06-14T23:12:30.650513+0000","last_clean":"2024-06-14T23:12:30.650513+0000","last_became_active":"2024-06-14T23:11:23.401625+0000","last_became_peered":"2024-06-14T23:11:23.401625+0000","last_unstale":"2024-06-14T23:12:30.650513+0000","last_undegraded":"2024-06-14T23:12:30.650513+0000","last_fullsized":"2024-06-14T23:12:30.650513+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.214300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.214300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:05:54.552362+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"337'10889","reported_seq":10952,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:50.392597+0000","last_change":"2024-06-14T23:11:24.651371+0000","last_active":"2024-06-14T23:12:50.392597+0000","last_peered":"2024-06-14T23:12:50.392597+0000","last_clean":"2024-06-14T23:10:37.020885+0000","last_became_active":"2024-06-14T23:11:24.646284+0000","last_became_peered":"2024-06-14T23:11:24.646284+0000","last_unstale":"2024-06-14T23:12:50.392597+0000","last_undegraded":"2024-06-14T23:11:24.405009+0000","last_fullsized":"2024-06-14T23:11:24.380990+0000","mapping_epoch":323,"log_start":"319'6542","ondisk_log_start":"319'6542","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4347,"log_dups_size":0,"ondisk_log_size":4347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:26:58.760076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246743040,"num_objects":3765,"num_object_clones":0,"num_object_copies":11295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10889,"num_write_kb":468928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1209}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":148,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.668234+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:12:30.668234+0000","last_peered":"2024-06-14T23:12:30.668234+0000","last_clean":"2024-06-14T23:12:30.668234+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:12:30.668234+0000","last_undegraded":"2024-06-14T23:12:30.668234+0000","last_fullsized":"2024-06-14T23:12:30.668234+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":322,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.658232+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:12:30.658232+0000","last_peered":"2024-06-14T23:12:30.658232+0000","last_clean":"2024-06-14T23:12:30.658232+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:12:30.658232+0000","last_undegraded":"2024-06-14T23:12:30.658232+0000","last_fullsized":"2024-06-14T23:12:30.658232+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":273,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645170+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:12:30.645170+0000","last_peered":"2024-06-14T23:12:30.645170+0000","last_clean":"2024-06-14T23:12:30.645170+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:12:30.645170+0000","last_undegraded":"2024-06-14T23:12:30.645170+0000","last_fullsized":"2024-06-14T23:12:30.645170+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":636,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.649350+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:12:30.649350+0000","last_peered":"2024-06-14T23:12:30.649350+0000","last_clean":"2024-06-14T23:12:30.649350+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:12:30.649350+0000","last_undegraded":"2024-06-14T23:12:30.649350+0000","last_fullsized":"2024-06-14T23:12:30.649350+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"337'10991","reported_seq":15576,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:50.381067+0000","last_change":"2024-06-14T23:11:24.647584+0000","last_active":"2024-06-14T23:12:50.381067+0000","last_peered":"2024-06-14T23:12:50.381067+0000","last_clean":"2024-06-14T23:10:25.009797+0000","last_became_active":"2024-06-14T23:11:24.638765+0000","last_became_peered":"2024-06-14T23:11:24.638765+0000","last_unstale":"2024-06-14T23:12:50.381067+0000","last_undegraded":"2024-06-14T23:11:24.535343+0000","last_fullsized":"2024-06-14T23:11:24.381001+0000","mapping_epoch":323,"log_start":"319'6672","ondisk_log_start":"319'6672","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4319,"log_dups_size":0,"ondisk_log_size":4319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:37.543254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247791616,"num_objects":3781,"num_object_clones":0,"num_object_copies":11343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10991,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2251,"num_bytes_recovered":147521536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[3,2],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,2,3","objects":655},{"shards":"1,3,4","objects":2529},{"shards":"2,3","objects":-1352}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":113,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.667501+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:12:30.667501+0000","last_peered":"2024-06-14T23:12:30.667501+0000","last_clean":"2024-06-14T23:12:30.667501+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:12:30.667501+0000","last_undegraded":"2024-06-14T23:12:30.667501+0000","last_fullsized":"2024-06-14T23:12:30.667501+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.644948+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:12:30.644948+0000","last_peered":"2024-06-14T23:12:30.644948+0000","last_clean":"2024-06-14T23:12:30.644948+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:12:30.644948+0000","last_undegraded":"2024-06-14T23:12:30.644948+0000","last_fullsized":"2024-06-14T23:12:30.644948+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":352,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.669159+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:12:30.669159+0000","last_peered":"2024-06-14T23:12:30.669159+0000","last_clean":"2024-06-14T23:12:30.669159+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:12:30.669159+0000","last_undegraded":"2024-06-14T23:12:30.669159+0000","last_fullsized":"2024-06-14T23:12:30.669159+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":616,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.647798+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:12:30.647798+0000","last_peered":"2024-06-14T23:12:30.647798+0000","last_clean":"2024-06-14T23:12:30.647798+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:12:30.647798+0000","last_undegraded":"2024-06-14T23:12:30.647798+0000","last_fullsized":"2024-06-14T23:12:30.647798+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"337'11109","reported_seq":16348,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:53.193219+0000","last_change":"2024-06-14T23:11:50.568053+0000","last_active":"2024-06-14T23:12:53.193219+0000","last_peered":"2024-06-14T23:12:53.193219+0000","last_clean":"2024-06-14T23:12:53.193219+0000","last_became_active":"2024-06-14T23:11:50.555786+0000","last_became_peered":"2024-06-14T23:11:50.555786+0000","last_unstale":"2024-06-14T23:12:53.193219+0000","last_undegraded":"2024-06-14T23:12:53.193219+0000","last_fullsized":"2024-06-14T23:12:53.193219+0000","mapping_epoch":328,"log_start":"318'5700","ondisk_log_start":"318'5700","created":295,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5409,"log_dups_size":0,"ondisk_log_size":5409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:36.382663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240189440,"num_objects":3665,"num_object_clones":0,"num_object_copies":10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11109,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2575,"num_bytes_recovered":168755200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":288,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.644456+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:12:30.644456+0000","last_peered":"2024-06-14T23:12:30.644456+0000","last_clean":"2024-06-14T23:12:30.644456+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:12:30.644456+0000","last_undegraded":"2024-06-14T23:12:30.644456+0000","last_fullsized":"2024-06-14T23:12:30.644456+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":335,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650505+0000","last_change":"2024-06-14T23:10:26.049033+0000","last_active":"2024-06-14T23:12:30.650505+0000","last_peered":"2024-06-14T23:12:30.650505+0000","last_clean":"2024-06-14T23:12:30.650505+0000","last_became_active":"2024-06-14T23:10:26.047775+0000","last_became_peered":"2024-06-14T23:10:26.047775+0000","last_unstale":"2024-06-14T23:12:30.650505+0000","last_undegraded":"2024-06-14T23:12:30.650505+0000","last_fullsized":"2024-06-14T23:12:30.650505+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:15:52.051858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":329,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.667732+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:12:30.667732+0000","last_peered":"2024-06-14T23:12:30.667732+0000","last_clean":"2024-06-14T23:12:30.667732+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:12:30.667732+0000","last_undegraded":"2024-06-14T23:12:30.667732+0000","last_fullsized":"2024-06-14T23:12:30.667732+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":730,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.667184+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:12:30.667184+0000","last_peered":"2024-06-14T23:12:30.667184+0000","last_clean":"2024-06-14T23:12:30.667184+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:12:30.667184+0000","last_undegraded":"2024-06-14T23:12:30.667184+0000","last_fullsized":"2024-06-14T23:12:30.667184+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"337'11051","reported_seq":11122,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:51.640021+0000","last_change":"2024-06-14T23:11:24.656131+0000","last_active":"2024-06-14T23:12:51.640021+0000","last_peered":"2024-06-14T23:12:51.640021+0000","last_clean":"2024-06-14T23:10:37.043860+0000","last_became_active":"2024-06-14T23:11:24.647995+0000","last_became_peered":"2024-06-14T23:11:24.647995+0000","last_unstale":"2024-06-14T23:12:51.640021+0000","last_undegraded":"2024-06-14T23:11:24.403317+0000","last_fullsized":"2024-06-14T23:11:24.380506+0000","mapping_epoch":323,"log_start":"318'6100","ondisk_log_start":"318'6100","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4951,"log_dups_size":0,"ondisk_log_size":4951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:32:10.827694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238878720,"num_objects":3645,"num_object_clones":0,"num_object_copies":10935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":470272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1144}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":297,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.669395+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:12:30.669395+0000","last_peered":"2024-06-14T23:12:30.669395+0000","last_clean":"2024-06-14T23:12:30.669395+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:12:30.669395+0000","last_undegraded":"2024-06-14T23:12:30.669395+0000","last_fullsized":"2024-06-14T23:12:30.669395+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":333,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.644482+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:12:30.644482+0000","last_peered":"2024-06-14T23:12:30.644482+0000","last_clean":"2024-06-14T23:12:30.644482+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:12:30.644482+0000","last_undegraded":"2024-06-14T23:12:30.644482+0000","last_fullsized":"2024-06-14T23:12:30.644482+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":138,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645956+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:12:30.645956+0000","last_peered":"2024-06-14T23:12:30.645956+0000","last_clean":"2024-06-14T23:12:30.645956+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:12:30.645956+0000","last_undegraded":"2024-06-14T23:12:30.645956+0000","last_fullsized":"2024-06-14T23:12:30.645956+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":624,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.643697+0000","last_change":"2024-06-14T23:11:23.413224+0000","last_active":"2024-06-14T23:12:30.643697+0000","last_peered":"2024-06-14T23:12:30.643697+0000","last_clean":"2024-06-14T23:12:30.643697+0000","last_became_active":"2024-06-14T23:11:23.412680+0000","last_became_peered":"2024-06-14T23:11:23.412680+0000","last_unstale":"2024-06-14T23:12:30.643697+0000","last_undegraded":"2024-06-14T23:12:30.643697+0000","last_fullsized":"2024-06-14T23:12:30.643697+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:48:10.967651+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.e","version":"337'10940","reported_seq":11000,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:53.374108+0000","last_change":"2024-06-14T23:09:41.994118+0000","last_active":"2024-06-14T23:12:53.374108+0000","last_peered":"2024-06-14T23:12:53.374108+0000","last_clean":"2024-06-14T23:12:53.374108+0000","last_became_active":"2024-06-14T23:09:41.993702+0000","last_became_peered":"2024-06-14T23:09:41.993702+0000","last_unstale":"2024-06-14T23:12:53.374108+0000","last_undegraded":"2024-06-14T23:12:53.374108+0000","last_fullsized":"2024-06-14T23:12:53.374108+0000","mapping_epoch":299,"log_start":"318'5600","ondisk_log_start":"318'5600","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:25.994252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235405312,"num_objects":3592,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":306,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.644798+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:12:30.644798+0000","last_peered":"2024-06-14T23:12:30.644798+0000","last_clean":"2024-06-14T23:12:30.644798+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:12:30.644798+0000","last_undegraded":"2024-06-14T23:12:30.644798+0000","last_fullsized":"2024-06-14T23:12:30.644798+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":185,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.641588+0000","last_change":"2024-06-14T23:09:41.991652+0000","last_active":"2024-06-14T23:12:30.641588+0000","last_peered":"2024-06-14T23:12:30.641588+0000","last_clean":"2024-06-14T23:12:30.641588+0000","last_became_active":"2024-06-14T23:09:41.991317+0000","last_became_peered":"2024-06-14T23:09:41.991317+0000","last_unstale":"2024-06-14T23:12:30.641588+0000","last_undegraded":"2024-06-14T23:12:30.641588+0000","last_fullsized":"2024-06-14T23:12:30.641588+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:16:18.209329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":302,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.649892+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:12:30.649892+0000","last_peered":"2024-06-14T23:12:30.649892+0000","last_clean":"2024-06-14T23:12:30.649892+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:12:30.649892+0000","last_undegraded":"2024-06-14T23:12:30.649892+0000","last_fullsized":"2024-06-14T23:12:30.649892+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":544,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.641510+0000","last_change":"2024-06-14T23:10:29.063981+0000","last_active":"2024-06-14T23:12:30.641510+0000","last_peered":"2024-06-14T23:12:30.641510+0000","last_clean":"2024-06-14T23:12:30.641510+0000","last_became_active":"2024-06-14T23:10:29.063546+0000","last_became_peered":"2024-06-14T23:10:29.063546+0000","last_unstale":"2024-06-14T23:12:30.641510+0000","last_undegraded":"2024-06-14T23:12:30.641510+0000","last_fullsized":"2024-06-14T23:12:30.641510+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:26:23.411322+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"337'10794","reported_seq":17894,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:48.734207+0000","last_change":"2024-06-14T23:11:24.664901+0000","last_active":"2024-06-14T23:12:48.734207+0000","last_peered":"2024-06-14T23:12:48.734207+0000","last_clean":"2024-06-14T23:10:24.989118+0000","last_became_active":"2024-06-14T23:11:24.647313+0000","last_became_peered":"2024-06-14T23:11:24.647313+0000","last_unstale":"2024-06-14T23:12:48.734207+0000","last_undegraded":"2024-06-14T23:11:24.555289+0000","last_fullsized":"2024-06-14T23:11:24.381668+0000","mapping_epoch":323,"log_start":"318'5900","ondisk_log_start":"318'5900","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:48:12.784805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247660568,"num_objects":3780,"num_object_clones":0,"num_object_copies":11340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10794,"num_write_kb":466305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3510,"num_bytes_recovered":230031360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,3],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,2,7","objects":677},{"shards":"1,3,7","objects":2484},{"shards":"3,7","objects":619}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":252,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.649153+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:12:30.649153+0000","last_peered":"2024-06-14T23:12:30.649153+0000","last_clean":"2024-06-14T23:12:30.649153+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:12:30.649153+0000","last_undegraded":"2024-06-14T23:12:30.649153+0000","last_fullsized":"2024-06-14T23:12:30.649153+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":258,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.667667+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:12:30.667667+0000","last_peered":"2024-06-14T23:12:30.667667+0000","last_clean":"2024-06-14T23:12:30.667667+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:12:30.667667+0000","last_undegraded":"2024-06-14T23:12:30.667667+0000","last_fullsized":"2024-06-14T23:12:30.667667+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":339,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.668295+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:12:30.668295+0000","last_peered":"2024-06-14T23:12:30.668295+0000","last_clean":"2024-06-14T23:12:30.668295+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:12:30.668295+0000","last_undegraded":"2024-06-14T23:12:30.668295+0000","last_fullsized":"2024-06-14T23:12:30.668295+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":616,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.644621+0000","last_change":"2024-06-14T23:11:23.406149+0000","last_active":"2024-06-14T23:12:30.644621+0000","last_peered":"2024-06-14T23:12:30.644621+0000","last_clean":"2024-06-14T23:12:30.644621+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:12:30.644621+0000","last_undegraded":"2024-06-14T23:12:30.644621+0000","last_fullsized":"2024-06-14T23:12:30.644621+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.751667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.751667+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.7","version":"337'10747","reported_seq":17839,"reported_epoch":337,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-14T23:12:48.727446+0000","last_change":"2024-06-14T23:11:24.669886+0000","last_active":"2024-06-14T23:12:48.727446+0000","last_peered":"2024-06-14T23:12:48.727446+0000","last_clean":"2024-06-14T23:10:25.005348+0000","last_became_active":"2024-06-14T23:11:24.634913+0000","last_became_peered":"2024-06-14T23:11:24.634913+0000","last_unstale":"2024-06-14T23:12:48.727446+0000","last_undegraded":"2024-06-14T23:12:48.727446+0000","last_fullsized":"2024-06-14T23:11:24.386154+0000","mapping_epoch":323,"log_start":"318'5900","ondisk_log_start":"318'5900","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:26:38.637559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246087680,"num_objects":3755,"num_object_clones":0,"num_object_copies":11265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3147,"num_objects_unfound":0,"num_objects_dirty":3755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10747,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3504,"num_bytes_recovered":229638144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,4],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,2,7","objects":653},{"shards":"1,4,7","objects":2494},{"shards":"4,7","objects":608}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":135,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:40.404625+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:12:40.404625+0000","last_peered":"2024-06-14T23:12:40.404625+0000","last_clean":"2024-06-14T23:12:40.404625+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:12:40.404625+0000","last_undegraded":"2024-06-14T23:12:40.404625+0000","last_fullsized":"2024-06-14T23:12:40.404625+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":306,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650926+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:12:30.650926+0000","last_peered":"2024-06-14T23:12:30.650926+0000","last_clean":"2024-06-14T23:12:30.650926+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:12:30.650926+0000","last_undegraded":"2024-06-14T23:12:30.650926+0000","last_fullsized":"2024-06-14T23:12:30.650926+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":728,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.667365+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:12:30.667365+0000","last_peered":"2024-06-14T23:12:30.667365+0000","last_clean":"2024-06-14T23:12:30.667365+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:12:30.667365+0000","last_undegraded":"2024-06-14T23:12:30.667365+0000","last_fullsized":"2024-06-14T23:12:30.667365+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"337'10776","reported_seq":11295,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:51.681341+0000","last_change":"2024-06-14T23:11:24.664127+0000","last_active":"2024-06-14T23:12:51.681341+0000","last_peered":"2024-06-14T23:12:51.681341+0000","last_clean":"2024-06-14T23:10:25.013628+0000","last_became_active":"2024-06-14T23:10:27.203384+0000","last_became_peered":"2024-06-14T23:10:27.203384+0000","last_unstale":"2024-06-14T23:12:51.681341+0000","last_undegraded":"2024-06-14T23:11:24.661184+0000","last_fullsized":"2024-06-14T23:10:27.026246+0000","mapping_epoch":310,"log_start":"318'5800","ondisk_log_start":"318'5800","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:40.932486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229113856,"num_objects":3496,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":14548992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"1,6,7","objects":513},{"shards":"6,7","objects":-435}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":324,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.652363+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:12:30.652363+0000","last_peered":"2024-06-14T23:12:30.652363+0000","last_clean":"2024-06-14T23:12:30.652363+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:12:30.652363+0000","last_undegraded":"2024-06-14T23:12:30.652363+0000","last_fullsized":"2024-06-14T23:12:30.652363+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":335,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.641803+0000","last_change":"2024-06-14T23:11:23.427812+0000","last_active":"2024-06-14T23:12:30.641803+0000","last_peered":"2024-06-14T23:12:30.641803+0000","last_clean":"2024-06-14T23:12:30.641803+0000","last_became_active":"2024-06-14T23:11:23.427563+0000","last_became_peered":"2024-06-14T23:11:23.427563+0000","last_unstale":"2024-06-14T23:12:30.641803+0000","last_undegraded":"2024-06-14T23:12:30.641803+0000","last_fullsized":"2024-06-14T23:12:30.641803+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.941006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":696,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.668165+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:12:30.668165+0000","last_peered":"2024-06-14T23:12:30.668165+0000","last_clean":"2024-06-14T23:12:30.668165+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:12:30.668165+0000","last_undegraded":"2024-06-14T23:12:30.668165+0000","last_fullsized":"2024-06-14T23:12:30.668165+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"337'10794","reported_seq":18123,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:51.681117+0000","last_change":"2024-06-14T23:11:24.656651+0000","last_active":"2024-06-14T23:12:51.681117+0000","last_peered":"2024-06-14T23:12:51.681117+0000","last_clean":"2024-06-14T23:10:25.016384+0000","last_became_active":"2024-06-14T23:11:24.643623+0000","last_became_peered":"2024-06-14T23:11:24.643623+0000","last_unstale":"2024-06-14T23:12:51.681117+0000","last_undegraded":"2024-06-14T23:11:24.519360+0000","last_fullsized":"2024-06-14T23:11:24.380111+0000","mapping_epoch":323,"log_start":"318'5800","ondisk_log_start":"318'5800","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4994,"log_dups_size":0,"ondisk_log_size":4994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:18:43.920069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232521728,"num_objects":3548,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10794,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3624,"num_bytes_recovered":237502464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,3],"avail_no_missing":["6"],"object_location_counts":[{"shards":"1,2,6","objects":524},{"shards":"1,3,6","objects":2461},{"shards":"3,6","objects":563}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":354,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650641+0000","last_change":"2024-06-14T23:11:23.416264+0000","last_active":"2024-06-14T23:12:30.650641+0000","last_peered":"2024-06-14T23:12:30.650641+0000","last_clean":"2024-06-14T23:12:30.650641+0000","last_became_active":"2024-06-14T23:11:23.415946+0000","last_became_peered":"2024-06-14T23:11:23.415946+0000","last_unstale":"2024-06-14T23:12:30.650641+0000","last_undegraded":"2024-06-14T23:12:30.650641+0000","last_fullsized":"2024-06-14T23:12:30.650641+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:22.668569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":205,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645688+0000","last_change":"2024-06-14T23:10:26.045363+0000","last_active":"2024-06-14T23:12:30.645688+0000","last_peered":"2024-06-14T23:12:30.645688+0000","last_clean":"2024-06-14T23:12:30.645688+0000","last_became_active":"2024-06-14T23:10:26.043844+0000","last_became_peered":"2024-06-14T23:10:26.043844+0000","last_unstale":"2024-06-14T23:12:30.645688+0000","last_undegraded":"2024-06-14T23:12:30.645688+0000","last_fullsized":"2024-06-14T23:12:30.645688+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:11:27.891184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":719,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.666683+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:12:30.666683+0000","last_peered":"2024-06-14T23:12:30.666683+0000","last_clean":"2024-06-14T23:12:30.666683+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:12:30.666683+0000","last_undegraded":"2024-06-14T23:12:30.666683+0000","last_fullsized":"2024-06-14T23:12:30.666683+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"337'10950","reported_seq":11242,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:51.559293+0000","last_change":"2024-06-14T23:10:39.484566+0000","last_active":"2024-06-14T23:12:51.559293+0000","last_peered":"2024-06-14T23:12:51.559293+0000","last_clean":"2024-06-14T23:10:24.998059+0000","last_became_active":"2024-06-14T23:10:27.239025+0000","last_became_peered":"2024-06-14T23:10:27.239025+0000","last_unstale":"2024-06-14T23:12:51.559293+0000","last_undegraded":"2024-06-14T23:10:27.071395+0000","last_fullsized":"2024-06-14T23:10:27.025672+0000","mapping_epoch":310,"log_start":"318'6000","ondisk_log_start":"318'6000","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4950,"log_dups_size":0,"ondisk_log_size":4950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:52:40.136978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236191744,"num_objects":3604,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":465728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":7602176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":494},{"shards":"3,6","objects":-449}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":314,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650284+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:12:30.650284+0000","last_peered":"2024-06-14T23:12:30.650284+0000","last_clean":"2024-06-14T23:12:30.650284+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:12:30.650284+0000","last_undegraded":"2024-06-14T23:12:30.650284+0000","last_fullsized":"2024-06-14T23:12:30.650284+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":135,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650101+0000","last_change":"2024-06-14T23:10:26.056252+0000","last_active":"2024-06-14T23:12:30.650101+0000","last_peered":"2024-06-14T23:12:30.650101+0000","last_clean":"2024-06-14T23:12:30.650101+0000","last_became_active":"2024-06-14T23:10:26.046629+0000","last_became_peered":"2024-06-14T23:10:26.046629+0000","last_unstale":"2024-06-14T23:12:30.650101+0000","last_undegraded":"2024-06-14T23:12:30.650101+0000","last_fullsized":"2024-06-14T23:12:30.650101+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:51:19.277386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":607,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645790+0000","last_change":"2024-06-14T23:11:23.428346+0000","last_active":"2024-06-14T23:12:30.645790+0000","last_peered":"2024-06-14T23:12:30.645790+0000","last_clean":"2024-06-14T23:12:30.645790+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:12:30.645790+0000","last_undegraded":"2024-06-14T23:12:30.645790+0000","last_fullsized":"2024-06-14T23:12:30.645790+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.656210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.656210+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"337'10857","reported_seq":10917,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:48.702637+0000","last_change":"2024-06-14T23:09:41.992020+0000","last_active":"2024-06-14T23:12:48.702637+0000","last_peered":"2024-06-14T23:12:48.702637+0000","last_clean":"2024-06-14T23:12:48.702637+0000","last_became_active":"2024-06-14T23:09:41.991722+0000","last_became_peered":"2024-06-14T23:09:41.991722+0000","last_unstale":"2024-06-14T23:12:48.702637+0000","last_undegraded":"2024-06-14T23:12:48.702637+0000","last_fullsized":"2024-06-14T23:12:48.702637+0000","mapping_epoch":299,"log_start":"318'6000","ondisk_log_start":"318'6000","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4857,"log_dups_size":0,"ondisk_log_size":4857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:35:54.751263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245170176,"num_objects":3741,"num_object_clones":0,"num_object_copies":11223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":467136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":134,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650047+0000","last_change":"2024-06-14T23:10:26.047111+0000","last_active":"2024-06-14T23:12:30.650047+0000","last_peered":"2024-06-14T23:12:30.650047+0000","last_clean":"2024-06-14T23:12:30.650047+0000","last_became_active":"2024-06-14T23:10:26.045540+0000","last_became_peered":"2024-06-14T23:10:26.045540+0000","last_unstale":"2024-06-14T23:12:30.650047+0000","last_undegraded":"2024-06-14T23:12:30.650047+0000","last_fullsized":"2024-06-14T23:12:30.650047+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:04.131400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":315,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.669000+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:12:30.669000+0000","last_peered":"2024-06-14T23:12:30.669000+0000","last_clean":"2024-06-14T23:12:30.669000+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:12:30.669000+0000","last_undegraded":"2024-06-14T23:12:30.669000+0000","last_fullsized":"2024-06-14T23:12:30.669000+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":575,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.651483+0000","last_change":"2024-06-14T23:09:17.382466+0000","last_active":"2024-06-14T23:12:30.651483+0000","last_peered":"2024-06-14T23:12:30.651483+0000","last_clean":"2024-06-14T23:12:30.651483+0000","last_became_active":"2024-06-14T23:09:12.030205+0000","last_became_peered":"2024-06-14T23:09:12.030205+0000","last_unstale":"2024-06-14T23:12:30.651483+0000","last_undegraded":"2024-06-14T23:12:30.651483+0000","last_fullsized":"2024-06-14T23:12:30.651483+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:09:17.382389+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:33.121472+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"337'10760","reported_seq":10810,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:53.342387+0000","last_change":"2024-06-14T23:09:29.408859+0000","last_active":"2024-06-14T23:12:53.342387+0000","last_peered":"2024-06-14T23:12:53.342387+0000","last_clean":"2024-06-14T23:12:53.342387+0000","last_became_active":"2024-06-14T23:09:29.408548+0000","last_became_peered":"2024-06-14T23:09:29.408548+0000","last_unstale":"2024-06-14T23:12:53.342387+0000","last_undegraded":"2024-06-14T23:12:53.342387+0000","last_fullsized":"2024-06-14T23:12:53.342387+0000","mapping_epoch":295,"log_start":"318'5400","ondisk_log_start":"318'5400","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:19.810173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229507072,"num_objects":3502,"num_object_clones":0,"num_object_copies":10506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10760,"num_write_kb":456384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":288,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.641928+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:12:30.641928+0000","last_peered":"2024-06-14T23:12:30.641928+0000","last_clean":"2024-06-14T23:12:30.641928+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:12:30.641928+0000","last_undegraded":"2024-06-14T23:12:30.641928+0000","last_fullsized":"2024-06-14T23:12:30.641928+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":232,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.668502+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:12:30.668502+0000","last_peered":"2024-06-14T23:12:30.668502+0000","last_clean":"2024-06-14T23:12:30.668502+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:12:30.668502+0000","last_undegraded":"2024-06-14T23:12:30.668502+0000","last_fullsized":"2024-06-14T23:12:30.668502+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":616,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.647867+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:12:30.647867+0000","last_peered":"2024-06-14T23:12:30.647867+0000","last_clean":"2024-06-14T23:12:30.647867+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:12:30.647867+0000","last_undegraded":"2024-06-14T23:12:30.647867+0000","last_fullsized":"2024-06-14T23:12:30.647867+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"337'10960","reported_seq":11010,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:51.680006+0000","last_change":"2024-06-14T23:09:29.403408+0000","last_active":"2024-06-14T23:12:51.680006+0000","last_peered":"2024-06-14T23:12:51.680006+0000","last_clean":"2024-06-14T23:12:51.680006+0000","last_became_active":"2024-06-14T23:09:29.403000+0000","last_became_peered":"2024-06-14T23:09:29.403000+0000","last_unstale":"2024-06-14T23:12:51.680006+0000","last_undegraded":"2024-06-14T23:12:51.680006+0000","last_fullsized":"2024-06-14T23:12:51.680006+0000","mapping_epoch":295,"log_start":"318'6000","ondisk_log_start":"318'6000","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4960,"log_dups_size":0,"ondisk_log_size":4960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:16.838865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233308160,"num_objects":3560,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":324,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.641724+0000","last_change":"2024-06-14T23:10:26.040201+0000","last_active":"2024-06-14T23:12:30.641724+0000","last_peered":"2024-06-14T23:12:30.641724+0000","last_clean":"2024-06-14T23:12:30.641724+0000","last_became_active":"2024-06-14T23:10:26.039865+0000","last_became_peered":"2024-06-14T23:10:26.039865+0000","last_unstale":"2024-06-14T23:12:30.641724+0000","last_undegraded":"2024-06-14T23:12:30.641724+0000","last_fullsized":"2024-06-14T23:12:30.641724+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:34.621202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":302,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.653675+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:12:30.653675+0000","last_peered":"2024-06-14T23:12:30.653675+0000","last_clean":"2024-06-14T23:12:30.653675+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:12:30.653675+0000","last_undegraded":"2024-06-14T23:12:30.653675+0000","last_fullsized":"2024-06-14T23:12:30.653675+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":744,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.647009+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:12:30.647009+0000","last_peered":"2024-06-14T23:12:30.647009+0000","last_clean":"2024-06-14T23:12:30.647009+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:12:30.647009+0000","last_undegraded":"2024-06-14T23:12:30.647009+0000","last_fullsized":"2024-06-14T23:12:30.647009+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"337'11084","reported_seq":11151,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:12:53.341795+0000","last_change":"2024-06-14T23:11:24.632384+0000","last_active":"2024-06-14T23:12:53.341795+0000","last_peered":"2024-06-14T23:12:53.341795+0000","last_clean":"2024-06-14T23:10:37.053680+0000","last_became_active":"2024-06-14T23:11:24.623684+0000","last_became_peered":"2024-06-14T23:11:24.623684+0000","last_unstale":"2024-06-14T23:12:53.341795+0000","last_undegraded":"2024-06-14T23:11:24.400366+0000","last_fullsized":"2024-06-14T23:11:24.379058+0000","mapping_epoch":323,"log_start":"318'5800","ondisk_log_start":"318'5800","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5284,"log_dups_size":0,"ondisk_log_size":5284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:34:53.331507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233832448,"num_objects":3568,"num_object_clones":0,"num_object_copies":10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11084,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":1014}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":297,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.666886+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:12:30.666886+0000","last_peered":"2024-06-14T23:12:30.666886+0000","last_clean":"2024-06-14T23:12:30.666886+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:12:30.666886+0000","last_undegraded":"2024-06-14T23:12:30.666886+0000","last_fullsized":"2024-06-14T23:12:30.666886+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":315,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.658037+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:12:30.658037+0000","last_peered":"2024-06-14T23:12:30.658037+0000","last_clean":"2024-06-14T23:12:30.658037+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:12:30.658037+0000","last_undegraded":"2024-06-14T23:12:30.658037+0000","last_fullsized":"2024-06-14T23:12:30.658037+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":770,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650432+0000","last_change":"2024-06-14T23:11:23.418239+0000","last_active":"2024-06-14T23:12:30.650432+0000","last_peered":"2024-06-14T23:12:30.650432+0000","last_clean":"2024-06-14T23:12:30.650432+0000","last_became_active":"2024-06-14T23:11:23.417873+0000","last_became_peered":"2024-06-14T23:11:23.417873+0000","last_unstale":"2024-06-14T23:12:30.650432+0000","last_undegraded":"2024-06-14T23:12:30.650432+0000","last_fullsized":"2024-06-14T23:12:30.650432+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:04:50.740175+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":323,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.640882+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:12:30.640882+0000","last_peered":"2024-06-14T23:12:30.640882+0000","last_clean":"2024-06-14T23:12:30.640882+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:12:30.640882+0000","last_undegraded":"2024-06-14T23:12:30.640882+0000","last_fullsized":"2024-06-14T23:12:30.640882+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":319,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.647272+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:12:30.647272+0000","last_peered":"2024-06-14T23:12:30.647272+0000","last_clean":"2024-06-14T23:12:30.647272+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:12:30.647272+0000","last_undegraded":"2024-06-14T23:12:30.647272+0000","last_fullsized":"2024-06-14T23:12:30.647272+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":692,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.667437+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:12:30.667437+0000","last_peered":"2024-06-14T23:12:30.667437+0000","last_clean":"2024-06-14T23:12:30.667437+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:12:30.667437+0000","last_undegraded":"2024-06-14T23:12:30.667437+0000","last_fullsized":"2024-06-14T23:12:30.667437+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":221,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.641449+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:12:30.641449+0000","last_peered":"2024-06-14T23:12:30.641449+0000","last_clean":"2024-06-14T23:12:30.641449+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:12:30.641449+0000","last_undegraded":"2024-06-14T23:12:30.641449+0000","last_fullsized":"2024-06-14T23:12:30.641449+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":200,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.649362+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:12:30.649362+0000","last_peered":"2024-06-14T23:12:30.649362+0000","last_clean":"2024-06-14T23:12:30.649362+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:12:30.649362+0000","last_undegraded":"2024-06-14T23:12:30.649362+0000","last_fullsized":"2024-06-14T23:12:30.649362+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":658,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.648003+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:12:30.648003+0000","last_peered":"2024-06-14T23:12:30.648003+0000","last_clean":"2024-06-14T23:12:30.648003+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:12:30.648003+0000","last_undegraded":"2024-06-14T23:12:30.648003+0000","last_fullsized":"2024-06-14T23:12:30.648003+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":38,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.650035+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:12:30.650035+0000","last_peered":"2024-06-14T23:12:30.650035+0000","last_clean":"2024-06-14T23:12:30.650035+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:12:30.650035+0000","last_undegraded":"2024-06-14T23:12:30.650035+0000","last_fullsized":"2024-06-14T23:12:30.650035+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":297,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.669022+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:12:30.669022+0000","last_peered":"2024-06-14T23:12:30.669022+0000","last_clean":"2024-06-14T23:12:30.669022+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:12:30.669022+0000","last_undegraded":"2024-06-14T23:12:30.669022+0000","last_fullsized":"2024-06-14T23:12:30.669022+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":656,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.666608+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:12:30.666608+0000","last_peered":"2024-06-14T23:12:30.666608+0000","last_clean":"2024-06-14T23:12:30.666608+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:12:30.666608+0000","last_undegraded":"2024-06-14T23:12:30.666608+0000","last_fullsized":"2024-06-14T23:12:30.666608+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":349,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.653080+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:12:30.653080+0000","last_peered":"2024-06-14T23:12:30.653080+0000","last_clean":"2024-06-14T23:12:30.653080+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:12:30.653080+0000","last_undegraded":"2024-06-14T23:12:30.653080+0000","last_fullsized":"2024-06-14T23:12:30.653080+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":38,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.645229+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:12:30.645229+0000","last_peered":"2024-06-14T23:12:30.645229+0000","last_clean":"2024-06-14T23:12:30.645229+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:12:30.645229+0000","last_undegraded":"2024-06-14T23:12:30.645229+0000","last_fullsized":"2024-06-14T23:12:30.645229+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":769,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-06-14T23:12:30.667097+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:12:30.667097+0000","last_peered":"2024-06-14T23:12:30.667097+0000","last_clean":"2024-06-14T23:12:30.667097+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:12:30.667097+0000","last_undegraded":"2024-06-14T23:12:30.667097+0000","last_fullsized":"2024-06-14T23:12:30.667097+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":3824025624,"num_objects":58351,"num_object_clones":0,"num_object_copies":175053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31862,"num_objects_misplaced":3147,"num_objects_unfound":0,"num_objects_dirty":58351,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174655,"num_write_kb":7456129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15803,"num_bytes_recovered":1035665408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12104577024,"data_stored":12104564808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78136,"ondisk_log_size":78136,"up":48,"acting":37,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":821296,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":103,"num_write":26,"num_write_kb":273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":365100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234465,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3339220,"kb_used_data":2139596,"kb_used_omap":1,"kb_used_meta":1199614,"kb_avail":90405932,"statfs":{"total":95995035648,"available":92575674368,"internally_reserved":0,"allocated":2190946304,"data_stored":2189199264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1228405080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,0,0,0,0,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":0.95599999999999996,"15min":0.99099999999999999},"min":{"1min":0.24199999999999999,"5min":0.218,"15min":0.19800000000000001},"max":{"1min":5.7229999999999999,"5min":5.7229999999999999,"15min":6.9539999999999997},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.169,"15min":1.1950000000000001},"min":{"1min":0.29599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":5.5350000000000001,"5min":5.5350000000000001,"15min":6.9550000000000001},"last":0.52800000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:12:11 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4410000000000001,"15min":1.3180000000000001},"min":{"1min":0.38700000000000001,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":3.9460000000000002,"5min":12.010999999999999,"15min":12.010999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4219999999999999,"15min":1.345},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.22900000000000001},"max":{"1min":3.738,"5min":13.531000000000001,"15min":13.531000000000001},"last":0.879}]},{"osd":2,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.1970000000000001,"5min":5.1970000000000001,"15min":5.1970000000000001},"last":0.432},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":0.46100000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5609999999999999,"15min":1.4179999999999999},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":5.2270000000000003,"5min":20.645,"15min":20.645},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.625,"15min":1.494},"min":{"1min":0.36499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.3639999999999999,"5min":22.042999999999999,"15min":22.042999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.127,"15min":1.077},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.3319999999999999,"5min":5.3319999999999999,"15min":11.342000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.208,"15min":1.1459999999999999},"min":{"1min":0.32600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3769999999999998,"5min":8.5899999999999999,"15min":16.879000000000001},"last":0.377}]},{"osd":5,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.026,"15min":1.109},"min":{"1min":0.39100000000000001,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":5.6849999999999996,"5min":5.6849999999999996,"15min":6.9000000000000004},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.407,"5min":1.222,"15min":1.1499999999999999},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.3949999999999996,"5min":5.3949999999999996,"15min":6.8529999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Fri Jun 14 23:12:00 2024","interfaces":[{"interface":"back","average":{"1min":2.7090000000000001,"5min":1.554,"15min":1.393},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":9.8949999999999996,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.431,"5min":1.3660000000000001,"15min":1.3759999999999999},"min":{"1min":0.57099999999999995,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":3.8879999999999999,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.48199999999999998}]}]},{"osd":6,"up_from":272,"seq":1168231104572,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2487936,"kb_used_data":2098744,"kb_used_omap":7,"kb_used_meta":388856,"kb_avail":91257216,"statfs":{"total":95995035648,"available":93447389184,"internally_reserved":0,"allocated":2149113856,"data_stored":2147386829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":398188986},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.2450000000000001,"15min":1.6659999999999999},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.1190000000000002,"5min":6.6440000000000001,"15min":6.6440000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.3089999999999999,"15min":1.7190000000000001},"min":{"1min":0.48599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.484,"5min":6.6790000000000003,"15min":6.6790000000000003},"last":0.83199999999999996}]},{"osd":1,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.5900000000000001,"15min":2.1800000000000002},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.4660000000000002,"5min":8.173,"15min":8.173},"last":2.0150000000000001},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.506,"15min":2.0859999999999999},"min":{"1min":0.36799999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.5209999999999999,"5min":10.380000000000001,"15min":10.380000000000001},"last":0.78300000000000003}]},{"osd":2,"last update":"Fri Jun 14 23:12:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":3.4489999999999998,"5min":3.4489999999999998,"15min":3.4489999999999998},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":1.673}]},{"osd":3,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.4279999999999999,"15min":1.827},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.5779999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.363,"5min":1.7190000000000001,"15min":2.0219999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.6239999999999997,"5min":8.4740000000000002,"15min":8.4740000000000002},"last":1.647}]},{"osd":4,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.3520000000000001,"15min":1.4259999999999999},"min":{"1min":0.28999999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":6.8920000000000003,"5min":11.010999999999999,"15min":11.010999999999999},"last":1.98},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.1599999999999999,"15min":1.266},"min":{"1min":0.39400000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.605,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":2.1629999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.139,"15min":1.371},"min":{"1min":0.34000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":3.5510000000000002,"5min":7.7469999999999999,"15min":7.7469999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.472,"5min":1.48,"15min":1.6020000000000001},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.5810000000000004,"5min":7.8680000000000003,"15min":7.8680000000000003},"last":1.694}]},{"osd":7,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.6930000000000001,"15min":1.7410000000000001},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.5419999999999998,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":2.1819999999999999},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.429,"15min":1.5389999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.9219999999999997,"5min":7.4619999999999997,"15min":7.4619999999999997},"last":2.0859999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757741,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1652412,"kb_used_data":472884,"kb_used_omap":1,"kb_used_meta":1179518,"kb_avail":92092740,"statfs":{"total":95995035648,"available":94302965760,"internally_reserved":0,"allocated":484233216,"data_stored":482506095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1207826776},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:11:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.93200000000000005,"15min":1.347},"min":{"1min":0.251,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.0270000000000001,"5min":6.1950000000000003,"15min":14.94},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.628,"5min":1.105,"15min":1.3700000000000001},"min":{"1min":0.29299999999999998,"5min":0.25800000000000001,"15min":0.219},"max":{"1min":2.3380000000000001,"5min":8.3520000000000003,"15min":9.3819999999999997},"last":0.40799999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.91900000000000004,"15min":0.99099999999999999},"min":{"1min":0.28999999999999998,"5min":0.245,"15min":0.23999999999999999},"max":{"1min":1.7509999999999999,"5min":6.9480000000000004,"15min":7.4370000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.96599999999999997,"15min":1.23},"min":{"1min":0.27400000000000002,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":3.0819999999999999,"5min":5.1920000000000002,"15min":9.5039999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.0579999999999998,"5min":3.0579999999999998,"15min":3.0579999999999998},"last":0.312},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.125,"5min":3.125,"15min":3.125},"last":0.434}]},{"osd":3,"last update":"Fri Jun 14 23:12:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.081,"15min":1.3979999999999999},"min":{"1min":0.31,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":1.9370000000000001,"5min":8.4350000000000005,"15min":10.358000000000001},"last":0.496},{"interface":"front","average":{"1min":0.629,"5min":1.006,"15min":1.4159999999999999},"min":{"1min":0.33600000000000002,"5min":0.318,"15min":0.28899999999999998},"max":{"1min":1.585,"5min":7.4290000000000003,"15min":14.914},"last":0.67600000000000005}]},{"osd":4,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.71599999999999997,"15min":0.89600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.1719999999999999,"5min":3.012,"15min":14.352},"last":1.573},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.80500000000000005,"15min":1.0489999999999999},"min":{"1min":0.34100000000000003,"5min":0.28799999999999998,"15min":0.253},"max":{"1min":1.9950000000000001,"5min":8.4039999999999999,"15min":9.5980000000000008},"last":0.38500000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.079,"15min":1.389},"min":{"1min":0.29599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.905,"5min":9.4710000000000001,"15min":9.4710000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.3600000000000001,"15min":1.746},"min":{"1min":0.379,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.1630000000000003,"5min":10.632,"15min":10.632},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.256,"15min":1.246},"min":{"1min":0.39100000000000001,"5min":0.29099999999999998,"15min":0.27900000000000003},"max":{"1min":26.981999999999999,"5min":26.981999999999999,"15min":26.981999999999999},"last":1.7589999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":1.333,"15min":1.222},"min":{"1min":0.42499999999999999,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":23.326000000000001,"5min":23.326000000000001,"15min":23.326000000000001},"last":0.53200000000000003}]}]},{"osd":4,"up_from":30,"seq":128849019377,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2212356,"kb_used_data":1321660,"kb_used_omap":1,"kb_used_meta":890686,"kb_avail":91532796,"statfs":{"total":95995035648,"available":93729583104,"internally_reserved":0,"allocated":1353379840,"data_stored":1351648281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912062808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.89000000000000001,"15min":1.135},"min":{"1min":0.251,"5min":0.217,"15min":0.17100000000000001},"max":{"1min":2.7490000000000001,"5min":6.875,"15min":10.567},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.80100000000000005,"15min":1.1100000000000001},"min":{"1min":0.25800000000000001,"5min":0.251,"15min":0.22700000000000001},"max":{"1min":1.7270000000000001,"5min":6.1379999999999999,"15min":8.3390000000000004},"last":0.42399999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.383,"15min":1.5},"min":{"1min":0.29299999999999998,"5min":0.26400000000000001,"15min":0.23100000000000001},"max":{"1min":4.476,"5min":42.656999999999996,"15min":56.917000000000002},"last":3.3759999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":1.343,"15min":1.5609999999999999},"min":{"1min":0.30399999999999999,"5min":0.249,"15min":0.249},"max":{"1min":2.7280000000000002,"5min":42.192999999999998,"15min":56.789999999999999},"last":0.48999999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:12:23 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":3.468},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":3.4209999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.6559999999999999,"15min":1.3859999999999999},"min":{"1min":0.30599999999999999,"5min":0.25,"15min":0.25},"max":{"1min":1.768,"5min":44.389000000000003,"15min":44.389000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.845,"15min":1.5309999999999999},"min":{"1min":0.40000000000000002,"5min":0.32500000000000001,"15min":0.27500000000000002},"max":{"1min":2.7040000000000002,"5min":46.101999999999997,"15min":46.101999999999997},"last":0.54700000000000004}]},{"osd":5,"last update":"Fri Jun 14 23:12:36 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.85799999999999998,"15min":1.5529999999999999},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.4159999999999999,"5min":4.04,"15min":139.13200000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":1.613},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.2360000000000002,"5min":6.9290000000000003,"15min":138.86000000000001},"last":5.077}]},{"osd":6,"last update":"Fri Jun 14 23:12:06 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.117,"15min":1.105},"min":{"1min":0.36599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.46,"5min":6.9989999999999997,"15min":6.9989999999999997},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.383,"5min":1.2230000000000001,"15min":1.5069999999999999},"min":{"1min":0.32200000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.1070000000000002,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":3.8519999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:12:17 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.1859999999999999,"15min":1.087},"min":{"1min":0.33800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.0439999999999996,"5min":6.5910000000000002,"15min":8.9489999999999998},"last":3.444},{"interface":"front","average":{"1min":1.583,"5min":1.2210000000000001,"15min":1.1659999999999999},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.4219999999999997,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":0.80800000000000005}]}]},{"osd":3,"up_from":138,"seq":592705487075,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3590648,"kb_used_data":2739056,"kb_used_omap":4,"kb_used_meta":851515,"kb_avail":90154504,"statfs":{"total":95995035648,"available":92318212096,"internally_reserved":0,"allocated":2804793344,"data_stored":2803069550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":871951838},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":1.083,"15min":1.347},"min":{"1min":0.187,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.77200000000000002,"5min":25.190000000000001,"15min":25.190000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":1.2050000000000001,"15min":1.3560000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.504,"5min":25.652000000000001,"15min":25.652000000000001},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.304,"15min":1.4530000000000001},"min":{"1min":0.245,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":2.649,"5min":25.754000000000001,"15min":25.754000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.548,"15min":1.623},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.25},"max":{"1min":2.6899999999999999,"5min":27.992999999999999,"15min":27.992999999999999},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.7549999999999999,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.972,"5min":2.972,"15min":2.972},"last":0.32000000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.5760000000000001,"15min":1.74},"min":{"1min":0.34100000000000003,"5min":0.28299999999999997,"15min":0.27900000000000003},"max":{"1min":2.1269999999999998,"5min":25.094000000000001,"15min":73.263000000000005},"last":0.623},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.3959999999999999,"15min":1.6659999999999999},"min":{"1min":0.34499999999999997,"5min":0.27600000000000002,"15min":0.27300000000000002},"max":{"1min":2.8559999999999999,"5min":25.5,"15min":73.5},"last":0.57199999999999995}]},{"osd":5,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":1.5760000000000001,"15min":1.7589999999999999},"min":{"1min":0.38500000000000001,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":2.0369999999999999,"5min":25.521000000000001,"15min":81.438999999999993},"last":0.371},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.6240000000000001,"15min":1.7989999999999999},"min":{"1min":0.41899999999999998,"5min":0.32800000000000001,"15min":0.29299999999999998},"max":{"1min":3.0739999999999998,"5min":25.84,"15min":81.626999999999995},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Jun 14 23:12:00 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":1.73,"15min":1.7170000000000001},"min":{"1min":0.39800000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":7.1379999999999999,"5min":25.869,"15min":25.869},"last":2.996},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.5289999999999999,"15min":1.516},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.4340000000000002,"5min":25.143000000000001,"15min":25.143000000000001},"last":0.55900000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.4379999999999999,"15min":1.4330000000000001},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.30499999999999999},"max":{"1min":2.4940000000000002,"5min":25.120000000000001,"15min":25.120000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.24,"5min":1.5940000000000001,"15min":1.3979999999999999},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":3.4300000000000002,"5min":25.472000000000001,"15min":25.472000000000001},"last":0.72199999999999998}]}]},{"osd":2,"up_from":322,"seq":1382979469331,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1014908,"kb_used_data":887860,"kb_used_omap":10,"kb_used_meta":127029,"kb_avail":92730244,"statfs":{"total":95995035648,"available":94955769856,"internally_reserved":0,"allocated":909168640,"data_stored":907441508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":130078246},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.51400000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.83599999999999997}]},{"osd":3,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.70499999999999996}]},{"osd":4,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":0.873},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0089999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.544,"5min":2.544,"15min":2.544},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.998}]},{"osd":6,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.6950000000000003,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":0.90600000000000003}]},{"osd":7,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":3.008,"5min":3.008,"15min":3.008},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.548,"5min":4.548,"15min":4.548},"last":3.8620000000000001}]}]},{"osd":1,"up_from":0,"seq":498216206619,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024274,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2335309824,"data_stored":2335309824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":890769408,"data_stored":890765336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2772504576,"data_stored":2772500504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1336803328,"data_stored":1336803328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":474480640,"data_stored":474480640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2122317824,"data_stored":2122317824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2172391424,"data_stored":2172387352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:12:55.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph status --format=json 2024-06-14T23:12:55.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:55.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:55.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:55.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:55.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:55.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:55.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:55.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:55.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:55.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:55.962 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:55.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:55.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:55 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:55.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3756820248' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:12:55.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:55 smithi137 ceph-mon[90634]: osdmap e338: 8 total, 8 up, 6 in 2024-06-14T23:12:56.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:55 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:56.024 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:55 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3756820248' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:12:56.024 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:55 smithi086 ceph-mon[97913]: osdmap e338: 8 total, 8 up, 6 in 2024-06-14T23:12:56.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:55 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:12:56.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:55 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3756820248' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:12:56.024 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:55 smithi086 ceph-mon[102754]: osdmap e338: 8 total, 8 up, 6 in 2024-06-14T23:12:56.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:56.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:56.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:56.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:56.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:56.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:56.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:56.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:56.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:56.522 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:56.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:56.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:56.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:56.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:56.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:56.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:56.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:56.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:56.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:56.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:56.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:56 smithi137 ceph-mon[90634]: pgmap v1261: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 803 op/s; 31618/169788 objects degraded (18.622%); 2926/169788 objects misplaced (1.723%); 9.2 MiB/s, 147 objects/s recovering 2024-06-14T23:12:57.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:57.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:57.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:57.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:57.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:56 smithi086 ceph-mon[97913]: pgmap v1261: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 803 op/s; 31618/169788 objects degraded (18.622%); 2926/169788 objects misplaced (1.723%); 9.2 MiB/s, 147 objects/s recovering 2024-06-14T23:12:57.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:56 smithi086 ceph-mon[102754]: pgmap v1261: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 803 op/s; 31618/169788 objects degraded (18.622%); 2926/169788 objects misplaced (1.723%); 9.2 MiB/s, 147 objects/s recovering 2024-06-14T23:12:57.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:57.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:57.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:57.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:57.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:57.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:57.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:57.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:57.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:57.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:57.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:57.639 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:12:57.640 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:12:57.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:57.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:57.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:57.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:12:57.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:57.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:57.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:12:57.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:58.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:58.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:58.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:12:58.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:58.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:12:58.230 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:12:58.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:58.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:58.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:58.298 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:12:58.298 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:12:58.333 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:12:58.333 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:12:58.372 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:12:58.372 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:12:58.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:58.406 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:12:58.407 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:12:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:58.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:58.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:58.441 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:12:58.441 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:12:58.469 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:12:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:12:58.501 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:12:58.502 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:12:58.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:58.537 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:12:58.537 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:12:58.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:12:58.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:58.560 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:58.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:58.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-14T23:12:58.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:58.657 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:58.658 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:58 smithi137 ceph-mon[90634]: pgmap v1262: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 770 op/s; 31373/166122 objects degraded (18.886%); 2926/166122 objects misplaced (1.761%); 8.8 MiB/s, 140 objects/s recovering 2024-06-14T23:12:58.658 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2678209342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:58.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:58.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:58.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:58.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:58.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:58.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:58 smithi086 ceph-mon[97913]: pgmap v1262: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 770 op/s; 31373/166122 objects degraded (18.886%); 2926/166122 objects misplaced (1.761%); 8.8 MiB/s, 140 objects/s recovering 2024-06-14T23:12:58.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2678209342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:58.733 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:12:58.733 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:12:58.733 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:58 smithi086 ceph-mon[102754]: pgmap v1262: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.4 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 770 op/s; 31373/166122 objects degraded (18.886%); 2926/166122 objects misplaced (1.761%); 8.8 MiB/s, 140 objects/s recovering 2024-06-14T23:12:58.733 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2678209342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:12:58.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:12:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:58.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:58.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:12:58.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:12:59.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:59.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:12:59.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:12:59.111 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:12:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:59.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:12:59.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:12:59.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:59.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:59.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:12:59.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:59.409 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:59.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:59.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:59.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:59.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:59.642 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:12:59 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 31618/169788 objects degraded (18.622%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:59.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:12:59.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:12:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:59.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:12:59.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:59.821 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:59.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:12:59.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:12:59 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 31618/169788 objects degraded (18.622%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:59.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:12:59 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 31618/169788 objects degraded (18.622%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:12:59.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:12:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:12:59.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:12:59.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:00.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:00.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:00.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:00.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:00.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:00.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:00.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:00.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:00.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:00.474 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:00 smithi137 ceph-mon[90634]: pgmap v1263: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 784 op/s; 31137/160926 objects degraded (19.349%); 2669/160926 objects misplaced (1.659%); 8.9 MiB/s, 142 objects/s recovering 2024-06-14T23:13:00.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:00.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:00.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:00.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:00.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:00.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:00.706 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:00 smithi086 ceph-mon[97913]: pgmap v1263: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 784 op/s; 31137/160926 objects degraded (19.349%); 2669/160926 objects misplaced (1.659%); 8.9 MiB/s, 142 objects/s recovering 2024-06-14T23:13:00.707 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:00 smithi086 ceph-mon[102754]: pgmap v1263: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.3 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 784 op/s; 31137/160926 objects degraded (19.349%); 2669/160926 objects misplaced (1.659%); 8.9 MiB/s, 142 objects/s recovering 2024-06-14T23:13:00.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:00.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:00.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:00.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:00.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:00.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:01.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:01.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:01.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:01.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:01.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:01.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:01.243 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252314 10.9 10:960f94a9:::benchmark_data_smithi086_189745_object121346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.011224+0000", 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 520.22752853199995, 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0900273949999999, 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.244 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252314 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011224+0000", 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011224+0000", 2024-06-14T23:13:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011223+0000", 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011230+0000", 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6009999999999997e-06 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011246+0000", 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6090999999999999e-05 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011252+0000", 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9660000000000001e-06 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011316+0000", 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4170000000000004e-05 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.011325+0000", 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1210000000000007e-06 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.041481+0000", 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0301560869999999 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.041565+0000", 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4305e-05 2024-06-14T23:13:01.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.041641+0000", 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6340000000000004e-05 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.059215+0000", 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017573808 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101090+0000", 2024-06-14T23:13:01.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041874407000000002 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101182+0000", 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2931999999999997e-05 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101222+0000", 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9650000000000002e-05 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101252+0000", 2024-06-14T23:13:01.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9663999999999999e-05 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252317 10.9 10:91aeda83:::benchmark_data_smithi086_189745_object121349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:21.022551+0000", 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 520.21620152699995, 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.078809973, 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:13:01.251 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252317 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022551+0000", 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022551+0000", 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022550+0000", 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022556+0000", 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4269999999999999e-06 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022566+0000", 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0010000000000001e-05 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022570+0000", 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2189999999999998e-06 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022606+0000", 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5809000000000002e-05 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:21.022610+0000", 2024-06-14T23:13:01.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8049999999999997e-06 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.045161+0000", 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0225505130000001 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.045235+0000", 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4148999999999998e-05 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.045310+0000", 2024-06-14T23:13:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4604000000000006e-05 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.059224+0000", 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013914517 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101136+0000", 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041911325999999999 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101303+0000", 2024-06-14T23:13:01.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016734900000000001 2024-06-14T23:13:01.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101339+0000", 2024-06-14T23:13:01.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6380000000000001e-05 2024-06-14T23:13:01.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:22.101361+0000", 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2092000000000001e-05 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252678 10.3 10:ceff32ef:::benchmark_data_smithi086_189745_object121710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.023839+0000", 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 518.21491439099998, 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98525532999999998, 2024-06-14T23:13:01.258 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252678 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023839+0000", 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023839+0000", 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023837+0000", 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023844+0000", 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.455e-06 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023858+0000", 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4619e-05 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023864+0000", 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3279999999999996e-06 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023971+0000", 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010751600000000001 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.023979+0000", 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4390000000000003e-06 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.997909+0000", 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97393003199999995 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.998001+0000", 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2132000000000005e-05 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.998077+0000", 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6552999999999997e-05 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002272+0000", 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041949149999999996 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.008946+0000", 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066735199999999996 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.008996+0000", 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0719e-05 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009049+0000", 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2899999999999998e-05 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009094+0000", 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4501e-05 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252708 10.3 10:c07e5468:::benchmark_data_smithi086_189745_object121740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.055163+0000", 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 518.18359006699995, 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95415895699999997, 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252708 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055163+0000", 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055163+0000", 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055162+0000", 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055167+0000", 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2140000000000004e-06 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055177+0000", 2024-06-14T23:13:01.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.013e-05 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055181+0000", 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8460000000000001e-06 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055222+0000", 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0878999999999999e-05 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.055230+0000", 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.276e-06 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002317+0000", 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94708661800000005 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002441+0000", 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124435 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:13:01.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.002570+0000", 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012846799999999999 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009132+0000", 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065624999999999998 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009196+0000", 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3718000000000002e-05 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009236+0000", 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0268000000000001e-05 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009286+0000", 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0078999999999999e-05 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009322+0000", 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5590999999999999e-05 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.272 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252709 10.3 10:c41e1665:::benchmark_data_smithi086_189745_object121741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:13:01.272 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.057035+0000", 2024-06-14T23:13:01.272 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 518.18171782100001, 2024-06-14T23:13:01.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95242822400000005, 2024-06-14T23:13:01.272 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.272 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.272 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.273 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:13:01.273 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:13:01.273 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252709 2024-06-14T23:13:01.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.273 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057035+0000", 2024-06-14T23:13:01.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057035+0000", 2024-06-14T23:13:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057034+0000", 2024-06-14T23:13:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057039+0000", 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1690000000000002e-06 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057049+0000", 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0148e-05 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057054+0000", 2024-06-14T23:13:01.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7590000000000003e-06 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057093+0000", 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8566000000000003e-05 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.057100+0000", 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9859999999999997e-06 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.005750+0000", 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94864968599999999 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.005829+0000", 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9124999999999999e-05 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.005919+0000", 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0216000000000005e-05 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009353+0000", 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034340680000000002 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009357+0000", 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8970000000000001e-06 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009391+0000", 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4400000000000003e-05 2024-06-14T23:13:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009429+0000", 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7137e-05 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009463+0000", 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4786000000000002e-05 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252721 10.3 10:cae392c3:::benchmark_data_smithi086_189745_object121753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.067872+0000", 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 518.17088094899998, 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95780526600000004, 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252721 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067872+0000", 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067872+0000", 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067870+0000", 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:13:01.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067876+0000", 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6359999999999999e-06 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067888+0000", 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1966e-05 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067894+0000", 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1739999999999997e-06 2024-06-14T23:13:01.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067944+0000", 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0259000000000001e-05 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.067951+0000", 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8650000000000003e-06 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009492+0000", 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94154064699999995 2024-06-14T23:13:01.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009567+0000", 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5333999999999994e-05 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.009654+0000", 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6735000000000002e-05 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017286+0000", 2024-06-14T23:13:01.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076317240000000003 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017292+0000", 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1759999999999998e-06 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025571+0000", 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082794340000000004 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025636+0000", 2024-06-14T23:13:01.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.444e-05 2024-06-14T23:13:01.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025677+0000", 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1607999999999999e-05 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252748 10.3 10:c31065d0:::benchmark_data_smithi086_189745_object121780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.118605+0000", 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 518.12014753699998, 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90724995399999997, 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.288 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252748 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118605+0000", 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118605+0000", 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118604+0000", 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118611+0000", 2024-06-14T23:13:01.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7259999999999996e-06 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118622+0000", 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1523e-05 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118628+0000", 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8119999999999996e-06 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118675+0000", 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6828e-05 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.118685+0000", 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7990000000000005e-06 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.012981+0000", 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89429594199999995 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.013114+0000", 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013327200000000001 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.013233+0000", 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119317 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017334+0000", 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041004850000000001 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025532+0000", 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081980309999999997 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025763+0000", 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023137200000000001 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025816+0000", 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2525000000000003e-05 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025855+0000", 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9762999999999999e-05 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252755 10.3 10:c538d66a:::benchmark_data_smithi086_189745_object121787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.127097+0000", 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 518.11165565800002, 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89934203300000004, 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252755 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127097+0000", 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127097+0000", 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127096+0000", 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:01.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127102+0000", 2024-06-14T23:13:01.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5610000000000004e-06 2024-06-14T23:13:01.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127117+0000", 2024-06-14T23:13:01.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4346e-05 2024-06-14T23:13:01.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127122+0000", 2024-06-14T23:13:01.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6799999999999998e-06 2024-06-14T23:13:01.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127178+0000", 2024-06-14T23:13:01.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5414000000000002e-05 2024-06-14T23:13:01.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.127189+0000", 2024-06-14T23:13:01.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1756000000000001e-05 2024-06-14T23:13:01.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017353+0000", 2024-06-14T23:13:01.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89016351900000001 2024-06-14T23:13:01.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017457+0000", 2024-06-14T23:13:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010399 2024-06-14T23:13:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:13:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.017571+0000", 2024-06-14T23:13:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113579 2024-06-14T23:13:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025689+0000", 2024-06-14T23:13:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081183660000000001 2024-06-14T23:13:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025731+0000", 2024-06-14T23:13:01.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1641000000000003e-05 2024-06-14T23:13:01.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026328+0000", 2024-06-14T23:13:01.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000597077 2024-06-14T23:13:01.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026402+0000", 2024-06-14T23:13:01.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4680999999999999e-05 2024-06-14T23:13:01.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026439+0000", 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7027000000000003e-05 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.45821.0:252760 10.3 10:cf81e509:::benchmark_data_smithi086_189745_object121792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 518.10747449899998, 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89530984300000005, 2024-06-14T23:13:01.306 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.45821", 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/4174109028", 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 252760 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131278+0000", 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131281+0000", 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6860000000000002e-06 2024-06-14T23:13:01.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131287+0000", 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1029999999999998e-06 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131291+0000", 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8959999999999996e-06 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131331+0000", 2024-06-14T23:13:01.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0021999999999997e-05 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:23.131336+0000", 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3900000000000001e-06 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.021730+0000", 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89039394699999996 2024-06-14T23:13:01.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.021825+0000", 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5179000000000005e-05 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.021931+0000", 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010621899999999999 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025902+0000", 2024-06-14T23:13:01.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039701329999999998 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.025913+0000", 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1089e-05 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026503+0000", 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00059051700000000004 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026545+0000", 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1745000000000001e-05 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:04:24.026588+0000", 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3331000000000002e-05 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136541 11.5 11:ac4784f5:::benchmark_data_smithi086_226760_object9220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.932519+0000", 2024-06-14T23:13:01.313 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.30623403599998, 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0219499169999999, 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136541 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932519+0000", 2024-06-14T23:13:01.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932519+0000", 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932518+0000", 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932523+0000", 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.356e-06 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932535+0000", 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2167999999999999e-05 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932541+0000", 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7400000000000001e-06 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932575+0000", 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3946000000000003e-05 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932684+0000", 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010943299999999999 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932803+0000", 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118439 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.946273+0000", 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013470650000000001 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.946281+0000", 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6260000000000003e-06 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.954404+0000", 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.008122773 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.954469+0000", 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4986000000000004e-05 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136539 11.8 11:1b96f922:::benchmark_data_smithi086_226760_object9218:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.28923691599999, 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.00353812, 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136539 2024-06-14T23:13:01.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949516+0000", 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949518+0000", 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.83e-06 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949525+0000", 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4980000000000003e-06 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949529+0000", 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8340000000000001e-06 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949580+0000", 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1199000000000002e-05 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949590+0000", 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9499999999999996e-06 2024-06-14T23:13:01.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.952967+0000", 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.003376877 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953054+0000", 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7299000000000006e-05 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136542 11.8 11:118834dc:::benchmark_data_smithi086_226760_object9221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.949549+0000", 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.28920434499997, 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.013177585, 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136542 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949549+0000", 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949549+0000", 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949548+0000", 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:01.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949551+0000", 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9459999999999998e-06 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949557+0000", 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6400000000000002e-06 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949560+0000", 2024-06-14T23:13:01.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6849999999999999e-06 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949608+0000", 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7905000000000003e-05 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949613+0000", 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5349999999999998e-06 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953079+0000", 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0034659319999999 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953172+0000", 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3556999999999999e-05 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962668+0000", 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094954789999999994 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962726+0000", 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8313999999999998e-05 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136550 11.a 11:5c3c9776:::benchmark_data_smithi086_226760_object9229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:13:01.330 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.28917753500002, 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0136263190000001, 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136550 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949575+0000", 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949577+0000", 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7889999999999999e-06 2024-06-14T23:13:01.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949583+0000", 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3689999999999998e-06 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.949586+0000", 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3380000000000001e-06 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.950035+0000", 2024-06-14T23:13:01.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044843299999999998 2024-06-14T23:13:01.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:01.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.950042+0000", 2024-06-14T23:13:01.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.922e-06 2024-06-14T23:13:01.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.952960+0000", 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.002918226 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953096+0000", 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000136355 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953224+0000", 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000128019 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962228+0000", 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090035859999999992 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963063+0000", 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000835453 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963118+0000", 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5297000000000003e-05 2024-06-14T23:13:01.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963202+0000", 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3250000000000004e-05 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136554 11.8 11:1bd3cba3:::benchmark_data_smithi086_226760_object9233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.957712+0000", 2024-06-14T23:13:01.338 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.281040781, 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.004780947, 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136554 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957712+0000", 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957712+0000", 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957710+0000", 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957717+0000", 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0480000000000001e-06 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957731+0000", 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3701e-05 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957738+0000", 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4860000000000002e-06 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957791+0000", 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3304999999999999e-05 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.957799+0000", 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9039999999999992e-06 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953252+0000", 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99545359899999997 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953371+0000", 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118045 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.953468+0000", 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7633000000000005e-05 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957490+0000", 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040219909999999999 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962312+0000", 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048222350000000002 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962399+0000", 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6630000000000002e-05 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962493+0000", 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4086999999999996e-05 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136561 11.a 11:5eddd569:::benchmark_data_smithi086_226760_object9240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.966732+0000", 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.27202078699997, 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0042532740000001, 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136561 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966732+0000", 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966732+0000", 2024-06-14T23:13:01.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966731+0000", 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966736+0000", 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0629999999999996e-06 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966747+0000", 2024-06-14T23:13:01.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0335e-05 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966751+0000", 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0840000000000002e-06 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966802+0000", 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0970999999999997e-05 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966812+0000", 2024-06-14T23:13:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9860000000000006e-06 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957486+0000", 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99067426400000003 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957592+0000", 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105921 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-06-14T23:13:01.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.957699+0000", 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106839 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962260+0000", 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045612719999999999 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.970899+0000", 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086394460000000003 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.970938+0000", 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9079000000000003e-05 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.970985+0000", 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6937999999999997e-05 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.351 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211914 11.5 11:a4813f22:::benchmark_data_smithi086_226760_object84593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.083822+0000", 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 273.15493048600001, 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97223605199999996, 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211914 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083822+0000", 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083822+0000", 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083821+0000", 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083827+0000", 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5289999999999999e-06 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083835+0000", 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4849999999999992e-06 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083840+0000", 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4460000000000003e-06 2024-06-14T23:13:01.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083879+0000", 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8899000000000001e-05 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.083885+0000", 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.93e-06 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.040652+0000", 2024-06-14T23:13:01.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95676742400000003 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.040790+0000", 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013773400000000001 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.041240+0000", 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00045052500000000001 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.047857+0000", 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066170179999999997 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.047945+0000", 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7732999999999995e-05 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.055720+0000", 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077744800000000003 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.055823+0000", 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103556 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.056058+0000", 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023532300000000001 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.358 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211934 11.5 11:a2c54b8c:::benchmark_data_smithi086_226760_object84613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.154477+0000", 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 273.08427583100001, 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90184410999999998, 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211934 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154477+0000", 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154477+0000", 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154476+0000", 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154483+0000", 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4680000000000003e-06 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154497+0000", 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4273999999999999e-05 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154504+0000", 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2550000000000003e-06 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154557+0000", 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3291999999999997e-05 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.154568+0000", 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1347e-05 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044096+0000", 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.88952724999999999 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044425+0000", 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000329408 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.045198+0000", 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000773363 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048409+0000", 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003210638 2024-06-14T23:13:01.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048463+0000", 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3637999999999997e-05 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.056114+0000", 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076511490000000003 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.056177+0000", 2024-06-14T23:13:01.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3575e-05 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.056321+0000", 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014390200000000001 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:131509 12.c 12:363e613a:::benchmark_data_smithi086_261492_object15004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:49.551046+0000", 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 71.687706551000005, 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0172304130000001, 2024-06-14T23:13:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 131509 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551046+0000", 2024-06-14T23:13:01.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551046+0000", 2024-06-14T23:13:01.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551044+0000", 2024-06-14T23:13:01.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999976 2024-06-14T23:13:01.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551055+0000", 2024-06-14T23:13:01.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0774000000000001e-05 2024-06-14T23:13:01.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551078+0000", 2024-06-14T23:13:01.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3337999999999999e-05 2024-06-14T23:13:01.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551087+0000", 2024-06-14T23:13:01.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8300000000000002e-06 2024-06-14T23:13:01.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551216+0000", 2024-06-14T23:13:01.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012913700000000001 2024-06-14T23:13:01.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551228+0000", 2024-06-14T23:13:01.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2318e-05 2024-06-14T23:13:01.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.564927+0000", 2024-06-14T23:13:01.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.013698679 2024-06-14T23:13:01.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.565051+0000", 2024-06-14T23:13:01.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012433 2024-06-14T23:13:01.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.565184+0000", 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013285700000000001 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568141+0000", 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029568350000000001 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568191+0000", 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9478000000000003e-05 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568201+0000", 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0181e-05 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568244+0000", 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3458000000000003e-05 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568277+0000", 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2592999999999998e-05 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:131550 12.c 12:34f200d1:::benchmark_data_smithi086_261492_object15045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:49.636597+0000", 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 71.602155702000005, 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94182777699999998, 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.376 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 131550 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636597+0000", 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636597+0000", 2024-06-14T23:13:01.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636595+0000", 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636604+0000", 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9789999999999999e-06 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636624+0000", 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9910000000000001e-05 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636633+0000", 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5720000000000003e-06 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636699+0000", 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6550000000000005e-05 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636712+0000", 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2938e-05 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568307+0000", 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93159522699999997 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568391+0000", 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3325999999999995e-05 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568460+0000", 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9091999999999995e-05 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.573215+0000", 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047549130000000004 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.573267+0000", 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2782999999999999e-05 2024-06-14T23:13:01.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578327+0000", 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050593189999999996 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578388+0000", 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1035000000000001e-05 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578425+0000", 2024-06-14T23:13:01.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7168000000000001e-05 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:131569 12.c 12:327400c1:::benchmark_data_smithi086_261492_object15064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:49.675496+0000", 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 71.563257327000002, 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90309976400000003, 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:13:01.383 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 131569 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.675496+0000", 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.675496+0000", 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.675493+0000", 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999976 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.675503+0000", 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6039999999999999e-06 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.675525+0000", 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2096999999999999e-05 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.675533+0000", 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5070000000000003e-06 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.675588+0000", 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5071000000000002e-05 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:13:01.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.675597+0000", 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2760000000000006e-06 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.570879+0000", 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89528197300000001 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.570979+0000", 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9735000000000006e-05 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:13:01.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.571050+0000", 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1605000000000003e-05 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.573262+0000", 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022121189999999998 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578278+0000", 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050153300000000001 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:01.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578527+0000", 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024904899999999999 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578565+0000", 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8115999999999999e-05 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578595+0000", 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0463e-05 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:01.390 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:01.390 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:13:01.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:01.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:01.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:01.562 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:01.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:01.629 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:01.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:01.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:01.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:01.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:01.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:01.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:01.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:01.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:01.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:01.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:02.011 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:13:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:{"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31137/157761 objects degraded (19.737%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1806,"monmap":{"epoch":4,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":338,"num_osds":8,"num_up_osds":8,"osd_up_since":1718406683,"num_in_osds":6,"osd_in_since":1718406625,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":108},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":119,"num_pools":5,"num_objects":52587,"data_bytes":3445464558,"bytes_used":13688090624,"bytes_avail":562282123264,"bytes_total":575970213888,"degraded_objects":31137,"degraded_total":157761,"degraded_ratio":0.19736817084070207,"misplaced_objects":2669,"misplaced_total":157761,"misplaced_ratio":0.016917996209456075,"recovering_objects_per_sec":120,"recovering_bytes_per_sec":7881066,"recovering_keys_per_sec":0,"num_objects_recovered":1203,"num_bytes_recovered":78839808,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":674},"fsmap":{"epoch":9,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi086.nhdjdb","status":"up:active","gid":24548}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.86:8443/","prometheus":"http://172.21.15.86:9283/"}},"servicemap":{"epoch":28,"modified":"2024-06-14T23:12:24.252974+0000","services":{}},"progress_events":{"af1960ac-63de-49e3-896b-122ab3eb5dd7":{"message":"Upgrade to 18.2.2-1443-g6f4b0956 (12m)\n [============................] (remaining: 15m)","progress":0.44999998807907104,"add_to_ceph_s":true},"c77b16e3-338f-456c-83d1-eff5a59f3260":{"message":"Global Recovery Event (2m)\n [=========================...] (remaining: 15s)","progress":0.90756303071975708,"add_to_ceph_s":true}}} 2024-06-14T23:13:02.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:02.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:02.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:02.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:02.258 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:02.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:02.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:02.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:02.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:02.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:02.525 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:02.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:02.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:02.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:02.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:02.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:02.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:02.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:02.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:02.952 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:02.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:03.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:03.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:13:03.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:03.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:03.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:02 smithi086 ceph-mon[102754]: pgmap v1264: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 674 op/s; 31137/157761 objects degraded (19.737%); 2669/157761 objects misplaced (1.692%); 7.5 MiB/s, 120 objects/s recovering 2024-06-14T23:13:03.086 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/792088103' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-14T23:13:03.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:02 smithi086 ceph-mon[97913]: pgmap v1264: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 674 op/s; 31137/157761 objects degraded (19.737%); 2669/157761 objects misplaced (1.692%); 7.5 MiB/s, 120 objects/s recovering 2024-06-14T23:13:03.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/792088103' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-14T23:13:03.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:03.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:03.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:03.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:02 smithi137 ceph-mon[90634]: pgmap v1264: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.2 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 674 op/s; 31137/157761 objects degraded (19.737%); 2669/157761 objects misplaced (1.692%); 7.5 MiB/s, 120 objects/s recovering 2024-06-14T23:13:03.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/792088103' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-14T23:13:03.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:03.363 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:03.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:03.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:03.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:03.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:03.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:03.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:03.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:03.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:03.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:03.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:03.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:03.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:03.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:03.963 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-06-14T23:13:03.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:13:04.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:04.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 31137/157761 objects degraded (19.737%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:04.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 31137/157761 objects degraded (19.737%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:04.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:04.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:04.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:04.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:04.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:04.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:04.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 31137/157761 objects degraded (19.737%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:04.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:04.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:04.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:04.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:04.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:04.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:04.517 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:04.540 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:04.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:04.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:04.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:04.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:04.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:04.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:04.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:04.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:04.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:05.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:05.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:05.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:04 smithi086 ceph-mon[97913]: pgmap v1265: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 805 op/s; 30884/150963 objects degraded (20.458%); 2669/150963 objects misplaced (1.768%); 9.1 MiB/s, 145 objects/s recovering 2024-06-14T23:13:05.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:04 smithi086 ceph-mon[102754]: pgmap v1265: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 805 op/s; 30884/150963 objects degraded (20.458%); 2669/150963 objects misplaced (1.768%); 9.1 MiB/s, 145 objects/s recovering 2024-06-14T23:13:05.084 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:13:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:13:04] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.43.0" 2024-06-14T23:13:05.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:05.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:05.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:05.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:04 smithi137 ceph-mon[90634]: pgmap v1265: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.1 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 805 op/s; 30884/150963 objects degraded (20.458%); 2669/150963 objects misplaced (1.768%); 9.1 MiB/s, 145 objects/s recovering 2024-06-14T23:13:05.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:05.305 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:05.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:05.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:05.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-14T23:13:05.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:05.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:05.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:05.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:05.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:05.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:05.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:05.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:05.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:05.810 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:13:05.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:05.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:05.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:05.918 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:13:05.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:06.023 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:06.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:06.063 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2482166843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:06.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2482166843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:06.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2482166843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:06.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:06.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:06.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:06.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:06.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:06.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:06.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:06.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:06.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:06.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:06.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:06.586 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:06.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:06.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:06.718 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:06.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:06.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:06.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:06.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:06.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:07.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:07.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:07.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:06 smithi086 ceph-mon[97913]: pgmap v1266: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 844 op/s; 30687/145221 objects degraded (21.131%); 2443/145221 objects misplaced (1.682%); 8.1 MiB/s, 128 objects/s recovering 2024-06-14T23:13:07.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:06 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:07.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:06 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2482166843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:13:07.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:06 smithi086 ceph-mon[97913]: osdmap e339: 8 total, 8 up, 6 in 2024-06-14T23:13:07.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:06 smithi086 ceph-mon[102754]: pgmap v1266: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 844 op/s; 30687/145221 objects degraded (21.131%); 2443/145221 objects misplaced (1.682%); 8.1 MiB/s, 128 objects/s recovering 2024-06-14T23:13:07.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:06 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:07.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:06 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2482166843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:13:07.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:06 smithi086 ceph-mon[102754]: osdmap e339: 8 total, 8 up, 6 in 2024-06-14T23:13:07.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:07.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:07.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:07.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:07.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:06 smithi137 ceph-mon[90634]: pgmap v1266: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 844 op/s; 30687/145221 objects degraded (21.131%); 2443/145221 objects misplaced (1.682%); 8.1 MiB/s, 128 objects/s recovering 2024-06-14T23:13:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:06 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:06 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2482166843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:13:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:06 smithi137 ceph-mon[90634]: osdmap e339: 8 total, 8 up, 6 in 2024-06-14T23:13:07.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:07.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:07.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:07.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:07.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:07.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:07.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:07.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:07.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:07.694 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:07.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:07.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:07.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:07.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:07.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:07.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:07.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:07.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:08.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:08.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:08.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:08.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:08.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:08.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:188920 12.7 12:ee03462b:::benchmark_data_smithi086_261492_object72415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.063489+0000", 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.027520406000000001, 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027552554999999999, 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:13:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188920 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.063489+0000", 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.063489+0000", 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.063488+0000", 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.063494+0000", 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7509999999999997e-06 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.063503+0000", 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.738e-06 2024-06-14T23:13:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.063507+0000", 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7659999999999998e-06 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.069803+0000", 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062951200000000004 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.069924+0000", 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121079 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.070045+0000", 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012137 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.084773+0000", 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014727629000000001 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:188927 12.4 e339/310 12:20f79b82:::benchmark_data_smithi086_261492_object72422:head v 339'11762, mlcod=305'116)", 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.070153+0000", 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.020857205, 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020968056999999998, 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.070153+0000", 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.070153+0000", 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.070152+0000", 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.070159+0000", 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5410000000000003e-06 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.070173+0000", 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3573e-05 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.070179+0000", 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049999999999999e-06 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.070231+0000", 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2432999999999998e-05 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.070250+0000", 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8771999999999999e-05 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:188928 12.4 e339/310 12:2c8c479a:::benchmark_data_smithi086_261492_object72423:head v 339'11763, mlcod=305'116)", 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.075420+0000", 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.015590342, 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015757134999999999, 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.075420+0000", 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.075420+0000", 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.075419+0000", 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.075428+0000", 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8699999999999998e-06 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.075443+0000", 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5803000000000001e-05 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.075449+0000", 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.93e-06 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.086195+0000", 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010745525000000001 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.086218+0000", 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3595e-05 2024-06-14T23:13:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:188930 12.4 e339/310)", 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.078315+0000", 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.012694530000000001, 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012909808, 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.078315+0000", 2024-06-14T23:13:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.078315+0000", 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.078315+0000", 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.078322+0000", 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7540000000000006e-06 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.078336+0000", 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3969e-05 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.078342+0000", 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4879999999999998e-06 2024-06-14T23:13:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:188936 12.d 12:bc9503b6:::benchmark_data_smithi086_261492_object72431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.081521+0000", 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0094892759999999996, 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097532220000000006, 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:13:08.354 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188936 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081521+0000", 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081521+0000", 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081519+0000", 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081525+0000", 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3389999999999998e-06 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081601+0000", 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5774999999999999e-05 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081608+0000", 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4649999999999999e-06 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.083999+0000", 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023917420000000001 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.086292+0000", 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022922630000000001 2024-06-14T23:13:08.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.086434+0000", 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014202 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089545+0000", 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031112420000000002 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089754+0000", 2024-06-14T23:13:08.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020955299999999999 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:188937 12.c e339/328)", 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.081958+0000", 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0090515609999999996, 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093849959999999996, 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081958+0000", 2024-06-14T23:13:08.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081958+0000", 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081958+0000", 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081963+0000", 2024-06-14T23:13:08.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.242e-06 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081972+0000", 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0890000000000006e-06 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.081976+0000", 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3810000000000001e-06 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.46469.0:188920 12.7 e339/323)", 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.085080+0000", 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.005929503, 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063042799999999998, 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.085080+0000", 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.085080+0000", 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.085079+0000", 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.085086+0000", 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4330000000000001e-06 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.085097+0000", 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1326e-05 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.085102+0000", 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6409999999999999e-06 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:188943 12.7 12.60778497 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.087257+0000", 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.003753171, 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041741979999999996, 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 188943 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.087257+0000", 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.087257+0000", 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.087255+0000", 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.087261+0000", 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7690000000000002e-06 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.087279+0000", 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7927e-05 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.087285+0000", 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0399999999999998e-06 2024-06-14T23:13:08.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:188940 12.6 e339/310 12:670f5644:::benchmark_data_smithi086_261492_object72435:head v 339'11606, mlcod=305'96)", 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.089511+0000", 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0014988460000000001, 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001974132, 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089511+0000", 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089511+0000", 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089510+0000", 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089519+0000", 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4849999999999998e-06 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089534+0000", 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4907e-05 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089541+0000", 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5429999999999999e-06 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089609+0000", 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8316000000000001e-05 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089634+0000", 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4508e-05 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:188941 12.2 e339/295 12:479db3b0:::benchmark_data_smithi086_261492_object72436:head v 339'11514, mlcod=339'11512)", 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.089830+0000", 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.001179746, 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001730877, 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089830+0000", 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089830+0000", 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089830+0000", 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089836+0000", 2024-06-14T23:13:08.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7120000000000003e-06 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089847+0000", 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1032e-05 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089852+0000", 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5000000000000001e-06 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089955+0000", 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103327 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.089972+0000", 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6942000000000001e-05 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:188942 12.2 e339/295)", 2024-06-14T23:13:08.375 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:08.091179+0000", 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4294967295.9998307, 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00043643600000000001, 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.091179+0000", 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:08.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.091179+0000", 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.091178+0000", 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.091186+0000", 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5349999999999999e-06 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.091199+0000", 2024-06-14T23:13:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3749e-05 2024-06-14T23:13:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:08.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:08.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:08.091205+0000", 2024-06-14T23:13:08.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5249999999999996e-06 2024-06-14T23:13:08.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.379 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:08.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:08.379 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T23:13:08.379 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 11 2024-06-14T23:13:08.379 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:13:08.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:08.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:08.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:08.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:08.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:08.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:08.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:08.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:08.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:08.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:13:08.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:08.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:08.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:08.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:08.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:08.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:08.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:08 smithi086 ceph-mon[97913]: pgmap v1268: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 837 op/s; 30661/140985 objects degraded (21.748%); 2443/140985 objects misplaced (1.733%); 7.5 MiB/s, 119 objects/s recovering 2024-06-14T23:13:08.988 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 30661/140985 objects degraded (21.748%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:08.988 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:08 smithi086 ceph-mon[102754]: pgmap v1268: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 837 op/s; 30661/140985 objects degraded (21.748%); 2443/140985 objects misplaced (1.733%); 7.5 MiB/s, 119 objects/s recovering 2024-06-14T23:13:08.988 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 30661/140985 objects degraded (21.748%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:09.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:09.103 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:13:09.103 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:13:09.105 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:13:09.105 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:13:09.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:09.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:09.158 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:13:09.158 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:13:09.201 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:13:09.201 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:13:09.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:09.241 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:13:09.241 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:13:09.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:08 smithi137 ceph-mon[90634]: pgmap v1268: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 837 op/s; 30661/140985 objects degraded (21.748%); 2443/140985 objects misplaced (1.733%); 7.5 MiB/s, 119 objects/s recovering 2024-06-14T23:13:09.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 30661/140985 objects degraded (21.748%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:09.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:09.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:09.276 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:13:09.277 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:13:09.311 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:13:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:13:09.356 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:13:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:13:09.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:09.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:09.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:09.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:09.512 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:13:09.515 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:13:09.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:09.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:09.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:09.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:09.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:09.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-14T23:13:09.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:09.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:09.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:09.831 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:09.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:09.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:09.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:09.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2743193047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:10.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:10.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:10.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2743193047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:10.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2743193047' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:10.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:10.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:10.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:10.157 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:13:10.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:10.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:10.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:10.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:10.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:10.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:10.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:10.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:10.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:10.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:10.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:10.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:10.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:10.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:10.957 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:10 smithi086 ceph-mon[97913]: from='client.36768 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:13:10.957 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:10 smithi086 ceph-mon[97913]: pgmap v1269: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 874 op/s; 30661/134694 objects degraded (22.763%); 2243/134694 objects misplaced (1.665%); 5.6 MiB/s, 90 objects/s recovering 2024-06-14T23:13:10.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:10 smithi086 ceph-mon[102754]: from='client.36768 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:13:10.957 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:10 smithi086 ceph-mon[102754]: pgmap v1269: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 874 op/s; 30661/134694 objects degraded (22.763%); 2243/134694 objects misplaced (1.665%); 5.6 MiB/s, 90 objects/s recovering 2024-06-14T23:13:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:10.972 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:10.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:10.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:10 smithi137 ceph-mon[90634]: from='client.36768 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:13:10.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:10 smithi137 ceph-mon[90634]: pgmap v1269: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 874 op/s; 30661/134694 objects degraded (22.763%); 2243/134694 objects misplaced (1.665%); 5.6 MiB/s, 90 objects/s recovering 2024-06-14T23:13:11.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:11.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:11.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:11.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:11.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:11.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:11.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:11.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:11.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:11.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:11.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:11.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:11.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:11.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:11.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:11.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:11.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:11.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:11.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:11.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:11.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:12.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:12.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:12.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:12.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:12.217 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:12.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:12.288 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1268,"stamp":"2024-06-14T23:13:07.724268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3078987246,"num_objects":46995,"num_object_clones":0,"num_object_copies":140985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30661,"num_objects_misplaced":2443,"num_objects_unfound":0,"num_objects_dirty":46995,"num_whiteouts":0,"num_read":56,"num_read_kb":104,"num_write":186123,"num_write_kb":7456431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17728,"num_bytes_recovered":1160877270,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83165,"ondisk_log_size":83165,"up":357,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":12309840,"kb_used_data":7674712,"kb_used_omap":27,"kb_used_meta":4634788,"kb_avail":550161072,"statfs":{"total":575970213888,"available":563364937728,"internally_reserved":0,"allocated":7858905088,"data_stored":7848458871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":4746023366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,9,7,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-549126144,"num_objects":-8379,"num_object_clones":0,"num_object_copies":-25137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-712,"num_objects_misplaced":-483,"num_objects_unfound":0,"num_objects_dirty":-8379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":78315520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003292"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":717,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.718759+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:12:43.718759+0000","last_peered":"2024-06-14T23:12:43.718759+0000","last_clean":"2024-06-14T23:12:43.718759+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:12:43.718759+0000","last_undegraded":"2024-06-14T23:12:43.718759+0000","last_fullsized":"2024-06-14T23:12:43.718759+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":155,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.710908+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:12:43.710908+0000","last_peered":"2024-06-14T23:12:43.710908+0000","last_clean":"2024-06-14T23:12:43.710908+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:12:43.710908+0000","last_undegraded":"2024-06-14T23:12:43.710908+0000","last_fullsized":"2024-06-14T23:12:43.710908+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":325,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.720032+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:12:43.720032+0000","last_peered":"2024-06-14T23:12:43.720032+0000","last_clean":"2024-06-14T23:12:43.720032+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:12:43.720032+0000","last_undegraded":"2024-06-14T23:12:43.720032+0000","last_fullsized":"2024-06-14T23:12:43.720032+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777778+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:12:54.777778+0000","last_peered":"2024-06-14T23:12:54.777778+0000","last_clean":"2024-06-14T23:12:54.777778+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:12:54.777778+0000","last_undegraded":"2024-06-14T23:12:54.777778+0000","last_fullsized":"2024-06-14T23:12:54.777778+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":121,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777781+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:12:54.777781+0000","last_peered":"2024-06-14T23:12:54.777781+0000","last_clean":"2024-06-14T23:12:54.777781+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:12:54.777781+0000","last_undegraded":"2024-06-14T23:12:54.777781+0000","last_fullsized":"2024-06-14T23:12:54.777781+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":742,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:53.411885+0000","last_change":"2024-06-14T23:12:53.411855+0000","last_active":"2024-06-14T23:12:53.411885+0000","last_peered":"2024-06-14T23:12:53.411885+0000","last_clean":"2024-06-14T23:12:53.411885+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:12:53.411885+0000","last_undegraded":"2024-06-14T23:12:53.411885+0000","last_fullsized":"2024-06-14T23:12:53.411885+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:09:56.196559+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:09:56.196559+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":301,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.708506+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:12:43.708506+0000","last_peered":"2024-06-14T23:12:43.708506+0000","last_clean":"2024-06-14T23:12:43.708506+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:12:43.708506+0000","last_undegraded":"2024-06-14T23:12:43.708506+0000","last_fullsized":"2024-06-14T23:12:43.708506+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":342,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716706+0000","last_change":"2024-06-14T23:11:23.413009+0000","last_active":"2024-06-14T23:12:43.716706+0000","last_peered":"2024-06-14T23:12:43.716706+0000","last_clean":"2024-06-14T23:12:43.716706+0000","last_became_active":"2024-06-14T23:11:23.412491+0000","last_became_peered":"2024-06-14T23:11:23.412491+0000","last_unstale":"2024-06-14T23:12:43.716706+0000","last_undegraded":"2024-06-14T23:12:43.716706+0000","last_fullsized":"2024-06-14T23:12:43.716706+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:57.263547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":488,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.720097+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:12:43.720097+0000","last_peered":"2024-06-14T23:12:43.720097+0000","last_clean":"2024-06-14T23:12:43.720097+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:12:43.720097+0000","last_undegraded":"2024-06-14T23:12:43.720097+0000","last_fullsized":"2024-06-14T23:12:43.720097+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":315,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.708369+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:12:43.708369+0000","last_peered":"2024-06-14T23:12:43.708369+0000","last_clean":"2024-06-14T23:12:43.708369+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:12:43.708369+0000","last_undegraded":"2024-06-14T23:12:43.708369+0000","last_fullsized":"2024-06-14T23:12:43.708369+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":342,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.720350+0000","last_change":"2024-06-14T23:11:23.427771+0000","last_active":"2024-06-14T23:12:43.720350+0000","last_peered":"2024-06-14T23:12:43.720350+0000","last_clean":"2024-06-14T23:12:43.720350+0000","last_became_active":"2024-06-14T23:11:23.427481+0000","last_became_peered":"2024-06-14T23:11:23.427481+0000","last_unstale":"2024-06-14T23:12:43.720350+0000","last_undegraded":"2024-06-14T23:12:43.720350+0000","last_fullsized":"2024-06-14T23:12:43.720350+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:01:04.162256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":131,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.720215+0000","last_change":"2024-06-14T23:10:26.053647+0000","last_active":"2024-06-14T23:12:43.720215+0000","last_peered":"2024-06-14T23:12:43.720215+0000","last_clean":"2024-06-14T23:12:43.720215+0000","last_became_active":"2024-06-14T23:10:26.053318+0000","last_became_peered":"2024-06-14T23:10:26.053318+0000","last_unstale":"2024-06-14T23:12:43.720215+0000","last_undegraded":"2024-06-14T23:12:43.720215+0000","last_fullsized":"2024-06-14T23:12:43.720215+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:32.647107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":131,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777849+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:12:54.777849+0000","last_peered":"2024-06-14T23:12:54.777849+0000","last_clean":"2024-06-14T23:12:54.777849+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:12:54.777849+0000","last_undegraded":"2024-06-14T23:12:54.777849+0000","last_fullsized":"2024-06-14T23:12:54.777849+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":642,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777322+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:12:54.777322+0000","last_peered":"2024-06-14T23:12:54.777322+0000","last_clean":"2024-06-14T23:12:54.777322+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:12:54.777322+0000","last_undegraded":"2024-06-14T23:12:54.777322+0000","last_fullsized":"2024-06-14T23:12:54.777322+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":364,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716792+0000","last_change":"2024-06-14T23:11:23.413241+0000","last_active":"2024-06-14T23:12:43.716792+0000","last_peered":"2024-06-14T23:12:43.716792+0000","last_clean":"2024-06-14T23:12:43.716792+0000","last_became_active":"2024-06-14T23:11:23.412936+0000","last_became_peered":"2024-06-14T23:11:23.412936+0000","last_unstale":"2024-06-14T23:12:43.716792+0000","last_undegraded":"2024-06-14T23:12:43.716792+0000","last_fullsized":"2024-06-14T23:12:43.716792+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:36.581716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":150,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.717055+0000","last_change":"2024-06-14T23:10:26.038549+0000","last_active":"2024-06-14T23:12:43.717055+0000","last_peered":"2024-06-14T23:12:43.717055+0000","last_clean":"2024-06-14T23:12:43.717055+0000","last_became_active":"2024-06-14T23:10:26.038260+0000","last_became_peered":"2024-06-14T23:10:26.038260+0000","last_unstale":"2024-06-14T23:12:43.717055+0000","last_undegraded":"2024-06-14T23:12:43.717055+0000","last_fullsized":"2024-06-14T23:12:43.717055+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:44.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":744,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.703779+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:12:43.703779+0000","last_peered":"2024-06-14T23:12:43.703779+0000","last_clean":"2024-06-14T23:12:43.703779+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:12:43.703779+0000","last_undegraded":"2024-06-14T23:12:43.703779+0000","last_fullsized":"2024-06-14T23:12:43.703779+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":135,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:50.411082+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:12:50.411082+0000","last_peered":"2024-06-14T23:12:50.411082+0000","last_clean":"2024-06-14T23:12:50.411082+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:12:50.411082+0000","last_undegraded":"2024-06-14T23:12:50.411082+0000","last_fullsized":"2024-06-14T23:12:50.411082+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":197,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.720029+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:12:43.720029+0000","last_peered":"2024-06-14T23:12:43.720029+0000","last_clean":"2024-06-14T23:12:43.720029+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:12:43.720029+0000","last_undegraded":"2024-06-14T23:12:43.720029+0000","last_fullsized":"2024-06-14T23:12:43.720029+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":692,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.705032+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:12:43.705032+0000","last_peered":"2024-06-14T23:12:43.705032+0000","last_clean":"2024-06-14T23:12:43.705032+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:12:43.705032+0000","last_undegraded":"2024-06-14T23:12:43.705032+0000","last_fullsized":"2024-06-14T23:12:43.705032+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.778036+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:12:54.778036+0000","last_peered":"2024-06-14T23:12:54.778036+0000","last_clean":"2024-06-14T23:12:54.778036+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:12:54.778036+0000","last_undegraded":"2024-06-14T23:12:54.778036+0000","last_fullsized":"2024-06-14T23:12:54.778036+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":290,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.779390+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:12:54.779390+0000","last_peered":"2024-06-14T23:12:54.779390+0000","last_clean":"2024-06-14T23:12:54.779390+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:12:54.779390+0000","last_undegraded":"2024-06-14T23:12:54.779390+0000","last_fullsized":"2024-06-14T23:12:54.779390+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":644,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.715147+0000","last_change":"2024-06-14T23:10:29.062659+0000","last_active":"2024-06-14T23:12:43.715147+0000","last_peered":"2024-06-14T23:12:43.715147+0000","last_clean":"2024-06-14T23:12:43.715147+0000","last_became_active":"2024-06-14T23:10:29.062256+0000","last_became_peered":"2024-06-14T23:10:29.062256+0000","last_unstale":"2024-06-14T23:12:43.715147+0000","last_undegraded":"2024-06-14T23:12:43.715147+0000","last_fullsized":"2024-06-14T23:12:43.715147+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:38.814157+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":321,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:45.407279+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:12:45.407279+0000","last_peered":"2024-06-14T23:12:45.407279+0000","last_clean":"2024-06-14T23:12:45.407279+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:12:45.407279+0000","last_undegraded":"2024-06-14T23:12:45.407279+0000","last_fullsized":"2024-06-14T23:12:45.407279+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":185,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.710986+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:12:43.710986+0000","last_peered":"2024-06-14T23:12:43.710986+0000","last_clean":"2024-06-14T23:12:43.710986+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:12:43.710986+0000","last_undegraded":"2024-06-14T23:12:43.710986+0000","last_fullsized":"2024-06-14T23:12:43.710986+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":644,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.708700+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:12:43.708700+0000","last_peered":"2024-06-14T23:12:43.708700+0000","last_clean":"2024-06-14T23:12:43.708700+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:12:43.708700+0000","last_undegraded":"2024-06-14T23:12:43.708700+0000","last_fullsized":"2024-06-14T23:12:43.708700+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":204,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.769041+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:12:54.769041+0000","last_peered":"2024-06-14T23:12:54.769041+0000","last_clean":"2024-06-14T23:12:54.769041+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:12:54.769041+0000","last_undegraded":"2024-06-14T23:12:54.769041+0000","last_fullsized":"2024-06-14T23:12:54.769041+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":309,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.709001+0000","last_change":"2024-06-14T23:10:26.053008+0000","last_active":"2024-06-14T23:12:43.709001+0000","last_peered":"2024-06-14T23:12:43.709001+0000","last_clean":"2024-06-14T23:12:43.709001+0000","last_became_active":"2024-06-14T23:10:26.052701+0000","last_became_peered":"2024-06-14T23:10:26.052701+0000","last_unstale":"2024-06-14T23:12:43.709001+0000","last_undegraded":"2024-06-14T23:12:43.709001+0000","last_fullsized":"2024-06-14T23:12:43.709001+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:15:03.929062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":669,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777775+0000","last_change":"2024-06-14T23:11:23.411926+0000","last_active":"2024-06-14T23:12:54.777775+0000","last_peered":"2024-06-14T23:12:54.777775+0000","last_clean":"2024-06-14T23:12:54.777775+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:12:54.777775+0000","last_undegraded":"2024-06-14T23:12:54.777775+0000","last_fullsized":"2024-06-14T23:12:54.777775+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":222,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:50.413810+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:12:50.413810+0000","last_peered":"2024-06-14T23:12:50.413810+0000","last_clean":"2024-06-14T23:12:50.413810+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:12:50.413810+0000","last_undegraded":"2024-06-14T23:12:50.413810+0000","last_fullsized":"2024-06-14T23:12:50.413810+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":352,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777463+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:12:54.777463+0000","last_peered":"2024-06-14T23:12:54.777463+0000","last_clean":"2024-06-14T23:12:54.777463+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:12:54.777463+0000","last_undegraded":"2024-06-14T23:12:54.777463+0000","last_fullsized":"2024-06-14T23:12:54.777463+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":514,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.710375+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:12:43.710375+0000","last_peered":"2024-06-14T23:12:43.710375+0000","last_clean":"2024-06-14T23:12:43.710375+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:12:43.710375+0000","last_undegraded":"2024-06-14T23:12:43.710375+0000","last_fullsized":"2024-06-14T23:12:43.710375+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":321,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716777+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:12:43.716777+0000","last_peered":"2024-06-14T23:12:43.716777+0000","last_clean":"2024-06-14T23:12:43.716777+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:12:43.716777+0000","last_undegraded":"2024-06-14T23:12:43.716777+0000","last_fullsized":"2024-06-14T23:12:43.716777+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":325,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.721340+0000","last_change":"2024-06-14T23:10:26.047451+0000","last_active":"2024-06-14T23:12:43.721340+0000","last_peered":"2024-06-14T23:12:43.721340+0000","last_clean":"2024-06-14T23:12:43.721340+0000","last_became_active":"2024-06-14T23:10:26.046781+0000","last_became_peered":"2024-06-14T23:10:26.046781+0000","last_unstale":"2024-06-14T23:12:43.721340+0000","last_undegraded":"2024-06-14T23:12:43.721340+0000","last_fullsized":"2024-06-14T23:12:43.721340+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:07:15.892864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":553,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.721061+0000","last_change":"2024-06-14T23:10:39.253251+0000","last_active":"2024-06-14T23:12:43.721061+0000","last_peered":"2024-06-14T23:12:43.721061+0000","last_clean":"2024-06-14T23:12:43.721061+0000","last_became_active":"2024-06-14T23:10:29.061386+0000","last_became_peered":"2024-06-14T23:10:29.061386+0000","last_unstale":"2024-06-14T23:12:43.721061+0000","last_undegraded":"2024-06-14T23:12:43.721061+0000","last_fullsized":"2024-06-14T23:12:43.721061+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.735431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.735431+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"338'11720","reported_seq":11783,"reported_epoch":338,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:05.334476+0000","last_change":"2024-06-14T23:11:24.652983+0000","last_active":"2024-06-14T23:13:05.334476+0000","last_peered":"2024-06-14T23:13:05.334476+0000","last_clean":"2024-06-14T23:10:37.054302+0000","last_became_active":"2024-06-14T23:11:24.646468+0000","last_became_peered":"2024-06-14T23:11:24.646468+0000","last_unstale":"2024-06-14T23:13:05.334476+0000","last_undegraded":"2024-06-14T23:11:24.406142+0000","last_fullsized":"2024-06-14T23:11:24.380989+0000","mapping_epoch":323,"log_start":"324'7405","ondisk_log_start":"324'7405","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:33:08.769394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187432960,"num_objects":2860,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11720,"num_write_kb":466560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":392}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":298,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.715201+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:12:43.715201+0000","last_peered":"2024-06-14T23:12:43.715201+0000","last_clean":"2024-06-14T23:12:43.715201+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:12:43.715201+0000","last_undegraded":"2024-06-14T23:12:43.715201+0000","last_fullsized":"2024-06-14T23:12:43.715201+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":311,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.779209+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:12:54.779209+0000","last_peered":"2024-06-14T23:12:54.779209+0000","last_clean":"2024-06-14T23:12:54.779209+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:12:54.779209+0000","last_undegraded":"2024-06-14T23:12:54.779209+0000","last_fullsized":"2024-06-14T23:12:54.779209+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":96,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.720776+0000","last_change":"2024-06-14T23:10:26.058423+0000","last_active":"2024-06-14T23:12:43.720776+0000","last_peered":"2024-06-14T23:12:43.720776+0000","last_clean":"2024-06-14T23:12:43.720776+0000","last_became_active":"2024-06-14T23:10:26.050244+0000","last_became_peered":"2024-06-14T23:10:26.050244+0000","last_unstale":"2024-06-14T23:12:43.720776+0000","last_undegraded":"2024-06-14T23:12:43.720776+0000","last_fullsized":"2024-06-14T23:12:43.720776+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:52.060325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":663,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:50.413769+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:12:50.413769+0000","last_peered":"2024-06-14T23:12:50.413769+0000","last_clean":"2024-06-14T23:12:50.413769+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:12:50.413769+0000","last_undegraded":"2024-06-14T23:12:50.413769+0000","last_fullsized":"2024-06-14T23:12:50.413769+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"338'11852","reported_seq":11927,"reported_epoch":338,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:05.373840+0000","last_change":"2024-06-14T23:11:24.639438+0000","last_active":"2024-06-14T23:13:05.373840+0000","last_peered":"2024-06-14T23:13:05.373840+0000","last_clean":"2024-06-14T23:10:37.032734+0000","last_became_active":"2024-06-14T23:11:24.634884+0000","last_became_peered":"2024-06-14T23:11:24.634884+0000","last_unstale":"2024-06-14T23:13:05.373840+0000","last_undegraded":"2024-06-14T23:11:24.403675+0000","last_fullsized":"2024-06-14T23:11:24.380974+0000","mapping_epoch":323,"log_start":"324'7500","ondisk_log_start":"324'7500","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:48:19.382266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188088320,"num_objects":2870,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11852,"num_write_kb":471104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":251,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716826+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:12:43.716826+0000","last_peered":"2024-06-14T23:12:43.716826+0000","last_clean":"2024-06-14T23:12:43.716826+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:12:43.716826+0000","last_undegraded":"2024-06-14T23:12:43.716826+0000","last_fullsized":"2024-06-14T23:12:43.716826+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":301,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.721186+0000","last_change":"2024-06-14T23:10:26.051450+0000","last_active":"2024-06-14T23:12:43.721186+0000","last_peered":"2024-06-14T23:12:43.721186+0000","last_clean":"2024-06-14T23:12:43.721186+0000","last_became_active":"2024-06-14T23:10:26.050830+0000","last_became_peered":"2024-06-14T23:10:26.050830+0000","last_unstale":"2024-06-14T23:12:43.721186+0000","last_undegraded":"2024-06-14T23:12:43.721186+0000","last_fullsized":"2024-06-14T23:12:43.721186+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:16:16.175014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":52,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.708354+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:12:43.708354+0000","last_peered":"2024-06-14T23:12:43.708354+0000","last_clean":"2024-06-14T23:12:43.708354+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:12:43.708354+0000","last_undegraded":"2024-06-14T23:12:43.708354+0000","last_fullsized":"2024-06-14T23:12:43.708354+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":752,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.717123+0000","last_change":"2024-06-14T23:11:23.401974+0000","last_active":"2024-06-14T23:12:43.717123+0000","last_peered":"2024-06-14T23:12:43.717123+0000","last_clean":"2024-06-14T23:12:43.717123+0000","last_became_active":"2024-06-14T23:11:23.401625+0000","last_became_peered":"2024-06-14T23:11:23.401625+0000","last_unstale":"2024-06-14T23:12:43.717123+0000","last_undegraded":"2024-06-14T23:12:43.717123+0000","last_fullsized":"2024-06-14T23:12:43.717123+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.214300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.214300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:05:54.552362+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"338'11674","reported_seq":11738,"reported_epoch":338,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:05.358144+0000","last_change":"2024-06-14T23:11:24.651371+0000","last_active":"2024-06-14T23:13:05.358144+0000","last_peered":"2024-06-14T23:13:05.358144+0000","last_clean":"2024-06-14T23:10:37.020885+0000","last_became_active":"2024-06-14T23:11:24.646284+0000","last_became_peered":"2024-06-14T23:11:24.646284+0000","last_unstale":"2024-06-14T23:13:05.358144+0000","last_undegraded":"2024-06-14T23:11:24.405009+0000","last_fullsized":"2024-06-14T23:11:24.380990+0000","mapping_epoch":323,"log_start":"324'7342","ondisk_log_start":"324'7342","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:26:58.760076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195297280,"num_objects":2980,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11674,"num_write_kb":468928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":424}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":149,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.705254+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:12:43.705254+0000","last_peered":"2024-06-14T23:12:43.705254+0000","last_clean":"2024-06-14T23:12:43.705254+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:12:43.705254+0000","last_undegraded":"2024-06-14T23:12:43.705254+0000","last_fullsized":"2024-06-14T23:12:43.705254+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":323,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.708765+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:12:43.708765+0000","last_peered":"2024-06-14T23:12:43.708765+0000","last_clean":"2024-06-14T23:12:43.708765+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:12:43.708765+0000","last_undegraded":"2024-06-14T23:12:43.708765+0000","last_fullsized":"2024-06-14T23:12:43.708765+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":275,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777936+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:12:54.777936+0000","last_peered":"2024-06-14T23:12:54.777936+0000","last_clean":"2024-06-14T23:12:54.777936+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:12:54.777936+0000","last_undegraded":"2024-06-14T23:12:54.777936+0000","last_fullsized":"2024-06-14T23:12:54.777936+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":638,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777751+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:12:54.777751+0000","last_peered":"2024-06-14T23:12:54.777751+0000","last_clean":"2024-06-14T23:12:54.777751+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:12:54.777751+0000","last_undegraded":"2024-06-14T23:12:54.777751+0000","last_fullsized":"2024-06-14T23:12:54.777751+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"338'11820","reported_seq":16407,"reported_epoch":338,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:05.406003+0000","last_change":"2024-06-14T23:11:24.647584+0000","last_active":"2024-06-14T23:13:05.406003+0000","last_peered":"2024-06-14T23:13:05.406003+0000","last_clean":"2024-06-14T23:10:25.009797+0000","last_became_active":"2024-06-14T23:11:24.638765+0000","last_became_peered":"2024-06-14T23:11:24.638765+0000","last_unstale":"2024-06-14T23:13:05.406003+0000","last_undegraded":"2024-06-14T23:11:24.535343+0000","last_fullsized":"2024-06-14T23:11:24.381001+0000","mapping_epoch":323,"log_start":"324'7472","ondisk_log_start":"324'7472","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:37.543254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193462272,"num_objects":2952,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11820,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2251,"num_bytes_recovered":147521536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[3,2],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3,4","objects":2529},{"shards":"2,3","objects":-2181}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":114,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.704457+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:12:43.704457+0000","last_peered":"2024-06-14T23:12:43.704457+0000","last_clean":"2024-06-14T23:12:43.704457+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:12:43.704457+0000","last_undegraded":"2024-06-14T23:12:43.704457+0000","last_fullsized":"2024-06-14T23:12:43.704457+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777694+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:12:54.777694+0000","last_peered":"2024-06-14T23:12:54.777694+0000","last_clean":"2024-06-14T23:12:54.777694+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:12:54.777694+0000","last_undegraded":"2024-06-14T23:12:54.777694+0000","last_fullsized":"2024-06-14T23:12:54.777694+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":353,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.708586+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:12:43.708586+0000","last_peered":"2024-06-14T23:12:43.708586+0000","last_clean":"2024-06-14T23:12:43.708586+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:12:43.708586+0000","last_undegraded":"2024-06-14T23:12:43.708586+0000","last_fullsized":"2024-06-14T23:12:43.708586+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":618,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.779271+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:12:54.779271+0000","last_peered":"2024-06-14T23:12:54.779271+0000","last_clean":"2024-06-14T23:12:54.779271+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:12:54.779271+0000","last_undegraded":"2024-06-14T23:12:54.779271+0000","last_fullsized":"2024-06-14T23:12:54.779271+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"338'11646","reported_seq":16884,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:13:03.247054+0000","last_change":"2024-06-14T23:11:50.568053+0000","last_active":"2024-06-14T23:13:03.247054+0000","last_peered":"2024-06-14T23:13:03.247054+0000","last_clean":"2024-06-14T23:13:03.247054+0000","last_became_active":"2024-06-14T23:11:50.555786+0000","last_became_peered":"2024-06-14T23:11:50.555786+0000","last_unstale":"2024-06-14T23:13:03.247054+0000","last_undegraded":"2024-06-14T23:13:03.247054+0000","last_fullsized":"2024-06-14T23:13:03.247054+0000","mapping_epoch":328,"log_start":"318'6200","ondisk_log_start":"318'6200","created":295,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5446,"log_dups_size":0,"ondisk_log_size":5446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:36.382663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204996608,"num_objects":3128,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2575,"num_bytes_recovered":168755200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":290,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777272+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:12:54.777272+0000","last_peered":"2024-06-14T23:12:54.777272+0000","last_clean":"2024-06-14T23:12:54.777272+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:12:54.777272+0000","last_undegraded":"2024-06-14T23:12:54.777272+0000","last_fullsized":"2024-06-14T23:12:54.777272+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":336,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.721456+0000","last_change":"2024-06-14T23:10:26.049033+0000","last_active":"2024-06-14T23:12:43.721456+0000","last_peered":"2024-06-14T23:12:43.721456+0000","last_clean":"2024-06-14T23:12:43.721456+0000","last_became_active":"2024-06-14T23:10:26.047775+0000","last_became_peered":"2024-06-14T23:10:26.047775+0000","last_unstale":"2024-06-14T23:12:43.721456+0000","last_undegraded":"2024-06-14T23:12:43.721456+0000","last_fullsized":"2024-06-14T23:12:43.721456+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:15:52.051858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":330,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.704718+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:12:43.704718+0000","last_peered":"2024-06-14T23:12:43.704718+0000","last_clean":"2024-06-14T23:12:43.704718+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:12:43.704718+0000","last_undegraded":"2024-06-14T23:12:43.704718+0000","last_fullsized":"2024-06-14T23:12:43.704718+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":731,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.704092+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:12:43.704092+0000","last_peered":"2024-06-14T23:12:43.704092+0000","last_clean":"2024-06-14T23:12:43.704092+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:12:43.704092+0000","last_undegraded":"2024-06-14T23:12:43.704092+0000","last_fullsized":"2024-06-14T23:12:43.704092+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"339'11849","reported_seq":11922,"reported_epoch":339,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:06.735759+0000","last_change":"2024-06-14T23:11:24.656131+0000","last_active":"2024-06-14T23:13:06.735759+0000","last_peered":"2024-06-14T23:13:06.735759+0000","last_clean":"2024-06-14T23:10:37.043860+0000","last_became_active":"2024-06-14T23:11:24.647995+0000","last_became_peered":"2024-06-14T23:11:24.647995+0000","last_unstale":"2024-06-14T23:13:06.735759+0000","last_undegraded":"2024-06-14T23:11:24.403317+0000","last_fullsized":"2024-06-14T23:11:24.380506+0000","mapping_epoch":323,"log_start":"319'6900","ondisk_log_start":"319'6900","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4949,"log_dups_size":0,"ondisk_log_size":4949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:32:10.827694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186580992,"num_objects":2847,"num_object_clones":0,"num_object_copies":8541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11849,"num_write_kb":470272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":346}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":298,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.710706+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:12:43.710706+0000","last_peered":"2024-06-14T23:12:43.710706+0000","last_clean":"2024-06-14T23:12:43.710706+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:12:43.710706+0000","last_undegraded":"2024-06-14T23:12:43.710706+0000","last_fullsized":"2024-06-14T23:12:43.710706+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":335,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777802+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:12:54.777802+0000","last_peered":"2024-06-14T23:12:54.777802+0000","last_clean":"2024-06-14T23:12:54.777802+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:12:54.777802+0000","last_undegraded":"2024-06-14T23:12:54.777802+0000","last_fullsized":"2024-06-14T23:12:54.777802+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":140,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777690+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:12:54.777690+0000","last_peered":"2024-06-14T23:12:54.777690+0000","last_clean":"2024-06-14T23:12:54.777690+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:12:54.777690+0000","last_undegraded":"2024-06-14T23:12:54.777690+0000","last_fullsized":"2024-06-14T23:12:54.777690+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":625,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716411+0000","last_change":"2024-06-14T23:11:23.413224+0000","last_active":"2024-06-14T23:12:43.716411+0000","last_peered":"2024-06-14T23:12:43.716411+0000","last_clean":"2024-06-14T23:12:43.716411+0000","last_became_active":"2024-06-14T23:11:23.412680+0000","last_became_peered":"2024-06-14T23:11:23.412680+0000","last_unstale":"2024-06-14T23:12:43.716411+0000","last_undegraded":"2024-06-14T23:12:43.716411+0000","last_fullsized":"2024-06-14T23:12:43.716411+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:48:10.967651+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.e","version":"338'11472","reported_seq":11533,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:13:03.365200+0000","last_change":"2024-06-14T23:09:41.994118+0000","last_active":"2024-06-14T23:13:03.365200+0000","last_peered":"2024-06-14T23:13:03.365200+0000","last_clean":"2024-06-14T23:13:03.365200+0000","last_became_active":"2024-06-14T23:09:41.993702+0000","last_became_peered":"2024-06-14T23:09:41.993702+0000","last_unstale":"2024-06-14T23:13:03.365200+0000","last_undegraded":"2024-06-14T23:13:03.365200+0000","last_fullsized":"2024-06-14T23:13:03.365200+0000","mapping_epoch":299,"log_start":"318'6200","ondisk_log_start":"318'6200","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5272,"log_dups_size":0,"ondisk_log_size":5272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:25.994252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200540160,"num_objects":3060,"num_object_clones":0,"num_object_copies":9180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11472,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":308,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777920+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:12:54.777920+0000","last_peered":"2024-06-14T23:12:54.777920+0000","last_clean":"2024-06-14T23:12:54.777920+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:12:54.777920+0000","last_undegraded":"2024-06-14T23:12:54.777920+0000","last_fullsized":"2024-06-14T23:12:54.777920+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":186,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.715046+0000","last_change":"2024-06-14T23:09:41.991652+0000","last_active":"2024-06-14T23:12:43.715046+0000","last_peered":"2024-06-14T23:12:43.715046+0000","last_clean":"2024-06-14T23:12:43.715046+0000","last_became_active":"2024-06-14T23:09:41.991317+0000","last_became_peered":"2024-06-14T23:09:41.991317+0000","last_unstale":"2024-06-14T23:12:43.715046+0000","last_undegraded":"2024-06-14T23:12:43.715046+0000","last_fullsized":"2024-06-14T23:12:43.715046+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:16:18.209329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":303,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716722+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:12:43.716722+0000","last_peered":"2024-06-14T23:12:43.716722+0000","last_clean":"2024-06-14T23:12:43.716722+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:12:43.716722+0000","last_undegraded":"2024-06-14T23:12:43.716722+0000","last_fullsized":"2024-06-14T23:12:43.716722+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":545,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.719329+0000","last_change":"2024-06-14T23:10:29.063981+0000","last_active":"2024-06-14T23:12:43.719329+0000","last_peered":"2024-06-14T23:12:43.719329+0000","last_clean":"2024-06-14T23:12:43.719329+0000","last_became_active":"2024-06-14T23:10:29.063546+0000","last_became_peered":"2024-06-14T23:10:29.063546+0000","last_unstale":"2024-06-14T23:12:43.719329+0000","last_undegraded":"2024-06-14T23:12:43.719329+0000","last_fullsized":"2024-06-14T23:12:43.719329+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:26:23.411322+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"338'11562","reported_seq":20020,"reported_epoch":338,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:03.748623+0000","last_change":"2024-06-14T23:11:24.664901+0000","last_active":"2024-06-14T23:13:03.748623+0000","last_peered":"2024-06-14T23:13:03.748623+0000","last_clean":"2024-06-14T23:10:24.989118+0000","last_became_active":"2024-06-14T23:11:24.647313+0000","last_became_peered":"2024-06-14T23:11:24.647313+0000","last_unstale":"2024-06-14T23:13:03.748623+0000","last_undegraded":"2024-06-14T23:11:24.555289+0000","last_fullsized":"2024-06-14T23:11:24.381668+0000","mapping_epoch":323,"log_start":"319'6700","ondisk_log_start":"319'6700","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4862,"log_dups_size":0,"ondisk_log_size":4862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:48:12.784805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197328920,"num_objects":3012,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11562,"num_write_kb":466305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4187,"num_bytes_recovered":274399232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"1,3,7","objects":2484},{"shards":"3,7","objects":528}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":254,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777968+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:12:54.777968+0000","last_peered":"2024-06-14T23:12:54.777968+0000","last_clean":"2024-06-14T23:12:54.777968+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:12:54.777968+0000","last_undegraded":"2024-06-14T23:12:54.777968+0000","last_fullsized":"2024-06-14T23:12:54.777968+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":259,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.704646+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:12:43.704646+0000","last_peered":"2024-06-14T23:12:43.704646+0000","last_clean":"2024-06-14T23:12:43.704646+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:12:43.704646+0000","last_undegraded":"2024-06-14T23:12:43.704646+0000","last_fullsized":"2024-06-14T23:12:43.704646+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":340,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.705316+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:12:43.705316+0000","last_peered":"2024-06-14T23:12:43.705316+0000","last_clean":"2024-06-14T23:12:43.705316+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:12:43.705316+0000","last_undegraded":"2024-06-14T23:12:43.705316+0000","last_fullsized":"2024-06-14T23:12:43.705316+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":618,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777953+0000","last_change":"2024-06-14T23:11:23.406149+0000","last_active":"2024-06-14T23:12:54.777953+0000","last_peered":"2024-06-14T23:12:54.777953+0000","last_clean":"2024-06-14T23:12:54.777953+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:12:54.777953+0000","last_undegraded":"2024-06-14T23:12:54.777953+0000","last_fullsized":"2024-06-14T23:12:54.777953+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.751667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.751667+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.7","version":"338'11492","reported_seq":19989,"reported_epoch":338,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-14T23:13:03.784061+0000","last_change":"2024-06-14T23:11:24.669886+0000","last_active":"2024-06-14T23:13:03.784061+0000","last_peered":"2024-06-14T23:13:03.784061+0000","last_clean":"2024-06-14T23:10:25.005348+0000","last_became_active":"2024-06-14T23:11:24.634913+0000","last_became_peered":"2024-06-14T23:11:24.634913+0000","last_unstale":"2024-06-14T23:13:03.784061+0000","last_undegraded":"2024-06-14T23:13:03.784061+0000","last_fullsized":"2024-06-14T23:11:24.386154+0000","mapping_epoch":323,"log_start":"319'6600","ondisk_log_start":"319'6600","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4892,"log_dups_size":0,"ondisk_log_size":4892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:26:38.637559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197263360,"num_objects":3010,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2443,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11492,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4208,"num_bytes_recovered":275775488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"1,4,7","objects":2443},{"shards":"4,7","objects":567}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":136,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.721617+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:12:43.721617+0000","last_peered":"2024-06-14T23:12:43.721617+0000","last_clean":"2024-06-14T23:12:43.721617+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:12:43.721617+0000","last_undegraded":"2024-06-14T23:12:43.721617+0000","last_fullsized":"2024-06-14T23:12:43.721617+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":307,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716951+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:12:43.716951+0000","last_peered":"2024-06-14T23:12:43.716951+0000","last_clean":"2024-06-14T23:12:43.716951+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:12:43.716951+0000","last_undegraded":"2024-06-14T23:12:43.716951+0000","last_fullsized":"2024-06-14T23:12:43.716951+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":729,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.704300+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:12:43.704300+0000","last_peered":"2024-06-14T23:12:43.704300+0000","last_clean":"2024-06-14T23:12:43.704300+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:12:43.704300+0000","last_undegraded":"2024-06-14T23:12:43.704300+0000","last_fullsized":"2024-06-14T23:12:43.704300+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"339'11521","reported_seq":12042,"reported_epoch":339,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:06.745072+0000","last_change":"2024-06-14T23:11:24.664127+0000","last_active":"2024-06-14T23:13:06.745072+0000","last_peered":"2024-06-14T23:13:06.745072+0000","last_clean":"2024-06-14T23:10:25.013628+0000","last_became_active":"2024-06-14T23:10:27.203384+0000","last_became_peered":"2024-06-14T23:10:27.203384+0000","last_unstale":"2024-06-14T23:13:06.745072+0000","last_undegraded":"2024-06-14T23:11:24.661184+0000","last_fullsized":"2024-06-14T23:10:27.026246+0000","mapping_epoch":310,"log_start":"319'6600","ondisk_log_start":"319'6600","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:40.932486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180289536,"num_objects":2751,"num_object_clones":0,"num_object_copies":8253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11521,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":14548992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"6,7","objects":-1180}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":325,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.721126+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:12:43.721126+0000","last_peered":"2024-06-14T23:12:43.721126+0000","last_clean":"2024-06-14T23:12:43.721126+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:12:43.721126+0000","last_undegraded":"2024-06-14T23:12:43.721126+0000","last_fullsized":"2024-06-14T23:12:43.721126+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":336,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.715428+0000","last_change":"2024-06-14T23:11:23.427812+0000","last_active":"2024-06-14T23:12:43.715428+0000","last_peered":"2024-06-14T23:12:43.715428+0000","last_clean":"2024-06-14T23:12:43.715428+0000","last_became_active":"2024-06-14T23:11:23.427563+0000","last_became_peered":"2024-06-14T23:11:23.427563+0000","last_unstale":"2024-06-14T23:12:43.715428+0000","last_undegraded":"2024-06-14T23:12:43.715428+0000","last_fullsized":"2024-06-14T23:12:43.715428+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.941006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":697,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.705175+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:12:43.705175+0000","last_peered":"2024-06-14T23:12:43.705175+0000","last_clean":"2024-06-14T23:12:43.705175+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:12:43.705175+0000","last_undegraded":"2024-06-14T23:12:43.705175+0000","last_fullsized":"2024-06-14T23:12:43.705175+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"339'11560","reported_seq":19940,"reported_epoch":339,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:06.751767+0000","last_change":"2024-06-14T23:11:24.656651+0000","last_active":"2024-06-14T23:13:06.751767+0000","last_peered":"2024-06-14T23:13:06.751767+0000","last_clean":"2024-06-14T23:10:25.016384+0000","last_became_active":"2024-06-14T23:11:24.643623+0000","last_became_peered":"2024-06-14T23:11:24.643623+0000","last_unstale":"2024-06-14T23:13:06.751767+0000","last_undegraded":"2024-06-14T23:11:24.519360+0000","last_fullsized":"2024-06-14T23:11:24.380111+0000","mapping_epoch":323,"log_start":"319'6600","ondisk_log_start":"319'6600","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4960,"log_dups_size":0,"ondisk_log_size":4960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:18:43.920069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182321152,"num_objects":2782,"num_object_clones":0,"num_object_copies":8346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4148,"num_bytes_recovered":271843328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"1,3,6","objects":2461},{"shards":"3,6","objects":321}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":355,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.717292+0000","last_change":"2024-06-14T23:11:23.416264+0000","last_active":"2024-06-14T23:12:43.717292+0000","last_peered":"2024-06-14T23:12:43.717292+0000","last_clean":"2024-06-14T23:12:43.717292+0000","last_became_active":"2024-06-14T23:11:23.415946+0000","last_became_peered":"2024-06-14T23:11:23.415946+0000","last_unstale":"2024-06-14T23:12:43.717292+0000","last_undegraded":"2024-06-14T23:12:43.717292+0000","last_fullsized":"2024-06-14T23:12:43.717292+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:22.668569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":206,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716628+0000","last_change":"2024-06-14T23:10:26.045363+0000","last_active":"2024-06-14T23:12:43.716628+0000","last_peered":"2024-06-14T23:12:43.716628+0000","last_clean":"2024-06-14T23:12:43.716628+0000","last_became_active":"2024-06-14T23:10:26.043844+0000","last_became_peered":"2024-06-14T23:10:26.043844+0000","last_unstale":"2024-06-14T23:12:43.716628+0000","last_undegraded":"2024-06-14T23:12:43.716628+0000","last_fullsized":"2024-06-14T23:12:43.716628+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:11:27.891184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":720,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.703655+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:12:43.703655+0000","last_peered":"2024-06-14T23:12:43.703655+0000","last_clean":"2024-06-14T23:12:43.703655+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:12:43.703655+0000","last_undegraded":"2024-06-14T23:12:43.703655+0000","last_fullsized":"2024-06-14T23:12:43.703655+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"339'11681","reported_seq":11975,"reported_epoch":339,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:06.685136+0000","last_change":"2024-06-14T23:10:39.484566+0000","last_active":"2024-06-14T23:13:06.685136+0000","last_peered":"2024-06-14T23:13:06.685136+0000","last_clean":"2024-06-14T23:10:24.998059+0000","last_became_active":"2024-06-14T23:10:27.239025+0000","last_became_peered":"2024-06-14T23:10:27.239025+0000","last_unstale":"2024-06-14T23:13:06.685136+0000","last_undegraded":"2024-06-14T23:10:27.071395+0000","last_fullsized":"2024-06-14T23:10:27.025672+0000","mapping_epoch":310,"log_start":"319'6700","ondisk_log_start":"319'6700","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:52:40.136978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188284928,"num_objects":2873,"num_object_clones":0,"num_object_copies":8619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11681,"num_write_kb":465728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":7602176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"3,6","objects":-1180}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":315,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716696+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:12:43.716696+0000","last_peered":"2024-06-14T23:12:43.716696+0000","last_clean":"2024-06-14T23:12:43.716696+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:12:43.716696+0000","last_undegraded":"2024-06-14T23:12:43.716696+0000","last_fullsized":"2024-06-14T23:12:43.716696+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":136,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.720073+0000","last_change":"2024-06-14T23:10:26.056252+0000","last_active":"2024-06-14T23:12:43.720073+0000","last_peered":"2024-06-14T23:12:43.720073+0000","last_clean":"2024-06-14T23:12:43.720073+0000","last_became_active":"2024-06-14T23:10:26.046629+0000","last_became_peered":"2024-06-14T23:10:26.046629+0000","last_unstale":"2024-06-14T23:12:43.720073+0000","last_undegraded":"2024-06-14T23:12:43.720073+0000","last_fullsized":"2024-06-14T23:12:43.720073+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:51:19.277386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":609,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777298+0000","last_change":"2024-06-14T23:11:23.428346+0000","last_active":"2024-06-14T23:12:54.777298+0000","last_peered":"2024-06-14T23:12:54.777298+0000","last_clean":"2024-06-14T23:12:54.777298+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:12:54.777298+0000","last_undegraded":"2024-06-14T23:12:54.777298+0000","last_fullsized":"2024-06-14T23:12:54.777298+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.656210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.656210+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"338'11586","reported_seq":11647,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:13:03.714021+0000","last_change":"2024-06-14T23:09:41.992020+0000","last_active":"2024-06-14T23:13:03.714021+0000","last_peered":"2024-06-14T23:13:03.714021+0000","last_clean":"2024-06-14T23:13:03.714021+0000","last_became_active":"2024-06-14T23:09:41.991722+0000","last_became_peered":"2024-06-14T23:09:41.991722+0000","last_unstale":"2024-06-14T23:13:03.714021+0000","last_undegraded":"2024-06-14T23:13:03.714021+0000","last_fullsized":"2024-06-14T23:13:03.714021+0000","mapping_epoch":299,"log_start":"319'6700","ondisk_log_start":"319'6700","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:35:54.751263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197394432,"num_objects":3012,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11586,"num_write_kb":467136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":135,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.719983+0000","last_change":"2024-06-14T23:10:26.047111+0000","last_active":"2024-06-14T23:12:43.719983+0000","last_peered":"2024-06-14T23:12:43.719983+0000","last_clean":"2024-06-14T23:12:43.719983+0000","last_became_active":"2024-06-14T23:10:26.045540+0000","last_became_peered":"2024-06-14T23:10:26.045540+0000","last_unstale":"2024-06-14T23:12:43.719983+0000","last_undegraded":"2024-06-14T23:12:43.719983+0000","last_fullsized":"2024-06-14T23:12:43.719983+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:04.131400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":316,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.708451+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:12:43.708451+0000","last_peered":"2024-06-14T23:12:43.708451+0000","last_clean":"2024-06-14T23:12:43.708451+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:12:43.708451+0000","last_undegraded":"2024-06-14T23:12:43.708451+0000","last_fullsized":"2024-06-14T23:12:43.708451+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":576,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.720696+0000","last_change":"2024-06-14T23:09:17.382466+0000","last_active":"2024-06-14T23:12:43.720696+0000","last_peered":"2024-06-14T23:12:43.720696+0000","last_clean":"2024-06-14T23:12:43.720696+0000","last_became_active":"2024-06-14T23:09:12.030205+0000","last_became_peered":"2024-06-14T23:09:12.030205+0000","last_unstale":"2024-06-14T23:12:43.720696+0000","last_undegraded":"2024-06-14T23:12:43.720696+0000","last_fullsized":"2024-06-14T23:12:43.720696+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:09:17.382389+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:33.121472+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"338'11250","reported_seq":11301,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:13:03.397196+0000","last_change":"2024-06-14T23:09:29.408859+0000","last_active":"2024-06-14T23:13:03.397196+0000","last_peered":"2024-06-14T23:13:03.397196+0000","last_clean":"2024-06-14T23:13:03.397196+0000","last_became_active":"2024-06-14T23:09:29.408548+0000","last_became_peered":"2024-06-14T23:09:29.408548+0000","last_unstale":"2024-06-14T23:13:03.397196+0000","last_undegraded":"2024-06-14T23:13:03.397196+0000","last_fullsized":"2024-06-14T23:13:03.397196+0000","mapping_epoch":295,"log_start":"318'5900","ondisk_log_start":"318'5900","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5350,"log_dups_size":0,"ondisk_log_size":5350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:19.810173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197394432,"num_objects":3012,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":456384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":290,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777234+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:12:54.777234+0000","last_peered":"2024-06-14T23:12:54.777234+0000","last_clean":"2024-06-14T23:12:54.777234+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:12:54.777234+0000","last_undegraded":"2024-06-14T23:12:54.777234+0000","last_fullsized":"2024-06-14T23:12:54.777234+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":233,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.708346+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:12:43.708346+0000","last_peered":"2024-06-14T23:12:43.708346+0000","last_clean":"2024-06-14T23:12:43.708346+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:12:43.708346+0000","last_undegraded":"2024-06-14T23:12:43.708346+0000","last_fullsized":"2024-06-14T23:12:43.708346+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":618,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.779333+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:12:54.779333+0000","last_peered":"2024-06-14T23:12:54.779333+0000","last_clean":"2024-06-14T23:12:54.779333+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:12:54.779333+0000","last_undegraded":"2024-06-14T23:12:54.779333+0000","last_fullsized":"2024-06-14T23:12:54.779333+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"339'11756","reported_seq":11808,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:06.734552+0000","last_change":"2024-06-14T23:09:29.403408+0000","last_active":"2024-06-14T23:13:06.734552+0000","last_peered":"2024-06-14T23:13:06.734552+0000","last_clean":"2024-06-14T23:13:06.734552+0000","last_became_active":"2024-06-14T23:09:29.403000+0000","last_became_peered":"2024-06-14T23:09:29.403000+0000","last_unstale":"2024-06-14T23:13:06.734552+0000","last_undegraded":"2024-06-14T23:13:06.734552+0000","last_fullsized":"2024-06-14T23:13:06.734552+0000","mapping_epoch":295,"log_start":"319'6800","ondisk_log_start":"319'6800","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:16.838865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181141504,"num_objects":2764,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11756,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":325,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.715366+0000","last_change":"2024-06-14T23:10:26.040201+0000","last_active":"2024-06-14T23:12:43.715366+0000","last_peered":"2024-06-14T23:12:43.715366+0000","last_clean":"2024-06-14T23:12:43.715366+0000","last_became_active":"2024-06-14T23:10:26.039865+0000","last_became_peered":"2024-06-14T23:10:26.039865+0000","last_unstale":"2024-06-14T23:12:43.715366+0000","last_undegraded":"2024-06-14T23:12:43.715366+0000","last_fullsized":"2024-06-14T23:12:43.715366+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:34.621202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":303,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.706694+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:12:43.706694+0000","last_peered":"2024-06-14T23:12:43.706694+0000","last_clean":"2024-06-14T23:12:43.706694+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:12:43.706694+0000","last_undegraded":"2024-06-14T23:12:43.706694+0000","last_fullsized":"2024-06-14T23:12:43.706694+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":745,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.717696+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:12:43.717696+0000","last_peered":"2024-06-14T23:12:43.717696+0000","last_clean":"2024-06-14T23:12:43.717696+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:12:43.717696+0000","last_undegraded":"2024-06-14T23:12:43.717696+0000","last_fullsized":"2024-06-14T23:12:43.717696+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"338'11595","reported_seq":11663,"reported_epoch":338,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:03.378466+0000","last_change":"2024-06-14T23:11:24.632384+0000","last_active":"2024-06-14T23:13:03.378466+0000","last_peered":"2024-06-14T23:13:03.378466+0000","last_clean":"2024-06-14T23:10:37.053680+0000","last_became_active":"2024-06-14T23:11:24.623684+0000","last_became_peered":"2024-06-14T23:11:24.623684+0000","last_unstale":"2024-06-14T23:13:03.378466+0000","last_undegraded":"2024-06-14T23:11:24.400366+0000","last_fullsized":"2024-06-14T23:11:24.379058+0000","mapping_epoch":323,"log_start":"319'6300","ondisk_log_start":"319'6300","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:34:53.331507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200343552,"num_objects":3057,"num_object_clones":0,"num_object_copies":9171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11595,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":503}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":298,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.703838+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:12:43.703838+0000","last_peered":"2024-06-14T23:12:43.703838+0000","last_clean":"2024-06-14T23:12:43.703838+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:12:43.703838+0000","last_undegraded":"2024-06-14T23:12:43.703838+0000","last_fullsized":"2024-06-14T23:12:43.703838+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":316,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.715102+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:12:43.715102+0000","last_peered":"2024-06-14T23:12:43.715102+0000","last_clean":"2024-06-14T23:12:43.715102+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:12:43.715102+0000","last_undegraded":"2024-06-14T23:12:43.715102+0000","last_fullsized":"2024-06-14T23:12:43.715102+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":771,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.716987+0000","last_change":"2024-06-14T23:11:23.418239+0000","last_active":"2024-06-14T23:12:43.716987+0000","last_peered":"2024-06-14T23:12:43.716987+0000","last_clean":"2024-06-14T23:12:43.716987+0000","last_became_active":"2024-06-14T23:11:23.417873+0000","last_became_peered":"2024-06-14T23:11:23.417873+0000","last_unstale":"2024-06-14T23:12:43.716987+0000","last_undegraded":"2024-06-14T23:12:43.716987+0000","last_fullsized":"2024-06-14T23:12:43.716987+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:04:50.740175+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":325,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.777597+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:12:54.777597+0000","last_peered":"2024-06-14T23:12:54.777597+0000","last_clean":"2024-06-14T23:12:54.777597+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:12:54.777597+0000","last_undegraded":"2024-06-14T23:12:54.777597+0000","last_fullsized":"2024-06-14T23:12:54.777597+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":320,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.719888+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:12:43.719888+0000","last_peered":"2024-06-14T23:12:43.719888+0000","last_clean":"2024-06-14T23:12:43.719888+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:12:43.719888+0000","last_undegraded":"2024-06-14T23:12:43.719888+0000","last_fullsized":"2024-06-14T23:12:43.719888+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":693,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.704381+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:12:43.704381+0000","last_peered":"2024-06-14T23:12:43.704381+0000","last_clean":"2024-06-14T23:12:43.704381+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:12:43.704381+0000","last_undegraded":"2024-06-14T23:12:43.704381+0000","last_fullsized":"2024-06-14T23:12:43.704381+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":222,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.715125+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:12:43.715125+0000","last_peered":"2024-06-14T23:12:43.715125+0000","last_clean":"2024-06-14T23:12:43.715125+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:12:43.715125+0000","last_undegraded":"2024-06-14T23:12:43.715125+0000","last_fullsized":"2024-06-14T23:12:43.715125+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":201,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.720072+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:12:43.720072+0000","last_peered":"2024-06-14T23:12:43.720072+0000","last_clean":"2024-06-14T23:12:43.720072+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:12:43.720072+0000","last_undegraded":"2024-06-14T23:12:43.720072+0000","last_fullsized":"2024-06-14T23:12:43.720072+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":659,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.717181+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:12:43.717181+0000","last_peered":"2024-06-14T23:12:43.717181+0000","last_clean":"2024-06-14T23:12:43.717181+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:12:43.717181+0000","last_undegraded":"2024-06-14T23:12:43.717181+0000","last_fullsized":"2024-06-14T23:12:43.717181+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":40,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.778782+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:12:54.778782+0000","last_peered":"2024-06-14T23:12:54.778782+0000","last_clean":"2024-06-14T23:12:54.778782+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:12:54.778782+0000","last_undegraded":"2024-06-14T23:12:54.778782+0000","last_fullsized":"2024-06-14T23:12:54.778782+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":298,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.708425+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:12:43.708425+0000","last_peered":"2024-06-14T23:12:43.708425+0000","last_clean":"2024-06-14T23:12:43.708425+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:12:43.708425+0000","last_undegraded":"2024-06-14T23:12:43.708425+0000","last_fullsized":"2024-06-14T23:12:43.708425+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":657,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.703587+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:12:43.703587+0000","last_peered":"2024-06-14T23:12:43.703587+0000","last_clean":"2024-06-14T23:12:43.703587+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:12:43.703587+0000","last_undegraded":"2024-06-14T23:12:43.703587+0000","last_fullsized":"2024-06-14T23:12:43.703587+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":350,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.706114+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:12:43.706114+0000","last_peered":"2024-06-14T23:12:43.706114+0000","last_clean":"2024-06-14T23:12:43.706114+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:12:43.706114+0000","last_undegraded":"2024-06-14T23:12:43.706114+0000","last_fullsized":"2024-06-14T23:12:43.706114+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":40,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-06-14T23:12:54.778097+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:12:54.778097+0000","last_peered":"2024-06-14T23:12:54.778097+0000","last_clean":"2024-06-14T23:12:54.778097+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:12:54.778097+0000","last_undegraded":"2024-06-14T23:12:54.778097+0000","last_fullsized":"2024-06-14T23:12:54.778097+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":770,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-06-14T23:12:43.704023+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:12:43.704023+0000","last_peered":"2024-06-14T23:12:43.704023+0000","last_clean":"2024-06-14T23:12:43.704023+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:12:43.704023+0000","last_undegraded":"2024-06-14T23:12:43.704023+0000","last_fullsized":"2024-06-14T23:12:43.704023+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":3078160408,"num_objects":46970,"num_object_clones":0,"num_object_copies":140910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30661,"num_objects_misplaced":2443,"num_objects_unfound":0,"num_objects_dirty":46970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186036,"num_write_kb":7456129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17708,"num_bytes_recovered":1160511488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10024529920,"data_stored":10024517704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78117,"ondisk_log_size":78117,"up":48,"acting":37,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":821296,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":103,"num_write":26,"num_write_kb":273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":365100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234468,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2948708,"kb_used_data":1761244,"kb_used_omap":1,"kb_used_meta":1187326,"kb_avail":90796444,"statfs":{"total":95995035648,"available":92975558656,"internally_reserved":0,"allocated":1803513856,"data_stored":1801758998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1215822168},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":0.95599999999999996,"15min":0.99099999999999999},"min":{"1min":0.24199999999999999,"5min":0.218,"15min":0.19800000000000001},"max":{"1min":5.7229999999999999,"5min":5.7229999999999999,"15min":6.9539999999999997},"last":1.923},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.169,"15min":1.1950000000000001},"min":{"1min":0.29599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":5.5350000000000001,"5min":5.5350000000000001,"15min":6.9550000000000001},"last":1.492}]},{"osd":1,"last update":"Fri Jun 14 23:12:11 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4410000000000001,"15min":1.3180000000000001},"min":{"1min":0.38700000000000001,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":3.9460000000000002,"5min":12.010999999999999,"15min":12.010999999999999},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4219999999999999,"15min":1.345},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.22900000000000001},"max":{"1min":3.738,"5min":13.531000000000001,"15min":13.531000000000001},"last":1.899}]},{"osd":2,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.1970000000000001,"5min":5.1970000000000001,"15min":5.1970000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":0.83399999999999996}]},{"osd":3,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5609999999999999,"15min":1.4179999999999999},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":5.2270000000000003,"5min":20.645,"15min":20.645},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.625,"15min":1.494},"min":{"1min":0.36499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.3639999999999999,"5min":22.042999999999999,"15min":22.042999999999999},"last":1.577}]},{"osd":4,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.127,"15min":1.077},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.3319999999999999,"5min":5.3319999999999999,"15min":11.342000000000001},"last":1.609},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.208,"15min":1.1459999999999999},"min":{"1min":0.32600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3769999999999998,"5min":8.5899999999999999,"15min":16.879000000000001},"last":1.9770000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.026,"15min":1.109},"min":{"1min":0.39100000000000001,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":5.6849999999999996,"5min":5.6849999999999996,"15min":6.9000000000000004},"last":1.859},{"interface":"front","average":{"1min":1.407,"5min":1.222,"15min":1.1499999999999999},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.3949999999999996,"5min":5.3949999999999996,"15min":6.8529999999999998},"last":1.526}]},{"osd":6,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":2.137,"5min":1.7190000000000001,"15min":1.448},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.7750000000000004,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":1.966},{"interface":"front","average":{"1min":3.0219999999999998,"5min":1.694,"15min":1.486},"min":{"1min":0.48199999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.087999999999999,"5min":10.087999999999999,"15min":10.087999999999999},"last":1.6319999999999999}]}]},{"osd":6,"up_from":272,"seq":1168231104575,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2047396,"kb_used_data":1650264,"kb_used_omap":7,"kb_used_meta":397048,"kb_avail":91697756,"statfs":{"total":95995035648,"available":93898502144,"internally_reserved":0,"allocated":1689870336,"data_stored":1688127673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":406577594},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.2450000000000001,"15min":1.6659999999999999},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.1190000000000002,"5min":6.6440000000000001,"15min":6.6440000000000001},"last":1.9219999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.3089999999999999,"15min":1.7190000000000001},"min":{"1min":0.48599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.484,"5min":6.6790000000000003,"15min":6.6790000000000003},"last":2.0129999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.5900000000000001,"15min":2.1800000000000002},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.4660000000000002,"5min":8.173,"15min":8.173},"last":0.441},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.506,"15min":2.0859999999999999},"min":{"1min":0.36799999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.5209999999999999,"5min":10.380000000000001,"15min":10.380000000000001},"last":1.9850000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:12:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":3.4489999999999998,"5min":3.4489999999999998,"15min":3.4489999999999998},"last":2.036},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":2.069}]},{"osd":3,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.4279999999999999,"15min":1.827},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.5779999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"last":1.956},{"interface":"front","average":{"1min":1.363,"5min":1.7190000000000001,"15min":2.0219999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.6239999999999997,"5min":8.4740000000000002,"15min":8.4740000000000002},"last":2.1070000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.3520000000000001,"15min":1.4259999999999999},"min":{"1min":0.28999999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":6.8920000000000003,"5min":11.010999999999999,"15min":11.010999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.1599999999999999,"15min":1.266},"min":{"1min":0.39400000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.605,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":2.4910000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.139,"15min":1.371},"min":{"1min":0.34000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":3.5510000000000002,"5min":7.7469999999999999,"15min":7.7469999999999999},"last":2.464},{"interface":"front","average":{"1min":1.472,"5min":1.48,"15min":1.6020000000000001},"min":{"1min":0.31900000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":5.5810000000000004,"5min":7.8680000000000003,"15min":7.8680000000000003},"last":4.9000000000000004}]},{"osd":7,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":1.6930000000000001,"15min":1.7410000000000001},"min":{"1min":0.36499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.5419999999999998,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":2.8570000000000002},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.429,"15min":1.5389999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.9219999999999997,"5min":7.4619999999999997,"15min":7.4619999999999997},"last":1.2569999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757743,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1589900,"kb_used_data":406276,"kb_used_omap":1,"kb_used_meta":1183614,"kb_avail":92155252,"statfs":{"total":95995035648,"available":94366978048,"internally_reserved":0,"allocated":416026624,"data_stored":414291685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1212021080},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.70299999999999996,"15min":1.2949999999999999},"min":{"1min":0.255,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":1.5189999999999999,"5min":4.0579999999999998,"15min":14.94},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.88200000000000001,"15min":1.3149999999999999},"min":{"1min":0.34799999999999998,"5min":0.26100000000000001,"15min":0.219},"max":{"1min":1.8680000000000001,"5min":8.3520000000000003,"15min":9.3819999999999997},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.91900000000000004,"15min":0.99099999999999999},"min":{"1min":0.28999999999999998,"5min":0.245,"15min":0.23999999999999999},"max":{"1min":1.7509999999999999,"5min":6.9480000000000004,"15min":7.4370000000000003},"last":1.8740000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.96599999999999997,"15min":1.23},"min":{"1min":0.27400000000000002,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":3.0819999999999999,"5min":5.1920000000000002,"15min":9.5039999999999996},"last":0.28399999999999997}]},{"osd":2,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.0579999999999998,"5min":3.0579999999999998,"15min":3.0579999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.125,"5min":3.125,"15min":3.125},"last":0.38400000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:12:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.081,"15min":1.3979999999999999},"min":{"1min":0.31,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":1.9370000000000001,"5min":8.4350000000000005,"15min":10.358000000000001},"last":0.432},{"interface":"front","average":{"1min":0.629,"5min":1.006,"15min":1.4159999999999999},"min":{"1min":0.33600000000000002,"5min":0.318,"15min":0.28899999999999998},"max":{"1min":1.585,"5min":7.4290000000000003,"15min":14.914},"last":0.35099999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.71599999999999997,"15min":0.89600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.1719999999999999,"5min":3.012,"15min":14.352},"last":1.8100000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.80500000000000005,"15min":1.0489999999999999},"min":{"1min":0.34100000000000003,"5min":0.28799999999999998,"15min":0.253},"max":{"1min":1.9950000000000001,"5min":8.4039999999999999,"15min":9.5980000000000008},"last":0.56799999999999995}]},{"osd":6,"last update":"Fri Jun 14 23:12:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.079,"15min":1.389},"min":{"1min":0.29599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.905,"5min":9.4710000000000001,"15min":9.4710000000000001},"last":4.0629999999999997},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.3600000000000001,"15min":1.746},"min":{"1min":0.379,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.1630000000000003,"5min":10.632,"15min":10.632},"last":0.41199999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.256,"15min":1.246},"min":{"1min":0.39100000000000001,"5min":0.29099999999999998,"15min":0.27900000000000003},"max":{"1min":26.981999999999999,"5min":26.981999999999999,"15min":26.981999999999999},"last":1.5209999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":1.333,"15min":1.222},"min":{"1min":0.42499999999999999,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":23.326000000000001,"5min":23.326000000000001,"15min":23.326000000000001},"last":1.8460000000000001}]}]},{"osd":4,"up_from":30,"seq":128849019379,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2079892,"kb_used_data":1193292,"kb_used_omap":1,"kb_used_meta":886590,"kb_avail":91665260,"statfs":{"total":95995035648,"available":93865226240,"internally_reserved":0,"allocated":1221931008,"data_stored":1220191631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907868504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.83699999999999997,"15min":1.1459999999999999},"min":{"1min":0.26500000000000001,"5min":0.217,"15min":0.17100000000000001},"max":{"1min":3.177,"5min":6.875,"15min":10.567},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.81200000000000006,"15min":1.1120000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":3.323,"5min":6.1379999999999999,"15min":8.3390000000000004},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.339,"15min":1.482},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23100000000000001},"max":{"1min":3.3759999999999999,"5min":42.656999999999996,"15min":56.917000000000002},"last":0.315},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.3109999999999999,"15min":1.5029999999999999},"min":{"1min":0.317,"5min":0.249,"15min":0.249},"max":{"1min":3.2090000000000001,"5min":42.192999999999998,"15min":56.789999999999999},"last":0.48999999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:12:23 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":0.378},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Jun 14 23:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.6930000000000001,"15min":1.3360000000000001},"min":{"1min":0.316,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":3.3809999999999998,"5min":44.389000000000003,"15min":44.389000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.8260000000000001,"15min":1.5},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":3.1000000000000001,"5min":46.101999999999997,"15min":46.101999999999997},"last":0.54000000000000004}]},{"osd":5,"last update":"Fri Jun 14 23:12:36 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.85799999999999998,"15min":1.5529999999999999},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.4159999999999999,"5min":4.04,"15min":139.13200000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":1.613},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.2360000000000002,"5min":6.9290000000000003,"15min":138.86000000000001},"last":0.45900000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:12:06 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.117,"15min":1.105},"min":{"1min":0.36599999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.46,"5min":6.9989999999999997,"15min":6.9989999999999997},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.2230000000000001,"15min":1.5069999999999999},"min":{"1min":0.32200000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.1070000000000002,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Jun 14 23:12:17 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.1859999999999999,"15min":1.087},"min":{"1min":0.33800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.0439999999999996,"5min":6.5910000000000002,"15min":8.9489999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.583,"5min":1.2210000000000001,"15min":1.1659999999999999},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.4219999999999997,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":0.65900000000000003}]}]},{"osd":3,"up_from":138,"seq":592705487078,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3131976,"kb_used_data":2274688,"kb_used_omap":4,"kb_used_meta":857275,"kb_avail":90613176,"statfs":{"total":95995035648,"available":92787892224,"internally_reserved":0,"allocated":2329280512,"data_stored":2327548900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":877850078},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":1.083,"15min":1.347},"min":{"1min":0.187,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.77200000000000002,"5min":25.190000000000001,"15min":25.190000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":1.2050000000000001,"15min":1.3560000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.504,"5min":25.652000000000001,"15min":25.652000000000001},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.304,"15min":1.4530000000000001},"min":{"1min":0.245,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":2.649,"5min":25.754000000000001,"15min":25.754000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.548,"15min":1.623},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.25},"max":{"1min":2.6899999999999999,"5min":27.992999999999999,"15min":27.992999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.7549999999999999,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.972,"5min":2.972,"15min":2.972},"last":0.72399999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.5760000000000001,"15min":1.74},"min":{"1min":0.34100000000000003,"5min":0.28299999999999997,"15min":0.27900000000000003},"max":{"1min":2.1269999999999998,"5min":25.094000000000001,"15min":73.263000000000005},"last":1.147},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.3959999999999999,"15min":1.6659999999999999},"min":{"1min":0.34499999999999997,"5min":0.27600000000000002,"15min":0.27300000000000002},"max":{"1min":2.8559999999999999,"5min":25.5,"15min":73.5},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":1.5760000000000001,"15min":1.7589999999999999},"min":{"1min":0.38500000000000001,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":2.0369999999999999,"5min":25.521000000000001,"15min":81.438999999999993},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.6240000000000001,"15min":1.7989999999999999},"min":{"1min":0.41899999999999998,"5min":0.32800000000000001,"15min":0.29299999999999998},"max":{"1min":3.0739999999999998,"5min":25.84,"15min":81.626999999999995},"last":0.54400000000000004}]},{"osd":6,"last update":"Fri Jun 14 23:13:04 2024","interfaces":[{"interface":"back","average":{"1min":2.2410000000000001,"5min":1.8360000000000001,"15min":1.752},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":17.141999999999999,"5min":25.869,"15min":25.869},"last":0.58299999999999996},{"interface":"front","average":{"1min":2.0680000000000001,"5min":1.641,"15min":1.5529999999999999},"min":{"1min":0.38100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":20.056999999999999,"5min":25.143000000000001,"15min":25.143000000000001},"last":0.82999999999999996}]},{"osd":7,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.4379999999999999,"15min":1.4330000000000001},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.30499999999999999},"max":{"1min":2.4940000000000002,"5min":25.120000000000001,"15min":25.120000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.24,"5min":1.5940000000000001,"15min":1.3979999999999999},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":3.4300000000000002,"5min":25.472000000000001,"15min":25.472000000000001},"last":0.68100000000000005}]}]},{"osd":2,"up_from":322,"seq":1382979469334,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":511968,"kb_used_data":388948,"kb_used_omap":10,"kb_used_meta":122933,"kb_avail":93233184,"statfs":{"total":95995035648,"available":95470780416,"internally_reserved":0,"allocated":398282752,"data_stored":396539984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":125883942},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.308},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.27600000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.32200000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.53700000000000003}]},{"osd":4,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.544,"5min":2.544,"15min":2.544},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.40799999999999997}]},{"osd":6,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.6950000000000003,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":0.82499999999999996}]},{"osd":7,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":3.008,"5min":3.008,"15min":3.008},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.548,"5min":4.548,"15min":4.548},"last":0.73699999999999999}]}]},{"osd":1,"up_from":0,"seq":498216206622,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024277,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2335309824,"data_stored":2335309824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371462144,"data_stored":371458072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2283212800,"data_stored":2283208728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1196752896,"data_stored":1196752896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":404029440,"data_stored":404029440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1662648320,"data_stored":1662648320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1771114496,"data_stored":1771110424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:13:12.304 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:13:12.304 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:13:12.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:12.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:12.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:12.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:12.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:13:12.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:12.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:12.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:12.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:12.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:12.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:12.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:12.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:12.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:12.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:12.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:12.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:12.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:12.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:12 smithi137 ceph-mon[90634]: pgmap v1270: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 884 op/s; 30661/131205 objects degraded (23.369%); 2243/131205 objects misplaced (1.710%); 5.6 MiB/s, 90 objects/s recovering 2024-06-14T23:13:12.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:12.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:13.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:13.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:13.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:12 smithi086 ceph-mon[97913]: pgmap v1270: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 884 op/s; 30661/131205 objects degraded (23.369%); 2243/131205 objects misplaced (1.710%); 5.6 MiB/s, 90 objects/s recovering 2024-06-14T23:13:13.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:13.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:12 smithi086 ceph-mon[102754]: pgmap v1270: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 884 op/s; 30661/131205 objects degraded (23.369%); 2243/131205 objects misplaced (1.710%); 5.6 MiB/s, 90 objects/s recovering 2024-06-14T23:13:13.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:13.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:13.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-14T23:13:13.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:13.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:13.255 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:13.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:13.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:13.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:13.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:13.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:13.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:13.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:13.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:13.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:13.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:13.683 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:13:13.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:13.754 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:13.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:13.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:13.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:13.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:13.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:13.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:14.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:14.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:14.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:14.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 30661/131205 objects degraded (23.369%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:14.055 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 30661/131205 objects degraded (23.369%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:14.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:14.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:14.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:14.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 30661/131205 objects degraded (23.369%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:14.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:14.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:14.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:14.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:14.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:14.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:14.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:14.605 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:14.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:14.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:14.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:14.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:14.838 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:13:14.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:14.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:14.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:14.963 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:14 smithi137 ceph-mon[90634]: pgmap v1271: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 920 op/s; 30661/123342 objects degraded (24.859%); 2243/123342 objects misplaced (1.819%); 4.1 MiB/s, 64 objects/s recovering 2024-06-14T23:13:14.963 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2008180289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:13:14.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:15.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:15.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:15.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:15.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:15.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:15.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:15.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:15.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:13:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:15.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:15.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:14 smithi086 ceph-mon[97913]: pgmap v1271: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 920 op/s; 30661/123342 objects degraded (24.859%); 2243/123342 objects misplaced (1.819%); 4.1 MiB/s, 64 objects/s recovering 2024-06-14T23:13:15.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2008180289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:13:15.339 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:13:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:13:14] "GET /metrics HTTP/1.1" 200 36713 "" "Prometheus/2.43.0" 2024-06-14T23:13:15.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:14 smithi086 ceph-mon[102754]: pgmap v1271: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 920 op/s; 30661/123342 objects degraded (24.859%); 2243/123342 objects misplaced (1.819%); 4.1 MiB/s, 64 objects/s recovering 2024-06-14T23:13:15.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2008180289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:13:15.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:15.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:15.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:15.551 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:15.582 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:15.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:15.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:15.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:15.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:15.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:15.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:15.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:15.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:15.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:15.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:15.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:15 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:15.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2008180289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:13:15.992 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:15 smithi137 ceph-mon[90634]: osdmap e340: 8 total, 8 up, 6 in 2024-06-14T23:13:16.045 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:13:16.045 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:13:16.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:15 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:16.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2008180289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:13:16.083 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:15 smithi086 ceph-mon[102754]: osdmap e340: 8 total, 8 up, 6 in 2024-06-14T23:13:16.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:15 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:16.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2008180289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:13:16.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:15 smithi086 ceph-mon[97913]: osdmap e340: 8 total, 8 up, 6 in 2024-06-14T23:13:16.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:16.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:16.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:16.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:16.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:16.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:16.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:16.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:16.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:16.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:16.442 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:16.446 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:13:16.447 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:13:16.480 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:13:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:13:16.510 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:13:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:13:16.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:16.542 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:13:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:13:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:16.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:16.581 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:13:16.581 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:13:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:13:16.614 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:13:16.646 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:13:16.646 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:13:16.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:16.675 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:13:16.675 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:13:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:16.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:16.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:16.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T23:13:16.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:16.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:16.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:16.961 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:16.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:16.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:16 smithi086 ceph-mon[97913]: pgmap v1273: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 935 op/s; 30484/117366 objects degraded (25.973%); 2049/117366 objects misplaced (1.746%); 3.8 MiB/s, 60 objects/s recovering 2024-06-14T23:13:16.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2290702485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:16.962 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:16 smithi086 ceph-mon[102754]: pgmap v1273: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 935 op/s; 30484/117366 objects degraded (25.973%); 2049/117366 objects misplaced (1.746%); 3.8 MiB/s, 60 objects/s recovering 2024-06-14T23:13:16.963 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:16 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2290702485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:17.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:17.086 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:17.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:17.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:17.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:17.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:17.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:16 smithi137 ceph-mon[90634]: pgmap v1273: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 935 op/s; 30484/117366 objects degraded (25.973%); 2049/117366 objects misplaced (1.746%); 3.8 MiB/s, 60 objects/s recovering 2024-06-14T23:13:17.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:16 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2290702485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:17.284 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:13:17.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:17.352 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:17.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:17.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:17.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:17.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:17.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:17.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:17.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:17.771 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:17.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:17.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:13:17.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:17.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:17.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:18.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:18.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:18.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:18.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:18.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:18.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:18.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:18.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:18.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:18.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:18.457 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:18.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:18.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:18.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:18.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:18.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:18.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:18.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:18.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:18.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:18.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:18.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:18 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:13:18.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:18 smithi137 ceph-mon[90634]: pgmap v1274: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 920 op/s; 30484/113355 objects degraded (26.893%); 2049/113355 objects misplaced (1.808%); 3.6 MiB/s, 57 objects/s recovering 2024-06-14T23:13:18.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 30484/113355 objects degraded (26.893%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:19.021 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:19.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:19.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:19.147 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:18 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:13:19.147 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:18 smithi086 ceph-mon[102754]: pgmap v1274: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 920 op/s; 30484/113355 objects degraded (26.893%); 2049/113355 objects misplaced (1.808%); 3.6 MiB/s, 57 objects/s recovering 2024-06-14T23:13:19.147 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 30484/113355 objects degraded (26.893%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:19.147 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:18 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:13:19.147 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:18 smithi086 ceph-mon[97913]: pgmap v1274: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.3 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 920 op/s; 30484/113355 objects degraded (26.893%); 2049/113355 objects misplaced (1.808%); 3.6 MiB/s, 57 objects/s recovering 2024-06-14T23:13:19.147 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 30484/113355 objects degraded (26.893%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:19.160 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:19.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:19.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:19.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:19.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:19.408 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:13:19.408 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:13:19.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.408 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:199361 12.5 12:a0f86ebe:::benchmark_data_smithi086_261492_object82856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-06-14T23:13:19.408 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:19.387320+0000", 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.014131023, 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014162109000000001, 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 199361 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.409 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.387320+0000", 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.387320+0000", 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.387318+0000", 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.387328+0000", 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7289999999999995e-06 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.387350+0000", 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2802999999999998e-05 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:19.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.387359+0000", 2024-06-14T23:13:19.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0790000000000007e-06 2024-06-14T23:13:19.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:19.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.387424+0000", 2024-06-14T23:13:19.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4207000000000004e-05 2024-06-14T23:13:19.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:19.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.387769+0000", 2024-06-14T23:13:19.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034567300000000001 2024-06-14T23:13:19.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:13:19.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.387871+0000", 2024-06-14T23:13:19.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101962 2024-06-14T23:13:19.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:19.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.391335+0000", 2024-06-14T23:13:19.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034637380000000001 2024-06-14T23:13:19.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.391991+0000", 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065631399999999996 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:199365 12.5 12:af8c3328:::benchmark_data_smithi086_261492_object82860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:19.391829+0000", 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0096220360000000005, 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097469110000000005, 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:19.415 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 199365 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.391829+0000", 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.391829+0000", 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.391827+0000", 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:19.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.391833+0000", 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4620000000000004e-06 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.391842+0000", 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0499999999999997e-06 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.391847+0000", 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0989999999999998e-06 2024-06-14T23:13:19.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.392020+0000", 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017343799999999999 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.392223+0000", 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020292800000000001 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.392461+0000", 2024-06-14T23:13:19.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000237383 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.397845+0000", 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053841160000000004 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.399001+0000", 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001156308 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:199367 12.5 12:a9a9dfba:::benchmark_data_smithi086_261492_object82862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:19.392282+0000", 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0091690190000000005, 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093763170000000003, 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 199367 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:19.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.392282+0000", 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.392282+0000", 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:19.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.392281+0000", 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.392286+0000", 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6810000000000003e-06 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.392297+0000", 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0671e-05 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.392303+0000", 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9200000000000001e-06 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.395184+0000", 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028807160000000002 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.395333+0000", 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014930100000000001 2024-06-14T23:13:19.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.395482+0000", 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014846099999999999 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.397953+0000", 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024715200000000001 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46469.0:199360 12.9 e340/323 12:92b7e0ea:::benchmark_data_smithi086_261492_object82855:head v 340'12526, mlcod=314'4822)", 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:19.398230+0000", 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0032205089999999999, 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035079099999999999, 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.398230+0000", 2024-06-14T23:13:19.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.398230+0000", 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.398229+0000", 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:19.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.398236+0000", 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4139999999999999e-06 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.398248+0000", 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2514000000000001e-05 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.398254+0000", 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8109999999999999e-06 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.398282+0000", 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8135000000000001e-05 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.398296+0000", 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4117999999999999e-05 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.46469.0:199361 12.5 e340/323)", 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:19.401711+0000", 2024-06-14T23:13:19.429 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4294967295.9997396, 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7539000000000003e-05, 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401711+0000", 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401711+0000", 2024-06-14T23:13:19.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401710+0000", 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401715+0000", 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9860000000000002e-06 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:19.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401729+0000", 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3243e-05 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401734+0000", 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6659999999999996e-06 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.46469.0:199365 12.5 e340/323)", 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:19.401749+0000", 2024-06-14T23:13:19.432 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4294967295.999702, 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9333000000000006e-05, 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401749+0000", 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401749+0000", 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401748+0000", 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401751+0000", 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6699999999999998e-06 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:19.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401758+0000", 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7340000000000002e-06 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401762+0000", 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1949999999999996e-06 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.435 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.46469.0:199367 12.5 e340/323)", 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:19.401766+0000", 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4294967295.9996848, 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012343900000000001, 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401766+0000", 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401766+0000", 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401766+0000", 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967296 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401768+0000", 2024-06-14T23:13:19.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1859999999999999e-06 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401773+0000", 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1390000000000003e-06 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:19.401776+0000", 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.332e-06 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.438 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:13:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:13:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T23:13:19.439 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 7 2024-06-14T23:13:19.439 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:13:19.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:19.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:19.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:19.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:19.624 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:19.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:19.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:19.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:19.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:19.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:19.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:20.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:20.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:20.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:20.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:20.166 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:20.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:20.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:20.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:20.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:20.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:13:20.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:20.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:20.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:20.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:20.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:20.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:20.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:20.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:20.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:20.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:20.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:20.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:20.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:20.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:21.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:21.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:21.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:21.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:20 smithi137 ceph-mon[90634]: pgmap v1275: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.2 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 912 op/s; 30484/107298 objects degraded (28.411%); 1817/107298 objects misplaced (1.693%); 3.8 MiB/s, 60 objects/s recovering 2024-06-14T23:13:21.270 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:21.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:21.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:20 smithi086 ceph-mon[102754]: pgmap v1275: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.2 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 912 op/s; 30484/107298 objects degraded (28.411%); 1817/107298 objects misplaced (1.693%); 3.8 MiB/s, 60 objects/s recovering 2024-06-14T23:13:21.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:20 smithi086 ceph-mon[97913]: pgmap v1275: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.2 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 912 op/s; 30484/107298 objects degraded (28.411%); 1817/107298 objects misplaced (1.693%); 3.8 MiB/s, 60 objects/s recovering 2024-06-14T23:13:21.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:21.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:21.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:21.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:21.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:21.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:21.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:21.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:21.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:21.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:21.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:21.890 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:21.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:21.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:22.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:22.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:22.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:22.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-14T23:13:22.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:22.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:22.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:22.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:22.408 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:22.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:22.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:22.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:22.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:22.627 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:13:22.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:22.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:22.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:22.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:22.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:22.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:22.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:22.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:22.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:23.028 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:22 smithi137 ceph-mon[90634]: pgmap v1276: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.1 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 918 op/s; 30181/103650 objects degraded (29.118%); 1817/103650 objects misplaced (1.753%); 5.7 MiB/s, 90 objects/s recovering 2024-06-14T23:13:23.042 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:22 smithi086 ceph-mon[97913]: pgmap v1276: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.1 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 918 op/s; 30181/103650 objects degraded (29.118%); 1817/103650 objects misplaced (1.753%); 5.7 MiB/s, 90 objects/s recovering 2024-06-14T23:13:23.042 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:22 smithi086 ceph-mon[102754]: pgmap v1276: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.1 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 918 op/s; 30181/103650 objects degraded (29.118%); 1817/103650 objects misplaced (1.753%); 5.7 MiB/s, 90 objects/s recovering 2024-06-14T23:13:23.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:23.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:23.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:23.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:23.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:23.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:23.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:23.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:23.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:23.405 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:13:23.407 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:13:23.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:23.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:23.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:23.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:23.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:23.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:23.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:23.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:23.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:23.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:23.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:23.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:24.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:24.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:24.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:24.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:23 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:13:24.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 30181/103650 objects degraded (29.118%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:24.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:23 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:13:24.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 30181/103650 objects degraded (29.118%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:24.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:24.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:23 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:13:24.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 30181/103650 objects degraded (29.118%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:24.198 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:24.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:24.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:24.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:24.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:24.434 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:24.480 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:24.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:24.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:24.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:24.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:24.720 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:13:24.720 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:13:24.720 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:13:24.720 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:13:24.721 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:13:24.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:24.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:24.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:24.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:24.870 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1276,"stamp":"2024-06-14T23:13:21.729570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2263391726,"num_objects":34550,"num_object_clones":0,"num_object_copies":103650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30181,"num_objects_misplaced":1817,"num_objects_unfound":0,"num_objects_dirty":34550,"num_whiteouts":0,"num_read":56,"num_read_kb":104,"num_write":198568,"num_write_kb":7456431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18834,"num_bytes_recovered":1233360086,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83310,"ondisk_log_size":83310,"up":357,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":10211500,"kb_used_data":5515640,"kb_used_omap":27,"kb_used_meta":4695460,"kb_avail":552259412,"statfs":{"total":575970213888,"available":565513637888,"internally_reserved":0,"allocated":5648015360,"data_stored":5637490963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":4808151494},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,7,11,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-601948160,"num_objects":-9185,"num_object_clones":0,"num_object_copies":-27555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-480,"num_objects_misplaced":-426,"num_objects_unfound":0,"num_objects_dirty":-9185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":59375616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003902"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":719,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.816772+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:13:05.816772+0000","last_peered":"2024-06-14T23:13:05.816772+0000","last_clean":"2024-06-14T23:13:05.816772+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:13:05.816772+0000","last_undegraded":"2024-06-14T23:13:05.816772+0000","last_fullsized":"2024-06-14T23:13:05.816772+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":157,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822280+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:13:05.822280+0000","last_peered":"2024-06-14T23:13:05.822280+0000","last_clean":"2024-06-14T23:13:05.822280+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:13:05.822280+0000","last_undegraded":"2024-06-14T23:13:05.822280+0000","last_fullsized":"2024-06-14T23:13:05.822280+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":327,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.820458+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:13:05.820458+0000","last_peered":"2024-06-14T23:13:05.820458+0000","last_clean":"2024-06-14T23:13:05.820458+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:13:05.820458+0000","last_undegraded":"2024-06-14T23:13:05.820458+0000","last_fullsized":"2024-06-14T23:13:05.820458+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":232,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.809783+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:13:05.809783+0000","last_peered":"2024-06-14T23:13:05.809783+0000","last_clean":"2024-06-14T23:13:05.809783+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:13:05.809783+0000","last_undegraded":"2024-06-14T23:13:05.809783+0000","last_fullsized":"2024-06-14T23:13:05.809783+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":122,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.810007+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:13:05.810007+0000","last_peered":"2024-06-14T23:13:05.810007+0000","last_clean":"2024-06-14T23:13:05.810007+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:13:05.810007+0000","last_undegraded":"2024-06-14T23:13:05.810007+0000","last_fullsized":"2024-06-14T23:13:05.810007+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":747,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:17.556502+0000","last_change":"2024-06-14T23:13:17.556502+0000","last_active":"2024-06-14T23:13:17.556502+0000","last_peered":"2024-06-14T23:13:17.556502+0000","last_clean":"2024-06-14T23:13:17.556502+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:13:17.556502+0000","last_undegraded":"2024-06-14T23:13:17.556502+0000","last_fullsized":"2024-06-14T23:13:17.556502+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:09:56.196559+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:09:56.196559+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":303,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821279+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:13:05.821279+0000","last_peered":"2024-06-14T23:13:05.821279+0000","last_clean":"2024-06-14T23:13:05.821279+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:13:05.821279+0000","last_undegraded":"2024-06-14T23:13:05.821279+0000","last_fullsized":"2024-06-14T23:13:05.821279+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":344,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822090+0000","last_change":"2024-06-14T23:11:23.413009+0000","last_active":"2024-06-14T23:13:05.822090+0000","last_peered":"2024-06-14T23:13:05.822090+0000","last_clean":"2024-06-14T23:13:05.822090+0000","last_became_active":"2024-06-14T23:11:23.412491+0000","last_became_peered":"2024-06-14T23:11:23.412491+0000","last_unstale":"2024-06-14T23:13:05.822090+0000","last_undegraded":"2024-06-14T23:13:05.822090+0000","last_fullsized":"2024-06-14T23:13:05.822090+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:57.263547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":491,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.857353+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:13:14.857353+0000","last_peered":"2024-06-14T23:13:14.857353+0000","last_clean":"2024-06-14T23:13:14.857353+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:13:14.857353+0000","last_undegraded":"2024-06-14T23:13:14.857353+0000","last_fullsized":"2024-06-14T23:13:14.857353+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":317,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821051+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:13:05.821051+0000","last_peered":"2024-06-14T23:13:05.821051+0000","last_clean":"2024-06-14T23:13:05.821051+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:13:05.821051+0000","last_undegraded":"2024-06-14T23:13:05.821051+0000","last_fullsized":"2024-06-14T23:13:05.821051+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":344,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.820129+0000","last_change":"2024-06-14T23:11:23.427771+0000","last_active":"2024-06-14T23:13:05.820129+0000","last_peered":"2024-06-14T23:13:05.820129+0000","last_clean":"2024-06-14T23:13:05.820129+0000","last_became_active":"2024-06-14T23:11:23.427481+0000","last_became_peered":"2024-06-14T23:11:23.427481+0000","last_unstale":"2024-06-14T23:13:05.820129+0000","last_undegraded":"2024-06-14T23:13:05.820129+0000","last_fullsized":"2024-06-14T23:13:05.820129+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:01:04.162256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":133,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819990+0000","last_change":"2024-06-14T23:10:26.053647+0000","last_active":"2024-06-14T23:13:05.819990+0000","last_peered":"2024-06-14T23:13:05.819990+0000","last_clean":"2024-06-14T23:13:05.819990+0000","last_became_active":"2024-06-14T23:10:26.053318+0000","last_became_peered":"2024-06-14T23:10:26.053318+0000","last_unstale":"2024-06-14T23:13:05.819990+0000","last_undegraded":"2024-06-14T23:13:05.819990+0000","last_fullsized":"2024-06-14T23:13:05.819990+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:32.647107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":132,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.812066+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:13:05.812066+0000","last_peered":"2024-06-14T23:13:05.812066+0000","last_clean":"2024-06-14T23:13:05.812066+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:13:05.812066+0000","last_undegraded":"2024-06-14T23:13:05.812066+0000","last_fullsized":"2024-06-14T23:13:05.812066+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":643,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817413+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:13:05.817413+0000","last_peered":"2024-06-14T23:13:05.817413+0000","last_clean":"2024-06-14T23:13:05.817413+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:13:05.817413+0000","last_undegraded":"2024-06-14T23:13:05.817413+0000","last_fullsized":"2024-06-14T23:13:05.817413+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":366,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822159+0000","last_change":"2024-06-14T23:11:23.413241+0000","last_active":"2024-06-14T23:13:05.822159+0000","last_peered":"2024-06-14T23:13:05.822159+0000","last_clean":"2024-06-14T23:13:05.822159+0000","last_became_active":"2024-06-14T23:11:23.412936+0000","last_became_peered":"2024-06-14T23:11:23.412936+0000","last_unstale":"2024-06-14T23:13:05.822159+0000","last_undegraded":"2024-06-14T23:13:05.822159+0000","last_fullsized":"2024-06-14T23:13:05.822159+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:36.581716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":152,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.820004+0000","last_change":"2024-06-14T23:10:26.038549+0000","last_active":"2024-06-14T23:13:05.820004+0000","last_peered":"2024-06-14T23:13:05.820004+0000","last_clean":"2024-06-14T23:13:05.820004+0000","last_became_active":"2024-06-14T23:10:26.038260+0000","last_became_peered":"2024-06-14T23:10:26.038260+0000","last_unstale":"2024-06-14T23:13:05.820004+0000","last_undegraded":"2024-06-14T23:13:05.820004+0000","last_fullsized":"2024-06-14T23:13:05.820004+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:44.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":746,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817905+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:13:05.817905+0000","last_peered":"2024-06-14T23:13:05.817905+0000","last_clean":"2024-06-14T23:13:05.817905+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:13:05.817905+0000","last_undegraded":"2024-06-14T23:13:05.817905+0000","last_fullsized":"2024-06-14T23:13:05.817905+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":137,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.816964+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:13:05.816964+0000","last_peered":"2024-06-14T23:13:05.816964+0000","last_clean":"2024-06-14T23:13:05.816964+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:13:05.816964+0000","last_undegraded":"2024-06-14T23:13:05.816964+0000","last_fullsized":"2024-06-14T23:13:05.816964+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":199,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817600+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:13:05.817600+0000","last_peered":"2024-06-14T23:13:05.817600+0000","last_clean":"2024-06-14T23:13:05.817600+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:13:05.817600+0000","last_undegraded":"2024-06-14T23:13:05.817600+0000","last_fullsized":"2024-06-14T23:13:05.817600+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":694,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819087+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:13:05.819087+0000","last_peered":"2024-06-14T23:13:05.819087+0000","last_clean":"2024-06-14T23:13:05.819087+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:13:05.819087+0000","last_undegraded":"2024-06-14T23:13:05.819087+0000","last_fullsized":"2024-06-14T23:13:05.819087+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":60,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.810129+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:13:05.810129+0000","last_peered":"2024-06-14T23:13:05.810129+0000","last_clean":"2024-06-14T23:13:05.810129+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:13:05.810129+0000","last_undegraded":"2024-06-14T23:13:05.810129+0000","last_fullsized":"2024-06-14T23:13:05.810129+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":291,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818824+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:13:05.818824+0000","last_peered":"2024-06-14T23:13:05.818824+0000","last_clean":"2024-06-14T23:13:05.818824+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:13:05.818824+0000","last_undegraded":"2024-06-14T23:13:05.818824+0000","last_fullsized":"2024-06-14T23:13:05.818824+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":646,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821778+0000","last_change":"2024-06-14T23:10:29.062659+0000","last_active":"2024-06-14T23:13:05.821778+0000","last_peered":"2024-06-14T23:13:05.821778+0000","last_clean":"2024-06-14T23:13:05.821778+0000","last_became_active":"2024-06-14T23:10:29.062256+0000","last_became_peered":"2024-06-14T23:10:29.062256+0000","last_unstale":"2024-06-14T23:13:05.821778+0000","last_undegraded":"2024-06-14T23:13:05.821778+0000","last_fullsized":"2024-06-14T23:13:05.821778+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:38.814157+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":323,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.816548+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:13:05.816548+0000","last_peered":"2024-06-14T23:13:05.816548+0000","last_clean":"2024-06-14T23:13:05.816548+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:13:05.816548+0000","last_undegraded":"2024-06-14T23:13:05.816548+0000","last_fullsized":"2024-06-14T23:13:05.816548+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":187,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822348+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:13:05.822348+0000","last_peered":"2024-06-14T23:13:05.822348+0000","last_clean":"2024-06-14T23:13:05.822348+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:13:05.822348+0000","last_undegraded":"2024-06-14T23:13:05.822348+0000","last_fullsized":"2024-06-14T23:13:05.822348+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":646,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.828544+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:13:05.828544+0000","last_peered":"2024-06-14T23:13:05.828544+0000","last_clean":"2024-06-14T23:13:05.828544+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:13:05.828544+0000","last_undegraded":"2024-06-14T23:13:05.828544+0000","last_fullsized":"2024-06-14T23:13:05.828544+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":206,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.816730+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:13:05.816730+0000","last_peered":"2024-06-14T23:13:05.816730+0000","last_clean":"2024-06-14T23:13:05.816730+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:13:05.816730+0000","last_undegraded":"2024-06-14T23:13:05.816730+0000","last_fullsized":"2024-06-14T23:13:05.816730+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":311,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821756+0000","last_change":"2024-06-14T23:10:26.053008+0000","last_active":"2024-06-14T23:13:05.821756+0000","last_peered":"2024-06-14T23:13:05.821756+0000","last_clean":"2024-06-14T23:13:05.821756+0000","last_became_active":"2024-06-14T23:10:26.052701+0000","last_became_peered":"2024-06-14T23:10:26.052701+0000","last_unstale":"2024-06-14T23:13:05.821756+0000","last_undegraded":"2024-06-14T23:13:05.821756+0000","last_fullsized":"2024-06-14T23:13:05.821756+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:15:03.929062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":670,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.812348+0000","last_change":"2024-06-14T23:11:23.411926+0000","last_active":"2024-06-14T23:13:05.812348+0000","last_peered":"2024-06-14T23:13:05.812348+0000","last_clean":"2024-06-14T23:13:05.812348+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:13:05.812348+0000","last_undegraded":"2024-06-14T23:13:05.812348+0000","last_fullsized":"2024-06-14T23:13:05.812348+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":224,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819227+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:13:05.819227+0000","last_peered":"2024-06-14T23:13:05.819227+0000","last_clean":"2024-06-14T23:13:05.819227+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:13:05.819227+0000","last_undegraded":"2024-06-14T23:13:05.819227+0000","last_fullsized":"2024-06-14T23:13:05.819227+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":353,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.812935+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:13:05.812935+0000","last_peered":"2024-06-14T23:13:05.812935+0000","last_clean":"2024-06-14T23:13:05.812935+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:13:05.812935+0000","last_undegraded":"2024-06-14T23:13:05.812935+0000","last_fullsized":"2024-06-14T23:13:05.812935+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":516,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821991+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:13:05.821991+0000","last_peered":"2024-06-14T23:13:05.821991+0000","last_clean":"2024-06-14T23:13:05.821991+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:13:05.821991+0000","last_undegraded":"2024-06-14T23:13:05.821991+0000","last_fullsized":"2024-06-14T23:13:05.821991+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":323,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.820138+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:13:05.820138+0000","last_peered":"2024-06-14T23:13:05.820138+0000","last_clean":"2024-06-14T23:13:05.820138+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:13:05.820138+0000","last_undegraded":"2024-06-14T23:13:05.820138+0000","last_fullsized":"2024-06-14T23:13:05.820138+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":327,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.820208+0000","last_change":"2024-06-14T23:10:26.047451+0000","last_active":"2024-06-14T23:13:05.820208+0000","last_peered":"2024-06-14T23:13:05.820208+0000","last_clean":"2024-06-14T23:13:05.820208+0000","last_became_active":"2024-06-14T23:10:26.046781+0000","last_became_peered":"2024-06-14T23:10:26.046781+0000","last_unstale":"2024-06-14T23:13:05.820208+0000","last_undegraded":"2024-06-14T23:13:05.820208+0000","last_fullsized":"2024-06-14T23:13:05.820208+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:07:15.892864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":555,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822669+0000","last_change":"2024-06-14T23:10:39.253251+0000","last_active":"2024-06-14T23:13:05.822669+0000","last_peered":"2024-06-14T23:13:05.822669+0000","last_clean":"2024-06-14T23:13:05.822669+0000","last_became_active":"2024-06-14T23:10:29.061386+0000","last_became_peered":"2024-06-14T23:10:29.061386+0000","last_unstale":"2024-06-14T23:13:05.822669+0000","last_undegraded":"2024-06-14T23:13:05.822669+0000","last_fullsized":"2024-06-14T23:13:05.822669+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.735431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.735431+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"340'12585","reported_seq":12650,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:20.494387+0000","last_change":"2024-06-14T23:11:24.652983+0000","last_active":"2024-06-14T23:13:20.494387+0000","last_peered":"2024-06-14T23:13:20.494387+0000","last_clean":"2024-06-14T23:10:37.054302+0000","last_became_active":"2024-06-14T23:11:24.646468+0000","last_became_peered":"2024-06-14T23:11:24.646468+0000","last_unstale":"2024-06-14T23:13:20.494387+0000","last_undegraded":"2024-06-14T23:11:24.406142+0000","last_fullsized":"2024-06-14T23:11:24.380989+0000","mapping_epoch":323,"log_start":"327'8205","ondisk_log_start":"327'8205","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:33:08.769394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130744320,"num_objects":1995,"num_object_clones":0,"num_object_copies":5985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12585,"num_write_kb":466560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-473}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":300,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819649+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:13:05.819649+0000","last_peered":"2024-06-14T23:13:05.819649+0000","last_clean":"2024-06-14T23:13:05.819649+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:13:05.819649+0000","last_undegraded":"2024-06-14T23:13:05.819649+0000","last_fullsized":"2024-06-14T23:13:05.819649+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":312,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818635+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:13:05.818635+0000","last_peered":"2024-06-14T23:13:05.818635+0000","last_clean":"2024-06-14T23:13:05.818635+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:13:05.818635+0000","last_undegraded":"2024-06-14T23:13:05.818635+0000","last_fullsized":"2024-06-14T23:13:05.818635+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":98,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822537+0000","last_change":"2024-06-14T23:10:26.058423+0000","last_active":"2024-06-14T23:13:05.822537+0000","last_peered":"2024-06-14T23:13:05.822537+0000","last_clean":"2024-06-14T23:13:05.822537+0000","last_became_active":"2024-06-14T23:10:26.050244+0000","last_became_peered":"2024-06-14T23:10:26.050244+0000","last_unstale":"2024-06-14T23:13:05.822537+0000","last_undegraded":"2024-06-14T23:13:05.822537+0000","last_fullsized":"2024-06-14T23:13:05.822537+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:52.060325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":665,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817781+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:13:05.817781+0000","last_peered":"2024-06-14T23:13:05.817781+0000","last_clean":"2024-06-14T23:13:05.817781+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:13:05.817781+0000","last_undegraded":"2024-06-14T23:13:05.817781+0000","last_fullsized":"2024-06-14T23:13:05.817781+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"340'12669","reported_seq":12746,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:20.458436+0000","last_change":"2024-06-14T23:11:24.639438+0000","last_active":"2024-06-14T23:13:20.458436+0000","last_peered":"2024-06-14T23:13:20.458436+0000","last_clean":"2024-06-14T23:10:37.032734+0000","last_became_active":"2024-06-14T23:11:24.634884+0000","last_became_peered":"2024-06-14T23:11:24.634884+0000","last_unstale":"2024-06-14T23:13:20.458436+0000","last_undegraded":"2024-06-14T23:11:24.403675+0000","last_fullsized":"2024-06-14T23:11:24.380974+0000","mapping_epoch":323,"log_start":"329'8300","ondisk_log_start":"329'8300","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:48:19.382266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134545408,"num_objects":2053,"num_object_clones":0,"num_object_copies":6159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12669,"num_write_kb":471104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":253,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817331+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:13:05.817331+0000","last_peered":"2024-06-14T23:13:05.817331+0000","last_clean":"2024-06-14T23:13:05.817331+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:13:05.817331+0000","last_undegraded":"2024-06-14T23:13:05.817331+0000","last_fullsized":"2024-06-14T23:13:05.817331+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":303,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822806+0000","last_change":"2024-06-14T23:10:26.051450+0000","last_active":"2024-06-14T23:13:05.822806+0000","last_peered":"2024-06-14T23:13:05.822806+0000","last_clean":"2024-06-14T23:13:05.822806+0000","last_became_active":"2024-06-14T23:10:26.050830+0000","last_became_peered":"2024-06-14T23:10:26.050830+0000","last_unstale":"2024-06-14T23:13:05.822806+0000","last_undegraded":"2024-06-14T23:13:05.822806+0000","last_fullsized":"2024-06-14T23:13:05.822806+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:16:16.175014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":54,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821908+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:13:05.821908+0000","last_peered":"2024-06-14T23:13:05.821908+0000","last_clean":"2024-06-14T23:13:05.821908+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:13:05.821908+0000","last_undegraded":"2024-06-14T23:13:05.821908+0000","last_fullsized":"2024-06-14T23:13:05.821908+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":754,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.820083+0000","last_change":"2024-06-14T23:11:23.401974+0000","last_active":"2024-06-14T23:13:05.820083+0000","last_peered":"2024-06-14T23:13:05.820083+0000","last_clean":"2024-06-14T23:13:05.820083+0000","last_became_active":"2024-06-14T23:11:23.401625+0000","last_became_peered":"2024-06-14T23:11:23.401625+0000","last_unstale":"2024-06-14T23:13:05.820083+0000","last_undegraded":"2024-06-14T23:13:05.820083+0000","last_fullsized":"2024-06-14T23:13:05.820083+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.214300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.214300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:05:54.552362+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"340'12584","reported_seq":12650,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:20.468314+0000","last_change":"2024-06-14T23:11:24.651371+0000","last_active":"2024-06-14T23:13:20.468314+0000","last_peered":"2024-06-14T23:13:20.468314+0000","last_clean":"2024-06-14T23:10:37.020885+0000","last_became_active":"2024-06-14T23:11:24.646284+0000","last_became_peered":"2024-06-14T23:11:24.646284+0000","last_unstale":"2024-06-14T23:13:20.468314+0000","last_undegraded":"2024-06-14T23:11:24.405009+0000","last_fullsized":"2024-06-14T23:11:24.380990+0000","mapping_epoch":323,"log_start":"327'8242","ondisk_log_start":"327'8242","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:26:58.760076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135659520,"num_objects":2070,"num_object_clones":0,"num_object_copies":6210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12584,"num_write_kb":468928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-486}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":151,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819271+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:13:05.819271+0000","last_peered":"2024-06-14T23:13:05.819271+0000","last_clean":"2024-06-14T23:13:05.819271+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:13:05.819271+0000","last_undegraded":"2024-06-14T23:13:05.819271+0000","last_fullsized":"2024-06-14T23:13:05.819271+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":325,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.828612+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:13:05.828612+0000","last_peered":"2024-06-14T23:13:05.828612+0000","last_clean":"2024-06-14T23:13:05.828612+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:13:05.828612+0000","last_undegraded":"2024-06-14T23:13:05.828612+0000","last_fullsized":"2024-06-14T23:13:05.828612+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":276,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.812155+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:13:05.812155+0000","last_peered":"2024-06-14T23:13:05.812155+0000","last_clean":"2024-06-14T23:13:05.812155+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:13:05.812155+0000","last_undegraded":"2024-06-14T23:13:05.812155+0000","last_fullsized":"2024-06-14T23:13:05.812155+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":639,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817184+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:13:05.817184+0000","last_peered":"2024-06-14T23:13:05.817184+0000","last_clean":"2024-06-14T23:13:05.817184+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:13:05.817184+0000","last_undegraded":"2024-06-14T23:13:05.817184+0000","last_fullsized":"2024-06-14T23:13:05.817184+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"340'12722","reported_seq":18270,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:20.501209+0000","last_change":"2024-06-14T23:11:24.647584+0000","last_active":"2024-06-14T23:13:20.501209+0000","last_peered":"2024-06-14T23:13:20.501209+0000","last_clean":"2024-06-14T23:10:25.009797+0000","last_became_active":"2024-06-14T23:11:24.638765+0000","last_became_peered":"2024-06-14T23:11:24.638765+0000","last_unstale":"2024-06-14T23:13:20.501209+0000","last_undegraded":"2024-06-14T23:11:24.535343+0000","last_fullsized":"2024-06-14T23:11:24.381001+0000","mapping_epoch":323,"log_start":"329'8372","ondisk_log_start":"329'8372","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:37.543254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134348800,"num_objects":2050,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12722,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2731,"num_bytes_recovered":178978816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[3,2],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3,4","objects":2049},{"shards":"2,3","objects":-2603}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":116,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818556+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:13:05.818556+0000","last_peered":"2024-06-14T23:13:05.818556+0000","last_clean":"2024-06-14T23:13:05.818556+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:13:05.818556+0000","last_undegraded":"2024-06-14T23:13:05.818556+0000","last_fullsized":"2024-06-14T23:13:05.818556+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":60,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.811828+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:13:05.811828+0000","last_peered":"2024-06-14T23:13:05.811828+0000","last_clean":"2024-06-14T23:13:05.811828+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:13:05.811828+0000","last_undegraded":"2024-06-14T23:13:05.811828+0000","last_fullsized":"2024-06-14T23:13:05.811828+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":355,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821354+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:13:05.821354+0000","last_peered":"2024-06-14T23:13:05.821354+0000","last_clean":"2024-06-14T23:13:05.821354+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:13:05.821354+0000","last_undegraded":"2024-06-14T23:13:05.821354+0000","last_fullsized":"2024-06-14T23:13:05.821354+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":619,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818698+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:13:05.818698+0000","last_peered":"2024-06-14T23:13:05.818698+0000","last_clean":"2024-06-14T23:13:05.818698+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:13:05.818698+0000","last_undegraded":"2024-06-14T23:13:05.818698+0000","last_fullsized":"2024-06-14T23:13:05.818698+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"340'12554","reported_seq":17796,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:18.192034+0000","last_change":"2024-06-14T23:11:50.568053+0000","last_active":"2024-06-14T23:13:18.192034+0000","last_peered":"2024-06-14T23:13:18.192034+0000","last_clean":"2024-06-14T23:13:18.192034+0000","last_became_active":"2024-06-14T23:11:50.555786+0000","last_became_peered":"2024-06-14T23:11:50.555786+0000","last_unstale":"2024-06-14T23:13:18.192034+0000","last_undegraded":"2024-06-14T23:13:18.192034+0000","last_fullsized":"2024-06-14T23:13:18.192034+0000","mapping_epoch":328,"log_start":"319'7100","ondisk_log_start":"319'7100","created":295,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5454,"log_dups_size":0,"ondisk_log_size":5454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:36.382663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145489920,"num_objects":2220,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12554,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2575,"num_bytes_recovered":168755200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":291,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.809728+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:13:05.809728+0000","last_peered":"2024-06-14T23:13:05.809728+0000","last_clean":"2024-06-14T23:13:05.809728+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:13:05.809728+0000","last_undegraded":"2024-06-14T23:13:05.809728+0000","last_fullsized":"2024-06-14T23:13:05.809728+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":338,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.820347+0000","last_change":"2024-06-14T23:10:26.049033+0000","last_active":"2024-06-14T23:13:05.820347+0000","last_peered":"2024-06-14T23:13:05.820347+0000","last_clean":"2024-06-14T23:13:05.820347+0000","last_became_active":"2024-06-14T23:10:26.047775+0000","last_became_peered":"2024-06-14T23:10:26.047775+0000","last_unstale":"2024-06-14T23:13:05.820347+0000","last_undegraded":"2024-06-14T23:13:05.820347+0000","last_fullsized":"2024-06-14T23:13:05.820347+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:15:52.051858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":332,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818780+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:13:05.818780+0000","last_peered":"2024-06-14T23:13:05.818780+0000","last_clean":"2024-06-14T23:13:05.818780+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:13:05.818780+0000","last_undegraded":"2024-06-14T23:13:05.818780+0000","last_fullsized":"2024-06-14T23:13:05.818780+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":733,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818235+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:13:05.818235+0000","last_peered":"2024-06-14T23:13:05.818235+0000","last_clean":"2024-06-14T23:13:05.818235+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:13:05.818235+0000","last_undegraded":"2024-06-14T23:13:05.818235+0000","last_fullsized":"2024-06-14T23:13:05.818235+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"340'12461","reported_seq":12535,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:16.752131+0000","last_change":"2024-06-14T23:11:24.656131+0000","last_active":"2024-06-14T23:13:16.752131+0000","last_peered":"2024-06-14T23:13:16.752131+0000","last_clean":"2024-06-14T23:10:37.043860+0000","last_became_active":"2024-06-14T23:11:24.647995+0000","last_became_peered":"2024-06-14T23:11:24.647995+0000","last_unstale":"2024-06-14T23:13:16.752131+0000","last_undegraded":"2024-06-14T23:11:24.403317+0000","last_fullsized":"2024-06-14T23:11:24.380506+0000","mapping_epoch":323,"log_start":"324'7500","ondisk_log_start":"324'7500","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4961,"log_dups_size":0,"ondisk_log_size":4961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:32:10.827694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146472960,"num_objects":2235,"num_object_clones":0,"num_object_copies":6705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":470272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":-266}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":300,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822117+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:13:05.822117+0000","last_peered":"2024-06-14T23:13:05.822117+0000","last_clean":"2024-06-14T23:13:05.822117+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:13:05.822117+0000","last_undegraded":"2024-06-14T23:13:05.822117+0000","last_fullsized":"2024-06-14T23:13:05.822117+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":336,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.809638+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:13:05.809638+0000","last_peered":"2024-06-14T23:13:05.809638+0000","last_clean":"2024-06-14T23:13:05.809638+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:13:05.809638+0000","last_undegraded":"2024-06-14T23:13:05.809638+0000","last_fullsized":"2024-06-14T23:13:05.809638+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":141,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.809930+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:13:05.809930+0000","last_peered":"2024-06-14T23:13:05.809930+0000","last_clean":"2024-06-14T23:13:05.809930+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:13:05.809930+0000","last_undegraded":"2024-06-14T23:13:05.809930+0000","last_fullsized":"2024-06-14T23:13:05.809930+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":627,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821851+0000","last_change":"2024-06-14T23:11:23.413224+0000","last_active":"2024-06-14T23:13:05.821851+0000","last_peered":"2024-06-14T23:13:05.821851+0000","last_clean":"2024-06-14T23:13:05.821851+0000","last_became_active":"2024-06-14T23:11:23.412680+0000","last_became_peered":"2024-06-14T23:11:23.412680+0000","last_unstale":"2024-06-14T23:13:05.821851+0000","last_undegraded":"2024-06-14T23:13:05.821851+0000","last_fullsized":"2024-06-14T23:13:05.821851+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:48:10.967651+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.e","version":"340'12307","reported_seq":12370,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:18.406244+0000","last_change":"2024-06-14T23:09:41.994118+0000","last_active":"2024-06-14T23:13:18.406244+0000","last_peered":"2024-06-14T23:13:18.406244+0000","last_clean":"2024-06-14T23:13:18.406244+0000","last_became_active":"2024-06-14T23:09:41.993702+0000","last_became_peered":"2024-06-14T23:09:41.993702+0000","last_unstale":"2024-06-14T23:13:18.406244+0000","last_undegraded":"2024-06-14T23:13:18.406244+0000","last_fullsized":"2024-06-14T23:13:18.406244+0000","mapping_epoch":299,"log_start":"319'7000","ondisk_log_start":"319'7000","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5307,"log_dups_size":0,"ondisk_log_size":5307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:25.994252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145817600,"num_objects":2225,"num_object_clones":0,"num_object_copies":6675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12307,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":309,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.810319+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:13:05.810319+0000","last_peered":"2024-06-14T23:13:05.810319+0000","last_clean":"2024-06-14T23:13:05.810319+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:13:05.810319+0000","last_undegraded":"2024-06-14T23:13:05.810319+0000","last_fullsized":"2024-06-14T23:13:05.810319+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":188,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819504+0000","last_change":"2024-06-14T23:09:41.991652+0000","last_active":"2024-06-14T23:13:05.819504+0000","last_peered":"2024-06-14T23:13:05.819504+0000","last_clean":"2024-06-14T23:13:05.819504+0000","last_became_active":"2024-06-14T23:09:41.991317+0000","last_became_peered":"2024-06-14T23:09:41.991317+0000","last_unstale":"2024-06-14T23:13:05.819504+0000","last_undegraded":"2024-06-14T23:13:05.819504+0000","last_fullsized":"2024-06-14T23:13:05.819504+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:16:18.209329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":305,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.816542+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:13:05.816542+0000","last_peered":"2024-06-14T23:13:05.816542+0000","last_clean":"2024-06-14T23:13:05.816542+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:13:05.816542+0000","last_undegraded":"2024-06-14T23:13:05.816542+0000","last_fullsized":"2024-06-14T23:13:05.816542+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":547,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.823453+0000","last_change":"2024-06-14T23:10:29.063981+0000","last_active":"2024-06-14T23:13:05.823453+0000","last_peered":"2024-06-14T23:13:05.823453+0000","last_clean":"2024-06-14T23:13:05.823453+0000","last_became_active":"2024-06-14T23:10:29.063546+0000","last_became_peered":"2024-06-14T23:10:29.063546+0000","last_unstale":"2024-06-14T23:13:05.823453+0000","last_undegraded":"2024-06-14T23:13:05.823453+0000","last_fullsized":"2024-06-14T23:13:05.823453+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:26:23.411322+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"340'12451","reported_seq":20911,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:18.752623+0000","last_change":"2024-06-14T23:11:24.664901+0000","last_active":"2024-06-14T23:13:18.752623+0000","last_peered":"2024-06-14T23:13:18.752623+0000","last_clean":"2024-06-14T23:10:24.989118+0000","last_became_active":"2024-06-14T23:11:24.647313+0000","last_became_peered":"2024-06-14T23:11:24.647313+0000","last_unstale":"2024-06-14T23:13:18.752623+0000","last_undegraded":"2024-06-14T23:11:24.555289+0000","last_fullsized":"2024-06-14T23:11:24.381668+0000","mapping_epoch":323,"log_start":"324'7600","ondisk_log_start":"324'7600","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:48:12.784805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139067416,"num_objects":2123,"num_object_clones":0,"num_object_copies":6369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12451,"num_write_kb":466305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4187,"num_bytes_recovered":274399232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"1,3,7","objects":2122},{"shards":"3,7","objects":-361}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":255,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818510+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:13:05.818510+0000","last_peered":"2024-06-14T23:13:05.818510+0000","last_clean":"2024-06-14T23:13:05.818510+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:13:05.818510+0000","last_undegraded":"2024-06-14T23:13:05.818510+0000","last_fullsized":"2024-06-14T23:13:05.818510+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":261,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818717+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:13:05.818717+0000","last_peered":"2024-06-14T23:13:05.818717+0000","last_clean":"2024-06-14T23:13:05.818717+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:13:05.818717+0000","last_undegraded":"2024-06-14T23:13:05.818717+0000","last_fullsized":"2024-06-14T23:13:05.818717+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":342,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819333+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:13:05.819333+0000","last_peered":"2024-06-14T23:13:05.819333+0000","last_clean":"2024-06-14T23:13:05.819333+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:13:05.819333+0000","last_undegraded":"2024-06-14T23:13:05.819333+0000","last_fullsized":"2024-06-14T23:13:05.819333+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":619,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.810118+0000","last_change":"2024-06-14T23:11:23.406149+0000","last_active":"2024-06-14T23:13:05.810118+0000","last_peered":"2024-06-14T23:13:05.810118+0000","last_clean":"2024-06-14T23:13:05.810118+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:13:05.810118+0000","last_undegraded":"2024-06-14T23:13:05.810118+0000","last_fullsized":"2024-06-14T23:13:05.810118+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.751667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.751667+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.7","version":"340'12376","reported_seq":22130,"reported_epoch":340,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-14T23:13:18.802763+0000","last_change":"2024-06-14T23:11:24.669886+0000","last_active":"2024-06-14T23:13:18.802763+0000","last_peered":"2024-06-14T23:13:18.802763+0000","last_clean":"2024-06-14T23:10:25.005348+0000","last_became_active":"2024-06-14T23:11:24.634913+0000","last_became_peered":"2024-06-14T23:11:24.634913+0000","last_unstale":"2024-06-14T23:13:18.802763+0000","last_undegraded":"2024-06-14T23:13:18.802763+0000","last_fullsized":"2024-06-14T23:11:24.386154+0000","mapping_epoch":323,"log_start":"324'7500","ondisk_log_start":"324'7500","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4876,"log_dups_size":0,"ondisk_log_size":4876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:26:38.637559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139329536,"num_objects":2126,"num_object_clones":0,"num_object_copies":6378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1817,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4834,"num_bytes_recovered":316801024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"1,4,7","objects":1817},{"shards":"4,7","objects":309}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":138,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817113+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:13:05.817113+0000","last_peered":"2024-06-14T23:13:05.817113+0000","last_clean":"2024-06-14T23:13:05.817113+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:13:05.817113+0000","last_undegraded":"2024-06-14T23:13:05.817113+0000","last_fullsized":"2024-06-14T23:13:05.817113+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":309,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817465+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:13:05.817465+0000","last_peered":"2024-06-14T23:13:05.817465+0000","last_clean":"2024-06-14T23:13:05.817465+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:13:05.817465+0000","last_undegraded":"2024-06-14T23:13:05.817465+0000","last_fullsized":"2024-06-14T23:13:05.817465+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":731,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818419+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:13:05.818419+0000","last_peered":"2024-06-14T23:13:05.818419+0000","last_clean":"2024-06-14T23:13:05.818419+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:13:05.818419+0000","last_undegraded":"2024-06-14T23:13:05.818419+0000","last_fullsized":"2024-06-14T23:13:05.818419+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"340'12061","reported_seq":12583,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:16.730900+0000","last_change":"2024-06-14T23:11:24.664127+0000","last_active":"2024-06-14T23:13:16.730900+0000","last_peered":"2024-06-14T23:13:16.730900+0000","last_clean":"2024-06-14T23:10:25.013628+0000","last_became_active":"2024-06-14T23:10:27.203384+0000","last_became_peered":"2024-06-14T23:10:27.203384+0000","last_unstale":"2024-06-14T23:13:16.730900+0000","last_undegraded":"2024-06-14T23:11:24.661184+0000","last_fullsized":"2024-06-14T23:10:27.026246+0000","mapping_epoch":310,"log_start":"320'7100","ondisk_log_start":"320'7100","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4961,"log_dups_size":0,"ondisk_log_size":4961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:40.932486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":6633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12061,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":14548992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"6,7","objects":-1720}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":327,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822737+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:13:05.822737+0000","last_peered":"2024-06-14T23:13:05.822737+0000","last_clean":"2024-06-14T23:13:05.822737+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:13:05.822737+0000","last_undegraded":"2024-06-14T23:13:05.822737+0000","last_fullsized":"2024-06-14T23:13:05.822737+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":338,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819873+0000","last_change":"2024-06-14T23:11:23.427812+0000","last_active":"2024-06-14T23:13:05.819873+0000","last_peered":"2024-06-14T23:13:05.819873+0000","last_clean":"2024-06-14T23:13:05.819873+0000","last_became_active":"2024-06-14T23:11:23.427563+0000","last_became_peered":"2024-06-14T23:11:23.427563+0000","last_unstale":"2024-06-14T23:13:05.819873+0000","last_undegraded":"2024-06-14T23:13:05.819873+0000","last_fullsized":"2024-06-14T23:13:05.819873+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.941006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":699,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819206+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:13:05.819206+0000","last_peered":"2024-06-14T23:13:05.819206+0000","last_clean":"2024-06-14T23:13:05.819206+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:13:05.819206+0000","last_undegraded":"2024-06-14T23:13:05.819206+0000","last_fullsized":"2024-06-14T23:13:05.819206+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"340'12104","reported_seq":20485,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:16.700533+0000","last_change":"2024-06-14T23:11:24.656651+0000","last_active":"2024-06-14T23:13:16.700533+0000","last_peered":"2024-06-14T23:13:16.700533+0000","last_clean":"2024-06-14T23:10:25.016384+0000","last_became_active":"2024-06-14T23:11:24.643623+0000","last_became_peered":"2024-06-14T23:11:24.643623+0000","last_unstale":"2024-06-14T23:13:16.700533+0000","last_undegraded":"2024-06-14T23:11:24.519360+0000","last_fullsized":"2024-06-14T23:11:24.380111+0000","mapping_epoch":323,"log_start":"320'7100","ondisk_log_start":"320'7100","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:18:43.920069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146669568,"num_objects":2238,"num_object_clones":0,"num_object_copies":6714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12104,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4148,"num_bytes_recovered":271843328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"1,3,6","objects":2238},{"shards":"3,6","objects":-223}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":357,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.820283+0000","last_change":"2024-06-14T23:11:23.416264+0000","last_active":"2024-06-14T23:13:05.820283+0000","last_peered":"2024-06-14T23:13:05.820283+0000","last_clean":"2024-06-14T23:13:05.820283+0000","last_became_active":"2024-06-14T23:11:23.415946+0000","last_became_peered":"2024-06-14T23:11:23.415946+0000","last_unstale":"2024-06-14T23:13:05.820283+0000","last_undegraded":"2024-06-14T23:13:05.820283+0000","last_fullsized":"2024-06-14T23:13:05.820283+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:22.668569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":208,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822031+0000","last_change":"2024-06-14T23:10:26.045363+0000","last_active":"2024-06-14T23:13:05.822031+0000","last_peered":"2024-06-14T23:13:05.822031+0000","last_clean":"2024-06-14T23:13:05.822031+0000","last_became_active":"2024-06-14T23:10:26.043844+0000","last_became_peered":"2024-06-14T23:10:26.043844+0000","last_unstale":"2024-06-14T23:13:05.822031+0000","last_undegraded":"2024-06-14T23:13:05.822031+0000","last_fullsized":"2024-06-14T23:13:05.822031+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:11:27.891184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":722,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817779+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:13:05.817779+0000","last_peered":"2024-06-14T23:13:05.817779+0000","last_clean":"2024-06-14T23:13:05.817779+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:13:05.817779+0000","last_undegraded":"2024-06-14T23:13:05.817779+0000","last_fullsized":"2024-06-14T23:13:05.817779+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"340'12247","reported_seq":12542,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:16.742919+0000","last_change":"2024-06-14T23:10:39.484566+0000","last_active":"2024-06-14T23:13:16.742919+0000","last_peered":"2024-06-14T23:13:16.742919+0000","last_clean":"2024-06-14T23:10:24.998059+0000","last_became_active":"2024-06-14T23:10:27.239025+0000","last_became_peered":"2024-06-14T23:10:27.239025+0000","last_unstale":"2024-06-14T23:13:16.742919+0000","last_undegraded":"2024-06-14T23:10:27.071395+0000","last_fullsized":"2024-06-14T23:10:27.025672+0000","mapping_epoch":310,"log_start":"324'7300","ondisk_log_start":"324'7300","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:52:40.136978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151191552,"num_objects":2307,"num_object_clones":0,"num_object_copies":6921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12247,"num_write_kb":465728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":7602176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"3,6","objects":-1746}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":317,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.820052+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:13:05.820052+0000","last_peered":"2024-06-14T23:13:05.820052+0000","last_clean":"2024-06-14T23:13:05.820052+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:13:05.820052+0000","last_undegraded":"2024-06-14T23:13:05.820052+0000","last_fullsized":"2024-06-14T23:13:05.820052+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":138,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819686+0000","last_change":"2024-06-14T23:10:26.056252+0000","last_active":"2024-06-14T23:13:05.819686+0000","last_peered":"2024-06-14T23:13:05.819686+0000","last_clean":"2024-06-14T23:13:05.819686+0000","last_became_active":"2024-06-14T23:10:26.046629+0000","last_became_peered":"2024-06-14T23:10:26.046629+0000","last_unstale":"2024-06-14T23:13:05.819686+0000","last_undegraded":"2024-06-14T23:13:05.819686+0000","last_fullsized":"2024-06-14T23:13:05.819686+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:51:19.277386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":610,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.809660+0000","last_change":"2024-06-14T23:11:23.428346+0000","last_active":"2024-06-14T23:13:05.809660+0000","last_peered":"2024-06-14T23:13:05.809660+0000","last_clean":"2024-06-14T23:13:05.809660+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:13:05.809660+0000","last_undegraded":"2024-06-14T23:13:05.809660+0000","last_fullsized":"2024-06-14T23:13:05.809660+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.656210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.656210+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"340'12455","reported_seq":12518,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:18.779430+0000","last_change":"2024-06-14T23:09:41.992020+0000","last_active":"2024-06-14T23:13:18.779430+0000","last_peered":"2024-06-14T23:13:18.779430+0000","last_clean":"2024-06-14T23:13:18.779430+0000","last_became_active":"2024-06-14T23:09:41.991722+0000","last_became_peered":"2024-06-14T23:09:41.991722+0000","last_unstale":"2024-06-14T23:13:18.779430+0000","last_undegraded":"2024-06-14T23:13:18.779430+0000","last_fullsized":"2024-06-14T23:13:18.779430+0000","mapping_epoch":299,"log_start":"324'7600","ondisk_log_start":"324'7600","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:35:54.751263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140443648,"num_objects":2143,"num_object_clones":0,"num_object_copies":6429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12455,"num_write_kb":467136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":137,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819595+0000","last_change":"2024-06-14T23:10:26.047111+0000","last_active":"2024-06-14T23:13:05.819595+0000","last_peered":"2024-06-14T23:13:05.819595+0000","last_clean":"2024-06-14T23:13:05.819595+0000","last_became_active":"2024-06-14T23:10:26.045540+0000","last_became_peered":"2024-06-14T23:10:26.045540+0000","last_unstale":"2024-06-14T23:13:05.819595+0000","last_undegraded":"2024-06-14T23:13:05.819595+0000","last_fullsized":"2024-06-14T23:13:05.819595+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:04.131400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":318,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821123+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:13:05.821123+0000","last_peered":"2024-06-14T23:13:05.821123+0000","last_clean":"2024-06-14T23:13:05.821123+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:13:05.821123+0000","last_undegraded":"2024-06-14T23:13:05.821123+0000","last_fullsized":"2024-06-14T23:13:05.821123+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":578,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.822451+0000","last_change":"2024-06-14T23:09:17.382466+0000","last_active":"2024-06-14T23:13:05.822451+0000","last_peered":"2024-06-14T23:13:05.822451+0000","last_clean":"2024-06-14T23:13:05.822451+0000","last_became_active":"2024-06-14T23:09:12.030205+0000","last_became_peered":"2024-06-14T23:09:12.030205+0000","last_unstale":"2024-06-14T23:13:05.822451+0000","last_undegraded":"2024-06-14T23:13:05.822451+0000","last_fullsized":"2024-06-14T23:13:05.822451+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:09:17.382389+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:33.121472+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"340'12137","reported_seq":12190,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:18.368539+0000","last_change":"2024-06-14T23:09:29.408859+0000","last_active":"2024-06-14T23:13:18.368539+0000","last_peered":"2024-06-14T23:13:18.368539+0000","last_clean":"2024-06-14T23:13:18.368539+0000","last_became_active":"2024-06-14T23:09:29.408548+0000","last_became_peered":"2024-06-14T23:09:29.408548+0000","last_unstale":"2024-06-14T23:13:18.368539+0000","last_undegraded":"2024-06-14T23:13:18.368539+0000","last_fullsized":"2024-06-14T23:13:18.368539+0000","mapping_epoch":295,"log_start":"319'6800","ondisk_log_start":"319'6800","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5337,"log_dups_size":0,"ondisk_log_size":5337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:19.810173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139264000,"num_objects":2125,"num_object_clones":0,"num_object_copies":6375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":456384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":291,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.812809+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:13:05.812809+0000","last_peered":"2024-06-14T23:13:05.812809+0000","last_clean":"2024-06-14T23:13:05.812809+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:13:05.812809+0000","last_undegraded":"2024-06-14T23:13:05.812809+0000","last_fullsized":"2024-06-14T23:13:05.812809+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":235,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821060+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:13:05.821060+0000","last_peered":"2024-06-14T23:13:05.821060+0000","last_clean":"2024-06-14T23:13:05.821060+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:13:05.821060+0000","last_undegraded":"2024-06-14T23:13:05.821060+0000","last_fullsized":"2024-06-14T23:13:05.821060+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":619,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818763+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:13:05.818763+0000","last_peered":"2024-06-14T23:13:05.818763+0000","last_clean":"2024-06-14T23:13:05.818763+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:13:05.818763+0000","last_undegraded":"2024-06-14T23:13:05.818763+0000","last_fullsized":"2024-06-14T23:13:05.818763+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"340'12327","reported_seq":12380,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:16.671028+0000","last_change":"2024-06-14T23:09:29.403408+0000","last_active":"2024-06-14T23:13:16.671028+0000","last_peered":"2024-06-14T23:13:16.671028+0000","last_clean":"2024-06-14T23:13:16.671028+0000","last_became_active":"2024-06-14T23:09:29.403000+0000","last_became_peered":"2024-06-14T23:09:29.403000+0000","last_unstale":"2024-06-14T23:13:16.671028+0000","last_undegraded":"2024-06-14T23:13:16.671028+0000","last_fullsized":"2024-06-14T23:13:16.671028+0000","mapping_epoch":295,"log_start":"324'7400","ondisk_log_start":"324'7400","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:16.838865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143720448,"num_objects":2193,"num_object_clones":0,"num_object_copies":6579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12327,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":327,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819828+0000","last_change":"2024-06-14T23:10:26.040201+0000","last_active":"2024-06-14T23:13:05.819828+0000","last_peered":"2024-06-14T23:13:05.819828+0000","last_clean":"2024-06-14T23:13:05.819828+0000","last_became_active":"2024-06-14T23:10:26.039865+0000","last_became_peered":"2024-06-14T23:10:26.039865+0000","last_unstale":"2024-06-14T23:13:05.819828+0000","last_undegraded":"2024-06-14T23:13:05.819828+0000","last_fullsized":"2024-06-14T23:13:05.819828+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:34.621202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":305,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.826904+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:13:05.826904+0000","last_peered":"2024-06-14T23:13:05.826904+0000","last_clean":"2024-06-14T23:13:05.826904+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:13:05.826904+0000","last_undegraded":"2024-06-14T23:13:05.826904+0000","last_fullsized":"2024-06-14T23:13:05.826904+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":747,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817863+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:13:05.817863+0000","last_peered":"2024-06-14T23:13:05.817863+0000","last_clean":"2024-06-14T23:13:05.817863+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:13:05.817863+0000","last_undegraded":"2024-06-14T23:13:05.817863+0000","last_fullsized":"2024-06-14T23:13:05.817863+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"340'12441","reported_seq":12511,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:18.386739+0000","last_change":"2024-06-14T23:11:24.632384+0000","last_active":"2024-06-14T23:13:18.386739+0000","last_peered":"2024-06-14T23:13:18.386739+0000","last_clean":"2024-06-14T23:10:37.053680+0000","last_became_active":"2024-06-14T23:11:24.623684+0000","last_became_peered":"2024-06-14T23:11:24.623684+0000","last_unstale":"2024-06-14T23:13:18.386739+0000","last_undegraded":"2024-06-14T23:11:24.400366+0000","last_fullsized":"2024-06-14T23:11:24.379058+0000","mapping_epoch":323,"log_start":"319'7100","ondisk_log_start":"319'7100","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:34:53.331507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":6633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12441,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":-343}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":300,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817977+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:13:05.817977+0000","last_peered":"2024-06-14T23:13:05.817977+0000","last_clean":"2024-06-14T23:13:05.817977+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:13:05.817977+0000","last_undegraded":"2024-06-14T23:13:05.817977+0000","last_fullsized":"2024-06-14T23:13:05.817977+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":318,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.829761+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:13:05.829761+0000","last_peered":"2024-06-14T23:13:05.829761+0000","last_clean":"2024-06-14T23:13:05.829761+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:13:05.829761+0000","last_undegraded":"2024-06-14T23:13:05.829761+0000","last_fullsized":"2024-06-14T23:13:05.829761+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":773,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819911+0000","last_change":"2024-06-14T23:11:23.418239+0000","last_active":"2024-06-14T23:13:05.819911+0000","last_peered":"2024-06-14T23:13:05.819911+0000","last_clean":"2024-06-14T23:13:05.819911+0000","last_became_active":"2024-06-14T23:11:23.417873+0000","last_became_peered":"2024-06-14T23:11:23.417873+0000","last_unstale":"2024-06-14T23:13:05.819911+0000","last_undegraded":"2024-06-14T23:13:05.819911+0000","last_fullsized":"2024-06-14T23:13:05.819911+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:04:50.740175+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":326,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.812668+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:13:05.812668+0000","last_peered":"2024-06-14T23:13:05.812668+0000","last_clean":"2024-06-14T23:13:05.812668+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:13:05.812668+0000","last_undegraded":"2024-06-14T23:13:05.812668+0000","last_fullsized":"2024-06-14T23:13:05.812668+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":322,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.816953+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:13:05.816953+0000","last_peered":"2024-06-14T23:13:05.816953+0000","last_clean":"2024-06-14T23:13:05.816953+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:13:05.816953+0000","last_undegraded":"2024-06-14T23:13:05.816953+0000","last_fullsized":"2024-06-14T23:13:05.816953+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":695,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818490+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:13:05.818490+0000","last_peered":"2024-06-14T23:13:05.818490+0000","last_clean":"2024-06-14T23:13:05.818490+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:13:05.818490+0000","last_undegraded":"2024-06-14T23:13:05.818490+0000","last_fullsized":"2024-06-14T23:13:05.818490+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":224,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.819584+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:13:05.819584+0000","last_peered":"2024-06-14T23:13:05.819584+0000","last_clean":"2024-06-14T23:13:05.819584+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:13:05.819584+0000","last_undegraded":"2024-06-14T23:13:05.819584+0000","last_fullsized":"2024-06-14T23:13:05.819584+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":203,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817737+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:13:05.817737+0000","last_peered":"2024-06-14T23:13:05.817737+0000","last_clean":"2024-06-14T23:13:05.817737+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:13:05.817737+0000","last_undegraded":"2024-06-14T23:13:05.817737+0000","last_fullsized":"2024-06-14T23:13:05.817737+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":661,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817147+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:13:05.817147+0000","last_peered":"2024-06-14T23:13:05.817147+0000","last_clean":"2024-06-14T23:13:05.817147+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:13:05.817147+0000","last_undegraded":"2024-06-14T23:13:05.817147+0000","last_fullsized":"2024-06-14T23:13:05.817147+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":41,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.809961+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:13:05.809961+0000","last_peered":"2024-06-14T23:13:05.809961+0000","last_clean":"2024-06-14T23:13:05.809961+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:13:05.809961+0000","last_undegraded":"2024-06-14T23:13:05.809961+0000","last_fullsized":"2024-06-14T23:13:05.809961+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":300,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.821198+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:13:05.821198+0000","last_peered":"2024-06-14T23:13:05.821198+0000","last_clean":"2024-06-14T23:13:05.821198+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:13:05.821198+0000","last_undegraded":"2024-06-14T23:13:05.821198+0000","last_fullsized":"2024-06-14T23:13:05.821198+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":659,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.817709+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:13:05.817709+0000","last_peered":"2024-06-14T23:13:05.817709+0000","last_clean":"2024-06-14T23:13:05.817709+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:13:05.817709+0000","last_undegraded":"2024-06-14T23:13:05.817709+0000","last_fullsized":"2024-06-14T23:13:05.817709+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":352,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.826281+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:13:05.826281+0000","last_peered":"2024-06-14T23:13:05.826281+0000","last_clean":"2024-06-14T23:13:05.826281+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:13:05.826281+0000","last_undegraded":"2024-06-14T23:13:05.826281+0000","last_fullsized":"2024-06-14T23:13:05.826281+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":41,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.812243+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:13:05.812243+0000","last_peered":"2024-06-14T23:13:05.812243+0000","last_clean":"2024-06-14T23:13:05.812243+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:13:05.812243+0000","last_undegraded":"2024-06-14T23:13:05.812243+0000","last_fullsized":"2024-06-14T23:13:05.812243+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":772,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-06-14T23:13:05.818156+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:13:05.818156+0000","last_peered":"2024-06-14T23:13:05.818156+0000","last_clean":"2024-06-14T23:13:05.818156+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:13:05.818156+0000","last_undegraded":"2024-06-14T23:13:05.818156+0000","last_fullsized":"2024-06-14T23:13:05.818156+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":2262564888,"num_objects":34525,"num_object_clones":0,"num_object_copies":103575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30181,"num_objects_misplaced":1817,"num_objects_unfound":0,"num_objects_dirty":34525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198481,"num_write_kb":7456129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18814,"num_bytes_recovered":1232994304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7837200384,"data_stored":7837188168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78262,"ondisk_log_size":78262,"up":48,"acting":37,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":821296,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":103,"num_write":26,"num_write_kb":273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":365100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234471,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2458692,"kb_used_data":1259004,"kb_used_omap":1,"kb_used_meta":1199614,"kb_avail":91286460,"statfs":{"total":95995035648,"available":93477335040,"internally_reserved":0,"allocated":1289220096,"data_stored":1287449602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1228405080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":0.95599999999999996,"15min":0.99099999999999999},"min":{"1min":0.24199999999999999,"5min":0.218,"15min":0.19800000000000001},"max":{"1min":5.7229999999999999,"5min":5.7229999999999999,"15min":6.9539999999999997},"last":3.7389999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.169,"15min":1.1950000000000001},"min":{"1min":0.29599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":5.5350000000000001,"5min":5.5350000000000001,"15min":6.9550000000000001},"last":2.1019999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.196,"15min":1.306},"min":{"1min":0.38600000000000001,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":5.5629999999999997,"5min":5.5629999999999997,"15min":12.010999999999999},"last":2.157},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.1950000000000001,"15min":1.337},"min":{"1min":0.34799999999999998,"5min":0.29599999999999999,"15min":0.22900000000000001},"max":{"1min":5.7480000000000002,"5min":5.7480000000000002,"15min":13.531000000000001},"last":3.7109999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.1970000000000001,"5min":5.1970000000000001,"15min":5.1970000000000001},"last":2.5670000000000002},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.2489999999999997,"5min":5.2489999999999997,"15min":5.2489999999999997},"last":1.3180000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5609999999999999,"15min":1.4179999999999999},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":5.2270000000000003,"5min":20.645,"15min":20.645},"last":2.6560000000000001},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.625,"15min":1.494},"min":{"1min":0.36499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.3639999999999999,"5min":22.042999999999999,"15min":22.042999999999999},"last":2.0739999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.127,"15min":1.077},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.3319999999999999,"5min":5.3319999999999999,"15min":11.342000000000001},"last":2.2519999999999998},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.208,"15min":1.1459999999999999},"min":{"1min":0.32600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3769999999999998,"5min":8.5899999999999999,"15min":16.879000000000001},"last":1.2789999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:12:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.026,"15min":1.109},"min":{"1min":0.39100000000000001,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":5.6849999999999996,"5min":5.6849999999999996,"15min":6.9000000000000004},"last":3.6709999999999998},{"interface":"front","average":{"1min":1.407,"5min":1.222,"15min":1.1499999999999999},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.3949999999999996,"5min":5.3949999999999996,"15min":6.8529999999999998},"last":2.0419999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":2.137,"5min":1.7190000000000001,"15min":1.448},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.7750000000000004,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":4.5},{"interface":"front","average":{"1min":3.0219999999999998,"5min":1.694,"15min":1.486},"min":{"1min":0.48199999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.087999999999999,"5min":10.087999999999999,"15min":10.087999999999999},"last":2.1269999999999998}]}]},{"osd":6,"up_from":272,"seq":1168231104577,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1728496,"kb_used_data":1327272,"kb_used_omap":7,"kb_used_meta":401144,"kb_avail":92016656,"statfs":{"total":95995035648,"available":94225055744,"internally_reserved":0,"allocated":1359126528,"data_stored":1357376047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":410771898},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.073,"15min":1.609},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.7330000000000001,"5min":6.6440000000000001,"15min":6.6440000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.095,"5min":1.143,"15min":1.6639999999999999},"min":{"1min":0.38800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.9430000000000001,"5min":6.6790000000000003,"15min":6.6790000000000003},"last":0.90100000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.3080000000000001,"15min":2.0859999999999999},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.0170000000000003,"5min":8.173,"15min":8.173},"last":1.157},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.238,"15min":1.996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.9039999999999999,"5min":10.380000000000001,"15min":10.380000000000001},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Jun 14 23:12:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":3.4489999999999998,"5min":3.4489999999999998,"15min":3.4489999999999998},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":3.3929999999999998},"last":0.80200000000000005}]},{"osd":3,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.2370000000000001,"15min":1.7629999999999999},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.7879999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.722,"5min":1.629,"15min":1.992},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.4969999999999999,"5min":8.4740000000000002,"15min":8.4740000000000002},"last":1.026}]},{"osd":4,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.4079999999999999,"15min":1.444},"min":{"1min":0.32600000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.5609999999999999,"5min":11.010999999999999,"15min":11.010999999999999},"last":1.135},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.1870000000000001,"15min":1.2749999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.2309999999999999,"5min":5.2309999999999999,"15min":5.2309999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0580000000000001,"15min":1.3440000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.8079999999999998,"5min":7.7469999999999999,"15min":7.7469999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.8680000000000001,"5min":1.5209999999999999,"15min":1.6160000000000001},"min":{"1min":0.38300000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.4359999999999999,"5min":7.8680000000000003,"15min":7.8680000000000003},"last":0.45400000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":2.0310000000000001,"5min":1.746,"15min":1.758},"min":{"1min":0.33100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":7.54,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":0.997},{"interface":"front","average":{"1min":1.8180000000000001,"5min":1.474,"15min":1.554},"min":{"1min":0.54400000000000004,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.9849999999999999,"5min":7.4619999999999997,"15min":7.4619999999999997},"last":1.1080000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757746,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1464428,"kb_used_data":288996,"kb_used_omap":1,"kb_used_meta":1175422,"kb_avail":92280724,"statfs":{"total":95995035648,"available":94495461376,"internally_reserved":0,"allocated":295931904,"data_stored":294181329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1203632472},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.70299999999999996,"15min":1.2949999999999999},"min":{"1min":0.255,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":1.5189999999999999,"5min":4.0579999999999998,"15min":14.94},"last":1.052},{"interface":"front","average":{"1min":0.505,"5min":0.88200000000000001,"15min":1.3149999999999999},"min":{"1min":0.34799999999999998,"5min":0.26100000000000001,"15min":0.219},"max":{"1min":1.8680000000000001,"5min":8.3520000000000003,"15min":9.3819999999999997},"last":0.39500000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.751,"15min":0.94899999999999995},"min":{"1min":0.30099999999999999,"5min":0.245,"15min":0.23999999999999999},"max":{"1min":2.79,"5min":4.1589999999999998,"15min":7.4370000000000003},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.82799999999999996,"15min":1.1479999999999999},"min":{"1min":0.28399999999999997,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":2.6240000000000001,"5min":5.1920000000000002,"15min":9.5039999999999996},"last":1.0880000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.0579999999999998,"5min":3.0579999999999998,"15min":3.0579999999999998},"last":1.131},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.125,"5min":3.125,"15min":3.125},"last":1.151}]},{"osd":3,"last update":"Fri Jun 14 23:12:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.081,"15min":1.3979999999999999},"min":{"1min":0.31,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":1.9370000000000001,"5min":8.4350000000000005,"15min":10.358000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.629,"5min":1.006,"15min":1.4159999999999999},"min":{"1min":0.33600000000000002,"5min":0.318,"15min":0.28899999999999998},"max":{"1min":1.585,"5min":7.4290000000000003,"15min":14.914},"last":1.1100000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.71599999999999997,"15min":0.89600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.1719999999999999,"5min":3.012,"15min":14.352},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.80500000000000005,"15min":1.0489999999999999},"min":{"1min":0.34100000000000003,"5min":0.28799999999999998,"15min":0.253},"max":{"1min":1.9950000000000001,"5min":8.4039999999999999,"15min":9.5980000000000008},"last":0.46700000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.95799999999999996,"15min":1.349},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.0629999999999997,"5min":9.4710000000000001,"15min":9.4710000000000001},"last":3.4119999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.1180000000000001,"15min":1.6659999999999999},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.028,"5min":10.632,"15min":10.632},"last":1.8149999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:12:27 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.256,"15min":1.246},"min":{"1min":0.39100000000000001,"5min":0.29099999999999998,"15min":0.27900000000000003},"max":{"1min":26.981999999999999,"5min":26.981999999999999,"15min":26.981999999999999},"last":1.264},{"interface":"front","average":{"1min":2.0670000000000002,"5min":1.333,"15min":1.222},"min":{"1min":0.42499999999999999,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":23.326000000000001,"5min":23.326000000000001,"15min":23.326000000000001},"last":0.67800000000000005}]}]},{"osd":4,"up_from":30,"seq":128849019382,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1746356,"kb_used_data":855596,"kb_used_omap":1,"kb_used_meta":890686,"kb_avail":91998796,"statfs":{"total":95995035648,"available":94206767104,"internally_reserved":0,"allocated":876130304,"data_stored":874375291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912062808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.83699999999999997,"15min":1.1459999999999999},"min":{"1min":0.26500000000000001,"5min":0.217,"15min":0.17100000000000001},"max":{"1min":3.177,"5min":6.875,"15min":10.567},"last":2.968},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.81200000000000006,"15min":1.1120000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":3.323,"5min":6.1379999999999999,"15min":8.3390000000000004},"last":2.0190000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.339,"15min":1.482},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23100000000000001},"max":{"1min":3.3759999999999999,"5min":42.656999999999996,"15min":56.917000000000002},"last":2.6640000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.3109999999999999,"15min":1.5029999999999999},"min":{"1min":0.317,"5min":0.249,"15min":0.249},"max":{"1min":3.2090000000000001,"5min":42.192999999999998,"15min":56.789999999999999},"last":3.0419999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:12:23 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.8130000000000002,"5min":2.8130000000000002,"15min":2.8130000000000002},"last":2.637},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":2.6120000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.6930000000000001,"15min":1.3360000000000001},"min":{"1min":0.316,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":3.3809999999999998,"5min":44.389000000000003,"15min":44.389000000000003},"last":2.044},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.8260000000000001,"15min":1.5},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":3.1000000000000001,"5min":46.101999999999997,"15min":46.101999999999997},"last":3.0209999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:12:36 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.85799999999999998,"15min":1.5529999999999999},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.4159999999999999,"5min":4.04,"15min":139.13200000000001},"last":1.948},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":1.613},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.2360000000000002,"5min":6.9290000000000003,"15min":138.86000000000001},"last":3.0960000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:13:09 2024","interfaces":[{"interface":"back","average":{"1min":7.6500000000000004,"5min":2.427,"15min":1.542},"min":{"1min":0.39200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":122.88500000000001,"5min":122.88500000000001,"15min":122.88500000000001},"last":2.9980000000000002},{"interface":"front","average":{"1min":7.8129999999999997,"5min":2.4550000000000001,"15min":1.9179999999999999},"min":{"1min":0.35699999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":122.21299999999999,"5min":122.21299999999999,"15min":122.21299999999999},"last":1.99}]},{"osd":7,"last update":"Fri Jun 14 23:12:17 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.1859999999999999,"15min":1.087},"min":{"1min":0.33800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.0439999999999996,"5min":6.5910000000000002,"15min":8.9489999999999998},"last":4.8789999999999996},{"interface":"front","average":{"1min":1.583,"5min":1.2210000000000001,"15min":1.1659999999999999},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.4219999999999997,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":4.9349999999999996}]}]},{"osd":3,"up_from":138,"seq":592705487081,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2492776,"kb_used_data":1613216,"kb_used_omap":4,"kb_used_meta":879419,"kb_avail":91252376,"statfs":{"total":95995035648,"available":93442433024,"internally_reserved":0,"allocated":1651933184,"data_stored":1650185936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":900525534},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":1.083,"15min":1.347},"min":{"1min":0.187,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.77200000000000002,"5min":25.190000000000001,"15min":25.190000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":1.2050000000000001,"15min":1.3560000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.504,"5min":25.652000000000001,"15min":25.652000000000001},"last":0.26400000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.304,"15min":1.4530000000000001},"min":{"1min":0.245,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":2.649,"5min":25.754000000000001,"15min":25.754000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.548,"15min":1.623},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.25},"max":{"1min":2.6899999999999999,"5min":27.992999999999999,"15min":27.992999999999999},"last":0.50700000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.7549999999999999,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.972,"5min":2.972,"15min":2.972},"last":0.52100000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.5760000000000001,"15min":1.74},"min":{"1min":0.34100000000000003,"5min":0.28299999999999997,"15min":0.27900000000000003},"max":{"1min":2.1269999999999998,"5min":25.094000000000001,"15min":73.263000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.3959999999999999,"15min":1.6659999999999999},"min":{"1min":0.34499999999999997,"5min":0.27600000000000002,"15min":0.27300000000000002},"max":{"1min":2.8559999999999999,"5min":25.5,"15min":73.5},"last":2.5089999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":1.5760000000000001,"15min":1.7589999999999999},"min":{"1min":0.38500000000000001,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":2.0369999999999999,"5min":25.521000000000001,"15min":81.438999999999993},"last":2.5579999999999998},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.6240000000000001,"15min":1.7989999999999999},"min":{"1min":0.41899999999999998,"5min":0.32800000000000001,"15min":0.29299999999999998},"max":{"1min":3.0739999999999998,"5min":25.84,"15min":81.626999999999995},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Jun 14 23:13:04 2024","interfaces":[{"interface":"back","average":{"1min":2.2410000000000001,"5min":1.8360000000000001,"15min":1.752},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":17.141999999999999,"5min":25.869,"15min":25.869},"last":0.66500000000000004},{"interface":"front","average":{"1min":2.0680000000000001,"5min":1.641,"15min":1.5529999999999999},"min":{"1min":0.38100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":20.056999999999999,"5min":25.143000000000001,"15min":25.143000000000001},"last":2.633}]},{"osd":7,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.4379999999999999,"15min":1.4330000000000001},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.30499999999999999},"max":{"1min":2.4940000000000002,"5min":25.120000000000001,"15min":25.120000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.24,"5min":1.5940000000000001,"15min":1.3979999999999999},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":3.4300000000000002,"5min":25.472000000000001,"15min":25.472000000000001},"last":2.927}]}]},{"osd":2,"up_from":322,"seq":1382979469336,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320752,"kb_used_data":171556,"kb_used_omap":10,"kb_used_meta":149173,"kb_avail":93424400,"statfs":{"total":95995035648,"available":95666585600,"internally_reserved":0,"allocated":175673344,"data_stored":173922758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":152753702},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.93700000000000006}]},{"osd":1,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.874},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.83999999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.54100000000000004}]},{"osd":5,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.544,"5min":2.544,"15min":2.544},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.90700000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.6950000000000003,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":0.499}]},{"osd":7,"last update":"Fri Jun 14 23:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":3.008,"5min":3.008,"15min":3.008},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.548,"5min":4.548,"15min":4.548},"last":0.58099999999999996}]}]},{"osd":1,"up_from":0,"seq":498216206625,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2335309824,"data_stored":2335309824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":168366080,"data_stored":168362008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1603604480,"data_stored":1603600408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":855506944,"data_stored":855506944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":285671424,"data_stored":285671424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1329659904,"data_stored":1329659904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1259081728,"data_stored":1259077656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:13:24.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph status --format=json 2024-06-14T23:13:24.892 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:24.911 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:24.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:25.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:25.041 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:25.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:25.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:24 smithi086 ceph-mon[97913]: from='client.46721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:13:25.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:24 smithi086 ceph-mon[97913]: pgmap v1277: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.0 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 913 op/s; 30181/95931 objects degraded (31.461%); 1817/95931 objects misplaced (1.894%); 5.7 MiB/s, 90 objects/s recovering 2024-06-14T23:13:25.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:13:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:13:24] "GET /metrics HTTP/1.1" 200 36713 "" "Prometheus/2.43.0" 2024-06-14T23:13:25.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:24 smithi086 ceph-mon[102754]: from='client.46721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:13:25.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:24 smithi086 ceph-mon[102754]: pgmap v1277: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.0 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 913 op/s; 30181/95931 objects degraded (31.461%); 1817/95931 objects misplaced (1.894%); 5.7 MiB/s, 90 objects/s recovering 2024-06-14T23:13:25.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:25.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:25.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:25.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:24 smithi137 ceph-mon[90634]: from='client.46721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:13:25.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:24 smithi137 ceph-mon[90634]: pgmap v1277: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 2.0 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 913 op/s; 30181/95931 objects degraded (31.461%); 1817/95931 objects misplaced (1.894%); 5.7 MiB/s, 90 objects/s recovering 2024-06-14T23:13:25.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:25.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:25.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:25.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:25.456 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:25.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:25.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-14T23:13:25.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:25.570 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:25.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:25.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:25.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:25.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:25.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:25.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:25.905 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:13:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:25.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:25.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:26.000 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:13:26.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:26.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:26.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:26.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:25 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:13:26.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/525073011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:26.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:25 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:26.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:25 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:13:26.089 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/525073011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:26.089 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:25 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:26.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:26.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:26.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:25 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:13:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/525073011' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:25 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:26.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:26.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:26.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:26.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:26.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:26.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:26.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:26.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:26.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:26.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:26.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:26.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:26.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:27.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:27.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:27.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:27.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:27.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:27.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:27.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:26 smithi137 ceph-mon[90634]: pgmap v1278: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.8 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 943 op/s; 29919/89865 objects degraded (33.293%); 1541/89865 objects misplaced (1.715%); 7.2 MiB/s, 114 objects/s recovering 2024-06-14T23:13:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:26 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:13:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:26 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:13:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:26 smithi137 ceph-mon[90634]: osdmap e341: 8 total, 8 up, 6 in 2024-06-14T23:13:27.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:27.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:26 smithi086 ceph-mon[97913]: pgmap v1278: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.8 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 943 op/s; 29919/89865 objects degraded (33.293%); 1541/89865 objects misplaced (1.715%); 7.2 MiB/s, 114 objects/s recovering 2024-06-14T23:13:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:26 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:13:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:26 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:13:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:26 smithi086 ceph-mon[97913]: osdmap e341: 8 total, 8 up, 6 in 2024-06-14T23:13:27.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:26 smithi086 ceph-mon[102754]: pgmap v1278: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.8 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 943 op/s; 29919/89865 objects degraded (33.293%); 1541/89865 objects misplaced (1.715%); 7.2 MiB/s, 114 objects/s recovering 2024-06-14T23:13:27.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:26 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:13:27.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:26 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:13:27.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:26 smithi086 ceph-mon[102754]: osdmap e341: 8 total, 8 up, 6 in 2024-06-14T23:13:27.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:27.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:27.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:27.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:27.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:27.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:27.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:27.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:27.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:27.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:27.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:27.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:27.981 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:27.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:28.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:28.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:28.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:28.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:28.258 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:28.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:28.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:28.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:28.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:28.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:28.655 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:28 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:13:28.655 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:28.655 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:28.655 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:28 smithi137 ceph-mon[90634]: pgmap v1280: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 922 op/s; 29919/85677 objects degraded (34.921%); 1541/85677 objects misplaced (1.799%); 6.7 MiB/s, 107 objects/s recovering 2024-06-14T23:13:28.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:28.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:28.775 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:28 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:13:28.775 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:28.775 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:28.775 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:28 smithi086 ceph-mon[102754]: pgmap v1280: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 922 op/s; 29919/85677 objects degraded (34.921%); 1541/85677 objects misplaced (1.799%); 6.7 MiB/s, 107 objects/s recovering 2024-06-14T23:13:28.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:28 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:13:28.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:28.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:28.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:28 smithi086 ceph-mon[97913]: pgmap v1280: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 922 op/s; 29919/85677 objects degraded (34.921%); 1541/85677 objects misplaced (1.799%); 6.7 MiB/s, 107 objects/s recovering 2024-06-14T23:13:28.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:28.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:28.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:28.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:28.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:29.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:29.144 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:29.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:29.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:29.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:29.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:29.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:29.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:13:29.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:29.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:29.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:29.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:29.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:29.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:29 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 29919/85677 objects degraded (34.921%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:29.609 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:29 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 29919/85677 objects degraded (34.921%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:29.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:29.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:29.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:29.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:29 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 29919/85677 objects degraded (34.921%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:29.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:29.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:29.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:29.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:29.923 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:13:29.923 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:13:29.956 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:29.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:29.963 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:13:29.963 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:13:30.004 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:13:30.004 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:13:30.046 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:13:30.046 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:13:30.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:30.079 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:13:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:13:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:30.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:30.115 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:13:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:13:30.146 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:13:30.146 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:13:30.183 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:13:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:13:30.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:30.202 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:30.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:30.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:30.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:30.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:30.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:30.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:30.537 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:30 smithi086 ceph-mon[97913]: pgmap v1281: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.6 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 928 op/s; 29919/79434 objects degraded (37.665%); 1265/79434 objects misplaced (1.593%); 7.0 MiB/s, 111 objects/s recovering 2024-06-14T23:13:30.537 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/743033857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:30.537 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:30 smithi086 ceph-mon[102754]: pgmap v1281: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.6 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 928 op/s; 29919/79434 objects degraded (37.665%); 1265/79434 objects misplaced (1.593%); 7.0 MiB/s, 111 objects/s recovering 2024-06-14T23:13:30.537 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/743033857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:30.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:30.599 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:13:30.599 INFO:teuthology.orchestra.run.smithi086.stdout:{"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29919/79434 objects degraded (37.665%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1835,"monmap":{"epoch":4,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":341,"num_osds":8,"num_up_osds":8,"osd_up_since":1718406683,"num_in_osds":6,"osd_in_since":1718406625,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":108},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":119,"num_pools":5,"num_objects":26478,"data_bytes":1734385134,"bytes_used":9465933824,"bytes_avail":566504280064,"bytes_total":575970213888,"degraded_objects":29919,"degraded_total":79434,"degraded_ratio":0.3766523151295415,"misplaced_objects":1265,"misplaced_total":79434,"misplaced_ratio":0.015925170581866707,"recovering_objects_per_sec":111,"recovering_bytes_per_sec":7317725,"recovering_keys_per_sec":0,"num_objects_recovered":1117,"num_bytes_recovered":73203712,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":928},"fsmap":{"epoch":9,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi086.nhdjdb","status":"up:active","gid":24548}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.86:8443/","prometheus":"http://172.21.15.86:9283/"}},"servicemap":{"epoch":28,"modified":"2024-06-14T23:12:24.252974+0000","services":{}},"progress_events":{"af1960ac-63de-49e3-896b-122ab3eb5dd7":{"message":"Upgrade to 18.2.2-1443-g6f4b0956 (12m)\n [============................] (remaining: 15m)","progress":0.44999998807907104,"add_to_ceph_s":true},"c77b16e3-338f-456c-83d1-eff5a59f3260":{"message":"Global Recovery Event (3m)\n [=========================...] (remaining: 18s)","progress":0.90756303071975708,"add_to_ceph_s":true}}} 2024-06-14T23:13:30.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:30.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:30 smithi137 ceph-mon[90634]: pgmap v1281: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.6 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 928 op/s; 29919/79434 objects degraded (37.665%); 1265/79434 objects misplaced (1.593%); 7.0 MiB/s, 111 objects/s recovering 2024-06-14T23:13:30.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/743033857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:30.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:30.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:30.880 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:30.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:30.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:31.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:31.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:31.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:31.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:31.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:31.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:31.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:31.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:31.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:31.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:31.451 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:31.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:31.541 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:13:31.541 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:13:31.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:31.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:31.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:31.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:31.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:31.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:31.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3061827866' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-14T23:13:31.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:31.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3061827866' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-14T23:13:31.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3061827866' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-14T23:13:31.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:31.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:31.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:32.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:32.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:32.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:32.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:32.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:32.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:32.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:32.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T23:13:32.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:32.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:32.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:32.292 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-06-14T23:13:32.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:13:32.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:32.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:32.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:32.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:32 smithi137 ceph-mon[90634]: pgmap v1282: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 922 op/s; 29630/75975 objects degraded (39.000%); 1265/75975 objects misplaced (1.665%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:13:32.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:32.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:32 smithi086 ceph-mon[97913]: pgmap v1282: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 922 op/s; 29630/75975 objects degraded (39.000%); 1265/75975 objects misplaced (1.665%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:13:32.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:32 smithi086 ceph-mon[102754]: pgmap v1282: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 922 op/s; 29630/75975 objects degraded (39.000%); 1265/75975 objects misplaced (1.665%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:13:32.618 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:32.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:32.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:32.756 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:32.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:32.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:32.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:32.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:32.948 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:32.967 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:33.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:33.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:13:33.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:33.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:33.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:33.230 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:33.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:33.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:33.370 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:33.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:33.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:33.508 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:33.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:33.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:33 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:13:33.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:33 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:13:33.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 29630/75975 objects degraded (39.000%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:33.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:33 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:13:33.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:33 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:13:33.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 29630/75975 objects degraded (39.000%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:33.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:33.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:33.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:33.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:33.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:33 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:13:33.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:33 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:13:33.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 29630/75975 objects degraded (39.000%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:33.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:33.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:33.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:33.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:33.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:34.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:34.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:34.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:34.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:34.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:34.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:34.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:34.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:34.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:34.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:34.467 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:13:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:34.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:34.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:34.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:34 smithi086 ceph-mon[97913]: pgmap v1283: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.4 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 909 op/s; 29630/68625 objects degraded (43.177%); 1265/68625 objects misplaced (1.843%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:13:34.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1520044577' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:13:34.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:34 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:13:34.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:34 smithi086 ceph-mon[102754]: pgmap v1283: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.4 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 909 op/s; 29630/68625 objects degraded (43.177%); 1265/68625 objects misplaced (1.843%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:13:34.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1520044577' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:13:34.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:34 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:13:34.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:34.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:34.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:34.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:34.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:34 smithi137 ceph-mon[90634]: pgmap v1283: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.4 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 909 op/s; 29630/68625 objects degraded (43.177%); 1265/68625 objects misplaced (1.843%); 6.9 MiB/s, 110 objects/s recovering 2024-06-14T23:13:34.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1520044577' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:13:34.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:34 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:13:34.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:34.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:34.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:34.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:34.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:35.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:35.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:35.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:35.135 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:13:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:13:34] "GET /metrics HTTP/1.1" 200 36712 "" "Prometheus/2.43.0" 2024-06-14T23:13:35.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:35.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:35.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:35.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:35.334 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:35.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:35.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:35.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:35.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:35.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:35.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:35.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:35.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:35.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:35 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:35.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:35 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:13:35.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:35 smithi137 ceph-mon[90634]: osdmap e342: 8 total, 8 up, 6 in 2024-06-14T23:13:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:35.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:35.762 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:35 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:35.762 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:35 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:13:35.762 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:35 smithi086 ceph-mon[97913]: osdmap e342: 8 total, 8 up, 6 in 2024-06-14T23:13:35.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:35 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:35.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:35 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:13:35.763 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:35 smithi086 ceph-mon[102754]: osdmap e342: 8 total, 8 up, 6 in 2024-06-14T23:13:35.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:35.892 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:35.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:35.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:36.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:36.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:36.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:36.165 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:36.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:36.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:36.292 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:36.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:36.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:36.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:36.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:36.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:36.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:36.579 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:36.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:36.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:36.722 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:36.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:36.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:36 smithi137 ceph-mon[90634]: pgmap v1285: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.3 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 926 op/s; 29338/62505 objects degraded (46.937%); 945/62505 objects misplaced (1.512%); 7.5 MiB/s, 119 objects/s recovering 2024-06-14T23:13:36.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:36.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:36 smithi086 ceph-mon[97913]: pgmap v1285: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.3 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 926 op/s; 29338/62505 objects degraded (46.937%); 945/62505 objects misplaced (1.512%); 7.5 MiB/s, 119 objects/s recovering 2024-06-14T23:13:36.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:36 smithi086 ceph-mon[102754]: pgmap v1285: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.3 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 926 op/s; 29338/62505 objects degraded (46.937%); 945/62505 objects misplaced (1.512%); 7.5 MiB/s, 119 objects/s recovering 2024-06-14T23:13:36.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:36.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:36.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:36.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:36.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:36.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:37.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:37.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:37.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:37.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:37.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:37.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:37.320 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:37.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:37.408 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:37.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:37.510 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:37.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:37.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:37.639 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:13:37.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:37.648 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:13:37.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:37.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:37.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:37.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:37.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:37.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:37.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:37.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:38.059 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:13:38.059 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:13:38.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:38.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:38.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:38.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:38.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:38.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:38.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:13:38.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:38.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:38.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:38.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:38.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:38.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:38.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:38.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:38.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:38.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:38.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:38.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:38.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:38.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:38.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:39.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:39.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:38 smithi086 ceph-mon[97913]: from='client.46745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:13:39.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:38 smithi086 ceph-mon[97913]: pgmap v1286: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.2 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 925 op/s; 29338/57894 objects degraded (50.675%); 945/57894 objects misplaced (1.632%); 7.4 MiB/s, 117 objects/s recovering 2024-06-14T23:13:39.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 29338/57894 objects degraded (50.675%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:39.046 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:38 smithi086 ceph-mon[102754]: from='client.46745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:13:39.046 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:38 smithi086 ceph-mon[102754]: pgmap v1286: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.2 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 925 op/s; 29338/57894 objects degraded (50.675%); 945/57894 objects misplaced (1.632%); 7.4 MiB/s, 117 objects/s recovering 2024-06-14T23:13:39.046 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 29338/57894 objects degraded (50.675%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:39.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:39.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:39.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-14T23:13:39.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:39.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:39.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:39.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:39.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:38 smithi137 ceph-mon[90634]: from='client.46745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:13:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:38 smithi137 ceph-mon[90634]: pgmap v1286: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.2 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 925 op/s; 29338/57894 objects degraded (50.675%); 945/57894 objects misplaced (1.632%); 7.4 MiB/s, 117 objects/s recovering 2024-06-14T23:13:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 29338/57894 objects degraded (50.675%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:39.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:39.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:39.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:39.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:39.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:39.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:39.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:39.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:39.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:39.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:39.734 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:39.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:39.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:39.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:39.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:39.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:40.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:40.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:40.096 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1285,"stamp":"2024-06-14T23:13:35.734617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1364565486,"num_objects":20835,"num_object_clones":0,"num_object_copies":62505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29338,"num_objects_misplaced":945,"num_objects_unfound":0,"num_objects_dirty":20835,"num_whiteouts":0,"num_read":56,"num_read_kb":104,"num_write":212283,"num_write_kb":7456431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20549,"num_bytes_recovered":1345754326,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83125,"ondisk_log_size":83125,"up":357,"acting":346,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":8336284,"kb_used_data":3307048,"kb_used_omap":27,"kb_used_meta":5029028,"kb_avail":554134628,"statfs":{"total":575970213888,"available":567433859072,"internally_reserved":0,"allocated":3386417152,"data_stored":3375838029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":5149725126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-597688320,"num_objects":-9120,"num_object_clones":0,"num_object_copies":-27360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-581,"num_objects_misplaced":-596,"num_objects_unfound":0,"num_objects_dirty":-9120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1177,"num_bytes_recovered":77135872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.838605"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":721,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.907255+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:13:25.907255+0000","last_peered":"2024-06-14T23:13:25.907255+0000","last_clean":"2024-06-14T23:13:25.907255+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:13:25.907255+0000","last_undegraded":"2024-06-14T23:13:25.907255+0000","last_fullsized":"2024-06-14T23:13:25.907255+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":158,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.861185+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:13:14.861185+0000","last_peered":"2024-06-14T23:13:14.861185+0000","last_clean":"2024-06-14T23:13:14.861185+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:13:14.861185+0000","last_undegraded":"2024-06-14T23:13:14.861185+0000","last_fullsized":"2024-06-14T23:13:14.861185+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":328,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855106+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:13:14.855106+0000","last_peered":"2024-06-14T23:13:14.855106+0000","last_clean":"2024-06-14T23:13:14.855106+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:13:14.855106+0000","last_undegraded":"2024-06-14T23:13:14.855106+0000","last_fullsized":"2024-06-14T23:13:14.855106+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":234,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:22.351046+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:13:22.351046+0000","last_peered":"2024-06-14T23:13:22.351046+0000","last_clean":"2024-06-14T23:13:22.351046+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:13:22.351046+0000","last_undegraded":"2024-06-14T23:13:22.351046+0000","last_fullsized":"2024-06-14T23:13:22.351046+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":123,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.836131+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:13:14.836131+0000","last_peered":"2024-06-14T23:13:14.836131+0000","last_clean":"2024-06-14T23:13:14.836131+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:13:14.836131+0000","last_undegraded":"2024-06-14T23:13:14.836131+0000","last_fullsized":"2024-06-14T23:13:14.836131+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":763,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:32.452266+0000","last_change":"2024-06-14T23:13:32.452266+0000","last_active":"2024-06-14T23:13:32.452266+0000","last_peered":"2024-06-14T23:13:32.452266+0000","last_clean":"2024-06-14T23:13:32.452266+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:13:32.452266+0000","last_undegraded":"2024-06-14T23:13:32.452266+0000","last_fullsized":"2024-06-14T23:13:32.452266+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:09:56.196559+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:09:56.196559+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":304,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859895+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:13:14.859895+0000","last_peered":"2024-06-14T23:13:14.859895+0000","last_clean":"2024-06-14T23:13:14.859895+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:13:14.859895+0000","last_undegraded":"2024-06-14T23:13:14.859895+0000","last_fullsized":"2024-06-14T23:13:14.859895+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":345,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.854056+0000","last_change":"2024-06-14T23:11:23.413009+0000","last_active":"2024-06-14T23:13:14.854056+0000","last_peered":"2024-06-14T23:13:14.854056+0000","last_clean":"2024-06-14T23:13:14.854056+0000","last_became_active":"2024-06-14T23:11:23.412491+0000","last_became_peered":"2024-06-14T23:11:23.412491+0000","last_unstale":"2024-06-14T23:13:14.854056+0000","last_undegraded":"2024-06-14T23:13:14.854056+0000","last_fullsized":"2024-06-14T23:13:14.854056+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:57.263547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":493,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.909366+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:13:25.909366+0000","last_peered":"2024-06-14T23:13:25.909366+0000","last_clean":"2024-06-14T23:13:25.909366+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:13:25.909366+0000","last_undegraded":"2024-06-14T23:13:25.909366+0000","last_fullsized":"2024-06-14T23:13:25.909366+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":318,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.860542+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:13:14.860542+0000","last_peered":"2024-06-14T23:13:14.860542+0000","last_clean":"2024-06-14T23:13:14.860542+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:13:14.860542+0000","last_undegraded":"2024-06-14T23:13:14.860542+0000","last_fullsized":"2024-06-14T23:13:14.860542+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":345,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855534+0000","last_change":"2024-06-14T23:11:23.427771+0000","last_active":"2024-06-14T23:13:14.855534+0000","last_peered":"2024-06-14T23:13:14.855534+0000","last_clean":"2024-06-14T23:13:14.855534+0000","last_became_active":"2024-06-14T23:11:23.427481+0000","last_became_peered":"2024-06-14T23:11:23.427481+0000","last_unstale":"2024-06-14T23:13:14.855534+0000","last_undegraded":"2024-06-14T23:13:14.855534+0000","last_fullsized":"2024-06-14T23:13:14.855534+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:01:04.162256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":134,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855469+0000","last_change":"2024-06-14T23:10:26.053647+0000","last_active":"2024-06-14T23:13:14.855469+0000","last_peered":"2024-06-14T23:13:14.855469+0000","last_clean":"2024-06-14T23:13:14.855469+0000","last_became_active":"2024-06-14T23:10:26.053318+0000","last_became_peered":"2024-06-14T23:10:26.053318+0000","last_unstale":"2024-06-14T23:13:14.855469+0000","last_undegraded":"2024-06-14T23:13:14.855469+0000","last_fullsized":"2024-06-14T23:13:14.855469+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:32.647107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":133,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.836513+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:13:14.836513+0000","last_peered":"2024-06-14T23:13:14.836513+0000","last_clean":"2024-06-14T23:13:14.836513+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:13:14.836513+0000","last_undegraded":"2024-06-14T23:13:14.836513+0000","last_fullsized":"2024-06-14T23:13:14.836513+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":644,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.844082+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:13:14.844082+0000","last_peered":"2024-06-14T23:13:14.844082+0000","last_clean":"2024-06-14T23:13:14.844082+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:13:14.844082+0000","last_undegraded":"2024-06-14T23:13:14.844082+0000","last_fullsized":"2024-06-14T23:13:14.844082+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":367,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.854128+0000","last_change":"2024-06-14T23:11:23.413241+0000","last_active":"2024-06-14T23:13:14.854128+0000","last_peered":"2024-06-14T23:13:14.854128+0000","last_clean":"2024-06-14T23:13:14.854128+0000","last_became_active":"2024-06-14T23:11:23.412936+0000","last_became_peered":"2024-06-14T23:11:23.412936+0000","last_unstale":"2024-06-14T23:13:14.854128+0000","last_undegraded":"2024-06-14T23:13:14.854128+0000","last_fullsized":"2024-06-14T23:13:14.854128+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:36.581716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":153,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.854706+0000","last_change":"2024-06-14T23:10:26.038549+0000","last_active":"2024-06-14T23:13:14.854706+0000","last_peered":"2024-06-14T23:13:14.854706+0000","last_clean":"2024-06-14T23:13:14.854706+0000","last_became_active":"2024-06-14T23:10:26.038260+0000","last_became_peered":"2024-06-14T23:10:26.038260+0000","last_unstale":"2024-06-14T23:13:14.854706+0000","last_undegraded":"2024-06-14T23:13:14.854706+0000","last_fullsized":"2024-06-14T23:13:14.854706+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:44.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":747,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.858721+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:13:14.858721+0000","last_peered":"2024-06-14T23:13:14.858721+0000","last_clean":"2024-06-14T23:13:14.858721+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:13:14.858721+0000","last_undegraded":"2024-06-14T23:13:14.858721+0000","last_fullsized":"2024-06-14T23:13:14.858721+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":139,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.906542+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:13:25.906542+0000","last_peered":"2024-06-14T23:13:25.906542+0000","last_clean":"2024-06-14T23:13:25.906542+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:13:25.906542+0000","last_undegraded":"2024-06-14T23:13:25.906542+0000","last_fullsized":"2024-06-14T23:13:25.906542+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":201,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.909294+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:13:25.909294+0000","last_peered":"2024-06-14T23:13:25.909294+0000","last_clean":"2024-06-14T23:13:25.909294+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:13:25.909294+0000","last_undegraded":"2024-06-14T23:13:25.909294+0000","last_fullsized":"2024-06-14T23:13:25.909294+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":695,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859595+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:13:14.859595+0000","last_peered":"2024-06-14T23:13:14.859595+0000","last_clean":"2024-06-14T23:13:14.859595+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:13:14.859595+0000","last_undegraded":"2024-06-14T23:13:14.859595+0000","last_fullsized":"2024-06-14T23:13:14.859595+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":61,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.835806+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:13:14.835806+0000","last_peered":"2024-06-14T23:13:14.835806+0000","last_clean":"2024-06-14T23:13:14.835806+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:13:14.835806+0000","last_undegraded":"2024-06-14T23:13:14.835806+0000","last_fullsized":"2024-06-14T23:13:14.835806+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":292,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.845405+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:13:14.845405+0000","last_peered":"2024-06-14T23:13:14.845405+0000","last_clean":"2024-06-14T23:13:14.845405+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:13:14.845405+0000","last_undegraded":"2024-06-14T23:13:14.845405+0000","last_fullsized":"2024-06-14T23:13:14.845405+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":647,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.853672+0000","last_change":"2024-06-14T23:10:29.062659+0000","last_active":"2024-06-14T23:13:14.853672+0000","last_peered":"2024-06-14T23:13:14.853672+0000","last_clean":"2024-06-14T23:13:14.853672+0000","last_became_active":"2024-06-14T23:10:29.062256+0000","last_became_peered":"2024-06-14T23:10:29.062256+0000","last_unstale":"2024-06-14T23:13:14.853672+0000","last_undegraded":"2024-06-14T23:13:14.853672+0000","last_fullsized":"2024-06-14T23:13:14.853672+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:38.814157+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":325,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.905731+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:13:25.905731+0000","last_peered":"2024-06-14T23:13:25.905731+0000","last_clean":"2024-06-14T23:13:25.905731+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:13:25.905731+0000","last_undegraded":"2024-06-14T23:13:25.905731+0000","last_fullsized":"2024-06-14T23:13:25.905731+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":188,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.861016+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:13:14.861016+0000","last_peered":"2024-06-14T23:13:14.861016+0000","last_clean":"2024-06-14T23:13:14.861016+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:13:14.861016+0000","last_undegraded":"2024-06-14T23:13:14.861016+0000","last_fullsized":"2024-06-14T23:13:14.861016+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":647,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855506+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:13:14.855506+0000","last_peered":"2024-06-14T23:13:14.855506+0000","last_clean":"2024-06-14T23:13:14.855506+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:13:14.855506+0000","last_undegraded":"2024-06-14T23:13:14.855506+0000","last_fullsized":"2024-06-14T23:13:14.855506+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":208,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.906284+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:13:25.906284+0000","last_peered":"2024-06-14T23:13:25.906284+0000","last_clean":"2024-06-14T23:13:25.906284+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:13:25.906284+0000","last_undegraded":"2024-06-14T23:13:25.906284+0000","last_fullsized":"2024-06-14T23:13:25.906284+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":312,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.860355+0000","last_change":"2024-06-14T23:10:26.053008+0000","last_active":"2024-06-14T23:13:14.860355+0000","last_peered":"2024-06-14T23:13:14.860355+0000","last_clean":"2024-06-14T23:13:14.860355+0000","last_became_active":"2024-06-14T23:10:26.052701+0000","last_became_peered":"2024-06-14T23:10:26.052701+0000","last_unstale":"2024-06-14T23:13:14.860355+0000","last_undegraded":"2024-06-14T23:13:14.860355+0000","last_fullsized":"2024-06-14T23:13:14.860355+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:15:03.929062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":671,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.836436+0000","last_change":"2024-06-14T23:11:23.411926+0000","last_active":"2024-06-14T23:13:14.836436+0000","last_peered":"2024-06-14T23:13:14.836436+0000","last_clean":"2024-06-14T23:13:14.836436+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:13:14.836436+0000","last_undegraded":"2024-06-14T23:13:14.836436+0000","last_fullsized":"2024-06-14T23:13:14.836436+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":226,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.907492+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:13:25.907492+0000","last_peered":"2024-06-14T23:13:25.907492+0000","last_clean":"2024-06-14T23:13:25.907492+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:13:25.907492+0000","last_undegraded":"2024-06-14T23:13:25.907492+0000","last_fullsized":"2024-06-14T23:13:25.907492+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":354,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.840167+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:13:14.840167+0000","last_peered":"2024-06-14T23:13:14.840167+0000","last_clean":"2024-06-14T23:13:14.840167+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:13:14.840167+0000","last_undegraded":"2024-06-14T23:13:14.840167+0000","last_fullsized":"2024-06-14T23:13:14.840167+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":517,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.860669+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:13:14.860669+0000","last_peered":"2024-06-14T23:13:14.860669+0000","last_clean":"2024-06-14T23:13:14.860669+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:13:14.860669+0000","last_undegraded":"2024-06-14T23:13:14.860669+0000","last_fullsized":"2024-06-14T23:13:14.860669+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":324,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.854458+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:13:14.854458+0000","last_peered":"2024-06-14T23:13:14.854458+0000","last_clean":"2024-06-14T23:13:14.854458+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:13:14.854458+0000","last_undegraded":"2024-06-14T23:13:14.854458+0000","last_fullsized":"2024-06-14T23:13:14.854458+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":328,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855613+0000","last_change":"2024-06-14T23:10:26.047451+0000","last_active":"2024-06-14T23:13:14.855613+0000","last_peered":"2024-06-14T23:13:14.855613+0000","last_clean":"2024-06-14T23:13:14.855613+0000","last_became_active":"2024-06-14T23:10:26.046781+0000","last_became_peered":"2024-06-14T23:10:26.046781+0000","last_unstale":"2024-06-14T23:13:14.855613+0000","last_undegraded":"2024-06-14T23:13:14.855613+0000","last_fullsized":"2024-06-14T23:13:14.855613+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:07:15.892864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":556,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855893+0000","last_change":"2024-06-14T23:10:39.253251+0000","last_active":"2024-06-14T23:13:14.855893+0000","last_peered":"2024-06-14T23:13:14.855893+0000","last_clean":"2024-06-14T23:13:14.855893+0000","last_became_active":"2024-06-14T23:10:29.061386+0000","last_became_peered":"2024-06-14T23:10:29.061386+0000","last_unstale":"2024-06-14T23:13:14.855893+0000","last_undegraded":"2024-06-14T23:13:14.855893+0000","last_fullsized":"2024-06-14T23:13:14.855893+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.735431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.735431+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"342'13449","reported_seq":13517,"reported_epoch":342,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:35.562587+0000","last_change":"2024-06-14T23:11:24.652983+0000","last_active":"2024-06-14T23:13:35.562587+0000","last_peered":"2024-06-14T23:13:35.562587+0000","last_clean":"2024-06-14T23:10:37.054302+0000","last_became_active":"2024-06-14T23:11:24.646468+0000","last_became_peered":"2024-06-14T23:11:24.646468+0000","last_unstale":"2024-06-14T23:13:35.562587+0000","last_undegraded":"2024-06-14T23:11:24.406142+0000","last_fullsized":"2024-06-14T23:11:24.380989+0000","mapping_epoch":323,"log_start":"333'9105","ondisk_log_start":"333'9105","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4344,"log_dups_size":0,"ondisk_log_size":4344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:33:08.769394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74121216,"num_objects":1131,"num_object_clones":0,"num_object_copies":3393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13449,"num_write_kb":466560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-1337}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":301,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.853787+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:13:14.853787+0000","last_peered":"2024-06-14T23:13:14.853787+0000","last_clean":"2024-06-14T23:13:14.853787+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:13:14.853787+0000","last_undegraded":"2024-06-14T23:13:14.853787+0000","last_fullsized":"2024-06-14T23:13:14.853787+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":313,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.845191+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:13:14.845191+0000","last_peered":"2024-06-14T23:13:14.845191+0000","last_clean":"2024-06-14T23:13:14.845191+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:13:14.845191+0000","last_undegraded":"2024-06-14T23:13:14.845191+0000","last_fullsized":"2024-06-14T23:13:14.845191+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":99,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855305+0000","last_change":"2024-06-14T23:10:26.058423+0000","last_active":"2024-06-14T23:13:14.855305+0000","last_peered":"2024-06-14T23:13:14.855305+0000","last_clean":"2024-06-14T23:13:14.855305+0000","last_became_active":"2024-06-14T23:10:26.050244+0000","last_became_peered":"2024-06-14T23:10:26.050244+0000","last_unstale":"2024-06-14T23:13:14.855305+0000","last_undegraded":"2024-06-14T23:13:14.855305+0000","last_fullsized":"2024-06-14T23:13:14.855305+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:52.060325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":667,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.907785+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:13:25.907785+0000","last_peered":"2024-06-14T23:13:25.907785+0000","last_clean":"2024-06-14T23:13:25.907785+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:13:25.907785+0000","last_undegraded":"2024-06-14T23:13:25.907785+0000","last_fullsized":"2024-06-14T23:13:25.907785+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"342'13566","reported_seq":13645,"reported_epoch":342,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:35.561425+0000","last_change":"2024-06-14T23:11:24.639438+0000","last_active":"2024-06-14T23:13:35.561425+0000","last_peered":"2024-06-14T23:13:35.561425+0000","last_clean":"2024-06-14T23:10:37.032734+0000","last_became_active":"2024-06-14T23:11:24.634884+0000","last_became_peered":"2024-06-14T23:11:24.634884+0000","last_unstale":"2024-06-14T23:13:35.561425+0000","last_undegraded":"2024-06-14T23:11:24.403675+0000","last_fullsized":"2024-06-14T23:11:24.380974+0000","mapping_epoch":323,"log_start":"333'9200","ondisk_log_start":"333'9200","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:48:19.382266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75759616,"num_objects":1156,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13566,"num_write_kb":471104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-1364}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":255,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.905825+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:13:25.905825+0000","last_peered":"2024-06-14T23:13:25.905825+0000","last_clean":"2024-06-14T23:13:25.905825+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:13:25.905825+0000","last_undegraded":"2024-06-14T23:13:25.905825+0000","last_fullsized":"2024-06-14T23:13:25.905825+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":304,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.856090+0000","last_change":"2024-06-14T23:10:26.051450+0000","last_active":"2024-06-14T23:13:14.856090+0000","last_peered":"2024-06-14T23:13:14.856090+0000","last_clean":"2024-06-14T23:13:14.856090+0000","last_became_active":"2024-06-14T23:10:26.050830+0000","last_became_peered":"2024-06-14T23:10:26.050830+0000","last_unstale":"2024-06-14T23:13:14.856090+0000","last_undegraded":"2024-06-14T23:13:14.856090+0000","last_fullsized":"2024-06-14T23:13:14.856090+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:16:16.175014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":55,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.860587+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:13:14.860587+0000","last_peered":"2024-06-14T23:13:14.860587+0000","last_clean":"2024-06-14T23:13:14.860587+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:13:14.860587+0000","last_undegraded":"2024-06-14T23:13:14.860587+0000","last_fullsized":"2024-06-14T23:13:14.860587+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":755,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.854771+0000","last_change":"2024-06-14T23:11:23.401974+0000","last_active":"2024-06-14T23:13:14.854771+0000","last_peered":"2024-06-14T23:13:14.854771+0000","last_clean":"2024-06-14T23:13:14.854771+0000","last_became_active":"2024-06-14T23:11:23.401625+0000","last_became_peered":"2024-06-14T23:11:23.401625+0000","last_unstale":"2024-06-14T23:13:14.854771+0000","last_undegraded":"2024-06-14T23:13:14.854771+0000","last_fullsized":"2024-06-14T23:13:14.854771+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.214300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.214300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:05:54.552362+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"342'13472","reported_seq":13540,"reported_epoch":342,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:35.556753+0000","last_change":"2024-06-14T23:11:24.651371+0000","last_active":"2024-06-14T23:13:35.556753+0000","last_peered":"2024-06-14T23:13:35.556753+0000","last_clean":"2024-06-14T23:10:37.020885+0000","last_became_active":"2024-06-14T23:11:24.646284+0000","last_became_peered":"2024-06-14T23:11:24.646284+0000","last_unstale":"2024-06-14T23:13:35.556753+0000","last_undegraded":"2024-06-14T23:11:24.405009+0000","last_fullsized":"2024-06-14T23:11:24.380990+0000","mapping_epoch":323,"log_start":"333'9142","ondisk_log_start":"333'9142","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4330,"log_dups_size":0,"ondisk_log_size":4330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:26:58.760076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77463552,"num_objects":1182,"num_object_clones":0,"num_object_copies":3546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13472,"num_write_kb":468928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-1374}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":152,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859682+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:13:14.859682+0000","last_peered":"2024-06-14T23:13:14.859682+0000","last_clean":"2024-06-14T23:13:14.859682+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:13:14.859682+0000","last_undegraded":"2024-06-14T23:13:14.859682+0000","last_fullsized":"2024-06-14T23:13:14.859682+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":326,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855568+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:13:14.855568+0000","last_peered":"2024-06-14T23:13:14.855568+0000","last_clean":"2024-06-14T23:13:14.855568+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:13:14.855568+0000","last_undegraded":"2024-06-14T23:13:14.855568+0000","last_fullsized":"2024-06-14T23:13:14.855568+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":278,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.911868+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:13:25.911868+0000","last_peered":"2024-06-14T23:13:25.911868+0000","last_clean":"2024-06-14T23:13:25.911868+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:13:25.911868+0000","last_undegraded":"2024-06-14T23:13:25.911868+0000","last_fullsized":"2024-06-14T23:13:25.911868+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":640,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.842182+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:13:14.842182+0000","last_peered":"2024-06-14T23:13:14.842182+0000","last_clean":"2024-06-14T23:13:14.842182+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:13:14.842182+0000","last_undegraded":"2024-06-14T23:13:14.842182+0000","last_fullsized":"2024-06-14T23:13:14.842182+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"342'13565","reported_seq":20801,"reported_epoch":342,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:35.561148+0000","last_change":"2024-06-14T23:11:24.647584+0000","last_active":"2024-06-14T23:13:35.561148+0000","last_peered":"2024-06-14T23:13:35.561148+0000","last_clean":"2024-06-14T23:10:25.009797+0000","last_became_active":"2024-06-14T23:11:24.638765+0000","last_became_peered":"2024-06-14T23:11:24.638765+0000","last_unstale":"2024-06-14T23:13:35.561148+0000","last_undegraded":"2024-06-14T23:11:24.535343+0000","last_fullsized":"2024-06-14T23:11:24.381001+0000","mapping_epoch":323,"log_start":"333'9272","ondisk_log_start":"333'9272","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4293,"log_dups_size":0,"ondisk_log_size":4293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:37.543254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79101952,"num_objects":1207,"num_object_clones":0,"num_object_copies":3621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13565,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3574,"num_bytes_recovered":234225664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[3,2],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3,4","objects":1206},{"shards":"2,3","objects":-2603}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":117,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859328+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:13:14.859328+0000","last_peered":"2024-06-14T23:13:14.859328+0000","last_clean":"2024-06-14T23:13:14.859328+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:13:14.859328+0000","last_undegraded":"2024-06-14T23:13:14.859328+0000","last_fullsized":"2024-06-14T23:13:14.859328+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":62,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.911648+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:13:25.911648+0000","last_peered":"2024-06-14T23:13:25.911648+0000","last_clean":"2024-06-14T23:13:25.911648+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:13:25.911648+0000","last_undegraded":"2024-06-14T23:13:25.911648+0000","last_fullsized":"2024-06-14T23:13:25.911648+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":356,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859978+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:13:14.859978+0000","last_peered":"2024-06-14T23:13:14.859978+0000","last_clean":"2024-06-14T23:13:14.859978+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:13:14.859978+0000","last_undegraded":"2024-06-14T23:13:14.859978+0000","last_fullsized":"2024-06-14T23:13:14.859978+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":620,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.845267+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:13:14.845267+0000","last_peered":"2024-06-14T23:13:14.845267+0000","last_clean":"2024-06-14T23:13:14.845267+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:13:14.845267+0000","last_undegraded":"2024-06-14T23:13:14.845267+0000","last_fullsized":"2024-06-14T23:13:14.845267+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"341'13426","reported_seq":18669,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:33.209207+0000","last_change":"2024-06-14T23:11:50.568053+0000","last_active":"2024-06-14T23:13:33.209207+0000","last_peered":"2024-06-14T23:13:33.209207+0000","last_clean":"2024-06-14T23:13:33.209207+0000","last_became_active":"2024-06-14T23:11:50.555786+0000","last_became_peered":"2024-06-14T23:11:50.555786+0000","last_unstale":"2024-06-14T23:13:33.209207+0000","last_undegraded":"2024-06-14T23:13:33.209207+0000","last_fullsized":"2024-06-14T23:13:33.209207+0000","mapping_epoch":328,"log_start":"325'8000","ondisk_log_start":"325'8000","created":295,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5426,"log_dups_size":0,"ondisk_log_size":5426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:36.382663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88342528,"num_objects":1348,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13426,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2575,"num_bytes_recovered":168755200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":293,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.911513+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:13:25.911513+0000","last_peered":"2024-06-14T23:13:25.911513+0000","last_clean":"2024-06-14T23:13:25.911513+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:13:25.911513+0000","last_undegraded":"2024-06-14T23:13:25.911513+0000","last_fullsized":"2024-06-14T23:13:25.911513+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":339,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855748+0000","last_change":"2024-06-14T23:10:26.049033+0000","last_active":"2024-06-14T23:13:14.855748+0000","last_peered":"2024-06-14T23:13:14.855748+0000","last_clean":"2024-06-14T23:13:14.855748+0000","last_became_active":"2024-06-14T23:10:26.047775+0000","last_became_peered":"2024-06-14T23:10:26.047775+0000","last_unstale":"2024-06-14T23:13:14.855748+0000","last_undegraded":"2024-06-14T23:13:14.855748+0000","last_fullsized":"2024-06-14T23:13:14.855748+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:15:52.051858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":333,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859437+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:13:14.859437+0000","last_peered":"2024-06-14T23:13:14.859437+0000","last_clean":"2024-06-14T23:13:14.859437+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:13:14.859437+0000","last_undegraded":"2024-06-14T23:13:14.859437+0000","last_fullsized":"2024-06-14T23:13:14.859437+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":734,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.858185+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:13:14.858185+0000","last_peered":"2024-06-14T23:13:14.858185+0000","last_clean":"2024-06-14T23:13:14.858185+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:13:14.858185+0000","last_undegraded":"2024-06-14T23:13:14.858185+0000","last_fullsized":"2024-06-14T23:13:14.858185+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"341'13247","reported_seq":13322,"reported_epoch":341,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:31.709139+0000","last_change":"2024-06-14T23:11:24.656131+0000","last_active":"2024-06-14T23:13:31.709139+0000","last_peered":"2024-06-14T23:13:31.709139+0000","last_clean":"2024-06-14T23:10:37.043860+0000","last_became_active":"2024-06-14T23:11:24.647995+0000","last_became_peered":"2024-06-14T23:11:24.647995+0000","last_unstale":"2024-06-14T23:13:31.709139+0000","last_undegraded":"2024-06-14T23:11:24.403317+0000","last_fullsized":"2024-06-14T23:11:24.380506+0000","mapping_epoch":323,"log_start":"327'8300","ondisk_log_start":"327'8300","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:32:10.827694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94961664,"num_objects":1449,"num_object_clones":0,"num_object_copies":4347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13247,"num_write_kb":470272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":-1052}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":301,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.860802+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:13:14.860802+0000","last_peered":"2024-06-14T23:13:14.860802+0000","last_clean":"2024-06-14T23:13:14.860802+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:13:14.860802+0000","last_undegraded":"2024-06-14T23:13:14.860802+0000","last_fullsized":"2024-06-14T23:13:14.860802+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":338,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:22.350906+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:13:22.350906+0000","last_peered":"2024-06-14T23:13:22.350906+0000","last_clean":"2024-06-14T23:13:22.350906+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:13:22.350906+0000","last_undegraded":"2024-06-14T23:13:22.350906+0000","last_fullsized":"2024-06-14T23:13:22.350906+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.910496+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:13:25.910496+0000","last_peered":"2024-06-14T23:13:25.910496+0000","last_clean":"2024-06-14T23:13:25.910496+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:13:25.910496+0000","last_undegraded":"2024-06-14T23:13:25.910496+0000","last_fullsized":"2024-06-14T23:13:25.910496+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":628,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.853744+0000","last_change":"2024-06-14T23:11:23.413224+0000","last_active":"2024-06-14T23:13:14.853744+0000","last_peered":"2024-06-14T23:13:14.853744+0000","last_clean":"2024-06-14T23:13:14.853744+0000","last_became_active":"2024-06-14T23:11:23.412680+0000","last_became_peered":"2024-06-14T23:11:23.412680+0000","last_unstale":"2024-06-14T23:13:14.853744+0000","last_undegraded":"2024-06-14T23:13:14.853744+0000","last_fullsized":"2024-06-14T23:13:14.853744+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:48:10.967651+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.e","version":"341'13179","reported_seq":13243,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:33.280728+0000","last_change":"2024-06-14T23:09:41.994118+0000","last_active":"2024-06-14T23:13:33.280728+0000","last_peered":"2024-06-14T23:13:33.280728+0000","last_clean":"2024-06-14T23:13:33.280728+0000","last_became_active":"2024-06-14T23:09:41.993702+0000","last_became_peered":"2024-06-14T23:09:41.993702+0000","last_unstale":"2024-06-14T23:13:33.280728+0000","last_undegraded":"2024-06-14T23:13:33.280728+0000","last_fullsized":"2024-06-14T23:13:33.280728+0000","mapping_epoch":299,"log_start":"326'7900","ondisk_log_start":"326'7900","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5279,"log_dups_size":0,"ondisk_log_size":5279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:25.994252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88670208,"num_objects":1353,"num_object_clones":0,"num_object_copies":4059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13179,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":311,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.910913+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:13:25.910913+0000","last_peered":"2024-06-14T23:13:25.910913+0000","last_clean":"2024-06-14T23:13:25.910913+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:13:25.910913+0000","last_undegraded":"2024-06-14T23:13:25.910913+0000","last_fullsized":"2024-06-14T23:13:25.910913+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":189,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.853637+0000","last_change":"2024-06-14T23:09:41.991652+0000","last_active":"2024-06-14T23:13:14.853637+0000","last_peered":"2024-06-14T23:13:14.853637+0000","last_clean":"2024-06-14T23:13:14.853637+0000","last_became_active":"2024-06-14T23:09:41.991317+0000","last_became_peered":"2024-06-14T23:09:41.991317+0000","last_unstale":"2024-06-14T23:13:14.853637+0000","last_undegraded":"2024-06-14T23:13:14.853637+0000","last_fullsized":"2024-06-14T23:13:14.853637+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:16:18.209329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":307,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.905661+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:13:25.905661+0000","last_peered":"2024-06-14T23:13:25.905661+0000","last_clean":"2024-06-14T23:13:25.905661+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:13:25.905661+0000","last_undegraded":"2024-06-14T23:13:25.905661+0000","last_fullsized":"2024-06-14T23:13:25.905661+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":548,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855197+0000","last_change":"2024-06-14T23:10:29.063981+0000","last_active":"2024-06-14T23:13:14.855197+0000","last_peered":"2024-06-14T23:13:14.855197+0000","last_clean":"2024-06-14T23:13:14.855197+0000","last_became_active":"2024-06-14T23:10:29.063546+0000","last_became_peered":"2024-06-14T23:10:29.063546+0000","last_unstale":"2024-06-14T23:13:14.855197+0000","last_undegraded":"2024-06-14T23:13:14.855197+0000","last_fullsized":"2024-06-14T23:13:14.855197+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:26:23.411322+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"341'13327","reported_seq":21788,"reported_epoch":341,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:33.824766+0000","last_change":"2024-06-14T23:11:24.664901+0000","last_active":"2024-06-14T23:13:33.824766+0000","last_peered":"2024-06-14T23:13:33.824766+0000","last_clean":"2024-06-14T23:10:24.989118+0000","last_became_active":"2024-06-14T23:11:24.647313+0000","last_became_peered":"2024-06-14T23:11:24.647313+0000","last_unstale":"2024-06-14T23:13:33.824766+0000","last_undegraded":"2024-06-14T23:11:24.555289+0000","last_fullsized":"2024-06-14T23:11:24.381668+0000","mapping_epoch":323,"log_start":"331'8400","ondisk_log_start":"331'8400","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:48:12.784805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81657880,"num_objects":1247,"num_object_clones":0,"num_object_copies":3741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13327,"num_write_kb":466305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4187,"num_bytes_recovered":274399232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"1,3,7","objects":1246},{"shards":"3,7","objects":-1237}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":256,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.846386+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:13:14.846386+0000","last_peered":"2024-06-14T23:13:14.846386+0000","last_clean":"2024-06-14T23:13:14.846386+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:13:14.846386+0000","last_undegraded":"2024-06-14T23:13:14.846386+0000","last_fullsized":"2024-06-14T23:13:14.846386+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":262,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859401+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:13:14.859401+0000","last_peered":"2024-06-14T23:13:14.859401+0000","last_clean":"2024-06-14T23:13:14.859401+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:13:14.859401+0000","last_undegraded":"2024-06-14T23:13:14.859401+0000","last_fullsized":"2024-06-14T23:13:14.859401+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":343,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859746+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:13:14.859746+0000","last_peered":"2024-06-14T23:13:14.859746+0000","last_clean":"2024-06-14T23:13:14.859746+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:13:14.859746+0000","last_undegraded":"2024-06-14T23:13:14.859746+0000","last_fullsized":"2024-06-14T23:13:14.859746+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":621,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:22.350932+0000","last_change":"2024-06-14T23:11:23.406149+0000","last_active":"2024-06-14T23:13:22.350932+0000","last_peered":"2024-06-14T23:13:22.350932+0000","last_clean":"2024-06-14T23:13:22.350932+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:13:22.350932+0000","last_undegraded":"2024-06-14T23:13:22.350932+0000","last_fullsized":"2024-06-14T23:13:22.350932+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.751667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.751667+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.7","version":"341'13212","reported_seq":24711,"reported_epoch":341,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-14T23:13:33.826648+0000","last_change":"2024-06-14T23:11:24.669886+0000","last_active":"2024-06-14T23:13:33.826648+0000","last_peered":"2024-06-14T23:13:33.826648+0000","last_clean":"2024-06-14T23:10:25.005348+0000","last_became_active":"2024-06-14T23:11:24.634913+0000","last_became_peered":"2024-06-14T23:11:24.634913+0000","last_unstale":"2024-06-14T23:13:33.826648+0000","last_undegraded":"2024-06-14T23:13:33.826648+0000","last_fullsized":"2024-06-14T23:11:24.386154+0000","mapping_epoch":323,"log_start":"330'8300","ondisk_log_start":"330'8300","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:26:38.637559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84541440,"num_objects":1290,"num_object_clones":0,"num_object_copies":3870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":945,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13212,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5706,"num_bytes_recovered":373948416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"1,4,7","objects":945},{"shards":"4,7","objects":345}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":140,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.907424+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:13:25.907424+0000","last_peered":"2024-06-14T23:13:25.907424+0000","last_clean":"2024-06-14T23:13:25.907424+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:13:25.907424+0000","last_undegraded":"2024-06-14T23:13:25.907424+0000","last_fullsized":"2024-06-14T23:13:25.907424+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":312,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-06-14T23:13:34.475306+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:13:34.475306+0000","last_peered":"2024-06-14T23:13:34.475306+0000","last_clean":"2024-06-14T23:13:34.475306+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:13:34.475306+0000","last_undegraded":"2024-06-14T23:13:34.475306+0000","last_fullsized":"2024-06-14T23:13:34.475306+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":732,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859186+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:13:14.859186+0000","last_peered":"2024-06-14T23:13:14.859186+0000","last_clean":"2024-06-14T23:13:14.859186+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:13:14.859186+0000","last_undegraded":"2024-06-14T23:13:14.859186+0000","last_fullsized":"2024-06-14T23:13:14.859186+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"341'12921","reported_seq":13444,"reported_epoch":341,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:31.755993+0000","last_change":"2024-06-14T23:11:24.664127+0000","last_active":"2024-06-14T23:13:31.755993+0000","last_peered":"2024-06-14T23:13:31.755993+0000","last_clean":"2024-06-14T23:10:25.013628+0000","last_became_active":"2024-06-14T23:10:27.203384+0000","last_became_peered":"2024-06-14T23:10:27.203384+0000","last_unstale":"2024-06-14T23:13:31.755993+0000","last_undegraded":"2024-06-14T23:11:24.661184+0000","last_fullsized":"2024-06-14T23:10:27.026246+0000","mapping_epoch":310,"log_start":"327'8000","ondisk_log_start":"327'8000","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:40.932486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88539136,"num_objects":1351,"num_object_clones":0,"num_object_copies":4053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12921,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":14548992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"6,7","objects":-2580}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":328,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.856005+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:13:14.856005+0000","last_peered":"2024-06-14T23:13:14.856005+0000","last_clean":"2024-06-14T23:13:14.856005+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:13:14.856005+0000","last_undegraded":"2024-06-14T23:13:14.856005+0000","last_fullsized":"2024-06-14T23:13:14.856005+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":339,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.854061+0000","last_change":"2024-06-14T23:11:23.427812+0000","last_active":"2024-06-14T23:13:14.854061+0000","last_peered":"2024-06-14T23:13:14.854061+0000","last_clean":"2024-06-14T23:13:14.854061+0000","last_became_active":"2024-06-14T23:11:23.427563+0000","last_became_peered":"2024-06-14T23:11:23.427563+0000","last_unstale":"2024-06-14T23:13:14.854061+0000","last_undegraded":"2024-06-14T23:13:14.854061+0000","last_fullsized":"2024-06-14T23:13:14.854061+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.941006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":700,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.860438+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:13:14.860438+0000","last_peered":"2024-06-14T23:13:14.860438+0000","last_clean":"2024-06-14T23:13:14.860438+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:13:14.860438+0000","last_undegraded":"2024-06-14T23:13:14.860438+0000","last_fullsized":"2024-06-14T23:13:14.860438+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"341'12935","reported_seq":21317,"reported_epoch":341,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:31.729657+0000","last_change":"2024-06-14T23:11:24.656651+0000","last_active":"2024-06-14T23:13:31.729657+0000","last_peered":"2024-06-14T23:13:31.729657+0000","last_clean":"2024-06-14T23:10:25.016384+0000","last_became_active":"2024-06-14T23:11:24.643623+0000","last_became_peered":"2024-06-14T23:11:24.643623+0000","last_unstale":"2024-06-14T23:13:31.729657+0000","last_undegraded":"2024-06-14T23:11:24.519360+0000","last_fullsized":"2024-06-14T23:11:24.380111+0000","mapping_epoch":323,"log_start":"327'8000","ondisk_log_start":"327'8000","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4935,"log_dups_size":0,"ondisk_log_size":4935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:18:43.920069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92209152,"num_objects":1407,"num_object_clones":0,"num_object_copies":4221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12935,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4148,"num_bytes_recovered":271843328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"1,3,6","objects":1407},{"shards":"3,6","objects":-1054}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":358,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.854943+0000","last_change":"2024-06-14T23:11:23.416264+0000","last_active":"2024-06-14T23:13:14.854943+0000","last_peered":"2024-06-14T23:13:14.854943+0000","last_clean":"2024-06-14T23:13:14.854943+0000","last_became_active":"2024-06-14T23:11:23.415946+0000","last_became_peered":"2024-06-14T23:11:23.415946+0000","last_unstale":"2024-06-14T23:13:14.854943+0000","last_undegraded":"2024-06-14T23:13:14.854943+0000","last_fullsized":"2024-06-14T23:13:14.854943+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:22.668569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":209,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.853994+0000","last_change":"2024-06-14T23:10:26.045363+0000","last_active":"2024-06-14T23:13:14.853994+0000","last_peered":"2024-06-14T23:13:14.853994+0000","last_clean":"2024-06-14T23:13:14.853994+0000","last_became_active":"2024-06-14T23:10:26.043844+0000","last_became_peered":"2024-06-14T23:10:26.043844+0000","last_unstale":"2024-06-14T23:13:14.853994+0000","last_undegraded":"2024-06-14T23:13:14.853994+0000","last_fullsized":"2024-06-14T23:13:14.853994+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:11:27.891184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":723,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.858580+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:13:14.858580+0000","last_peered":"2024-06-14T23:13:14.858580+0000","last_clean":"2024-06-14T23:13:14.858580+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:13:14.858580+0000","last_undegraded":"2024-06-14T23:13:14.858580+0000","last_fullsized":"2024-06-14T23:13:14.858580+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"341'13154","reported_seq":13450,"reported_epoch":341,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:31.749276+0000","last_change":"2024-06-14T23:10:39.484566+0000","last_active":"2024-06-14T23:13:31.749276+0000","last_peered":"2024-06-14T23:13:31.749276+0000","last_clean":"2024-06-14T23:10:24.998059+0000","last_became_active":"2024-06-14T23:10:27.239025+0000","last_became_peered":"2024-06-14T23:10:27.239025+0000","last_unstale":"2024-06-14T23:13:31.749276+0000","last_undegraded":"2024-06-14T23:10:27.071395+0000","last_fullsized":"2024-06-14T23:10:27.025672+0000","mapping_epoch":310,"log_start":"327'8200","ondisk_log_start":"327'8200","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:52:40.136978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91750400,"num_objects":1400,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13154,"num_write_kb":465728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":7602176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"3,6","objects":-2653}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":318,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.854383+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:13:14.854383+0000","last_peered":"2024-06-14T23:13:14.854383+0000","last_clean":"2024-06-14T23:13:14.854383+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:13:14.854383+0000","last_undegraded":"2024-06-14T23:13:14.854383+0000","last_fullsized":"2024-06-14T23:13:14.854383+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":139,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855341+0000","last_change":"2024-06-14T23:10:26.056252+0000","last_active":"2024-06-14T23:13:14.855341+0000","last_peered":"2024-06-14T23:13:14.855341+0000","last_clean":"2024-06-14T23:13:14.855341+0000","last_became_active":"2024-06-14T23:10:26.046629+0000","last_became_peered":"2024-06-14T23:10:26.046629+0000","last_unstale":"2024-06-14T23:13:14.855341+0000","last_undegraded":"2024-06-14T23:13:14.855341+0000","last_fullsized":"2024-06-14T23:13:14.855341+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:51:19.277386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":612,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.910415+0000","last_change":"2024-06-14T23:11:23.428346+0000","last_active":"2024-06-14T23:13:25.910415+0000","last_peered":"2024-06-14T23:13:25.910415+0000","last_clean":"2024-06-14T23:13:25.910415+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:13:25.910415+0000","last_undegraded":"2024-06-14T23:13:25.910415+0000","last_fullsized":"2024-06-14T23:13:25.910415+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.656210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.656210+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"341'13339","reported_seq":13403,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:33.808813+0000","last_change":"2024-06-14T23:09:41.992020+0000","last_active":"2024-06-14T23:13:33.808813+0000","last_peered":"2024-06-14T23:13:33.808813+0000","last_clean":"2024-06-14T23:13:33.808813+0000","last_became_active":"2024-06-14T23:09:41.991722+0000","last_became_peered":"2024-06-14T23:09:41.991722+0000","last_unstale":"2024-06-14T23:13:33.808813+0000","last_undegraded":"2024-06-14T23:13:33.808813+0000","last_fullsized":"2024-06-14T23:13:33.808813+0000","mapping_epoch":299,"log_start":"331'8500","ondisk_log_start":"331'8500","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:35:54.751263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82509824,"num_objects":1259,"num_object_clones":0,"num_object_copies":3777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13339,"num_write_kb":467136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":138,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855266+0000","last_change":"2024-06-14T23:10:26.047111+0000","last_active":"2024-06-14T23:13:14.855266+0000","last_peered":"2024-06-14T23:13:14.855266+0000","last_clean":"2024-06-14T23:13:14.855266+0000","last_became_active":"2024-06-14T23:10:26.045540+0000","last_became_peered":"2024-06-14T23:10:26.045540+0000","last_unstale":"2024-06-14T23:13:14.855266+0000","last_undegraded":"2024-06-14T23:13:14.855266+0000","last_fullsized":"2024-06-14T23:13:14.855266+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:04.131400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":319,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.860615+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:13:14.860615+0000","last_peered":"2024-06-14T23:13:14.860615+0000","last_clean":"2024-06-14T23:13:14.860615+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:13:14.860615+0000","last_undegraded":"2024-06-14T23:13:14.860615+0000","last_fullsized":"2024-06-14T23:13:14.860615+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":579,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.855219+0000","last_change":"2024-06-14T23:09:17.382466+0000","last_active":"2024-06-14T23:13:14.855219+0000","last_peered":"2024-06-14T23:13:14.855219+0000","last_clean":"2024-06-14T23:13:14.855219+0000","last_became_active":"2024-06-14T23:09:12.030205+0000","last_became_peered":"2024-06-14T23:09:12.030205+0000","last_unstale":"2024-06-14T23:13:14.855219+0000","last_undegraded":"2024-06-14T23:13:14.855219+0000","last_fullsized":"2024-06-14T23:13:14.855219+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:09:17.382389+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:33.121472+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"341'12960","reported_seq":13011,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:33.416764+0000","last_change":"2024-06-14T23:09:29.408859+0000","last_active":"2024-06-14T23:13:33.416764+0000","last_peered":"2024-06-14T23:13:33.416764+0000","last_clean":"2024-06-14T23:13:33.416764+0000","last_became_active":"2024-06-14T23:09:29.408548+0000","last_became_peered":"2024-06-14T23:09:29.408548+0000","last_unstale":"2024-06-14T23:13:33.416764+0000","last_undegraded":"2024-06-14T23:13:33.416764+0000","last_fullsized":"2024-06-14T23:13:33.416764+0000","mapping_epoch":295,"log_start":"325'7600","ondisk_log_start":"325'7600","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:19.810173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85327872,"num_objects":1302,"num_object_clones":0,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12960,"num_write_kb":456384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":292,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.840046+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:13:14.840046+0000","last_peered":"2024-06-14T23:13:14.840046+0000","last_clean":"2024-06-14T23:13:14.840046+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:13:14.840046+0000","last_undegraded":"2024-06-14T23:13:14.840046+0000","last_fullsized":"2024-06-14T23:13:14.840046+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":236,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859752+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:13:14.859752+0000","last_peered":"2024-06-14T23:13:14.859752+0000","last_clean":"2024-06-14T23:13:14.859752+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:13:14.859752+0000","last_undegraded":"2024-06-14T23:13:14.859752+0000","last_fullsized":"2024-06-14T23:13:14.859752+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":620,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.845338+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:13:14.845338+0000","last_peered":"2024-06-14T23:13:14.845338+0000","last_clean":"2024-06-14T23:13:14.845338+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:13:14.845338+0000","last_undegraded":"2024-06-14T23:13:14.845338+0000","last_fullsized":"2024-06-14T23:13:14.845338+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"341'13139","reported_seq":13193,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:31.770981+0000","last_change":"2024-06-14T23:09:29.403408+0000","last_active":"2024-06-14T23:13:31.770981+0000","last_peered":"2024-06-14T23:13:31.770981+0000","last_clean":"2024-06-14T23:13:31.770981+0000","last_became_active":"2024-06-14T23:09:29.403000+0000","last_became_peered":"2024-06-14T23:09:29.403000+0000","last_unstale":"2024-06-14T23:13:31.770981+0000","last_undegraded":"2024-06-14T23:13:31.770981+0000","last_fullsized":"2024-06-14T23:13:31.770981+0000","mapping_epoch":295,"log_start":"328'8200","ondisk_log_start":"328'8200","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:16.838865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90505216,"num_objects":1381,"num_object_clones":0,"num_object_copies":4143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13139,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":328,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.853986+0000","last_change":"2024-06-14T23:10:26.040201+0000","last_active":"2024-06-14T23:13:14.853986+0000","last_peered":"2024-06-14T23:13:14.853986+0000","last_clean":"2024-06-14T23:13:14.853986+0000","last_became_active":"2024-06-14T23:10:26.039865+0000","last_became_peered":"2024-06-14T23:10:26.039865+0000","last_unstale":"2024-06-14T23:13:14.853986+0000","last_undegraded":"2024-06-14T23:13:14.853986+0000","last_fullsized":"2024-06-14T23:13:14.853986+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:34.621202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":306,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.861741+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:13:14.861741+0000","last_peered":"2024-06-14T23:13:14.861741+0000","last_clean":"2024-06-14T23:13:14.861741+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:13:14.861741+0000","last_undegraded":"2024-06-14T23:13:14.861741+0000","last_fullsized":"2024-06-14T23:13:14.861741+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":749,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.907869+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:13:25.907869+0000","last_peered":"2024-06-14T23:13:25.907869+0000","last_clean":"2024-06-14T23:13:25.907869+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:13:25.907869+0000","last_undegraded":"2024-06-14T23:13:25.907869+0000","last_fullsized":"2024-06-14T23:13:25.907869+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"341'13305","reported_seq":13376,"reported_epoch":341,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:33.355569+0000","last_change":"2024-06-14T23:11:24.632384+0000","last_active":"2024-06-14T23:13:33.355569+0000","last_peered":"2024-06-14T23:13:33.355569+0000","last_clean":"2024-06-14T23:10:37.053680+0000","last_became_active":"2024-06-14T23:11:24.623684+0000","last_became_peered":"2024-06-14T23:11:24.623684+0000","last_unstale":"2024-06-14T23:13:33.355569+0000","last_undegraded":"2024-06-14T23:11:24.400366+0000","last_fullsized":"2024-06-14T23:11:24.379058+0000","mapping_epoch":323,"log_start":"326'8000","ondisk_log_start":"326'8000","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5305,"log_dups_size":0,"ondisk_log_size":5305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:34:53.331507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88276992,"num_objects":1347,"num_object_clones":0,"num_object_copies":4041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13305,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":-1207}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":301,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.858790+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:13:14.858790+0000","last_peered":"2024-06-14T23:13:14.858790+0000","last_clean":"2024-06-14T23:13:14.858790+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:13:14.858790+0000","last_undegraded":"2024-06-14T23:13:14.858790+0000","last_fullsized":"2024-06-14T23:13:14.858790+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":319,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.857114+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:13:14.857114+0000","last_peered":"2024-06-14T23:13:14.857114+0000","last_clean":"2024-06-14T23:13:14.857114+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:13:14.857114+0000","last_undegraded":"2024-06-14T23:13:14.857114+0000","last_fullsized":"2024-06-14T23:13:14.857114+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":774,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.854642+0000","last_change":"2024-06-14T23:11:23.418239+0000","last_active":"2024-06-14T23:13:14.854642+0000","last_peered":"2024-06-14T23:13:14.854642+0000","last_clean":"2024-06-14T23:13:14.854642+0000","last_became_active":"2024-06-14T23:11:23.417873+0000","last_became_peered":"2024-06-14T23:11:23.417873+0000","last_unstale":"2024-06-14T23:13:14.854642+0000","last_undegraded":"2024-06-14T23:13:14.854642+0000","last_fullsized":"2024-06-14T23:13:14.854642+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:04:50.740175+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":327,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.841658+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:13:14.841658+0000","last_peered":"2024-06-14T23:13:14.841658+0000","last_clean":"2024-06-14T23:13:14.841658+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:13:14.841658+0000","last_undegraded":"2024-06-14T23:13:14.841658+0000","last_fullsized":"2024-06-14T23:13:14.841658+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":325,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.907251+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:13:25.907251+0000","last_peered":"2024-06-14T23:13:25.907251+0000","last_clean":"2024-06-14T23:13:25.907251+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:13:25.907251+0000","last_undegraded":"2024-06-14T23:13:25.907251+0000","last_fullsized":"2024-06-14T23:13:25.907251+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":696,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859265+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:13:14.859265+0000","last_peered":"2024-06-14T23:13:14.859265+0000","last_clean":"2024-06-14T23:13:14.859265+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:13:14.859265+0000","last_undegraded":"2024-06-14T23:13:14.859265+0000","last_fullsized":"2024-06-14T23:13:14.859265+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":225,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.853715+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:13:14.853715+0000","last_peered":"2024-06-14T23:13:14.853715+0000","last_clean":"2024-06-14T23:13:14.853715+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:13:14.853715+0000","last_undegraded":"2024-06-14T23:13:14.853715+0000","last_fullsized":"2024-06-14T23:13:14.853715+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":206,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-06-14T23:13:34.472401+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:13:34.472401+0000","last_peered":"2024-06-14T23:13:34.472401+0000","last_clean":"2024-06-14T23:13:34.472401+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:13:34.472401+0000","last_undegraded":"2024-06-14T23:13:34.472401+0000","last_fullsized":"2024-06-14T23:13:34.472401+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":663,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-14T23:13:25.909563+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:13:25.909563+0000","last_peered":"2024-06-14T23:13:25.909563+0000","last_clean":"2024-06-14T23:13:25.909563+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:13:25.909563+0000","last_undegraded":"2024-06-14T23:13:25.909563+0000","last_fullsized":"2024-06-14T23:13:25.909563+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.835980+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:13:14.835980+0000","last_peered":"2024-06-14T23:13:14.835980+0000","last_clean":"2024-06-14T23:13:14.835980+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:13:14.835980+0000","last_undegraded":"2024-06-14T23:13:14.835980+0000","last_fullsized":"2024-06-14T23:13:14.835980+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":301,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.859821+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:13:14.859821+0000","last_peered":"2024-06-14T23:13:14.859821+0000","last_clean":"2024-06-14T23:13:14.859821+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:13:14.859821+0000","last_undegraded":"2024-06-14T23:13:14.859821+0000","last_fullsized":"2024-06-14T23:13:14.859821+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":660,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.858489+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:13:14.858489+0000","last_peered":"2024-06-14T23:13:14.858489+0000","last_clean":"2024-06-14T23:13:14.858489+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:13:14.858489+0000","last_undegraded":"2024-06-14T23:13:14.858489+0000","last_fullsized":"2024-06-14T23:13:14.858489+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":353,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.853631+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:13:14.853631+0000","last_peered":"2024-06-14T23:13:14.853631+0000","last_clean":"2024-06-14T23:13:14.853631+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:13:14.853631+0000","last_undegraded":"2024-06-14T23:13:14.853631+0000","last_fullsized":"2024-06-14T23:13:14.853631+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":42,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.836672+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:13:14.836672+0000","last_peered":"2024-06-14T23:13:14.836672+0000","last_clean":"2024-06-14T23:13:14.836672+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:13:14.836672+0000","last_undegraded":"2024-06-14T23:13:14.836672+0000","last_fullsized":"2024-06-14T23:13:14.836672+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":773,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-14T23:13:14.858102+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:13:14.858102+0000","last_peered":"2024-06-14T23:13:14.858102+0000","last_clean":"2024-06-14T23:13:14.858102+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:13:14.858102+0000","last_undegraded":"2024-06-14T23:13:14.858102+0000","last_fullsized":"2024-06-14T23:13:14.858102+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":1363738648,"num_objects":20810,"num_object_clones":0,"num_object_copies":62430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29338,"num_objects_misplaced":945,"num_objects_unfound":0,"num_objects_dirty":20810,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212196,"num_write_kb":7456129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20529,"num_bytes_recovered":1345388544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5639188480,"data_stored":5639176264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78077,"ondisk_log_size":78077,"up":48,"acting":37,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":821296,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":103,"num_write":26,"num_write_kb":273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":365100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234474,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2273108,"kb_used_data":766156,"kb_used_omap":1,"kb_used_meta":1506878,"kb_avail":91472044,"statfs":{"total":95995035648,"available":93667373056,"internally_reserved":0,"allocated":784543744,"data_stored":782765432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1543043416},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":0.95599999999999996,"15min":0.99099999999999999},"min":{"1min":0.24199999999999999,"5min":0.218,"15min":0.19800000000000001},"max":{"1min":5.7229999999999999,"5min":5.7229999999999999,"15min":6.9539999999999997},"last":5.8259999999999996},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.169,"15min":1.1950000000000001},"min":{"1min":0.29599999999999999,"5min":0.224,"15min":0.224},"max":{"1min":5.5350000000000001,"5min":5.5350000000000001,"15min":6.9550000000000001},"last":5.9870000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.196,"15min":1.306},"min":{"1min":0.38600000000000001,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":5.5629999999999997,"5min":5.5629999999999997,"15min":12.010999999999999},"last":6.0229999999999997},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.1950000000000001,"15min":1.337},"min":{"1min":0.34799999999999998,"5min":0.29599999999999999,"15min":0.22900000000000001},"max":{"1min":5.7480000000000002,"5min":5.7480000000000002,"15min":13.531000000000001},"last":5.8490000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.4970000000000001,"15min":1.4490000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":10.539,"5min":10.539,"15min":10.539},"last":2.0529999999999999},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.431,"15min":1.379},"min":{"1min":0.35899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":10.56,"5min":10.56,"15min":10.56},"last":2.125}]},{"osd":3,"last update":"Fri Jun 14 23:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.5609999999999999,"15min":1.4179999999999999},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.28199999999999997},"max":{"1min":5.2270000000000003,"5min":20.645,"15min":20.645},"last":2.1019999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":1.625,"15min":1.494},"min":{"1min":0.36499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.3639999999999999,"5min":22.042999999999999,"15min":22.042999999999999},"last":6.069}]},{"osd":4,"last update":"Fri Jun 14 23:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.2749999999999999,"15min":1.107},"min":{"1min":0.42299999999999999,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":3.871,"5min":5.3319999999999999,"15min":11.342000000000001},"last":6.0490000000000004},{"interface":"front","average":{"1min":1.776,"5min":1.452,"15min":1.1539999999999999},"min":{"1min":0.316,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":10.499000000000001,"5min":10.499000000000001,"15min":12.651},"last":2.081}]},{"osd":5,"last update":"Fri Jun 14 23:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.1200000000000001,"15min":1.1180000000000001},"min":{"1min":0.32800000000000001,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":4.0019999999999998,"5min":5.6849999999999996,"15min":6.9000000000000004},"last":5.734},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.2849999999999999,"15min":1.135},"min":{"1min":0.45100000000000001,"5min":0.33100000000000002,"15min":0.24399999999999999},"max":{"1min":3.9020000000000001,"5min":5.3949999999999996,"15min":6.2809999999999997},"last":6.0899999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":2.137,"5min":1.7190000000000001,"15min":1.448},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.7750000000000004,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":7.1970000000000001},{"interface":"front","average":{"1min":3.0219999999999998,"5min":1.694,"15min":1.486},"min":{"1min":0.48199999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.087999999999999,"5min":10.087999999999999,"15min":10.087999999999999},"last":6.9569999999999999}]}]},{"osd":6,"up_from":272,"seq":1168231104580,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1207936,"kb_used_data":810872,"kb_used_omap":7,"kb_used_meta":397048,"kb_avail":92537216,"statfs":{"total":95995035648,"available":94758109184,"internally_reserved":0,"allocated":830332928,"data_stored":828574629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":406577594},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.073,"15min":1.609},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.7330000000000001,"5min":6.6440000000000001,"15min":6.6440000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.095,"5min":1.143,"15min":1.6639999999999999},"min":{"1min":0.38800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.9430000000000001,"5min":6.6790000000000003,"15min":6.6790000000000003},"last":0.64800000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.3080000000000001,"15min":2.0859999999999999},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.0170000000000003,"5min":8.173,"15min":8.173},"last":1.7689999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.238,"15min":1.996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.9039999999999999,"5min":10.380000000000001,"15min":10.380000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.1970000000000001,"15min":1.143},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.476,"5min":7.476,"15min":7.476},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.782,"5min":1.304,"15min":1.224},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.7590000000000003,"5min":7.7590000000000003,"15min":7.7590000000000003},"last":0.76100000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.2370000000000001,"15min":1.7629999999999999},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.7879999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.722,"5min":1.629,"15min":1.992},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.4969999999999999,"5min":8.4740000000000002,"15min":8.4740000000000002},"last":0.46600000000000003}]},{"osd":4,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.4079999999999999,"15min":1.444},"min":{"1min":0.32600000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.5609999999999999,"5min":11.010999999999999,"15min":11.010999999999999},"last":2.5289999999999999},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.1870000000000001,"15min":1.2749999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.2309999999999999,"5min":5.2309999999999999,"15min":5.2309999999999999},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0580000000000001,"15min":1.3440000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.8079999999999998,"5min":7.7469999999999999,"15min":7.7469999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.8680000000000001,"5min":1.5209999999999999,"15min":1.6160000000000001},"min":{"1min":0.38300000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.4359999999999999,"5min":7.8680000000000003,"15min":7.8680000000000003},"last":0.496}]},{"osd":7,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":2.0310000000000001,"5min":1.746,"15min":1.758},"min":{"1min":0.33100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":7.54,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.8180000000000001,"5min":1.474,"15min":1.554},"min":{"1min":0.54400000000000004,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.9849999999999999,"5min":7.4619999999999997,"15min":7.4619999999999997},"last":4.4790000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757749,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1366652,"kb_used_data":183028,"kb_used_omap":1,"kb_used_meta":1183614,"kb_avail":92378500,"statfs":{"total":95995035648,"available":94595584000,"internally_reserved":0,"allocated":187420672,"data_stored":185662279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1212021080},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.70299999999999996,"15min":1.2949999999999999},"min":{"1min":0.255,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":1.5189999999999999,"5min":4.0579999999999998,"15min":14.94},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.88200000000000001,"15min":1.3149999999999999},"min":{"1min":0.34799999999999998,"5min":0.26100000000000001,"15min":0.219},"max":{"1min":1.8680000000000001,"5min":8.3520000000000003,"15min":9.3819999999999997},"last":0.41899999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.751,"15min":0.94899999999999995},"min":{"1min":0.30099999999999999,"5min":0.245,"15min":0.23999999999999999},"max":{"1min":2.79,"5min":4.1589999999999998,"15min":7.4370000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.82799999999999996,"15min":1.1479999999999999},"min":{"1min":0.28399999999999997,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":2.6240000000000001,"5min":5.1920000000000002,"15min":9.5039999999999996},"last":0.33300000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.74099999999999999,"15min":0.67900000000000005},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.9800000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.30099999999999999},{"interface":"front","average":{"1min":1.272,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.0659999999999998,"5min":6.0659999999999998,"15min":6.0659999999999998},"last":0.51600000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:12:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.081,"15min":1.3979999999999999},"min":{"1min":0.31,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":1.9370000000000001,"5min":8.4350000000000005,"15min":10.358000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.629,"5min":1.006,"15min":1.4159999999999999},"min":{"1min":0.33600000000000002,"5min":0.318,"15min":0.28899999999999998},"max":{"1min":1.585,"5min":7.4290000000000003,"15min":14.914},"last":0.68500000000000005}]},{"osd":4,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.81899999999999995,"15min":0.90100000000000002},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":3.032,"5min":3.032,"15min":14.352},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.23,"5min":0.83299999999999996,"15min":1.0660000000000001},"min":{"1min":0.34999999999999998,"5min":0.30399999999999999,"15min":0.253},"max":{"1min":6.3630000000000004,"5min":6.3630000000000004,"15min":9.5980000000000008},"last":0.46999999999999997}]},{"osd":6,"last update":"Fri Jun 14 23:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.95799999999999996,"15min":1.349},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.0629999999999997,"5min":9.4710000000000001,"15min":9.4710000000000001},"last":0.496},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.1180000000000001,"15min":1.6659999999999999},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.028,"5min":10.632,"15min":10.632},"last":0.39600000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.4019999999999999,"15min":1.282},"min":{"1min":0.32500000000000001,"5min":0.29099999999999998,"15min":0.27900000000000003},"max":{"1min":6.8010000000000002,"5min":26.981999999999999,"15min":26.981999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.512,"15min":1.262},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.2629999999999999,"5min":23.326000000000001,"15min":23.326000000000001},"last":0.55600000000000005}]}]},{"osd":4,"up_from":30,"seq":128849019385,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418180,"kb_used_data":523388,"kb_used_omap":1,"kb_used_meta":894782,"kb_avail":92326972,"statfs":{"total":95995035648,"available":94542819328,"internally_reserved":0,"allocated":535949312,"data_stored":534186481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916257112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.83699999999999997,"15min":1.1459999999999999},"min":{"1min":0.26500000000000001,"5min":0.217,"15min":0.17100000000000001},"max":{"1min":3.177,"5min":6.875,"15min":10.567},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.81200000000000006,"15min":1.1120000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":3.323,"5min":6.1379999999999999,"15min":8.3390000000000004},"last":0.76700000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.339,"15min":1.482},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23100000000000001},"max":{"1min":3.3759999999999999,"5min":42.656999999999996,"15min":56.917000000000002},"last":0.434},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.3109999999999999,"15min":1.5029999999999999},"min":{"1min":0.317,"5min":0.249,"15min":0.249},"max":{"1min":3.2090000000000001,"5min":42.192999999999998,"15min":56.789999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.88900000000000001,"15min":0.83799999999999997},"min":{"1min":0.29999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.468,"5min":3.468,"15min":3.468},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.93899999999999995,"15min":0.90000000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.4209999999999998,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":1.0229999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.6930000000000001,"15min":1.3360000000000001},"min":{"1min":0.316,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":3.3809999999999998,"5min":44.389000000000003,"15min":44.389000000000003},"last":1.069},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.8260000000000001,"15min":1.5},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":3.1000000000000001,"5min":46.101999999999997,"15min":46.101999999999997},"last":0.58599999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:12:36 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.85799999999999998,"15min":1.5529999999999999},"min":{"1min":0.28399999999999997,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.4159999999999999,"5min":4.04,"15min":139.13200000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.88900000000000001,"15min":1.613},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.2360000000000002,"5min":6.9290000000000003,"15min":138.86000000000001},"last":0.44800000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:13:09 2024","interfaces":[{"interface":"back","average":{"1min":7.6500000000000004,"5min":2.427,"15min":1.542},"min":{"1min":0.39200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":122.88500000000001,"5min":122.88500000000001,"15min":122.88500000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":7.8129999999999997,"5min":2.4550000000000001,"15min":1.9179999999999999},"min":{"1min":0.35699999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":122.21299999999999,"5min":122.21299999999999,"15min":122.21299999999999},"last":0.96099999999999997}]},{"osd":7,"last update":"Fri Jun 14 23:13:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.351,"15min":1.127},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":4.8789999999999996,"5min":6.5910000000000002,"15min":8.9489999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.397,"15min":1.181},"min":{"1min":0.48199999999999998,"5min":0.32400000000000001,"15min":0.27300000000000002},"max":{"1min":4.9349999999999996,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":1.0369999999999999}]}]},{"osd":3,"up_from":138,"seq":592705487084,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1794892,"kb_used_data":907200,"kb_used_omap":4,"kb_used_meta":887611,"kb_avail":91950260,"statfs":{"total":95995035648,"available":94157066240,"internally_reserved":0,"allocated":928972800,"data_stored":927209916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":908914142},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":1.083,"15min":1.347},"min":{"1min":0.187,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.77200000000000002,"5min":25.190000000000001,"15min":25.190000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":1.2050000000000001,"15min":1.3560000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.17799999999999999},"max":{"1min":2.504,"5min":25.652000000000001,"15min":25.652000000000001},"last":0.21299999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.304,"15min":1.4530000000000001},"min":{"1min":0.245,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":2.649,"5min":25.754000000000001,"15min":25.754000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.548,"15min":1.623},"min":{"1min":0.27700000000000002,"5min":0.251,"15min":0.25},"max":{"1min":2.6899999999999999,"5min":27.992999999999999,"15min":27.992999999999999},"last":1.0940000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.877,"15min":0.89900000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.5230000000000001,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.72899999999999998,"15min":0.71099999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.8730000000000002,"5min":2.972,"15min":2.972},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.5760000000000001,"15min":1.74},"min":{"1min":0.34100000000000003,"5min":0.28299999999999997,"15min":0.27900000000000003},"max":{"1min":2.1269999999999998,"5min":25.094000000000001,"15min":73.263000000000005},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.3959999999999999,"15min":1.6659999999999999},"min":{"1min":0.34499999999999997,"5min":0.27600000000000002,"15min":0.27300000000000002},"max":{"1min":2.8559999999999999,"5min":25.5,"15min":73.5},"last":0.58799999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":1.5760000000000001,"15min":1.7589999999999999},"min":{"1min":0.38500000000000001,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":2.0369999999999999,"5min":25.521000000000001,"15min":81.438999999999993},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.6240000000000001,"15min":1.7989999999999999},"min":{"1min":0.41899999999999998,"5min":0.32800000000000001,"15min":0.29299999999999998},"max":{"1min":3.0739999999999998,"5min":25.84,"15min":81.626999999999995},"last":1.117}]},{"osd":6,"last update":"Fri Jun 14 23:13:04 2024","interfaces":[{"interface":"back","average":{"1min":2.2410000000000001,"5min":1.8360000000000001,"15min":1.752},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":17.141999999999999,"5min":25.869,"15min":25.869},"last":1.0609999999999999},{"interface":"front","average":{"1min":2.0680000000000001,"5min":1.641,"15min":1.5529999999999999},"min":{"1min":0.38100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":20.056999999999999,"5min":25.143000000000001,"15min":25.143000000000001},"last":1.137}]},{"osd":7,"last update":"Fri Jun 14 23:12:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.4379999999999999,"15min":1.4330000000000001},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.30499999999999999},"max":{"1min":2.4940000000000002,"5min":25.120000000000001,"15min":25.120000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.5940000000000001,"15min":1.3979999999999999},"min":{"1min":0.39500000000000002,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":3.4300000000000002,"5min":25.472000000000001,"15min":25.472000000000001},"last":1.266}]}]},{"osd":2,"up_from":322,"seq":1382979469339,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":275516,"kb_used_data":116404,"kb_used_omap":10,"kb_used_meta":159093,"kb_avail":93469636,"statfs":{"total":95995035648,"available":95712907264,"internally_reserved":0,"allocated":119197696,"data_stored":117439292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":162911782},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.41099999999999998,"15min":0.35099999999999998},"min":{"1min":0.19600000000000001,"5min":0.192,"15min":0.192},"max":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.45600000000000002,"15min":0.44400000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.314}]},{"osd":1,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.433,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.47599999999999998,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.4089999999999998,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.28999999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.67600000000000005,"15min":0.61899999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.7519999999999998,"5min":3.7519999999999998,"15min":3.7519999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.60799999999999998,"15min":0.54600000000000004},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.26300000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":0.96299999999999997,"15min":0.92300000000000004},"min":{"1min":0.34699999999999998,"5min":0.314,"15min":0.314},"max":{"1min":4.0250000000000004,"5min":4.0250000000000004,"15min":4.0250000000000004},"last":2.9399999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":0.76400000000000001,"15min":0.69799999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.3680000000000003,"5min":4.3680000000000003,"15min":4.3680000000000003},"last":2.8069999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.79300000000000004,"15min":0.78300000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.3519999999999999,"5min":3.3519999999999999,"15min":3.3519999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.86299999999999999,"15min":0.86799999999999999},"min":{"1min":0.377,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.4750000000000001,"5min":3.4750000000000001,"15min":3.4750000000000001},"last":0.434}]},{"osd":6,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":0.89300000000000002,"15min":0.81200000000000006},"min":{"1min":0.45500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":4.8099999999999996,"5min":4.8099999999999996,"15min":4.8099999999999996},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.98099999999999998,"15min":0.97499999999999998},"min":{"1min":0.35599999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.298,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":1.867}]},{"osd":7,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94699999999999995,"15min":0.94299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":2.5089999999999999,"5min":3.008,"15min":3.008},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.8,"5min":1.24,"15min":1.147},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":498216206628,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024282,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2335309824,"data_stored":2335309824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":112398336,"data_stored":112394264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":916721664,"data_stored":916717592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":518651904,"data_stored":518651904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":177143808,"data_stored":177143808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":824573952,"data_stored":824573952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":754388992,"data_stored":754384920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:13:40.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:40.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:40.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:40.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:40.297 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:40.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:40.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:40.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:40.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:40.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:40.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:40.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:40.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:40.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:40.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:40.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:40.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:40.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:40.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:40.981 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:40.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:41.049 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:40 smithi137 ceph-mon[90634]: pgmap v1287: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.0 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 936 op/s; 29338/51336 objects degraded (57.149%); 576/51336 objects misplaced (1.122%); 7.9 MiB/s, 126 objects/s recovering 2024-06-14T23:13:41.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:41.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:40 smithi086 ceph-mon[97913]: pgmap v1287: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.0 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 936 op/s; 29338/51336 objects degraded (57.149%); 576/51336 objects misplaced (1.122%); 7.9 MiB/s, 126 objects/s recovering 2024-06-14T23:13:41.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:40 smithi086 ceph-mon[102754]: pgmap v1287: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.0 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 936 op/s; 29338/51336 objects degraded (57.149%); 576/51336 objects misplaced (1.122%); 7.9 MiB/s, 126 objects/s recovering 2024-06-14T23:13:41.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:41.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:41.176 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:41.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:41.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:41.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:41.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:41.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:41.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:41.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:41.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:41.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:41.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:41.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:41.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:41.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:41.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:41.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:41.932 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:41.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:41.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:42.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:42.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:42.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:42.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:42.213 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:42.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:42.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:42.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:42.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:42.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:42.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:42.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:42.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:42.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:42.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:42.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:42.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:42.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:42.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:42.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:42.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:42.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:43.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:43.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:43.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:43.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:42 smithi086 ceph-mon[97913]: pgmap v1288: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 989 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 947 op/s; 29018/47529 objects degraded (61.053%); 576/47529 objects misplaced (1.212%); 8.1 MiB/s, 130 objects/s recovering 2024-06-14T23:13:43.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:43.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:42 smithi086 ceph-mon[102754]: pgmap v1288: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 989 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 947 op/s; 29018/47529 objects degraded (61.053%); 576/47529 objects misplaced (1.212%); 8.1 MiB/s, 130 objects/s recovering 2024-06-14T23:13:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:43.092 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:42 smithi137 ceph-mon[90634]: pgmap v1288: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 989 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 947 op/s; 29018/47529 objects degraded (61.053%); 576/47529 objects misplaced (1.212%); 8.1 MiB/s, 130 objects/s recovering 2024-06-14T23:13:43.092 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:43.092 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:43.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:13:43.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:43.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:43.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:43.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:43.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:43.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:43.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:43.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:43.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:43.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:43.618 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:43.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:43.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:43.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:43.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:43.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:43.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:43.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:44.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:44.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:44.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:44.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 29018/47529 objects degraded (61.053%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:44.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 29018/47529 objects degraded (61.053%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:44.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:44.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:44.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:44.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:44.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 29018/47529 objects degraded (61.053%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:44.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:44.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:44.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:44.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:44.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:44.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:44.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:44.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:44.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:44.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:44.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:44.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:44.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:44.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:44.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:44.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:44.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:44.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:44.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:44.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:45.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:45.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:44 smithi086 ceph-mon[97913]: pgmap v1289: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 812 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 985 op/s; 29018/39039 objects degraded (74.331%); 576/39039 objects misplaced (1.475%); 8.1 MiB/s, 130 objects/s recovering 2024-06-14T23:13:45.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:44 smithi086 ceph-mon[102754]: pgmap v1289: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 812 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 985 op/s; 29018/39039 objects degraded (74.331%); 576/39039 objects misplaced (1.475%); 8.1 MiB/s, 130 objects/s recovering 2024-06-14T23:13:45.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:13:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:13:44] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.43.0" 2024-06-14T23:13:45.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:45.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:45.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:45.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:45.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:45.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:44 smithi137 ceph-mon[90634]: pgmap v1289: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 812 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 985 op/s; 29018/39039 objects degraded (74.331%); 576/39039 objects misplaced (1.475%); 8.1 MiB/s, 130 objects/s recovering 2024-06-14T23:13:45.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:45.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:45.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:45.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:45.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:45.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:45.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:45.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:45.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:45.646 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:45.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:45.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:45.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:45.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:45.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:45.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:45.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:46.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:46.104 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:46.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:46.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:46.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:46.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:46.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:46.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:46.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:46.374 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:13:46.374 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:13:46.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:46.482 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:46.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:46.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:46.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:46.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:46.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:46.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:46.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:46.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:46.841 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:13:46.856 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:46.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:46.861 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:13:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:13:46.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:13:46.887 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:13:46.919 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:13:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:13:46.949 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:13:46.949 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:13:46.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:46.988 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:13:46.989 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:13:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:46.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:46.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:47.019 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:13:47.020 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:13:47.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:46 smithi086 ceph-mon[102754]: pgmap v1290: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 675 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 994 op/s; 28691/32427 objects degraded (88.479%); 159/32427 objects misplaced (0.490%); 9.6 MiB/s, 152 objects/s recovering 2024-06-14T23:13:47.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:46 smithi086 ceph-mon[102754]: osdmap e343: 8 total, 8 up, 6 in 2024-06-14T23:13:47.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3638705678' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:47.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:46 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:47.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:46 smithi086 ceph-mon[97913]: pgmap v1290: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 675 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 994 op/s; 28691/32427 objects degraded (88.479%); 159/32427 objects misplaced (0.490%); 9.6 MiB/s, 152 objects/s recovering 2024-06-14T23:13:47.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:46 smithi086 ceph-mon[97913]: osdmap e343: 8 total, 8 up, 6 in 2024-06-14T23:13:47.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3638705678' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:47.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:46 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:47.051 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:13:47.051 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:13:47.073 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:13:47.073 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:13:47.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:47.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:47.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:47.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:47.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:46 smithi137 ceph-mon[90634]: pgmap v1290: 119 pgs: 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 675 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 994 op/s; 28691/32427 objects degraded (88.479%); 159/32427 objects misplaced (0.490%); 9.6 MiB/s, 152 objects/s recovering 2024-06-14T23:13:47.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:46 smithi137 ceph-mon[90634]: osdmap e343: 8 total, 8 up, 6 in 2024-06-14T23:13:47.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3638705678' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:47.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:46 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:13:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:47.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:47.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:47.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:47.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:47.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:47.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:47.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:47.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:47.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:47.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:47.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:47.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:47.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:47.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:47.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:47.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:47.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:47.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:48.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-14T23:13:48.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:48.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:47 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:48.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:47 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:13:48.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:47 smithi086 ceph-mon[102754]: osdmap e344: 8 total, 8 up, 6 in 2024-06-14T23:13:48.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2722629432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:48.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:47 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:48.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:47 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:13:48.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:47 smithi086 ceph-mon[97913]: osdmap e344: 8 total, 8 up, 6 in 2024-06-14T23:13:48.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2722629432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:48.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:48.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:48.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:48.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:48.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:47 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:13:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:47 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:13:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:47 smithi137 ceph-mon[90634]: osdmap e344: 8 total, 8 up, 6 in 2024-06-14T23:13:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2722629432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:13:48.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:48.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:48.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:48.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:48.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:48.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:48.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:48.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:48.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:48.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:48.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:48.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:48.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:48.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:48.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:48.932 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:48.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:49.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:49.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:49.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:49.072 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:48 smithi137 ceph-mon[90634]: pgmap v1293: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 589 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 959 op/s; 28176/28290 objects degraded (99.597%); 521/28290 objects misplaced (1.842%); 9.5 MiB/s, 152 objects/s recovering 2024-06-14T23:13:49.072 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:48 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:13:49.072 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:48 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 28176/28290 objects degraded (99.597%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:48 smithi086 ceph-mon[97913]: pgmap v1293: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 589 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 959 op/s; 28176/28290 objects degraded (99.597%); 521/28290 objects misplaced (1.842%); 9.5 MiB/s, 152 objects/s recovering 2024-06-14T23:13:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:13:49.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:48 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 28176/28290 objects degraded (99.597%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:49.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:48 smithi086 ceph-mon[102754]: pgmap v1293: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 589 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 959 op/s; 28176/28290 objects degraded (99.597%); 521/28290 objects misplaced (1.842%); 9.5 MiB/s, 152 objects/s recovering 2024-06-14T23:13:49.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:13:49.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:48 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 28176/28290 objects degraded (99.597%), 10 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:49.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:49.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:49.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:49.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:49.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:49.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:49.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:49.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:49.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:49.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:49.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:49.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:49.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:49.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:49.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:49.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:49.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:49.925 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:49.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:50.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:50.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:50.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:50.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:50.210 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:50.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:50.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:50.342 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:50.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:50.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:50.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:50.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:50.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:50.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:50.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:50.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:50.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:50.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:50.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:50.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:50.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:50.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:51.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:51.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:51.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:51.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:51.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:51.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:50 smithi137 ceph-mon[90634]: pgmap v1294: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 467 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 28176/22461 objects degraded (125.444%); 362/22461 objects misplaced (1.612%); 8.2 MiB/s, 131 objects/s recovering 2024-06-14T23:13:51.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:51.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:51.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:50 smithi086 ceph-mon[97913]: pgmap v1294: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 467 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 28176/22461 objects degraded (125.444%); 362/22461 objects misplaced (1.612%); 8.2 MiB/s, 131 objects/s recovering 2024-06-14T23:13:51.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:50 smithi086 ceph-mon[102754]: pgmap v1294: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 467 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 1.04k op/s; 28176/22461 objects degraded (125.444%); 362/22461 objects misplaced (1.612%); 8.2 MiB/s, 131 objects/s recovering 2024-06-14T23:13:51.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:51.451 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:51.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:51.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:51.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:51.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:51.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:51.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:51.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:51.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:51.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:51.860 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:51.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:51.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:51.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:51.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:52.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:52.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:52.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:52.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:52.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:52.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:52.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:52.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:52.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:52.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:52.496 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:52.561 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:52.564 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:52.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:52.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:52.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:52.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:52.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:52.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:52.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:52.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:52.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:52.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:53.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:53.114 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:53.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:53.123 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:52 smithi086 ceph-mon[97913]: pgmap v1295: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 403 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 818 op/s; 27913/19392 objects degraded (143.941%); 362/19392 objects misplaced (1.867%); 10 MiB/s, 164 objects/s recovering 2024-06-14T23:13:53.123 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:52 smithi086 ceph-mon[102754]: pgmap v1295: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 403 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 818 op/s; 27913/19392 objects degraded (143.941%); 362/19392 objects misplaced (1.867%); 10 MiB/s, 164 objects/s recovering 2024-06-14T23:13:53.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:53.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:52 smithi137 ceph-mon[90634]: pgmap v1295: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 403 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 818 op/s; 27913/19392 objects degraded (143.941%); 362/19392 objects misplaced (1.867%); 10 MiB/s, 164 objects/s recovering 2024-06-14T23:13:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:53.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:53.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:53.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:53.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:53.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:53.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:53.510 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:53.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:53.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:53.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:53.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:53.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:53.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:53.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:53.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:53.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:13:53.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:53.909 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:53.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:54.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:54.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:54.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 27913/19392 objects degraded (143.941%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:54.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 27913/19392 objects degraded (143.941%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:54.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:54.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:54.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:54.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 27913/19392 objects degraded (143.941%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:54.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:54.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:54.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:54.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:54.470 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:54.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:54.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:13:54.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:54.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:54.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:54.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:54.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:54.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:54.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:54.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:54.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:55.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:55.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:55.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:55.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:55.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:55.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:55.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:54 smithi137 ceph-mon[90634]: pgmap v1296: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 237 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 876 op/s; 27913/11394 objects degraded (244.980%); 72/11394 objects misplaced (0.632%); 6.8 MiB/s, 108 objects/s recovering 2024-06-14T23:13:55.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:55.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:55.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:54 smithi086 ceph-mon[102754]: pgmap v1296: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 237 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 876 op/s; 27913/11394 objects degraded (244.980%); 72/11394 objects misplaced (0.632%); 6.8 MiB/s, 108 objects/s recovering 2024-06-14T23:13:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:54 smithi086 ceph-mon[97913]: pgmap v1296: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 237 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 876 op/s; 27913/11394 objects degraded (244.980%); 72/11394 objects misplaced (0.632%); 6.8 MiB/s, 108 objects/s recovering 2024-06-14T23:13:55.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:13:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:13:54] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.43.0" 2024-06-14T23:13:55.394 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 116502 objects 2024-06-14T23:13:55.395 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :150.71 2024-06-14T23:13:55.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:55.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:55.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:55.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:55.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:55.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:55.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:55.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:55.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:55.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:55.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:55.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:55.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:55.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:55.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:56.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:56.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:56.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:56.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:56.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:56.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:56.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:56.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:56.315 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:56.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:56.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:56.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:56.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:56.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:56.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:56.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:56.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:56.651 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:13:56.651 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:13:56.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:13:56.651 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:13:56.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.652 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:184594 12.f e338/323 12:f09910e4:::benchmark_data_smithi086_261492_object68089:head v 338'11672, mlcod=314'4847)", 2024-06-14T23:13:56.652 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:03.411166+0000", 2024-06-14T23:13:56.652 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 53.236205628, 2024-06-14T23:13:56.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036949754000000001, 2024-06-14T23:13:56.652 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.652 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.652 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.411166+0000", 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.411166+0000", 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.411165+0000", 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.411172+0000", 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1250000000000004e-06 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.411183+0000", 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0531e-05 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.411188+0000", 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8490000000000002e-06 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.419843+0000", 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0086546939999999992 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.419862+0000", 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9363000000000001e-05 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.448084+0000", 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.028221162000000001 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:03.448115+0000", 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1769e-05 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.656 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:203700 12.8 e340/323 12:1fae30d4:::benchmark_data_smithi086_261492_object87195:head v 340'12895, mlcod=314'4841)", 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:24.163312+0000", 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 32.484059416000001, 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039314658000000002, 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.163312+0000", 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.163312+0000", 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.163311+0000", 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.163316+0000", 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1590000000000001e-06 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.163326+0000", 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9000000000000001e-06 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.163332+0000", 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1430000000000003e-06 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.171215+0000", 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0078826950000000003 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.171250+0000", 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5423000000000003e-05 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.202580+0000", 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031329425000000001 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:24.202627+0000", 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6718999999999998e-05 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.661 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:208423 12.9 e341/323 12:90766ffc:::benchmark_data_smithi086_261492_object91918:head v 341'13090, mlcod=314'4822)", 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:29.318545+0000", 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.328826331999998, 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.043467239999999997, 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.318545+0000", 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.318545+0000", 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.318544+0000", 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.318552+0000", 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7849999999999992e-06 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.318563+0000", 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0998999999999999e-05 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.318570+0000", 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8859999999999998e-06 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.318715+0000", 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145308 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.318745+0000", 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9941999999999999e-05 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.361809+0000", 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.043063717000000001 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.362012+0000", 2024-06-14T23:13:56.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000203375 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:208427 12.9 e341/323 12:9d285065:::benchmark_data_smithi086_261492_object91922:head v 341'13091, mlcod=314'4822)", 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:29.324939+0000", 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.322432464999999, 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037163873, 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.324939+0000", 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.324939+0000", 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.324937+0000", 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.324948+0000", 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0611999999999999e-05 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.324961+0000", 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3155e-05 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.324969+0000", 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4359999999999996e-06 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.330931+0000", 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059625679999999997 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.330954+0000", 2024-06-14T23:13:56.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2912999999999999e-05 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.362016+0000", 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.031061603 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.362103+0000", 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7009999999999995e-05 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:208595 12.8 e341/323 12:1490067a:::benchmark_data_smithi086_261492_object92090:head v 341'13218, mlcod=314'4841)", 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:29.543024+0000", 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.104346939999999, 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.053290469, 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.543024+0000", 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.543024+0000", 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.543023+0000", 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.543031+0000", 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0430000000000008e-06 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.543044+0000", 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2609000000000001e-05 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.543051+0000", 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0849999999999999e-06 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.548972+0000", 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005921506 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.548997+0000", 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4403e-05 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.596232+0000", 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047235444000000001 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.596315+0000", 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2581000000000001e-05 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:208602 12.8 e341/323 12:1d3fba58:::benchmark_data_smithi086_261492_object92097:head v 341'13219, mlcod=314'4841)", 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:29.553022+0000", 2024-06-14T23:13:56.676 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.094349360999999, 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.043381284999999999, 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.553022+0000", 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.553022+0000", 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.553022+0000", 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.553025+0000", 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8720000000000004e-06 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.553033+0000", 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9060000000000002e-06 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.553038+0000", 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1330000000000001e-06 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.559951+0000", 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069130270000000004 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.559978+0000", 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7566e-05 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.596352+0000", 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.036373638999999999 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.596403+0000", 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1465000000000003e-05 2024-06-14T23:13:56.680 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:208601 12.0 e341/323 12:050171b6:::benchmark_data_smithi086_261492_object92096:head v 341'13084, mlcod=314'4772)", 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:29.554627+0000", 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.092743892000001, 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.041722051000000003, 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.554627+0000", 2024-06-14T23:13:56.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.554627+0000", 2024-06-14T23:13:56.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.554627+0000", 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.554634+0000", 2024-06-14T23:13:56.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7729999999999999e-06 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.554649+0000", 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4531e-05 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.554655+0000", 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6109999999999999e-06 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.560884+0000", 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0062286290000000003 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.560950+0000", 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5777000000000004e-05 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.596319+0000", 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.035369376000000001 2024-06-14T23:13:56.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.596349+0000", 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0193999999999999e-05 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:208610 12.8 e341/323 12:1c62dea9:::benchmark_data_smithi086_261492_object92105:head v 341'13220, mlcod=314'4841)", 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:29.558850+0000", 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.088521450999998, 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.042602795999999998, 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.687 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.558850+0000", 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.558850+0000", 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.558849+0000", 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.558856+0000", 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5889999999999999e-06 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.558871+0000", 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5141e-05 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.558877+0000", 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7360000000000004e-06 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.567795+0000", 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089172299999999999 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.567847+0000", 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2584999999999999e-05 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.601384+0000", 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.033536449000000003 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.601453+0000", 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8889999999999999e-05 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.691 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:208612 12.0 e341/323 12:0eea3107:::benchmark_data_smithi086_261492_object92107:head v 341'13085, mlcod=314'4772)", 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:29.561654+0000", 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.085717292999998, 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.039726760999999999, 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.561654+0000", 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.561654+0000", 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.561653+0000", 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.561663+0000", 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0091999999999999e-05 2024-06-14T23:13:56.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.561680+0000", 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7027000000000001e-05 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.561688+0000", 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7000000000000008e-06 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.568785+0000", 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070968489999999997 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.568825+0000", 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0295e-05 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.601336+0000", 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.032511467000000002 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.601381+0000", 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4409000000000001e-05 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:208613 12.8 e341/323 12:16e07f41:::benchmark_data_smithi086_261492_object92108:head v 341'13221, mlcod=314'4841)", 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:29.563680+0000", 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.083691372000001, 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.037840225999999998, 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.563680+0000", 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.563680+0000", 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.563679+0000", 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:56.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.563685+0000", 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0279999999999997e-06 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.563704+0000", 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8403e-05 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.563711+0000", 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5179999999999998e-06 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.580289+0000", 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016577788999999999 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.580307+0000", 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8386999999999999e-05 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.601455+0000", 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021147943999999998 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.601520+0000", 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4814000000000007e-05 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:208626 12.0 e341/323 12:06ba0eb1:::benchmark_data_smithi086_261492_object92121:head v 341'13086, mlcod=314'4772)", 2024-06-14T23:13:56.700 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:29.577605+0000", 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.069765939, 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038787738000000002, 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.577605+0000", 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.577605+0000", 2024-06-14T23:13:56.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.577604+0000", 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.577613+0000", 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2449999999999998e-06 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.577628+0000", 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5642999999999999e-05 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.577636+0000", 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6240000000000002e-06 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.596430+0000", 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018794424000000001 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.596456+0000", 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5996999999999999e-05 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.616358+0000", 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019901295999999999 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:29.616393+0000", 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5472999999999997e-05 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDPGPush(12.5 344/323 [PushOp(12:ad4af19e:::benchmark_data_smithi086_261492_object111008:head, version: 319'6857, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(12:ad4af19e:::benchmark_data_smithi086_261492_object111008:head@319'6857, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:47.372136+0000", 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.2752353309999993, 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044493534000000001, 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:47.372136+0000", 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:47.372136+0000", 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:47.372135+0000", 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:47.372199+0000", 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4505000000000007e-05 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:47.372218+0000", 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9202999999999999e-05 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:47.372236+0000", 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7591000000000001e-05 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:47.414040+0000", 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.041804015999999999 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:47.414051+0000", 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1003e-05 2024-06-14T23:13:56.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:47.416629+0000", 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025786049999999999 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232556 12.d e344/323 12:bf0779b7:::benchmark_data_smithi086_261492_object116051:head v 344'14541, mlcod=314'4802)", 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.887331+0000", 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.7600398689999999, 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059639240000000003, 2024-06-14T23:13:56.709 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887331+0000", 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887331+0000", 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887331+0000", 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887345+0000", 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.454e-05 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887356+0000", 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1266e-05 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887363+0000", 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2820000000000002e-06 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887424+0000", 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1023999999999997e-05 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887448+0000", 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4295e-05 2024-06-14T23:13:56.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.946945+0000", 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059496926999999998 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.946971+0000", 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5774000000000001e-05 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232561 12.d e344/323 12:bd780168:::benchmark_data_smithi086_261492_object116056:head v 344'14542, mlcod=314'4802)", 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.890945+0000", 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.7564266040000001, 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.056041514000000001, 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.890945+0000", 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.890945+0000", 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.890944+0000", 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.890951+0000", 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5170000000000002e-06 2024-06-14T23:13:56.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.890962+0000", 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0433e-05 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.890968+0000", 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6109999999999999e-06 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.892554+0000", 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001585248 2024-06-14T23:13:56.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.892573+0000", 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9508999999999999e-05 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.946974+0000", 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.054400794000000002 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.946986+0000", 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2184999999999999e-05 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232565 12.f e344/323 12:f78b7fe0:::benchmark_data_smithi086_261492_object116060:head v 344'14674, mlcod=314'4847)", 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.891008+0000", 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.756363315, 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.045796768000000002, 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.891008+0000", 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.891008+0000", 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.891008+0000", 2024-06-14T23:13:56.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.891013+0000", 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8950000000000003e-06 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.891021+0000", 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.233e-06 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.891026+0000", 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6729999999999997e-06 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.892327+0000", 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013006109999999999 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.892350+0000", 2024-06-14T23:13:56.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3221e-05 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936770+0000", 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044420039000000001 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936805+0000", 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4557999999999997e-05 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.722 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232567 12.5 e344/323 12:add5c34b:::benchmark_data_smithi086_261492_object116062:head v 344'14312, mlcod=339'11983)", 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.894472+0000", 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.752898812, 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.052546051000000003, 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.894472+0000", 2024-06-14T23:13:56.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.894472+0000", 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.894472+0000", 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.894479+0000", 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0299999999999996e-06 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.894489+0000", 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8079999999999999e-06 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.894495+0000", 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0299999999999999e-06 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.894525+0000", 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0772999999999998e-05 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.894541+0000", 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5308e-05 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.946989+0000", 2024-06-14T23:13:56.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.052447832 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.947019+0000", 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9943000000000001e-05 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232564 12.7 e344/343 12:ef9bcbad:::benchmark_data_smithi086_261492_object116059:head v 344'14474, mlcod=344'14472)", 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.895327+0000", 2024-06-14T23:13:56.727 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.752044715, 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.041504996000000002, 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895327+0000", 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895327+0000", 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895326+0000", 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895333+0000", 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8950000000000001e-06 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895343+0000", 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0121e-05 2024-06-14T23:13:56.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895349+0000", 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7429999999999999e-06 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895377+0000", 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8438000000000001e-05 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895392+0000", 2024-06-14T23:13:56.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4538e-05 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936809+0000", 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.041416970999999997 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936832+0000", 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2986e-05 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232558 12.a e344/323 12:5a2cede9:::benchmark_data_smithi086_261492_object116053:head v 344'14753, mlcod=0'0)", 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.895433+0000", 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.751938177, 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.041410781000000001, 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895433+0000", 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895433+0000", 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895432+0000", 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895439+0000", 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.737e-06 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895449+0000", 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4409999999999999e-06 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895454+0000", 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7810000000000002e-06 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895622+0000", 2024-06-14T23:13:56.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016728899999999999 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.895635+0000", 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3551e-05 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936817+0000", 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.041182350999999999 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936844+0000", 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6398999999999999e-05 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232566 12.d e344/323 12:b387c827:::benchmark_data_smithi086_261492_object116061:head v 344'14543, mlcod=314'4802)", 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.898063+0000", 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.7493078550000001, 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.062928186999999997, 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.898063+0000", 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.898063+0000", 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.898063+0000", 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.898069+0000", 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8710000000000002e-06 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936761+0000", 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038692336000000001 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936771+0000", 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0461e-05 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.941716+0000", 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049446239999999999 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.941739+0000", 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3218e-05 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.960919+0000", 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019180049000000001 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.960992+0000", 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2385999999999994e-05 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232572 12.5 e344/323 12:aea5441a:::benchmark_data_smithi086_261492_object116067:head v 344'14313, mlcod=339'11983)", 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.900903+0000", 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.7464679169999999, 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.046127250000000002, 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.900903+0000", 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.900903+0000", 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.900903+0000", 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.900913+0000", 2024-06-14T23:13:56.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0633e-05 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936752+0000", 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.035838442999999998 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936767+0000", 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5591e-05 2024-06-14T23:13:56.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.937048+0000", 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00028075799999999998 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.937079+0000", 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0759000000000001e-05 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.947021+0000", 2024-06-14T23:13:56.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099417749999999999 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.947031+0000", 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0056e-05 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:13:56.745 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:13:56.766 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:56.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:56.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:56.790 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-06-14T23:13:56.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-06-14T23:13:56.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:56.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:56.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:56.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:57.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:57.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:57.029 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:56 smithi086 ceph-mon[102754]: pgmap v1297: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 115 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 904 op/s; 27617/5547 objects degraded (497.873%); 72/5547 objects misplaced (1.298%); 7.3 MiB/s, 117 objects/s recovering 2024-06-14T23:13:57.029 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:57.029 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:57.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:56 smithi086 ceph-mon[97913]: pgmap v1297: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 115 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 904 op/s; 27617/5547 objects degraded (497.873%); 72/5547 objects misplaced (1.298%); 7.3 MiB/s, 117 objects/s recovering 2024-06-14T23:13:57.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:57.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:57.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:57.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:57.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:57.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:57.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:56 smithi137 ceph-mon[90634]: pgmap v1297: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 115 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 904 op/s; 27617/5547 objects degraded (497.873%); 72/5547 objects misplaced (1.298%); 7.3 MiB/s, 117 objects/s recovering 2024-06-14T23:13:57.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:57.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:57.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:57.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:57.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:57.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:57.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:57.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:57.437 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:57.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:57.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:57.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:57.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:57.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:57.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:57.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:57.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:57.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:57.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:57.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:57.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:57.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:58.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:58.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:58.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:58.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:58.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:58.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:58.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:58.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:58.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:58.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:58.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:58.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:58.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:58.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:13:58.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:13:58.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:58.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:58.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:58.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:13:58.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:58.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:58.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:13:58.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:58.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:58.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:58.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:58.626 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:58.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:58.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:58.750 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:58.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:58.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:58.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:58.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:58.918 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:13:58.920 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:13:58.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:13:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:58.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:58.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:13:59.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:13:59.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:59.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:13:59.190 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:59 smithi086 ceph-mon[97913]: pgmap v1298: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 47 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 793 op/s; 27138/2319 objects degraded (1170.246%); 6.2 MiB/s, 142 objects/s recovering 2024-06-14T23:13:59.191 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:13:59 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 27138/2319 objects degraded (1170.246%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:59.191 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:59 smithi086 ceph-mon[102754]: pgmap v1298: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 47 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 793 op/s; 27138/2319 objects degraded (1170.246%); 6.2 MiB/s, 142 objects/s recovering 2024-06-14T23:13:59.191 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:13:59 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 27138/2319 objects degraded (1170.246%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:59.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:59.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:59.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:13:59.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:59 smithi137 ceph-mon[90634]: pgmap v1298: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 47 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 793 op/s; 27138/2319 objects degraded (1170.246%); 6.2 MiB/s, 142 objects/s recovering 2024-06-14T23:13:59.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:13:59 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 27138/2319 objects degraded (1170.246%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-14T23:13:59.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:13:59.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:59.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:13:59.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:59.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:59.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:59.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:13:59.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:59.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:13:59.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-14T23:13:59.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:13:59.747 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:59.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:59.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:13:59.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:13:59.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:13:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:59.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:13:59.902 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:13:59.972 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:13:59.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:13:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:13:59.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:00.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:00.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:00.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:00.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:00.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:00.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:00.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:00.294 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:00 smithi086 ceph-mon[97913]: from='client.36828 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:14:00.294 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:00 smithi086 ceph-mon[102754]: from='client.36828 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:14:00.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:00.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:00.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:00.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:00.492 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:00.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:00.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:00.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:00 smithi137 ceph-mon[90634]: from='client.36828 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:14:00.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:00.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:00.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:00.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:00.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:00.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:00.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:14:00.810 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:14:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:14:00.829 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:14:00.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:00.847 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:14:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:14:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:14:00.866 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:14:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:00.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:14:00.888 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:14:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:14:00.911 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:14:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:14:00.934 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:14:00.956 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:14:00.957 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:14:00.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:00.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:00.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:01.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:01.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:01.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:01.237 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:01.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:01.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:01.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:01 smithi086 ceph-mon[102754]: pgmap v1299: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 783 op/s; 27138/75 objects degraded (36184.000%); 5.6 MiB/s, 129 objects/s recovering 2024-06-14T23:14:01.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2843124135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:01.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:01 smithi086 ceph-mon[97913]: pgmap v1299: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 783 op/s; 27138/75 objects degraded (36184.000%); 5.6 MiB/s, 129 objects/s recovering 2024-06-14T23:14:01.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2843124135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:01.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:01.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:01.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:01.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:01.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:01.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:01 smithi137 ceph-mon[90634]: pgmap v1299: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 783 op/s; 27138/75 objects degraded (36184.000%); 5.6 MiB/s, 129 objects/s recovering 2024-06-14T23:14:01.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2843124135' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:01.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:01.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:01.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:01.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:01.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:01.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:01.856 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:01.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:01.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:01.979 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:01.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:02.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:02.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:02.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:02.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:02.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:02.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:02.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:02.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:02 smithi086 ceph-mon[102754]: osdmap e345: 8 total, 8 up, 6 in 2024-06-14T23:14:02.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:02 smithi086 ceph-mon[97913]: osdmap e345: 8 total, 8 up, 6 in 2024-06-14T23:14:02.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:02.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:02.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:02.478 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:02.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:02.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:02 smithi137 ceph-mon[90634]: osdmap e345: 8 total, 8 up, 6 in 2024-06-14T23:14:02.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:02.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:02.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:02.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:02.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:02.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:02.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:02.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:02.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:02.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:02.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:02.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:02.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:03.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:03.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:03.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:03.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1298,"stamp":"2024-06-14T23:13:57.743290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":49782254,"num_objects":773,"num_object_clones":0,"num_object_copies":2319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":56,"num_read_kb":104,"num_write":232345,"num_write_kb":7456431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23694,"num_bytes_recovered":1520473302,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83187,"ondisk_log_size":83187,"up":357,"acting":347,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5217956,"kb_used_data":172212,"kb_used_omap":27,"kb_used_meta":5045668,"kb_avail":557252956,"statfs":{"total":575970213888,"available":570627026944,"internally_reserved":0,"allocated":176345088,"data_stored":165637005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":5166764486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,6,8,10],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-567345152,"num_objects":-8657,"num_object_clones":0,"num_object_copies":-25971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1038,"num_objects_misplaced":-521,"num_objects_unfound":0,"num_objects_dirty":-8657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1559,"num_bytes_recovered":70778880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.908169"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":723,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835060+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:13:45.835060+0000","last_peered":"2024-06-14T23:13:45.835060+0000","last_clean":"2024-06-14T23:13:45.835060+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:13:45.835060+0000","last_undegraded":"2024-06-14T23:13:45.835060+0000","last_fullsized":"2024-06-14T23:13:45.835060+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":161,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.840864+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:13:45.840864+0000","last_peered":"2024-06-14T23:13:45.840864+0000","last_clean":"2024-06-14T23:13:45.840864+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:13:45.840864+0000","last_undegraded":"2024-06-14T23:13:45.840864+0000","last_fullsized":"2024-06-14T23:13:45.840864+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":331,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.840182+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:13:45.840182+0000","last_peered":"2024-06-14T23:13:45.840182+0000","last_clean":"2024-06-14T23:13:45.840182+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:13:45.840182+0000","last_undegraded":"2024-06-14T23:13:45.840182+0000","last_fullsized":"2024-06-14T23:13:45.840182+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":237,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.830758+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:13:45.830758+0000","last_peered":"2024-06-14T23:13:45.830758+0000","last_clean":"2024-06-14T23:13:45.830758+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:13:45.830758+0000","last_undegraded":"2024-06-14T23:13:45.830758+0000","last_fullsized":"2024-06-14T23:13:45.830758+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":127,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-06-14T23:13:46.840973+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:13:46.840973+0000","last_peered":"2024-06-14T23:13:46.840973+0000","last_clean":"2024-06-14T23:13:46.840973+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:13:46.840973+0000","last_undegraded":"2024-06-14T23:13:46.840973+0000","last_fullsized":"2024-06-14T23:13:46.840973+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":766,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838069+0000","last_change":"2024-06-14T23:13:32.452266+0000","last_active":"2024-06-14T23:13:45.838069+0000","last_peered":"2024-06-14T23:13:45.838069+0000","last_clean":"2024-06-14T23:13:45.838069+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:13:45.838069+0000","last_undegraded":"2024-06-14T23:13:45.838069+0000","last_fullsized":"2024-06-14T23:13:45.838069+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:09:56.196559+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:09:56.196559+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54614,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":307,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839156+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:13:45.839156+0000","last_peered":"2024-06-14T23:13:45.839156+0000","last_clean":"2024-06-14T23:13:45.839156+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:13:45.839156+0000","last_undegraded":"2024-06-14T23:13:45.839156+0000","last_fullsized":"2024-06-14T23:13:45.839156+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":348,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839452+0000","last_change":"2024-06-14T23:11:23.413009+0000","last_active":"2024-06-14T23:13:45.839452+0000","last_peered":"2024-06-14T23:13:45.839452+0000","last_clean":"2024-06-14T23:13:45.839452+0000","last_became_active":"2024-06-14T23:11:23.412491+0000","last_became_peered":"2024-06-14T23:11:23.412491+0000","last_unstale":"2024-06-14T23:13:45.839452+0000","last_undegraded":"2024-06-14T23:13:45.839452+0000","last_fullsized":"2024-06-14T23:13:45.839452+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:57.263547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":495,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838959+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:13:45.838959+0000","last_peered":"2024-06-14T23:13:45.838959+0000","last_clean":"2024-06-14T23:13:45.838959+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:13:45.838959+0000","last_undegraded":"2024-06-14T23:13:45.838959+0000","last_fullsized":"2024-06-14T23:13:45.838959+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":321,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838767+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:13:45.838767+0000","last_peered":"2024-06-14T23:13:45.838767+0000","last_clean":"2024-06-14T23:13:45.838767+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:13:45.838767+0000","last_undegraded":"2024-06-14T23:13:45.838767+0000","last_fullsized":"2024-06-14T23:13:45.838767+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":348,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838010+0000","last_change":"2024-06-14T23:11:23.427771+0000","last_active":"2024-06-14T23:13:45.838010+0000","last_peered":"2024-06-14T23:13:45.838010+0000","last_clean":"2024-06-14T23:13:45.838010+0000","last_became_active":"2024-06-14T23:11:23.427481+0000","last_became_peered":"2024-06-14T23:11:23.427481+0000","last_unstale":"2024-06-14T23:13:45.838010+0000","last_undegraded":"2024-06-14T23:13:45.838010+0000","last_fullsized":"2024-06-14T23:13:45.838010+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:01:04.162256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":137,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834892+0000","last_change":"2024-06-14T23:10:26.053647+0000","last_active":"2024-06-14T23:13:45.834892+0000","last_peered":"2024-06-14T23:13:45.834892+0000","last_clean":"2024-06-14T23:13:45.834892+0000","last_became_active":"2024-06-14T23:10:26.053318+0000","last_became_peered":"2024-06-14T23:10:26.053318+0000","last_unstale":"2024-06-14T23:13:45.834892+0000","last_undegraded":"2024-06-14T23:13:45.834892+0000","last_fullsized":"2024-06-14T23:13:45.834892+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:32.647107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":137,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-06-14T23:13:46.841447+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:13:46.841447+0000","last_peered":"2024-06-14T23:13:46.841447+0000","last_clean":"2024-06-14T23:13:46.841447+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:13:46.841447+0000","last_undegraded":"2024-06-14T23:13:46.841447+0000","last_fullsized":"2024-06-14T23:13:46.841447+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":647,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834413+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:13:45.834413+0000","last_peered":"2024-06-14T23:13:45.834413+0000","last_clean":"2024-06-14T23:13:45.834413+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:13:45.834413+0000","last_undegraded":"2024-06-14T23:13:45.834413+0000","last_fullsized":"2024-06-14T23:13:45.834413+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":370,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839520+0000","last_change":"2024-06-14T23:11:23.413241+0000","last_active":"2024-06-14T23:13:45.839520+0000","last_peered":"2024-06-14T23:13:45.839520+0000","last_clean":"2024-06-14T23:13:45.839520+0000","last_became_active":"2024-06-14T23:11:23.412936+0000","last_became_peered":"2024-06-14T23:11:23.412936+0000","last_unstale":"2024-06-14T23:13:45.839520+0000","last_undegraded":"2024-06-14T23:13:45.839520+0000","last_fullsized":"2024-06-14T23:13:45.839520+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:36.581716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":156,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835078+0000","last_change":"2024-06-14T23:10:26.038549+0000","last_active":"2024-06-14T23:13:45.835078+0000","last_peered":"2024-06-14T23:13:45.835078+0000","last_clean":"2024-06-14T23:13:45.835078+0000","last_became_active":"2024-06-14T23:10:26.038260+0000","last_became_peered":"2024-06-14T23:10:26.038260+0000","last_unstale":"2024-06-14T23:13:45.835078+0000","last_undegraded":"2024-06-14T23:13:45.835078+0000","last_fullsized":"2024-06-14T23:13:45.835078+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:44.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":750,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.836383+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:13:45.836383+0000","last_peered":"2024-06-14T23:13:45.836383+0000","last_clean":"2024-06-14T23:13:45.836383+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:13:45.836383+0000","last_undegraded":"2024-06-14T23:13:45.836383+0000","last_fullsized":"2024-06-14T23:13:45.836383+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":141,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839718+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:13:45.839718+0000","last_peered":"2024-06-14T23:13:45.839718+0000","last_clean":"2024-06-14T23:13:45.839718+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:13:45.839718+0000","last_undegraded":"2024-06-14T23:13:45.839718+0000","last_fullsized":"2024-06-14T23:13:45.839718+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":203,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838932+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:13:45.838932+0000","last_peered":"2024-06-14T23:13:45.838932+0000","last_clean":"2024-06-14T23:13:45.838932+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:13:45.838932+0000","last_undegraded":"2024-06-14T23:13:45.838932+0000","last_fullsized":"2024-06-14T23:13:45.838932+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":698,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838068+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:13:45.838068+0000","last_peered":"2024-06-14T23:13:45.838068+0000","last_clean":"2024-06-14T23:13:45.838068+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:13:45.838068+0000","last_undegraded":"2024-06-14T23:13:45.838068+0000","last_fullsized":"2024-06-14T23:13:45.838068+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":65,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-06-14T23:13:46.840935+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:13:46.840935+0000","last_peered":"2024-06-14T23:13:46.840935+0000","last_clean":"2024-06-14T23:13:46.840935+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:13:46.840935+0000","last_undegraded":"2024-06-14T23:13:46.840935+0000","last_fullsized":"2024-06-14T23:13:46.840935+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":295,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835798+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:13:45.835798+0000","last_peered":"2024-06-14T23:13:45.835798+0000","last_clean":"2024-06-14T23:13:45.835798+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:13:45.835798+0000","last_undegraded":"2024-06-14T23:13:45.835798+0000","last_fullsized":"2024-06-14T23:13:45.835798+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":650,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839114+0000","last_change":"2024-06-14T23:10:29.062659+0000","last_active":"2024-06-14T23:13:45.839114+0000","last_peered":"2024-06-14T23:13:45.839114+0000","last_clean":"2024-06-14T23:13:45.839114+0000","last_became_active":"2024-06-14T23:10:29.062256+0000","last_became_peered":"2024-06-14T23:10:29.062256+0000","last_unstale":"2024-06-14T23:13:45.839114+0000","last_undegraded":"2024-06-14T23:13:45.839114+0000","last_fullsized":"2024-06-14T23:13:45.839114+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:38.814157+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":327,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835533+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:13:45.835533+0000","last_peered":"2024-06-14T23:13:45.835533+0000","last_clean":"2024-06-14T23:13:45.835533+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:13:45.835533+0000","last_undegraded":"2024-06-14T23:13:45.835533+0000","last_fullsized":"2024-06-14T23:13:45.835533+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":191,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.840943+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:13:45.840943+0000","last_peered":"2024-06-14T23:13:45.840943+0000","last_clean":"2024-06-14T23:13:45.840943+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:13:45.840943+0000","last_undegraded":"2024-06-14T23:13:45.840943+0000","last_fullsized":"2024-06-14T23:13:45.840943+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":650,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.833138+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:13:45.833138+0000","last_peered":"2024-06-14T23:13:45.833138+0000","last_clean":"2024-06-14T23:13:45.833138+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:13:45.833138+0000","last_undegraded":"2024-06-14T23:13:45.833138+0000","last_fullsized":"2024-06-14T23:13:45.833138+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":210,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839531+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:13:45.839531+0000","last_peered":"2024-06-14T23:13:45.839531+0000","last_clean":"2024-06-14T23:13:45.839531+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:13:45.839531+0000","last_undegraded":"2024-06-14T23:13:45.839531+0000","last_fullsized":"2024-06-14T23:13:45.839531+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":315,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839582+0000","last_change":"2024-06-14T23:10:26.053008+0000","last_active":"2024-06-14T23:13:45.839582+0000","last_peered":"2024-06-14T23:13:45.839582+0000","last_clean":"2024-06-14T23:13:45.839582+0000","last_became_active":"2024-06-14T23:10:26.052701+0000","last_became_peered":"2024-06-14T23:10:26.052701+0000","last_unstale":"2024-06-14T23:13:45.839582+0000","last_undegraded":"2024-06-14T23:13:45.839582+0000","last_fullsized":"2024-06-14T23:13:45.839582+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:15:03.929062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":675,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.831775+0000","last_change":"2024-06-14T23:11:23.411926+0000","last_active":"2024-06-14T23:13:45.831775+0000","last_peered":"2024-06-14T23:13:45.831775+0000","last_clean":"2024-06-14T23:13:45.831775+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:13:45.831775+0000","last_undegraded":"2024-06-14T23:13:45.831775+0000","last_fullsized":"2024-06-14T23:13:45.831775+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:09.356524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:09.356524+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":228,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835312+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:13:45.835312+0000","last_peered":"2024-06-14T23:13:45.835312+0000","last_clean":"2024-06-14T23:13:45.835312+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:13:45.835312+0000","last_undegraded":"2024-06-14T23:13:45.835312+0000","last_fullsized":"2024-06-14T23:13:45.835312+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":357,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834169+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:13:45.834169+0000","last_peered":"2024-06-14T23:13:45.834169+0000","last_clean":"2024-06-14T23:13:45.834169+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:13:45.834169+0000","last_undegraded":"2024-06-14T23:13:45.834169+0000","last_fullsized":"2024-06-14T23:13:45.834169+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":520,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.840551+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:13:45.840551+0000","last_peered":"2024-06-14T23:13:45.840551+0000","last_clean":"2024-06-14T23:13:45.840551+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:13:45.840551+0000","last_undegraded":"2024-06-14T23:13:45.840551+0000","last_fullsized":"2024-06-14T23:13:45.840551+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":327,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835457+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:13:45.835457+0000","last_peered":"2024-06-14T23:13:45.835457+0000","last_clean":"2024-06-14T23:13:45.835457+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:13:45.835457+0000","last_undegraded":"2024-06-14T23:13:45.835457+0000","last_fullsized":"2024-06-14T23:13:45.835457+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":331,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838080+0000","last_change":"2024-06-14T23:10:26.047451+0000","last_active":"2024-06-14T23:13:45.838080+0000","last_peered":"2024-06-14T23:13:45.838080+0000","last_clean":"2024-06-14T23:13:45.838080+0000","last_became_active":"2024-06-14T23:10:26.046781+0000","last_became_peered":"2024-06-14T23:10:26.046781+0000","last_unstale":"2024-06-14T23:13:45.838080+0000","last_undegraded":"2024-06-14T23:13:45.838080+0000","last_fullsized":"2024-06-14T23:13:45.838080+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:07:15.892864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":559,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837294+0000","last_change":"2024-06-14T23:10:39.253251+0000","last_active":"2024-06-14T23:13:45.837294+0000","last_peered":"2024-06-14T23:13:45.837294+0000","last_clean":"2024-06-14T23:13:45.837294+0000","last_became_active":"2024-06-14T23:10:29.061386+0000","last_became_peered":"2024-06-14T23:10:29.061386+0000","last_unstale":"2024-06-14T23:13:45.837294+0000","last_undegraded":"2024-06-14T23:13:45.837294+0000","last_fullsized":"2024-06-14T23:13:45.837294+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.735431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.735431+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"344'14580","reported_seq":14650,"reported_epoch":344,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:55.376865+0000","last_change":"2024-06-14T23:11:24.652983+0000","last_active":"2024-06-14T23:13:55.376865+0000","last_peered":"2024-06-14T23:13:55.376865+0000","last_clean":"2024-06-14T23:10:37.054302+0000","last_became_active":"2024-06-14T23:11:24.646468+0000","last_became_peered":"2024-06-14T23:11:24.646468+0000","last_unstale":"2024-06-14T23:13:55.376865+0000","last_undegraded":"2024-06-14T23:11:24.406142+0000","last_fullsized":"2024-06-14T23:11:24.380989+0000","mapping_epoch":323,"log_start":"336'10205","ondisk_log_start":"336'10205","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4375,"log_dups_size":0,"ondisk_log_size":4375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:33:08.769394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14580,"num_write_kb":466560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-2468}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":304,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834163+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:13:45.834163+0000","last_peered":"2024-06-14T23:13:45.834163+0000","last_clean":"2024-06-14T23:13:45.834163+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:13:45.834163+0000","last_undegraded":"2024-06-14T23:13:45.834163+0000","last_fullsized":"2024-06-14T23:13:45.834163+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":316,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835603+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:13:45.835603+0000","last_peered":"2024-06-14T23:13:45.835603+0000","last_clean":"2024-06-14T23:13:45.835603+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:13:45.835603+0000","last_undegraded":"2024-06-14T23:13:45.835603+0000","last_fullsized":"2024-06-14T23:13:45.835603+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":102,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837144+0000","last_change":"2024-06-14T23:10:26.058423+0000","last_active":"2024-06-14T23:13:45.837144+0000","last_peered":"2024-06-14T23:13:45.837144+0000","last_clean":"2024-06-14T23:13:45.837144+0000","last_became_active":"2024-06-14T23:10:26.050244+0000","last_became_peered":"2024-06-14T23:10:26.050244+0000","last_unstale":"2024-06-14T23:13:45.837144+0000","last_undegraded":"2024-06-14T23:13:45.837144+0000","last_fullsized":"2024-06-14T23:13:45.837144+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:52.060325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":669,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834849+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:13:45.834849+0000","last_peered":"2024-06-14T23:13:45.834849+0000","last_clean":"2024-06-14T23:13:45.834849+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:13:45.834849+0000","last_undegraded":"2024-06-14T23:13:45.834849+0000","last_fullsized":"2024-06-14T23:13:45.834849+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"344'14722","reported_seq":14804,"reported_epoch":344,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:55.355982+0000","last_change":"2024-06-14T23:11:24.639438+0000","last_active":"2024-06-14T23:13:55.355982+0000","last_peered":"2024-06-14T23:13:55.355982+0000","last_clean":"2024-06-14T23:10:37.032734+0000","last_became_active":"2024-06-14T23:11:24.634884+0000","last_became_peered":"2024-06-14T23:11:24.634884+0000","last_unstale":"2024-06-14T23:13:55.355982+0000","last_undegraded":"2024-06-14T23:11:24.403675+0000","last_fullsized":"2024-06-14T23:11:24.380974+0000","mapping_epoch":323,"log_start":"336'10400","ondisk_log_start":"336'10400","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4322,"log_dups_size":0,"ondisk_log_size":4322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:48:19.382266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":471104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-2520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":257,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.836808+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:13:45.836808+0000","last_peered":"2024-06-14T23:13:45.836808+0000","last_clean":"2024-06-14T23:13:45.836808+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:13:45.836808+0000","last_undegraded":"2024-06-14T23:13:45.836808+0000","last_fullsized":"2024-06-14T23:13:45.836808+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":307,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837437+0000","last_change":"2024-06-14T23:10:26.051450+0000","last_active":"2024-06-14T23:13:45.837437+0000","last_peered":"2024-06-14T23:13:45.837437+0000","last_clean":"2024-06-14T23:13:45.837437+0000","last_became_active":"2024-06-14T23:10:26.050830+0000","last_became_peered":"2024-06-14T23:10:26.050830+0000","last_unstale":"2024-06-14T23:13:45.837437+0000","last_undegraded":"2024-06-14T23:13:45.837437+0000","last_fullsized":"2024-06-14T23:13:45.837437+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:16:16.175014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":58,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.840460+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:13:45.840460+0000","last_peered":"2024-06-14T23:13:45.840460+0000","last_clean":"2024-06-14T23:13:45.840460+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:13:45.840460+0000","last_undegraded":"2024-06-14T23:13:45.840460+0000","last_fullsized":"2024-06-14T23:13:45.840460+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":758,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835160+0000","last_change":"2024-06-14T23:11:23.401974+0000","last_active":"2024-06-14T23:13:45.835160+0000","last_peered":"2024-06-14T23:13:45.835160+0000","last_clean":"2024-06-14T23:13:45.835160+0000","last_became_active":"2024-06-14T23:11:23.401625+0000","last_became_peered":"2024-06-14T23:11:23.401625+0000","last_unstale":"2024-06-14T23:13:45.835160+0000","last_undegraded":"2024-06-14T23:13:45.835160+0000","last_fullsized":"2024-06-14T23:13:45.835160+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.214300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.214300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:05:54.552362+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"344'14654","reported_seq":14725,"reported_epoch":344,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:55.364727+0000","last_change":"2024-06-14T23:11:24.651371+0000","last_active":"2024-06-14T23:13:55.364727+0000","last_peered":"2024-06-14T23:13:55.364727+0000","last_clean":"2024-06-14T23:10:37.020885+0000","last_became_active":"2024-06-14T23:11:24.646284+0000","last_became_peered":"2024-06-14T23:11:24.646284+0000","last_unstale":"2024-06-14T23:13:55.364727+0000","last_undegraded":"2024-06-14T23:11:24.405009+0000","last_fullsized":"2024-06-14T23:11:24.380990+0000","mapping_epoch":323,"log_start":"336'10342","ondisk_log_start":"336'10342","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:26:58.760076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14654,"num_write_kb":468928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-2556}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":155,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838265+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:13:45.838265+0000","last_peered":"2024-06-14T23:13:45.838265+0000","last_clean":"2024-06-14T23:13:45.838265+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:13:45.838265+0000","last_undegraded":"2024-06-14T23:13:45.838265+0000","last_fullsized":"2024-06-14T23:13:45.838265+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":329,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.833202+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:13:45.833202+0000","last_peered":"2024-06-14T23:13:45.833202+0000","last_clean":"2024-06-14T23:13:45.833202+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:13:45.833202+0000","last_undegraded":"2024-06-14T23:13:45.833202+0000","last_fullsized":"2024-06-14T23:13:45.833202+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":281,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.831984+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:13:45.831984+0000","last_peered":"2024-06-14T23:13:45.831984+0000","last_clean":"2024-06-14T23:13:45.831984+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:13:45.831984+0000","last_undegraded":"2024-06-14T23:13:45.831984+0000","last_fullsized":"2024-06-14T23:13:45.831984+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":643,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.840327+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:13:45.840327+0000","last_peered":"2024-06-14T23:13:45.840327+0000","last_clean":"2024-06-14T23:13:45.840327+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:13:45.840327+0000","last_undegraded":"2024-06-14T23:13:45.840327+0000","last_fullsized":"2024-06-14T23:13:45.840327+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.a","version":"344'14772","reported_seq":24423,"reported_epoch":344,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:55.367374+0000","last_change":"2024-06-14T23:11:24.647584+0000","last_active":"2024-06-14T23:13:55.367374+0000","last_peered":"2024-06-14T23:13:55.367374+0000","last_clean":"2024-06-14T23:10:25.009797+0000","last_became_active":"2024-06-14T23:11:24.638765+0000","last_became_peered":"2024-06-14T23:11:24.638765+0000","last_unstale":"2024-06-14T23:13:55.367374+0000","last_undegraded":"2024-06-14T23:11:24.535343+0000","last_fullsized":"2024-06-14T23:11:24.381001+0000","mapping_epoch":323,"log_start":"336'10472","ondisk_log_start":"336'10472","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:37.543254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14772,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4780,"num_bytes_recovered":313262080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[3,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"2,3","objects":-2604}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":120,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837190+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:13:45.837190+0000","last_peered":"2024-06-14T23:13:45.837190+0000","last_clean":"2024-06-14T23:13:45.837190+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:13:45.837190+0000","last_undegraded":"2024-06-14T23:13:45.837190+0000","last_fullsized":"2024-06-14T23:13:45.837190+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":65,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.831703+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:13:45.831703+0000","last_peered":"2024-06-14T23:13:45.831703+0000","last_clean":"2024-06-14T23:13:45.831703+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:13:45.831703+0000","last_undegraded":"2024-06-14T23:13:45.831703+0000","last_fullsized":"2024-06-14T23:13:45.831703+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":359,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839222+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:13:45.839222+0000","last_peered":"2024-06-14T23:13:45.839222+0000","last_clean":"2024-06-14T23:13:45.839222+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:13:45.839222+0000","last_undegraded":"2024-06-14T23:13:45.839222+0000","last_fullsized":"2024-06-14T23:13:45.839222+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":623,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835669+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:13:45.835669+0000","last_peered":"2024-06-14T23:13:45.835669+0000","last_clean":"2024-06-14T23:13:45.835669+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:13:45.835669+0000","last_undegraded":"2024-06-14T23:13:45.835669+0000","last_fullsized":"2024-06-14T23:13:45.835669+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.c","version":"344'14658","reported_seq":19904,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-06-14T23:13:53.234415+0000","last_change":"2024-06-14T23:11:50.568053+0000","last_active":"2024-06-14T23:13:53.234415+0000","last_peered":"2024-06-14T23:13:53.234415+0000","last_clean":"2024-06-14T23:13:53.234415+0000","last_became_active":"2024-06-14T23:11:50.555786+0000","last_became_peered":"2024-06-14T23:11:50.555786+0000","last_unstale":"2024-06-14T23:13:53.234415+0000","last_undegraded":"2024-06-14T23:13:53.234415+0000","last_fullsized":"2024-06-14T23:13:53.234415+0000","mapping_epoch":328,"log_start":"333'9300","ondisk_log_start":"333'9300","created":295,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5358,"log_dups_size":0,"ondisk_log_size":5358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:36.382663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14658,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2575,"num_bytes_recovered":168755200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":296,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.831574+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:13:45.831574+0000","last_peered":"2024-06-14T23:13:45.831574+0000","last_clean":"2024-06-14T23:13:45.831574+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:13:45.831574+0000","last_undegraded":"2024-06-14T23:13:45.831574+0000","last_fullsized":"2024-06-14T23:13:45.831574+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":342,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838239+0000","last_change":"2024-06-14T23:10:26.049033+0000","last_active":"2024-06-14T23:13:45.838239+0000","last_peered":"2024-06-14T23:13:45.838239+0000","last_clean":"2024-06-14T23:13:45.838239+0000","last_became_active":"2024-06-14T23:10:26.047775+0000","last_became_peered":"2024-06-14T23:10:26.047775+0000","last_unstale":"2024-06-14T23:13:45.838239+0000","last_undegraded":"2024-06-14T23:13:45.838239+0000","last_fullsized":"2024-06-14T23:13:45.838239+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:15:52.051858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":336,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837435+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:13:45.837435+0000","last_peered":"2024-06-14T23:13:45.837435+0000","last_clean":"2024-06-14T23:13:45.837435+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:13:45.837435+0000","last_undegraded":"2024-06-14T23:13:45.837435+0000","last_fullsized":"2024-06-14T23:13:45.837435+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":737,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.836849+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:13:45.836849+0000","last_peered":"2024-06-14T23:13:45.836849+0000","last_clean":"2024-06-14T23:13:45.836849+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:13:45.836849+0000","last_undegraded":"2024-06-14T23:13:45.836849+0000","last_fullsized":"2024-06-14T23:13:45.836849+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.f","version":"344'14696","reported_seq":14774,"reported_epoch":344,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:55.340121+0000","last_change":"2024-06-14T23:11:24.656131+0000","last_active":"2024-06-14T23:13:55.340121+0000","last_peered":"2024-06-14T23:13:55.340121+0000","last_clean":"2024-06-14T23:10:37.043860+0000","last_became_active":"2024-06-14T23:11:24.647995+0000","last_became_peered":"2024-06-14T23:11:24.647995+0000","last_unstale":"2024-06-14T23:13:55.340121+0000","last_undegraded":"2024-06-14T23:11:24.403317+0000","last_fullsized":"2024-06-14T23:11:24.380506+0000","mapping_epoch":323,"log_start":"335'9700","ondisk_log_start":"335'9700","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4996,"log_dups_size":0,"ondisk_log_size":4996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:32:10.827694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14696,"num_write_kb":470272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":-2501}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":304,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.840699+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:13:45.840699+0000","last_peered":"2024-06-14T23:13:45.840699+0000","last_clean":"2024-06-14T23:13:45.840699+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:13:45.840699+0000","last_undegraded":"2024-06-14T23:13:45.840699+0000","last_fullsized":"2024-06-14T23:13:45.840699+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":341,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835707+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:13:45.835707+0000","last_peered":"2024-06-14T23:13:45.835707+0000","last_clean":"2024-06-14T23:13:45.835707+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:13:45.835707+0000","last_undegraded":"2024-06-14T23:13:45.835707+0000","last_fullsized":"2024-06-14T23:13:45.835707+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835035+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:13:45.835035+0000","last_peered":"2024-06-14T23:13:45.835035+0000","last_clean":"2024-06-14T23:13:45.835035+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:13:45.835035+0000","last_undegraded":"2024-06-14T23:13:45.835035+0000","last_fullsized":"2024-06-14T23:13:45.835035+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":631,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839200+0000","last_change":"2024-06-14T23:11:23.413224+0000","last_active":"2024-06-14T23:13:45.839200+0000","last_peered":"2024-06-14T23:13:45.839200+0000","last_clean":"2024-06-14T23:13:45.839200+0000","last_became_active":"2024-06-14T23:11:23.412680+0000","last_became_peered":"2024-06-14T23:11:23.412680+0000","last_unstale":"2024-06-14T23:13:45.839200+0000","last_undegraded":"2024-06-14T23:13:45.839200+0000","last_fullsized":"2024-06-14T23:13:45.839200+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:48:10.967651+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.e","version":"344'14407","reported_seq":14474,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-06-14T23:13:53.422837+0000","last_change":"2024-06-14T23:09:41.994118+0000","last_active":"2024-06-14T23:13:53.422837+0000","last_peered":"2024-06-14T23:13:53.422837+0000","last_clean":"2024-06-14T23:13:53.422837+0000","last_became_active":"2024-06-14T23:09:41.993702+0000","last_became_peered":"2024-06-14T23:09:41.993702+0000","last_unstale":"2024-06-14T23:13:53.422837+0000","last_undegraded":"2024-06-14T23:13:53.422837+0000","last_fullsized":"2024-06-14T23:13:53.422837+0000","mapping_epoch":299,"log_start":"333'9100","ondisk_log_start":"333'9100","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5307,"log_dups_size":0,"ondisk_log_size":5307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:25.994252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8192000,"num_objects":125,"num_object_clones":0,"num_object_copies":375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14407,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":314,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.830891+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:13:45.830891+0000","last_peered":"2024-06-14T23:13:45.830891+0000","last_clean":"2024-06-14T23:13:45.830891+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:13:45.830891+0000","last_undegraded":"2024-06-14T23:13:45.830891+0000","last_fullsized":"2024-06-14T23:13:45.830891+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":192,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834037+0000","last_change":"2024-06-14T23:09:41.991652+0000","last_active":"2024-06-14T23:13:45.834037+0000","last_peered":"2024-06-14T23:13:45.834037+0000","last_clean":"2024-06-14T23:13:45.834037+0000","last_became_active":"2024-06-14T23:09:41.991317+0000","last_became_peered":"2024-06-14T23:09:41.991317+0000","last_unstale":"2024-06-14T23:13:45.834037+0000","last_undegraded":"2024-06-14T23:13:45.834037+0000","last_fullsized":"2024-06-14T23:13:45.834037+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:16:18.209329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":309,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835415+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:13:45.835415+0000","last_peered":"2024-06-14T23:13:45.835415+0000","last_clean":"2024-06-14T23:13:45.835415+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:13:45.835415+0000","last_undegraded":"2024-06-14T23:13:45.835415+0000","last_fullsized":"2024-06-14T23:13:45.835415+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":551,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834608+0000","last_change":"2024-06-14T23:10:29.063981+0000","last_active":"2024-06-14T23:13:45.834608+0000","last_peered":"2024-06-14T23:13:45.834608+0000","last_clean":"2024-06-14T23:13:45.834608+0000","last_became_active":"2024-06-14T23:10:29.063546+0000","last_became_peered":"2024-06-14T23:10:29.063546+0000","last_unstale":"2024-06-14T23:13:45.834608+0000","last_undegraded":"2024-06-14T23:13:45.834608+0000","last_fullsized":"2024-06-14T23:13:45.834608+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:26:23.411322+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.d","version":"344'14473","reported_seq":22937,"reported_epoch":344,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:53.836651+0000","last_change":"2024-06-14T23:11:24.664901+0000","last_active":"2024-06-14T23:13:53.836651+0000","last_peered":"2024-06-14T23:13:53.836651+0000","last_clean":"2024-06-14T23:10:24.989118+0000","last_became_active":"2024-06-14T23:11:24.647313+0000","last_became_peered":"2024-06-14T23:11:24.647313+0000","last_unstale":"2024-06-14T23:13:53.836651+0000","last_undegraded":"2024-06-14T23:11:24.555289+0000","last_fullsized":"2024-06-14T23:11:24.381668+0000","mapping_epoch":323,"log_start":"335'9600","ondisk_log_start":"335'9600","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:48:12.784805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553624,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14473,"num_write_kb":466305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4187,"num_bytes_recovered":274399232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"1,3,7","objects":100},{"shards":"3,7","objects":-2383}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":259,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835575+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:13:45.835575+0000","last_peered":"2024-06-14T23:13:45.835575+0000","last_clean":"2024-06-14T23:13:45.835575+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:13:45.835575+0000","last_undegraded":"2024-06-14T23:13:45.835575+0000","last_fullsized":"2024-06-14T23:13:45.835575+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":265,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837362+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:13:45.837362+0000","last_peered":"2024-06-14T23:13:45.837362+0000","last_clean":"2024-06-14T23:13:45.837362+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:13:45.837362+0000","last_undegraded":"2024-06-14T23:13:45.837362+0000","last_fullsized":"2024-06-14T23:13:45.837362+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":346,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838325+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:13:45.838325+0000","last_peered":"2024-06-14T23:13:45.838325+0000","last_clean":"2024-06-14T23:13:45.838325+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:13:45.838325+0000","last_undegraded":"2024-06-14T23:13:45.838325+0000","last_fullsized":"2024-06-14T23:13:45.838325+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":624,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835788+0000","last_change":"2024-06-14T23:11:23.406149+0000","last_active":"2024-06-14T23:13:45.835788+0000","last_peered":"2024-06-14T23:13:45.835788+0000","last_clean":"2024-06-14T23:13:45.835788+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:13:45.835788+0000","last_undegraded":"2024-06-14T23:13:45.835788+0000","last_fullsized":"2024-06-14T23:13:45.835788+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.751667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.751667+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.7","version":"344'14421","reported_seq":27824,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-06-14T23:13:53.850353+0000","last_change":"2024-06-14T23:13:46.895025+0000","last_active":"2024-06-14T23:13:53.850353+0000","last_peered":"2024-06-14T23:13:53.850353+0000","last_clean":"2024-06-14T23:13:53.850353+0000","last_became_active":"2024-06-14T23:13:46.841502+0000","last_became_peered":"2024-06-14T23:13:46.841502+0000","last_unstale":"2024-06-14T23:13:53.850353+0000","last_undegraded":"2024-06-14T23:13:53.850353+0000","last_fullsized":"2024-06-14T23:13:53.850353+0000","mapping_epoch":343,"log_start":"335'9500","ondisk_log_start":"335'9500","created":295,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:52.190691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14421,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6651,"num_bytes_recovered":435879936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":142,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835257+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:13:45.835257+0000","last_peered":"2024-06-14T23:13:45.835257+0000","last_clean":"2024-06-14T23:13:45.835257+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:13:45.835257+0000","last_undegraded":"2024-06-14T23:13:45.835257+0000","last_fullsized":"2024-06-14T23:13:45.835257+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":314,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837402+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:13:45.837402+0000","last_peered":"2024-06-14T23:13:45.837402+0000","last_clean":"2024-06-14T23:13:45.837402+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:13:45.837402+0000","last_undegraded":"2024-06-14T23:13:45.837402+0000","last_fullsized":"2024-06-14T23:13:45.837402+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":735,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837040+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:13:45.837040+0000","last_peered":"2024-06-14T23:13:45.837040+0000","last_clean":"2024-06-14T23:13:45.837040+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:13:45.837040+0000","last_undegraded":"2024-06-14T23:13:45.837040+0000","last_fullsized":"2024-06-14T23:13:45.837040+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"344'14272","reported_seq":14798,"reported_epoch":344,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:55.376135+0000","last_change":"2024-06-14T23:11:24.664127+0000","last_active":"2024-06-14T23:13:55.376135+0000","last_peered":"2024-06-14T23:13:55.376135+0000","last_clean":"2024-06-14T23:10:25.013628+0000","last_became_active":"2024-06-14T23:10:27.203384+0000","last_became_peered":"2024-06-14T23:10:27.203384+0000","last_unstale":"2024-06-14T23:13:55.376135+0000","last_undegraded":"2024-06-14T23:11:24.661184+0000","last_fullsized":"2024-06-14T23:10:27.026246+0000","mapping_epoch":310,"log_start":"335'9300","ondisk_log_start":"335'9300","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:40.932486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3931,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14272,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":14548992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"6,7","objects":-3931}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":331,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837369+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:13:45.837369+0000","last_peered":"2024-06-14T23:13:45.837369+0000","last_clean":"2024-06-14T23:13:45.837369+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:13:45.837369+0000","last_undegraded":"2024-06-14T23:13:45.837369+0000","last_fullsized":"2024-06-14T23:13:45.837369+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":342,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834376+0000","last_change":"2024-06-14T23:11:23.427812+0000","last_active":"2024-06-14T23:13:45.834376+0000","last_peered":"2024-06-14T23:13:45.834376+0000","last_clean":"2024-06-14T23:13:45.834376+0000","last_became_active":"2024-06-14T23:11:23.427563+0000","last_became_peered":"2024-06-14T23:11:23.427563+0000","last_unstale":"2024-06-14T23:13:45.834376+0000","last_undegraded":"2024-06-14T23:13:45.834376+0000","last_fullsized":"2024-06-14T23:13:45.834376+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.941006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":703,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838199+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:13:45.838199+0000","last_peered":"2024-06-14T23:13:45.838199+0000","last_clean":"2024-06-14T23:13:45.838199+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:13:45.838199+0000","last_undegraded":"2024-06-14T23:13:45.838199+0000","last_fullsized":"2024-06-14T23:13:45.838199+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.5","version":"344'14342","reported_seq":24716,"reported_epoch":344,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:56.790174+0000","last_change":"2024-06-14T23:13:45.329953+0000","last_active":"2024-06-14T23:13:56.790174+0000","last_peered":"2024-06-14T23:13:56.790174+0000","last_clean":"2024-06-14T23:10:25.016384+0000","last_became_active":"2024-06-14T23:11:24.643623+0000","last_became_peered":"2024-06-14T23:11:24.643623+0000","last_unstale":"2024-06-14T23:13:56.790174+0000","last_undegraded":"2024-06-14T23:11:24.519360+0000","last_fullsized":"2024-06-14T23:11:24.380111+0000","mapping_epoch":323,"log_start":"335'9400","ondisk_log_start":"335'9400","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4942,"log_dups_size":0,"ondisk_log_size":4942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:18:43.920069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14342,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5142,"num_bytes_recovered":305594368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":-1467}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":361,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835334+0000","last_change":"2024-06-14T23:11:23.416264+0000","last_active":"2024-06-14T23:13:45.835334+0000","last_peered":"2024-06-14T23:13:45.835334+0000","last_clean":"2024-06-14T23:13:45.835334+0000","last_became_active":"2024-06-14T23:11:23.415946+0000","last_became_peered":"2024-06-14T23:11:23.415946+0000","last_unstale":"2024-06-14T23:13:45.835334+0000","last_undegraded":"2024-06-14T23:13:45.835334+0000","last_fullsized":"2024-06-14T23:13:45.835334+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:22.668569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":212,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839392+0000","last_change":"2024-06-14T23:10:26.045363+0000","last_active":"2024-06-14T23:13:45.839392+0000","last_peered":"2024-06-14T23:13:45.839392+0000","last_clean":"2024-06-14T23:13:45.839392+0000","last_became_active":"2024-06-14T23:10:26.043844+0000","last_became_peered":"2024-06-14T23:10:26.043844+0000","last_unstale":"2024-06-14T23:13:45.839392+0000","last_undegraded":"2024-06-14T23:13:45.839392+0000","last_fullsized":"2024-06-14T23:13:45.839392+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:11:27.891184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":726,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.836315+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:13:45.836315+0000","last_peered":"2024-06-14T23:13:45.836315+0000","last_clean":"2024-06-14T23:13:45.836315+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:13:45.836315+0000","last_undegraded":"2024-06-14T23:13:45.836315+0000","last_fullsized":"2024-06-14T23:13:45.836315+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.4","version":"344'14554","reported_seq":14853,"reported_epoch":344,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:55.391651+0000","last_change":"2024-06-14T23:10:39.484566+0000","last_active":"2024-06-14T23:13:55.391651+0000","last_peered":"2024-06-14T23:13:55.391651+0000","last_clean":"2024-06-14T23:10:24.998059+0000","last_became_active":"2024-06-14T23:10:27.239025+0000","last_became_peered":"2024-06-14T23:10:27.239025+0000","last_unstale":"2024-06-14T23:13:55.391651+0000","last_undegraded":"2024-06-14T23:10:27.071395+0000","last_fullsized":"2024-06-14T23:10:27.025672+0000","mapping_epoch":310,"log_start":"335'9600","ondisk_log_start":"335'9600","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:52:40.136978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14554,"num_write_kb":465728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":7602176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"3,6","objects":-4053}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":321,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834758+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:13:45.834758+0000","last_peered":"2024-06-14T23:13:45.834758+0000","last_clean":"2024-06-14T23:13:45.834758+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:13:45.834758+0000","last_undegraded":"2024-06-14T23:13:45.834758+0000","last_fullsized":"2024-06-14T23:13:45.834758+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":142,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834755+0000","last_change":"2024-06-14T23:10:26.056252+0000","last_active":"2024-06-14T23:13:45.834755+0000","last_peered":"2024-06-14T23:13:45.834755+0000","last_clean":"2024-06-14T23:13:45.834755+0000","last_became_active":"2024-06-14T23:10:26.046629+0000","last_became_peered":"2024-06-14T23:10:26.046629+0000","last_unstale":"2024-06-14T23:13:45.834755+0000","last_undegraded":"2024-06-14T23:13:45.834755+0000","last_fullsized":"2024-06-14T23:13:45.834755+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:51:19.277386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":615,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834858+0000","last_change":"2024-06-14T23:11:23.428346+0000","last_active":"2024-06-14T23:13:45.834858+0000","last_peered":"2024-06-14T23:13:45.834858+0000","last_clean":"2024-06-14T23:13:45.834858+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:13:45.834858+0000","last_undegraded":"2024-06-14T23:13:45.834858+0000","last_fullsized":"2024-06-14T23:13:45.834858+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.656210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.656210+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"344'14507","reported_seq":14574,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-06-14T23:13:53.851867+0000","last_change":"2024-06-14T23:09:41.992020+0000","last_active":"2024-06-14T23:13:53.851867+0000","last_peered":"2024-06-14T23:13:53.851867+0000","last_clean":"2024-06-14T23:13:53.851867+0000","last_became_active":"2024-06-14T23:09:41.991722+0000","last_became_peered":"2024-06-14T23:09:41.991722+0000","last_unstale":"2024-06-14T23:13:53.851867+0000","last_undegraded":"2024-06-14T23:13:53.851867+0000","last_fullsized":"2024-06-14T23:13:53.851867+0000","mapping_epoch":299,"log_start":"335'9600","ondisk_log_start":"335'9600","created":295,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4907,"log_dups_size":0,"ondisk_log_size":4907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:35:54.751263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5963776,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14507,"num_write_kb":467136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":141,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834680+0000","last_change":"2024-06-14T23:10:26.047111+0000","last_active":"2024-06-14T23:13:45.834680+0000","last_peered":"2024-06-14T23:13:45.834680+0000","last_clean":"2024-06-14T23:13:45.834680+0000","last_became_active":"2024-06-14T23:10:26.045540+0000","last_became_peered":"2024-06-14T23:10:26.045540+0000","last_unstale":"2024-06-14T23:13:45.834680+0000","last_undegraded":"2024-06-14T23:13:45.834680+0000","last_fullsized":"2024-06-14T23:13:45.834680+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:04.131400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":322,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.838872+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:13:45.838872+0000","last_peered":"2024-06-14T23:13:45.838872+0000","last_clean":"2024-06-14T23:13:45.838872+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:13:45.838872+0000","last_undegraded":"2024-06-14T23:13:45.838872+0000","last_fullsized":"2024-06-14T23:13:45.838872+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":582,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835990+0000","last_change":"2024-06-14T23:09:17.382466+0000","last_active":"2024-06-14T23:13:45.835990+0000","last_peered":"2024-06-14T23:13:45.835990+0000","last_clean":"2024-06-14T23:13:45.835990+0000","last_became_active":"2024-06-14T23:09:12.030205+0000","last_became_peered":"2024-06-14T23:09:12.030205+0000","last_unstale":"2024-06-14T23:13:45.835990+0000","last_undegraded":"2024-06-14T23:13:45.835990+0000","last_fullsized":"2024-06-14T23:13:45.835990+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:09:17.382389+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:33.121472+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"344'14150","reported_seq":14207,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-06-14T23:13:53.441631+0000","last_change":"2024-06-14T23:09:29.408859+0000","last_active":"2024-06-14T23:13:53.441631+0000","last_peered":"2024-06-14T23:13:53.441631+0000","last_clean":"2024-06-14T23:13:53.441631+0000","last_became_active":"2024-06-14T23:09:29.408548+0000","last_became_peered":"2024-06-14T23:09:29.408548+0000","last_unstale":"2024-06-14T23:13:53.441631+0000","last_undegraded":"2024-06-14T23:13:53.441631+0000","last_fullsized":"2024-06-14T23:13:53.441631+0000","mapping_epoch":295,"log_start":"333'8800","ondisk_log_start":"333'8800","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5350,"log_dups_size":0,"ondisk_log_size":5350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:19.810173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14150,"num_write_kb":456384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":295,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834101+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:13:45.834101+0000","last_peered":"2024-06-14T23:13:45.834101+0000","last_clean":"2024-06-14T23:13:45.834101+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:13:45.834101+0000","last_undegraded":"2024-06-14T23:13:45.834101+0000","last_fullsized":"2024-06-14T23:13:45.834101+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":239,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839023+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:13:45.839023+0000","last_peered":"2024-06-14T23:13:45.839023+0000","last_clean":"2024-06-14T23:13:45.839023+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:13:45.839023+0000","last_undegraded":"2024-06-14T23:13:45.839023+0000","last_fullsized":"2024-06-14T23:13:45.839023+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":623,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835735+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:13:45.835735+0000","last_peered":"2024-06-14T23:13:45.835735+0000","last_clean":"2024-06-14T23:13:45.835735+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:13:45.835735+0000","last_undegraded":"2024-06-14T23:13:45.835735+0000","last_fullsized":"2024-06-14T23:13:45.835735+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1","version":"344'14520","reported_seq":14577,"reported_epoch":344,"state":"active+clean","last_fresh":"2024-06-14T23:13:55.381489+0000","last_change":"2024-06-14T23:09:29.403408+0000","last_active":"2024-06-14T23:13:55.381489+0000","last_peered":"2024-06-14T23:13:55.381489+0000","last_clean":"2024-06-14T23:13:55.381489+0000","last_became_active":"2024-06-14T23:09:29.403000+0000","last_became_peered":"2024-06-14T23:09:29.403000+0000","last_unstale":"2024-06-14T23:13:55.381489+0000","last_undegraded":"2024-06-14T23:13:55.381489+0000","last_fullsized":"2024-06-14T23:13:55.381489+0000","mapping_epoch":295,"log_start":"335'9600","ondisk_log_start":"335'9600","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:16.838865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14520,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":331,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834309+0000","last_change":"2024-06-14T23:10:26.040201+0000","last_active":"2024-06-14T23:13:45.834309+0000","last_peered":"2024-06-14T23:13:45.834309+0000","last_clean":"2024-06-14T23:13:45.834309+0000","last_became_active":"2024-06-14T23:10:26.039865+0000","last_became_peered":"2024-06-14T23:10:26.039865+0000","last_unstale":"2024-06-14T23:13:45.834309+0000","last_undegraded":"2024-06-14T23:13:45.834309+0000","last_fullsized":"2024-06-14T23:13:45.834309+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:34.621202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":309,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835328+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:13:45.835328+0000","last_peered":"2024-06-14T23:13:45.835328+0000","last_clean":"2024-06-14T23:13:45.835328+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:13:45.835328+0000","last_undegraded":"2024-06-14T23:13:45.835328+0000","last_fullsized":"2024-06-14T23:13:45.835328+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":752,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835377+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:13:45.835377+0000","last_peered":"2024-06-14T23:13:45.835377+0000","last_clean":"2024-06-14T23:13:45.835377+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:13:45.835377+0000","last_undegraded":"2024-06-14T23:13:45.835377+0000","last_fullsized":"2024-06-14T23:13:45.835377+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"344'14530","reported_seq":14604,"reported_epoch":344,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:13:53.423259+0000","last_change":"2024-06-14T23:11:24.632384+0000","last_active":"2024-06-14T23:13:53.423259+0000","last_peered":"2024-06-14T23:13:53.423259+0000","last_clean":"2024-06-14T23:10:37.053680+0000","last_became_active":"2024-06-14T23:11:24.623684+0000","last_became_peered":"2024-06-14T23:11:24.623684+0000","last_unstale":"2024-06-14T23:13:53.423259+0000","last_undegraded":"2024-06-14T23:11:24.400366+0000","last_fullsized":"2024-06-14T23:11:24.379058+0000","mapping_epoch":323,"log_start":"333'9200","ondisk_log_start":"333'9200","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:09:28.385974+0000","last_clean_scrub_stamp":"2024-06-14T23:09:28.385974+0000","objects_scrubbed":0,"log_size":5330,"log_dups_size":0,"ondisk_log_size":5330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:34:53.331507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7995392,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14530,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":-2432}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":304,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.836424+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:13:45.836424+0000","last_peered":"2024-06-14T23:13:45.836424+0000","last_clean":"2024-06-14T23:13:45.836424+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:13:45.836424+0000","last_undegraded":"2024-06-14T23:13:45.836424+0000","last_fullsized":"2024-06-14T23:13:45.836424+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":322,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839501+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:13:45.839501+0000","last_peered":"2024-06-14T23:13:45.839501+0000","last_clean":"2024-06-14T23:13:45.839501+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:13:45.839501+0000","last_undegraded":"2024-06-14T23:13:45.839501+0000","last_fullsized":"2024-06-14T23:13:45.839501+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":777,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835007+0000","last_change":"2024-06-14T23:11:23.418239+0000","last_active":"2024-06-14T23:13:45.835007+0000","last_peered":"2024-06-14T23:13:45.835007+0000","last_clean":"2024-06-14T23:13:45.835007+0000","last_became_active":"2024-06-14T23:11:23.417873+0000","last_became_peered":"2024-06-14T23:11:23.417873+0000","last_unstale":"2024-06-14T23:13:45.835007+0000","last_undegraded":"2024-06-14T23:13:45.835007+0000","last_fullsized":"2024-06-14T23:13:45.835007+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:04:50.740175+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":330,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834260+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:13:45.834260+0000","last_peered":"2024-06-14T23:13:45.834260+0000","last_clean":"2024-06-14T23:13:45.834260+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:13:45.834260+0000","last_undegraded":"2024-06-14T23:13:45.834260+0000","last_fullsized":"2024-06-14T23:13:45.834260+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":327,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.835134+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:13:45.835134+0000","last_peered":"2024-06-14T23:13:45.835134+0000","last_clean":"2024-06-14T23:13:45.835134+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:13:45.835134+0000","last_undegraded":"2024-06-14T23:13:45.835134+0000","last_fullsized":"2024-06-14T23:13:45.835134+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":699,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.837119+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:13:45.837119+0000","last_peered":"2024-06-14T23:13:45.837119+0000","last_clean":"2024-06-14T23:13:45.837119+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:13:45.837119+0000","last_undegraded":"2024-06-14T23:13:45.837119+0000","last_fullsized":"2024-06-14T23:13:45.837119+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":228,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.834106+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:13:45.834106+0000","last_peered":"2024-06-14T23:13:45.834106+0000","last_clean":"2024-06-14T23:13:45.834106+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:13:45.834106+0000","last_undegraded":"2024-06-14T23:13:45.834106+0000","last_fullsized":"2024-06-14T23:13:45.834106+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":208,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839009+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:13:45.839009+0000","last_peered":"2024-06-14T23:13:45.839009+0000","last_clean":"2024-06-14T23:13:45.839009+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:13:45.839009+0000","last_undegraded":"2024-06-14T23:13:45.839009+0000","last_fullsized":"2024-06-14T23:13:45.839009+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":665,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.836405+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:13:45.836405+0000","last_peered":"2024-06-14T23:13:45.836405+0000","last_clean":"2024-06-14T23:13:45.836405+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:13:45.836405+0000","last_undegraded":"2024-06-14T23:13:45.836405+0000","last_fullsized":"2024-06-14T23:13:45.836405+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.832138+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:13:45.832138+0000","last_peered":"2024-06-14T23:13:45.832138+0000","last_clean":"2024-06-14T23:13:45.832138+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:13:45.832138+0000","last_undegraded":"2024-06-14T23:13:45.832138+0000","last_fullsized":"2024-06-14T23:13:45.832138+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":304,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.839086+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:13:45.839086+0000","last_peered":"2024-06-14T23:13:45.839086+0000","last_clean":"2024-06-14T23:13:45.839086+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:13:45.839086+0000","last_undegraded":"2024-06-14T23:13:45.839086+0000","last_fullsized":"2024-06-14T23:13:45.839086+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":663,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.836243+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:13:45.836243+0000","last_peered":"2024-06-14T23:13:45.836243+0000","last_clean":"2024-06-14T23:13:45.836243+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:13:45.836243+0000","last_undegraded":"2024-06-14T23:13:45.836243+0000","last_fullsized":"2024-06-14T23:13:45.836243+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":356,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.832613+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:13:45.832613+0000","last_peered":"2024-06-14T23:13:45.832613+0000","last_clean":"2024-06-14T23:13:45.832613+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:13:45.832613+0000","last_undegraded":"2024-06-14T23:13:45.832613+0000","last_fullsized":"2024-06-14T23:13:45.832613+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":46,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.832051+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:13:45.832051+0000","last_peered":"2024-06-14T23:13:45.832051+0000","last_clean":"2024-06-14T23:13:45.832051+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:13:45.832051+0000","last_undegraded":"2024-06-14T23:13:45.832051+0000","last_fullsized":"2024-06-14T23:13:45.832051+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":776,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-06-14T23:13:45.836528+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:13:45.836528+0000","last_peered":"2024-06-14T23:13:45.836528+0000","last_clean":"2024-06-14T23:13:45.836528+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:13:45.836528+0000","last_undegraded":"2024-06-14T23:13:45.836528+0000","last_fullsized":"2024-06-14T23:13:45.836528+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":48955416,"num_objects":748,"num_object_clones":0,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":232258,"num_write_kb":7456129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23674,"num_bytes_recovered":1520107520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2431193088,"data_stored":2431189016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78139,"ondisk_log_size":78139,"up":48,"acting":38,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":821296,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":103,"num_write":26,"num_write_kb":273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":365100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234478,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1586052,"kb_used_data":75068,"kb_used_omap":1,"kb_used_meta":1510974,"kb_avail":92159100,"statfs":{"total":95995035648,"available":94370918400,"internally_reserved":0,"allocated":76869632,"data_stored":75067845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1547237720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:34 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.0960000000000001,"15min":1.016},"min":{"1min":0.36599999999999999,"5min":0.24199999999999999,"15min":0.19800000000000001},"max":{"1min":5.8259999999999996,"5min":5.8259999999999996,"15min":6.9539999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.196,"15min":1.1930000000000001},"min":{"1min":0.30299999999999999,"5min":0.254,"15min":0.224},"max":{"1min":6.5149999999999997,"5min":6.5149999999999997,"15min":6.9550000000000001},"last":0.35599999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.196,"15min":1.306},"min":{"1min":0.38600000000000001,"5min":0.23599999999999999,"15min":0.22600000000000001},"max":{"1min":5.5629999999999997,"5min":5.5629999999999997,"15min":12.010999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.1950000000000001,"15min":1.337},"min":{"1min":0.34799999999999998,"5min":0.29599999999999999,"15min":0.22900000000000001},"max":{"1min":5.7480000000000002,"5min":5.7480000000000002,"15min":13.531000000000001},"last":0.48899999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.4970000000000001,"15min":1.4490000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":10.539,"5min":10.539,"15min":10.539},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.431,"15min":1.379},"min":{"1min":0.35899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":10.56,"5min":10.56,"15min":10.56},"last":0.57199999999999995}]},{"osd":3,"last update":"Fri Jun 14 23:13:34 2024","interfaces":[{"interface":"back","average":{"1min":2.7789999999999999,"5min":1.546,"15min":1.4410000000000001},"min":{"1min":0.40600000000000003,"5min":0.30599999999999999,"15min":0.28199999999999997},"max":{"1min":17.991,"5min":17.991,"15min":20.645},"last":0.58999999999999997},{"interface":"front","average":{"1min":2.5339999999999998,"5min":1.5640000000000001,"15min":1.532},"min":{"1min":0.443,"5min":0.29699999999999999,"15min":0.28299999999999997},"max":{"1min":17.824999999999999,"5min":17.824999999999999,"15min":22.042999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Fri Jun 14 23:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.2749999999999999,"15min":1.107},"min":{"1min":0.42299999999999999,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":3.871,"5min":5.3319999999999999,"15min":11.342000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.776,"5min":1.452,"15min":1.1539999999999999},"min":{"1min":0.316,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":10.499000000000001,"5min":10.499000000000001,"15min":12.651},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Jun 14 23:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.1200000000000001,"15min":1.1180000000000001},"min":{"1min":0.32800000000000001,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":4.0019999999999998,"5min":5.6849999999999996,"15min":6.9000000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.2849999999999999,"15min":1.135},"min":{"1min":0.45100000000000001,"5min":0.33100000000000002,"15min":0.24399999999999999},"max":{"1min":3.9020000000000001,"5min":5.3949999999999996,"15min":6.2809999999999997},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":2.137,"5min":1.7190000000000001,"15min":1.448},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.7750000000000004,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":3.0219999999999998,"5min":1.694,"15min":1.486},"min":{"1min":0.48199999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.087999999999999,"5min":10.087999999999999,"15min":10.087999999999999},"last":0.70999999999999996}]}]},{"osd":6,"up_from":272,"seq":1168231104585,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397616,"kb_used_data":4648,"kb_used_omap":7,"kb_used_meta":392952,"kb_avail":93347536,"statfs":{"total":95995035648,"available":95587876864,"internally_reserved":0,"allocated":4759552,"data_stored":2977778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":402383290},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.073,"15min":1.609},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.7330000000000001,"5min":6.6440000000000001,"15min":6.6440000000000001},"last":2.145},{"interface":"front","average":{"1min":1.095,"5min":1.143,"15min":1.6639999999999999},"min":{"1min":0.38800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.9430000000000001,"5min":6.6790000000000003,"15min":6.6790000000000003},"last":2.3639999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.3080000000000001,"15min":2.0859999999999999},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.0170000000000003,"5min":8.173,"15min":8.173},"last":2.806},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.238,"15min":1.996},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.9039999999999999,"5min":10.380000000000001,"15min":10.380000000000001},"last":2.46}]},{"osd":2,"last update":"Fri Jun 14 23:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.1970000000000001,"15min":1.143},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.476,"5min":7.476,"15min":7.476},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.782,"5min":1.304,"15min":1.224},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.7590000000000003,"5min":7.7590000000000003,"15min":7.7590000000000003},"last":0.45000000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":1.2370000000000001,"15min":1.7629999999999999},"min":{"1min":0.34200000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.7879999999999998,"5min":6.7089999999999996,"15min":6.7089999999999996},"last":2.4020000000000001},{"interface":"front","average":{"1min":1.722,"5min":1.629,"15min":1.992},"min":{"1min":0.38900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.4969999999999999,"5min":8.4740000000000002,"15min":8.4740000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.4079999999999999,"15min":1.444},"min":{"1min":0.32600000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.5609999999999999,"5min":11.010999999999999,"15min":11.010999999999999},"last":2.7730000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.1870000000000001,"15min":1.2749999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.2309999999999999,"5min":5.2309999999999999,"15min":5.2309999999999999},"last":2.4319999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0580000000000001,"15min":1.3440000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.8079999999999998,"5min":7.7469999999999999,"15min":7.7469999999999999},"last":2.1080000000000001},{"interface":"front","average":{"1min":1.8680000000000001,"5min":1.5209999999999999,"15min":1.6160000000000001},"min":{"1min":0.38300000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.4359999999999999,"5min":7.8680000000000003,"15min":7.8680000000000003},"last":0.504}]},{"osd":7,"last update":"Fri Jun 14 23:13:07 2024","interfaces":[{"interface":"back","average":{"1min":2.0310000000000001,"5min":1.746,"15min":1.758},"min":{"1min":0.33100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":7.54,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":3.4140000000000001},{"interface":"front","average":{"1min":1.8180000000000001,"5min":1.474,"15min":1.554},"min":{"1min":0.54400000000000004,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.9849999999999999,"5min":7.4619999999999997,"15min":7.4619999999999997},"last":3.98}]}]},{"osd":5,"up_from":38,"seq":163208757753,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1195820,"kb_used_data":20388,"kb_used_omap":1,"kb_used_meta":1175422,"kb_avail":92549332,"statfs":{"total":95995035648,"available":94770515968,"internally_reserved":0,"allocated":20877312,"data_stored":19095444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1203632472},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.70299999999999996,"15min":1.2949999999999999},"min":{"1min":0.255,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":1.5189999999999999,"5min":4.0579999999999998,"15min":14.94},"last":1.1830000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.88200000000000001,"15min":1.3149999999999999},"min":{"1min":0.34799999999999998,"5min":0.26100000000000001,"15min":0.219},"max":{"1min":1.8680000000000001,"5min":8.3520000000000003,"15min":9.3819999999999997},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.751,"15min":0.94899999999999995},"min":{"1min":0.30099999999999999,"5min":0.245,"15min":0.23999999999999999},"max":{"1min":2.79,"5min":4.1589999999999998,"15min":7.4370000000000003},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.82799999999999996,"15min":1.1479999999999999},"min":{"1min":0.28399999999999997,"5min":0.24299999999999999,"15min":0.22900000000000001},"max":{"1min":2.6240000000000001,"5min":5.1920000000000002,"15min":9.5039999999999996},"last":1.0640000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.74099999999999999,"15min":0.67900000000000005},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.9800000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.272,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.0659999999999998,"5min":6.0659999999999998,"15min":6.0659999999999998},"last":1.03}]},{"osd":3,"last update":"Fri Jun 14 23:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.78100000000000003,"15min":1.363},"min":{"1min":0.29999999999999999,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":4.1600000000000001,"5min":4.3949999999999996,"15min":10.358000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.2290000000000001,"5min":0.93500000000000005,"15min":1.377},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.28899999999999998},"max":{"1min":6.0099999999999998,"5min":6.0099999999999998,"15min":14.914},"last":0.97899999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.81899999999999995,"15min":0.90100000000000002},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":3.032,"5min":3.032,"15min":14.352},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.23,"5min":0.83299999999999996,"15min":1.0660000000000001},"min":{"1min":0.34999999999999998,"5min":0.30399999999999999,"15min":0.253},"max":{"1min":6.3630000000000004,"5min":6.3630000000000004,"15min":9.5980000000000008},"last":0.33400000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.95799999999999996,"15min":1.349},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.0629999999999997,"5min":9.4710000000000001,"15min":9.4710000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.1180000000000001,"15min":1.6659999999999999},"min":{"1min":0.27900000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.028,"5min":10.632,"15min":10.632},"last":0.61599999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.4019999999999999,"15min":1.282},"min":{"1min":0.32500000000000001,"5min":0.29099999999999998,"15min":0.27900000000000003},"max":{"1min":6.8010000000000002,"5min":26.981999999999999,"15min":26.981999999999999},"last":5.4420000000000002},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.512,"15min":1.262},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.2629999999999999,"5min":23.326000000000001,"15min":23.326000000000001},"last":0.34399999999999997}]}]},{"osd":4,"up_from":30,"seq":128849019389,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":949300,"kb_used_data":62700,"kb_used_omap":1,"kb_used_meta":886590,"kb_avail":92795852,"statfs":{"total":95995035648,"available":95022952448,"internally_reserved":0,"allocated":64204800,"data_stored":62418494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907868504},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.83699999999999997,"15min":1.1459999999999999},"min":{"1min":0.26500000000000001,"5min":0.217,"15min":0.17100000000000001},"max":{"1min":3.177,"5min":6.875,"15min":10.567},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.81200000000000006,"15min":1.1120000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":3.323,"5min":6.1379999999999999,"15min":8.3390000000000004},"last":3.9500000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.339,"15min":1.482},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23100000000000001},"max":{"1min":3.3759999999999999,"5min":42.656999999999996,"15min":56.917000000000002},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.3109999999999999,"15min":1.5029999999999999},"min":{"1min":0.317,"5min":0.249,"15min":0.249},"max":{"1min":3.2090000000000001,"5min":42.192999999999998,"15min":56.789999999999999},"last":1.2370000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.88900000000000001,"15min":0.83799999999999997},"min":{"1min":0.29999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.468,"5min":3.468,"15min":3.468},"last":1.462},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.93899999999999995,"15min":0.90000000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.4209999999999998,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":3.4980000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.6930000000000001,"15min":1.3360000000000001},"min":{"1min":0.316,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":3.3809999999999998,"5min":44.389000000000003,"15min":44.389000000000003},"last":3.089},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.8260000000000001,"15min":1.5},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.27600000000000002},"max":{"1min":3.1000000000000001,"5min":46.101999999999997,"15min":46.101999999999997},"last":0.50600000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.89200000000000002,"15min":1.5389999999999999},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.20300000000000001},"max":{"1min":3.694,"5min":4.04,"15min":139.13200000000001},"last":2.8450000000000002},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.94999999999999996,"15min":1.585},"min":{"1min":0.29099999999999998,"5min":0.28100000000000003,"15min":0.23300000000000001},"max":{"1min":5.077,"5min":6.9290000000000003,"15min":138.86000000000001},"last":0.47999999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:13:09 2024","interfaces":[{"interface":"back","average":{"1min":7.6500000000000004,"5min":2.427,"15min":1.542},"min":{"1min":0.39200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":122.88500000000001,"5min":122.88500000000001,"15min":122.88500000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":7.8129999999999997,"5min":2.4550000000000001,"15min":1.9179999999999999},"min":{"1min":0.35699999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":122.21299999999999,"5min":122.21299999999999,"15min":122.21299999999999},"last":4.8200000000000003}]},{"osd":7,"last update":"Fri Jun 14 23:13:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.351,"15min":1.127},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":4.8789999999999996,"5min":6.5910000000000002,"15min":8.9489999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.397,"15min":1.181},"min":{"1min":0.48199999999999998,"5min":0.32400000000000001,"15min":0.27300000000000002},"max":{"1min":4.9349999999999996,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":4.8659999999999997}]}]},{"osd":3,"up_from":138,"seq":592705487088,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910504,"kb_used_data":4832,"kb_used_omap":4,"kb_used_meta":905659,"kb_avail":92834648,"statfs":{"total":95995035648,"available":95062679552,"internally_reserved":0,"allocated":4947968,"data_stored":3169427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":927395294},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.75700000000000001,"15min":1.2829999999999999},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.175,"5min":25.190000000000001,"15min":25.190000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.044,"15min":1.329},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.17799999999999999},"max":{"1min":2.4020000000000001,"5min":25.652000000000001,"15min":25.652000000000001},"last":0.215}]},{"osd":1,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.218,"15min":1.393},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":2.431,"5min":25.754000000000001,"15min":25.754000000000001},"last":0.314},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.274,"15min":1.5660000000000001},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.25},"max":{"1min":2.4649999999999999,"5min":27.992999999999999,"15min":27.992999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.877,"15min":0.89900000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.5230000000000001,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.72899999999999998,"15min":0.71099999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.8730000000000002,"5min":2.972,"15min":2.972},"last":0.59999999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":1.9019999999999999,"5min":1.6619999999999999,"15min":1.798},"min":{"1min":0.30399999999999999,"5min":0.28299999999999997,"15min":0.27900000000000003},"max":{"1min":17.018999999999998,"5min":25.094000000000001,"15min":73.263000000000005},"last":0.438},{"interface":"front","average":{"1min":2.1890000000000001,"5min":1.516,"15min":1.714},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.27300000000000002},"max":{"1min":17.074000000000002,"5min":25.5,"15min":73.5},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.3759999999999999,"15min":1.7270000000000001},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":3.4740000000000002,"5min":25.521000000000001,"15min":81.438999999999993},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.4950000000000001,"15min":1.7509999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.29299999999999998},"max":{"1min":2.3199999999999998,"5min":25.84,"15min":81.626999999999995},"last":0.45300000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:13:04 2024","interfaces":[{"interface":"back","average":{"1min":2.2410000000000001,"5min":1.8360000000000001,"15min":1.752},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":17.141999999999999,"5min":25.869,"15min":25.869},"last":2.1030000000000002},{"interface":"front","average":{"1min":2.0680000000000001,"5min":1.641,"15min":1.5529999999999999},"min":{"1min":0.38100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":20.056999999999999,"5min":25.143000000000001,"15min":25.143000000000001},"last":1.1819999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":3.468,"5min":1.8080000000000001,"15min":1.571},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":44.887999999999998,"5min":44.887999999999998,"15min":44.887999999999998},"last":1.081},{"interface":"front","average":{"1min":3.6760000000000002,"5min":1.982,"15min":1.5309999999999999},"min":{"1min":0.39800000000000002,"5min":0.36099999999999999,"15min":0.32400000000000001},"max":{"1min":45.835000000000001,"5min":45.835000000000001,"15min":45.835000000000001},"last":1.27}]}]},{"osd":2,"up_from":322,"seq":1382979469344,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178664,"kb_used_data":4576,"kb_used_omap":10,"kb_used_meta":174069,"kb_avail":93566488,"statfs":{"total":95995035648,"available":95812083712,"internally_reserved":0,"allocated":4685824,"data_stored":2908017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":178247206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.41099999999999998,"15min":0.35099999999999998},"min":{"1min":0.19600000000000001,"5min":0.192,"15min":0.192},"max":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.45600000000000002,"15min":0.44400000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.21299999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.433,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.47599999999999998,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.4089999999999998,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.22700000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.67600000000000005,"15min":0.61899999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.7519999999999998,"5min":3.7519999999999998,"15min":3.7519999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.60799999999999998,"15min":0.54600000000000004},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":0.96299999999999997,"15min":0.92300000000000004},"min":{"1min":0.34699999999999998,"5min":0.314,"15min":0.314},"max":{"1min":4.0250000000000004,"5min":4.0250000000000004,"15min":4.0250000000000004},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.1639999999999999,"5min":0.76400000000000001,"15min":0.69799999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.3680000000000003,"5min":4.3680000000000003,"15min":4.3680000000000003},"last":0.49199999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.79300000000000004,"15min":0.78300000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.3519999999999999,"5min":3.3519999999999999,"15min":3.3519999999999999},"last":2.2690000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.86299999999999999,"15min":0.86799999999999999},"min":{"1min":0.377,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.4750000000000001,"5min":3.4750000000000001,"15min":3.4750000000000001},"last":0.56799999999999995}]},{"osd":6,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":0.89300000000000002,"15min":0.81200000000000006},"min":{"1min":0.45500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":4.8099999999999996,"5min":4.8099999999999996,"15min":4.8099999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.98099999999999998,"15min":0.97499999999999998},"min":{"1min":0.35599999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.298,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":2.4740000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94699999999999995,"15min":0.94299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":2.5089999999999999,"5min":3.008,"15min":3.008},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.8,"5min":1.24,"15min":1.147},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":2.395}]}]},{"osd":1,"up_from":0,"seq":498216206632,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024287,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2317680640,"data_stored":2317680640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45350912,"data_stored":45350912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16056320,"data_stored":16056320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":52105216,"data_stored":52101144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:14:03.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph status --format=json 2024-06-14T23:14:03.293 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:03.317 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:03 smithi086 ceph-mon[97913]: pgmap v1301: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 643 op/s; 27138/75 objects degraded (36184.000%); 4.1 MiB/s, 113 objects/s recovering 2024-06-14T23:14:03.317 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:03 smithi086 ceph-mon[97913]: osdmap e346: 8 total, 8 up, 6 in 2024-06-14T23:14:03.318 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:03 smithi086 ceph-mon[102754]: pgmap v1301: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 643 op/s; 27138/75 objects degraded (36184.000%); 4.1 MiB/s, 113 objects/s recovering 2024-06-14T23:14:03.318 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:03 smithi086 ceph-mon[102754]: osdmap e346: 8 total, 8 up, 6 in 2024-06-14T23:14:03.327 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:03.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:03.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:03.353 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:03 smithi137 ceph-mon[90634]: pgmap v1301: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 643 op/s; 27138/75 objects degraded (36184.000%); 4.1 MiB/s, 113 objects/s recovering 2024-06-14T23:14:03.353 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:03 smithi137 ceph-mon[90634]: osdmap e346: 8 total, 8 up, 6 in 2024-06-14T23:14:03.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:03.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:03.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:03.562 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:03.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:03.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:03.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:03.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:03.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:03.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:03.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:03.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:03.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:03.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:03.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:03.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:03.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:04.069 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:14:04.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:04.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:04.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:04.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:04 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 27138/75 objects degraded (36184.000%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-14T23:14:04.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:04 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/641048054' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:14:04.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:04 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:14:04.188 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:04 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 27138/75 objects degraded (36184.000%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-14T23:14:04.188 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:04 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/641048054' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:14:04.188 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:04 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:14:04.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:04.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:04.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:04.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:04.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:04.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:04.470 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:04.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:04 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 27138/75 objects degraded (36184.000%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-06-14T23:14:04.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:04 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/641048054' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:14:04.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:04 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:14:04.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:14:04.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:04.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:04.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:04.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:04.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:04.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:04.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:04.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:04.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:04.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:04.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:04.966 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:04.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:05.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:05.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:05.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:05.213 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:05.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:05.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:05.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:14:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:14:04] "GET /metrics HTTP/1.1" 200 36725 "" "Prometheus/2.43.0" 2024-06-14T23:14:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[97913]: pgmap v1303: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 227 op/s; 25109/75 objects degraded (33478.667%); 576 KiB/s, 322 objects/s recovering 2024-06-14T23:14:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:14:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:14:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[97913]: osdmap e347: 8 total, 8 up, 6 in 2024-06-14T23:14:05.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:14:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[102754]: pgmap v1303: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 227 op/s; 25109/75 objects degraded (33478.667%); 576 KiB/s, 322 objects/s recovering 2024-06-14T23:14:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:14:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:14:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[102754]: osdmap e347: 8 total, 8 up, 6 in 2024-06-14T23:14:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:05 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:14:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:05.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:05.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:05.440 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:05.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:05.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:05.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:05 smithi137 ceph-mon[90634]: pgmap v1303: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 227 op/s; 25109/75 objects degraded (33478.667%); 576 KiB/s, 322 objects/s recovering 2024-06-14T23:14:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:05 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:14:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:05 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:14:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:05 smithi137 ceph-mon[90634]: osdmap e347: 8 total, 8 up, 6 in 2024-06-14T23:14:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:05 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:14:05.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:05.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:05.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:05.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:05.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:05.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:05.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:05.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:05.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:05.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:05.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:05.955 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:05.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:06.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:06.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:06.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:06.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:06.202 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:06.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:06.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:06.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:06.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:06.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:06.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:06.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:06.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:06.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:06.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:06.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:06.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:06.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:06.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:06.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:06.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:06.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:06.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:06.948 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:06.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:06.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:14:07.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:07.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:07.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:07.155 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:06 smithi086 ceph-mon[102754]: pgmap v1305: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 25109/75 objects degraded (33478.667%); 0 B/s, 338 objects/s recovering 2024-06-14T23:14:07.155 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:07.155 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:07.155 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:06 smithi086 ceph-mon[97913]: pgmap v1305: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 25109/75 objects degraded (33478.667%); 0 B/s, 338 objects/s recovering 2024-06-14T23:14:07.155 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:07.155 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:07.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:07.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:07.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:07.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:06 smithi137 ceph-mon[90634]: pgmap v1305: 119 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 109 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 25109/75 objects degraded (33478.667%); 0 B/s, 338 objects/s recovering 2024-06-14T23:14:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:07.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:07.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:07.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:07.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:07.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:07.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:07.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:07.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:07.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:07.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:07.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:07.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:07.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:07.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:07.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:07.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:08.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:08.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:08.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:08.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:08.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:08.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:08.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:07 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:14:08.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:08.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:07 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:14:08.291 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:07 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:14:08.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:08.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:08.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:08.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:08.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:08.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:08.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:08.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:08.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:08.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:08.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:08.755 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:14:08.755 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:14:08.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:08.806 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:08.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:08.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:08.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:08.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:09.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:09.054 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:09.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:09.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:09.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:09.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:09.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:08 smithi137 ceph-mon[90634]: pgmap v1306: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 24433/75 objects degraded (32577.333%); 0 B/s, 404 objects/s recovering 2024-06-14T23:14:09.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:08 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:14:09.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 24433/75 objects degraded (32577.333%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-06-14T23:14:09.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:09.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:09.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:09.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:08 smithi086 ceph-mon[97913]: pgmap v1306: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 24433/75 objects degraded (32577.333%); 0 B/s, 404 objects/s recovering 2024-06-14T23:14:09.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:08 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:14:09.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 24433/75 objects degraded (32577.333%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-06-14T23:14:09.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:08 smithi086 ceph-mon[102754]: pgmap v1306: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 24433/75 objects degraded (32577.333%); 0 B/s, 404 objects/s recovering 2024-06-14T23:14:09.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:08 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:14:09.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 24433/75 objects degraded (32577.333%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-06-14T23:14:09.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:09.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:09.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:09.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:09.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:09.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:09.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-14T23:14:09.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:09.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:09.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:09.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:09.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:09.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:09.926 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:09.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:10.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:10.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:10.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:10.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:10.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:10.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:10.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:10.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:10.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:10.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:10.419 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:10.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:10.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:10.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:10.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:10.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:10.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:10.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:10.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:10.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:10.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:10.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:10.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:10.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:11.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:11.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:11.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:11.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:11.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:11.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:11.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:10 smithi137 ceph-mon[90634]: pgmap v1307: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 23161/75 objects degraded (30881.333%); 0 B/s, 496 objects/s recovering 2024-06-14T23:14:11.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:10 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:14:11.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:10 smithi137 ceph-mon[90634]: 1.16 scrub starts 2024-06-14T23:14:11.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:11.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:11.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:11.323 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:10 smithi086 ceph-mon[97913]: pgmap v1307: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 23161/75 objects degraded (30881.333%); 0 B/s, 496 objects/s recovering 2024-06-14T23:14:11.323 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:10 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:14:11.323 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:10 smithi086 ceph-mon[97913]: 1.16 scrub starts 2024-06-14T23:14:11.324 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:10 smithi086 ceph-mon[102754]: pgmap v1307: 119 pgs: 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 23161/75 objects degraded (30881.333%); 0 B/s, 496 objects/s recovering 2024-06-14T23:14:11.324 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:10 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:14:11.324 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:10 smithi086 ceph-mon[102754]: 1.16 scrub starts 2024-06-14T23:14:11.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:11.413 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:11.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:11.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:11.539 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:11.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:11.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:11.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:11.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:11.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:11.939 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_7' removed 2024-06-14T23:14:12.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[97913]: osdmap e348: 8 total, 8 up, 6 in 2024-06-14T23:14:12.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1136767778' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:14:12.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:14:12.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:14:12.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[97913]: osdmap e349: 8 total, 8 up, 6 in 2024-06-14T23:14:12.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[102754]: osdmap e348: 8 total, 8 up, 6 in 2024-06-14T23:14:12.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1136767778' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:14:12.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:14:12.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:14:12.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:11 smithi086 ceph-mon[102754]: osdmap e349: 8 total, 8 up, 6 in 2024-06-14T23:14:12.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:12.071 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:12.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:12.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:12.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:12.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:12.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:12.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:12.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:11 smithi137 ceph-mon[90634]: osdmap e348: 8 total, 8 up, 6 in 2024-06-14T23:14:12.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1136767778' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:14:12.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:11 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:14:12.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:11 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:14:12.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:11 smithi137 ceph-mon[90634]: osdmap e349: 8 total, 8 up, 6 in 2024-06-14T23:14:12.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:12.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:12.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:12.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:12.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:12.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:12.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:12.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:12.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:12.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:12.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:12.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:12.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:12.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:12.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:12.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:12.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:12.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:12.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:13.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:13.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:13.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:13.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:13.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:13.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:12 smithi137 ceph-mon[90634]: pgmap v1309: 119 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 23145/75 objects degraded (30860.000%); 0 B/s, 245 objects/s recovering 2024-06-14T23:14:13.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:13.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:12 smithi137 ceph-mon[90634]: 1.16 scrub starts 2024-06-14T23:14:13.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:12 smithi137 ceph-mon[90634]: 1.16 scrub ok 2024-06-14T23:14:13.275 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:12 smithi086 ceph-mon[102754]: pgmap v1309: 119 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 23145/75 objects degraded (30860.000%); 0 B/s, 245 objects/s recovering 2024-06-14T23:14:13.275 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:13.276 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:12 smithi086 ceph-mon[102754]: 1.16 scrub starts 2024-06-14T23:14:13.276 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:12 smithi086 ceph-mon[102754]: 1.16 scrub ok 2024-06-14T23:14:13.276 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:12 smithi086 ceph-mon[97913]: pgmap v1309: 119 pgs: 3 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 807 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 23145/75 objects degraded (30860.000%); 0 B/s, 245 objects/s recovering 2024-06-14T23:14:13.276 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:13.276 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:12 smithi086 ceph-mon[97913]: 1.16 scrub starts 2024-06-14T23:14:13.276 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:12 smithi086 ceph-mon[97913]: 1.16 scrub ok 2024-06-14T23:14:13.280 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:13.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:13.306 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T23:14:13.307 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T23:14:13.307 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_8 2024-06-14T23:14:13.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_8 16 2024-06-14T23:14:13.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:13.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:13.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:13.436 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:13.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:13.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:13.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:13.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:13.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:13.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:13.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:13.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:13.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:13.807 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:13.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:13.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:13.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:13.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:14.026 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:14:14.027 INFO:teuthology.orchestra.run.smithi086.stdout:{"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1878,"monmap":{"epoch":4,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":349,"num_osds":8,"num_up_osds":8,"osd_up_since":1718406683,"num_in_osds":6,"osd_in_since":1718406625,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":103}],"num_pgs":103,"num_pools":4,"num_objects":25,"data_bytes":813420,"bytes_used":5495115776,"bytes_avail":570475098112,"bytes_total":575970213888},"fsmap":{"epoch":9,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi086.nhdjdb","status":"up:active","gid":24548}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.86:8443/","prometheus":"http://172.21.15.86:9283/"}},"servicemap":{"epoch":29,"modified":"2024-06-14T23:13:55.742895+0000","services":{}},"progress_events":{"af1960ac-63de-49e3-896b-122ab3eb5dd7":{"message":"Upgrade to 18.2.2-1443-g6f4b0956 (12m)\n [============................] (remaining: 15m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-06-14T23:14:14.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:14.053 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:14.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:14.063 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 23145/75 objects degraded (30860.000%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-06-14T23:14:14.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 23145/75 objects degraded (30860.000%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-06-14T23:14:14.066 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:14.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:14.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:14.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:14.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 23145/75 objects degraded (30860.000%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-06-14T23:14:14.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:14.300 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:14.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:14.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:14.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:14.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:14.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:14.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:14.548 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:14.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:14.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:14.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:14.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:14.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:14.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:14.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:14.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:14.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:14.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:15.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:15.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:15.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:15.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:15.165 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:15.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:15.204 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:14:14] "GET /metrics HTTP/1.1" 200 34271 "" "Prometheus/2.43.0" 2024-06-14T23:14:15.205 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[97913]: pgmap v1311: 103 pgs: 103 active+clean; 794 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:15.205 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[97913]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23145/75 objects degraded (30860.000%), 9 pgs degraded, 9 pgs undersized) 2024-06-14T23:14:15.205 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3591700073' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-14T23:14:15.205 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[97913]: 1.9 scrub starts 2024-06-14T23:14:15.205 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[97913]: 1.9 scrub ok 2024-06-14T23:14:15.205 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[102754]: pgmap v1311: 103 pgs: 103 active+clean; 794 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:15.205 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[102754]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23145/75 objects degraded (30860.000%), 9 pgs degraded, 9 pgs undersized) 2024-06-14T23:14:15.206 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3591700073' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-14T23:14:15.206 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[102754]: 1.9 scrub starts 2024-06-14T23:14:15.206 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:14 smithi086 ceph-mon[102754]: 1.9 scrub ok 2024-06-14T23:14:15.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:14 smithi137 ceph-mon[90634]: pgmap v1311: 103 pgs: 103 active+clean; 794 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:15.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:14 smithi137 ceph-mon[90634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23145/75 objects degraded (30860.000%), 9 pgs degraded, 9 pgs undersized) 2024-06-14T23:14:15.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3591700073' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-14T23:14:15.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:14 smithi137 ceph-mon[90634]: 1.9 scrub starts 2024-06-14T23:14:15.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:14 smithi137 ceph-mon[90634]: 1.9 scrub ok 2024-06-14T23:14:15.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:15.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:15.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:15.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:15.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:15.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:15.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:15.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:15.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:15.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:15.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:15.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:15.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:15.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:15.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:15.842 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-06-14T23:14:15.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:14:15.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:15.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:16.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:16.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:16.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:16.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:16.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:16.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:16.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:16.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:16.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:16.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:16.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:16.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:16.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:16.549 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:16.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:16.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:16.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:16.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:16.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:16.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:16.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:16.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:16.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:16.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:17.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:17.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:17.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:17.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:17.180 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:17.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:17.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:16 smithi137 ceph-mon[90634]: pgmap v1312: 103 pgs: 103 active+clean; 794 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:17.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:16 smithi137 ceph-mon[90634]: 1.1c scrub starts 2024-06-14T23:14:17.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:16 smithi137 ceph-mon[90634]: 1.1c scrub ok 2024-06-14T23:14:17.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:16 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:17.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:16 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:17.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:16 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:14:17.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:16 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:14:17.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:16 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:17.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:17.305 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:17.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[97913]: pgmap v1312: 103 pgs: 103 active+clean; 794 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[97913]: 1.1c scrub starts 2024-06-14T23:14:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[97913]: 1.1c scrub ok 2024-06-14T23:14:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:17.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:17.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:14:17.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:14:17.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:17.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[102754]: pgmap v1312: 103 pgs: 103 active+clean; 794 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:17.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[102754]: 1.1c scrub starts 2024-06-14T23:14:17.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[102754]: 1.1c scrub ok 2024-06-14T23:14:17.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:17.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:17.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:14:17.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:14:17.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:16 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:17.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:17.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:17.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:17.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:17.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:17.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:17.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:17.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:17.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:17.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:17.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:17.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:17.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:17.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:17.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:17.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:17.981 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:14:18.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:18.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:18.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:18.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3470916507' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:18.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:18.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[97913]: 1.14 scrub starts 2024-06-14T23:14:18.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[97913]: 1.14 scrub ok 2024-06-14T23:14:18.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:18.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3470916507' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:18.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:18.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[102754]: 1.14 scrub starts 2024-06-14T23:14:18.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[102754]: 1.14 scrub ok 2024-06-14T23:14:18.073 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:18.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:18.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:18.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:18.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:18.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3470916507' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:18.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:17 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:18.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:17 smithi137 ceph-mon[90634]: 1.14 scrub starts 2024-06-14T23:14:18.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:17 smithi137 ceph-mon[90634]: 1.14 scrub ok 2024-06-14T23:14:18.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:18.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:18.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:18.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:18.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:18.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:18.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:18.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:18.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:18.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:18.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:18.566 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:14:18.566 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:14:18.566 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:14:18.566 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:14:18.573 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:14:18.574 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:14:18.591 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:14:18.592 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:14:18.600 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:18.609 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:14:18.609 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:14:18.627 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:14:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:14:18.644 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:14:18.645 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:14:18.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:18.661 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:14:18.661 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:14:18.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:18.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:18.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:18.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:18.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:18.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:18.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:18.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:19.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:19.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:19.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:19.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:19.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:19.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:19.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:18 smithi137 ceph-mon[90634]: pgmap v1313: 103 pgs: 103 active+clean; 661 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:18 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:14:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:18 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:14:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:18 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:14:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:18 smithi137 ceph-mon[90634]: osdmap e350: 8 total, 8 up, 6 in 2024-06-14T23:14:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:18 smithi137 ceph-mon[90634]: 1.0 scrub starts 2024-06-14T23:14:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:18 smithi137 ceph-mon[90634]: 1.0 scrub ok 2024-06-14T23:14:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3935739913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:19.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[97913]: pgmap v1313: 103 pgs: 103 active+clean; 661 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:19.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:14:19.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:14:19.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[97913]: osdmap e350: 8 total, 8 up, 6 in 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[97913]: 1.0 scrub starts 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[97913]: 1.0 scrub ok 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3935739913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[102754]: pgmap v1313: 103 pgs: 103 active+clean; 661 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[102754]: osdmap e350: 8 total, 8 up, 6 in 2024-06-14T23:14:19.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[102754]: 1.0 scrub starts 2024-06-14T23:14:19.256 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[102754]: 1.0 scrub ok 2024-06-14T23:14:19.256 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3935739913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:19.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:19.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:19.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:19.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:19.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:19.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:19.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:19.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:19.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:19.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:19.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:19.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:19.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:19.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:19.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:19.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:20.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:20.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:20.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:20.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:20.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:20.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:20.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:20.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:20.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:20.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:20.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:20.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:20.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:20.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:20.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:20.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:20.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:20.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:20.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:20.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:20.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:20.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:20.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:20.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:20.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:21.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:21.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:21.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:21.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:21.098 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:21.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:14:21.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:21.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:21.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:21.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:21.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:20 smithi137 ceph-mon[90634]: pgmap v1315: 103 pgs: 103 active+clean; 608 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:21.284 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:20 smithi086 ceph-mon[97913]: pgmap v1315: 103 pgs: 103 active+clean; 608 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:21.284 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:20 smithi086 ceph-mon[102754]: pgmap v1315: 103 pgs: 103 active+clean; 608 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:21.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:21.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:21.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:21.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:21.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:21.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:21.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:21.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:21.586 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:21.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:21.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:21.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:21.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:21.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:21.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:21.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:21.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:21.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:21.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:14:22.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:22.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:22.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:22.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:22.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:22.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:22.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:22.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:22.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:22.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:22.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:22.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:22.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:22.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:22.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:22.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:22.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:22.710 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:22.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:22.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:22.834 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:22.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:22.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:22.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:22.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:23.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:23.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:23.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:23.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:23.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:23.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:23.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:23.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:22 smithi137 ceph-mon[90634]: pgmap v1316: 103 pgs: 103 active+clean; 608 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:23.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:23.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:23.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:23.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:22 smithi086 ceph-mon[97913]: pgmap v1316: 103 pgs: 103 active+clean; 608 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:23.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:22 smithi086 ceph-mon[102754]: pgmap v1316: 103 pgs: 103 active+clean; 608 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:23.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:23.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:23.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:23.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:23.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:23.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:23.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:23.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:23.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:23.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:23.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:23.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:23.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:23.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:23.956 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:23.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:24.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:24.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:24.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:24.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:24.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:24.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:24.333 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:24.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:24.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:24.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:24.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:24.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:24.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:24.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:24.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:24.699 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:14:24.701 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:14:24.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:24.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:24.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:24.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:24.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:24.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:24.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:24.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:24.975 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:14:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:14:24] "GET /metrics HTTP/1.1" 200 34271 "" "Prometheus/2.43.0" 2024-06-14T23:14:25.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:25.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:25.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:25.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:25.211 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:25.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:25.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:24 smithi137 ceph-mon[90634]: pgmap v1317: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:25.295 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:24 smithi086 ceph-mon[97913]: pgmap v1317: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:25.295 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:24 smithi086 ceph-mon[102754]: pgmap v1317: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:25.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:25.334 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:25.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:25.394 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:14:25.395 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:14:25.395 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:14:25.395 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:14:25.395 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:14:25.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:25.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:25.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:25.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:25.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:25.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:25.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:25.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:25.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:25.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:25.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:25.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:25.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:25.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:25.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:26.048 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1317,"stamp":"2024-06-14T23:14:23.750206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5048,"ondisk_log_size":5048,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5403420,"kb_used_data":28244,"kb_used_omap":27,"kb_used_meta":5375076,"kb_avail":557067492,"statfs":{"total":575970213888,"available":570437111808,"internally_reserved":0,"allocated":28921856,"data_stored":17958961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":5504078278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-217510,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-44,"num_read_kb":-82,"num_write":-20,"num_write_kb":-218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-324848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002590"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":729,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.964843+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:14:11.964843+0000","last_peered":"2024-06-14T23:14:11.964843+0000","last_clean":"2024-06-14T23:14:11.964843+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:14:11.964843+0000","last_undegraded":"2024-06-14T23:14:11.964843+0000","last_fullsized":"2024-06-14T23:14:11.964843+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":167,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370843+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:14:12.370843+0000","last_peered":"2024-06-14T23:14:12.370843+0000","last_clean":"2024-06-14T23:14:12.370843+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:14:12.370843+0000","last_undegraded":"2024-06-14T23:14:12.370843+0000","last_fullsized":"2024-06-14T23:14:12.370843+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":337,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936358+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:14:11.936358+0000","last_peered":"2024-06-14T23:14:11.936358+0000","last_clean":"2024-06-14T23:14:11.936358+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:14:11.936358+0000","last_undegraded":"2024-06-14T23:14:11.936358+0000","last_fullsized":"2024-06-14T23:14:11.936358+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":243,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937362+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:14:11.937362+0000","last_peered":"2024-06-14T23:14:11.937362+0000","last_clean":"2024-06-14T23:14:11.937362+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:14:11.937362+0000","last_undegraded":"2024-06-14T23:14:11.937362+0000","last_fullsized":"2024-06-14T23:14:11.937362+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":133,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936418+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:14:11.936418+0000","last_peered":"2024-06-14T23:14:11.936418+0000","last_clean":"2024-06-14T23:14:11.936418+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:14:11.936418+0000","last_undegraded":"2024-06-14T23:14:11.936418+0000","last_fullsized":"2024-06-14T23:14:11.936418+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":794,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:16.297984+0000","last_change":"2024-06-14T23:14:16.297931+0000","last_active":"2024-06-14T23:14:16.297984+0000","last_peered":"2024-06-14T23:14:16.297984+0000","last_clean":"2024-06-14T23:14:16.297984+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:14:16.297984+0000","last_undegraded":"2024-06-14T23:14:16.297984+0000","last_fullsized":"2024-06-14T23:14:16.297984+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:14:16.297860+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:14:16.297860+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:46.938506+0000","scrub_duration":0.0016077540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":313,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370471+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:14:12.370471+0000","last_peered":"2024-06-14T23:14:12.370471+0000","last_clean":"2024-06-14T23:14:12.370471+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:14:12.370471+0000","last_undegraded":"2024-06-14T23:14:12.370471+0000","last_fullsized":"2024-06-14T23:14:12.370471+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":354,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.938833+0000","last_change":"2024-06-14T23:11:23.413009+0000","last_active":"2024-06-14T23:14:11.938833+0000","last_peered":"2024-06-14T23:14:11.938833+0000","last_clean":"2024-06-14T23:14:11.938833+0000","last_became_active":"2024-06-14T23:11:23.412491+0000","last_became_peered":"2024-06-14T23:11:23.412491+0000","last_unstale":"2024-06-14T23:14:11.938833+0000","last_undegraded":"2024-06-14T23:14:11.938833+0000","last_fullsized":"2024-06-14T23:14:11.938833+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:57.263547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":501,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.965931+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:14:11.965931+0000","last_peered":"2024-06-14T23:14:11.965931+0000","last_clean":"2024-06-14T23:14:11.965931+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:14:11.965931+0000","last_undegraded":"2024-06-14T23:14:11.965931+0000","last_fullsized":"2024-06-14T23:14:11.965931+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":327,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370393+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:14:12.370393+0000","last_peered":"2024-06-14T23:14:12.370393+0000","last_clean":"2024-06-14T23:14:12.370393+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:14:12.370393+0000","last_undegraded":"2024-06-14T23:14:12.370393+0000","last_fullsized":"2024-06-14T23:14:12.370393+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":354,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937525+0000","last_change":"2024-06-14T23:11:23.427771+0000","last_active":"2024-06-14T23:14:11.937525+0000","last_peered":"2024-06-14T23:14:11.937525+0000","last_clean":"2024-06-14T23:14:11.937525+0000","last_became_active":"2024-06-14T23:11:23.427481+0000","last_became_peered":"2024-06-14T23:11:23.427481+0000","last_unstale":"2024-06-14T23:14:11.937525+0000","last_undegraded":"2024-06-14T23:14:11.937525+0000","last_fullsized":"2024-06-14T23:14:11.937525+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:01:04.162256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":143,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937386+0000","last_change":"2024-06-14T23:10:26.053647+0000","last_active":"2024-06-14T23:14:11.937386+0000","last_peered":"2024-06-14T23:14:11.937386+0000","last_clean":"2024-06-14T23:14:11.937386+0000","last_became_active":"2024-06-14T23:10:26.053318+0000","last_became_peered":"2024-06-14T23:10:26.053318+0000","last_unstale":"2024-06-14T23:14:11.937386+0000","last_undegraded":"2024-06-14T23:14:11.937386+0000","last_fullsized":"2024-06-14T23:14:11.937386+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:32.647107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":143,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937157+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:14:11.937157+0000","last_peered":"2024-06-14T23:14:11.937157+0000","last_clean":"2024-06-14T23:14:11.937157+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:14:11.937157+0000","last_undegraded":"2024-06-14T23:14:11.937157+0000","last_fullsized":"2024-06-14T23:14:11.937157+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":653,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.934562+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:14:11.934562+0000","last_peered":"2024-06-14T23:14:11.934562+0000","last_clean":"2024-06-14T23:14:11.934562+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:14:11.934562+0000","last_undegraded":"2024-06-14T23:14:11.934562+0000","last_fullsized":"2024-06-14T23:14:11.934562+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":376,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.938950+0000","last_change":"2024-06-14T23:11:23.413241+0000","last_active":"2024-06-14T23:14:11.938950+0000","last_peered":"2024-06-14T23:14:11.938950+0000","last_clean":"2024-06-14T23:14:11.938950+0000","last_became_active":"2024-06-14T23:11:23.412936+0000","last_became_peered":"2024-06-14T23:11:23.412936+0000","last_unstale":"2024-06-14T23:14:11.938950+0000","last_undegraded":"2024-06-14T23:14:11.938950+0000","last_fullsized":"2024-06-14T23:14:11.938950+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:36.581716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":162,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939230+0000","last_change":"2024-06-14T23:10:26.038549+0000","last_active":"2024-06-14T23:14:11.939230+0000","last_peered":"2024-06-14T23:14:11.939230+0000","last_clean":"2024-06-14T23:14:11.939230+0000","last_became_active":"2024-06-14T23:10:26.038260+0000","last_became_peered":"2024-06-14T23:10:26.038260+0000","last_unstale":"2024-06-14T23:14:11.939230+0000","last_undegraded":"2024-06-14T23:14:11.939230+0000","last_fullsized":"2024-06-14T23:14:11.939230+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:44.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":756,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370040+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:14:12.370040+0000","last_peered":"2024-06-14T23:14:12.370040+0000","last_clean":"2024-06-14T23:14:12.370040+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:14:12.370040+0000","last_undegraded":"2024-06-14T23:14:12.370040+0000","last_fullsized":"2024-06-14T23:14:12.370040+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":147,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.960668+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:14:11.960668+0000","last_peered":"2024-06-14T23:14:11.960668+0000","last_clean":"2024-06-14T23:14:11.960668+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:14:11.960668+0000","last_undegraded":"2024-06-14T23:14:11.960668+0000","last_fullsized":"2024-06-14T23:14:11.960668+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":210,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-06-14T23:14:17.979100+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:14:17.979100+0000","last_peered":"2024-06-14T23:14:17.979100+0000","last_clean":"2024-06-14T23:14:17.979100+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:14:17.979100+0000","last_undegraded":"2024-06-14T23:14:17.979100+0000","last_fullsized":"2024-06-14T23:14:17.979100+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":704,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370122+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:14:12.370122+0000","last_peered":"2024-06-14T23:14:12.370122+0000","last_clean":"2024-06-14T23:14:12.370122+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:14:12.370122+0000","last_undegraded":"2024-06-14T23:14:12.370122+0000","last_fullsized":"2024-06-14T23:14:12.370122+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":71,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937692+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:14:11.937692+0000","last_peered":"2024-06-14T23:14:11.937692+0000","last_clean":"2024-06-14T23:14:11.937692+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:14:11.937692+0000","last_undegraded":"2024-06-14T23:14:11.937692+0000","last_fullsized":"2024-06-14T23:14:11.937692+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":301,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939145+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:14:11.939145+0000","last_peered":"2024-06-14T23:14:11.939145+0000","last_clean":"2024-06-14T23:14:11.939145+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:14:11.939145+0000","last_undegraded":"2024-06-14T23:14:11.939145+0000","last_fullsized":"2024-06-14T23:14:11.939145+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":656,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.938120+0000","last_change":"2024-06-14T23:10:29.062659+0000","last_active":"2024-06-14T23:14:11.938120+0000","last_peered":"2024-06-14T23:14:11.938120+0000","last_clean":"2024-06-14T23:14:11.938120+0000","last_became_active":"2024-06-14T23:10:29.062256+0000","last_became_peered":"2024-06-14T23:10:29.062256+0000","last_unstale":"2024-06-14T23:14:11.938120+0000","last_undegraded":"2024-06-14T23:14:11.938120+0000","last_fullsized":"2024-06-14T23:14:11.938120+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:38.814157+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":333,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.958967+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:14:11.958967+0000","last_peered":"2024-06-14T23:14:11.958967+0000","last_clean":"2024-06-14T23:14:11.958967+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:14:11.958967+0000","last_undegraded":"2024-06-14T23:14:11.958967+0000","last_fullsized":"2024-06-14T23:14:11.958967+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":197,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370930+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:14:12.370930+0000","last_peered":"2024-06-14T23:14:12.370930+0000","last_clean":"2024-06-14T23:14:12.370930+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:14:12.370930+0000","last_undegraded":"2024-06-14T23:14:12.370930+0000","last_fullsized":"2024-06-14T23:14:12.370930+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":656,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.962346+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:14:11.962346+0000","last_peered":"2024-06-14T23:14:11.962346+0000","last_clean":"2024-06-14T23:14:11.962346+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:14:11.962346+0000","last_undegraded":"2024-06-14T23:14:11.962346+0000","last_fullsized":"2024-06-14T23:14:11.962346+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":216,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.959896+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:14:11.959896+0000","last_peered":"2024-06-14T23:14:11.959896+0000","last_clean":"2024-06-14T23:14:11.959896+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:14:11.959896+0000","last_undegraded":"2024-06-14T23:14:11.959896+0000","last_fullsized":"2024-06-14T23:14:11.959896+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":321,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370660+0000","last_change":"2024-06-14T23:10:26.053008+0000","last_active":"2024-06-14T23:14:12.370660+0000","last_peered":"2024-06-14T23:14:12.370660+0000","last_clean":"2024-06-14T23:14:12.370660+0000","last_became_active":"2024-06-14T23:10:26.052701+0000","last_became_peered":"2024-06-14T23:10:26.052701+0000","last_unstale":"2024-06-14T23:14:12.370660+0000","last_undegraded":"2024-06-14T23:14:12.370660+0000","last_fullsized":"2024-06-14T23:14:12.370660+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:15:03.929062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":693,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:17.053346+0000","last_change":"2024-06-14T23:14:17.053346+0000","last_active":"2024-06-14T23:14:17.053346+0000","last_peered":"2024-06-14T23:14:17.053346+0000","last_clean":"2024-06-14T23:14:17.053346+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:14:17.053346+0000","last_undegraded":"2024-06-14T23:14:17.053346+0000","last_fullsized":"2024-06-14T23:14:17.053346+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:17.053258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:17.053258+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:03.297247+0000","scrub_duration":0.0014699820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":234,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.965099+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:14:11.965099+0000","last_peered":"2024-06-14T23:14:11.965099+0000","last_clean":"2024-06-14T23:14:11.965099+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:14:11.965099+0000","last_undegraded":"2024-06-14T23:14:11.965099+0000","last_fullsized":"2024-06-14T23:14:11.965099+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":363,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936064+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:14:11.936064+0000","last_peered":"2024-06-14T23:14:11.936064+0000","last_clean":"2024-06-14T23:14:11.936064+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:14:11.936064+0000","last_undegraded":"2024-06-14T23:14:11.936064+0000","last_fullsized":"2024-06-14T23:14:11.936064+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":526,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370589+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:14:12.370589+0000","last_peered":"2024-06-14T23:14:12.370589+0000","last_clean":"2024-06-14T23:14:12.370589+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:14:12.370589+0000","last_undegraded":"2024-06-14T23:14:12.370589+0000","last_fullsized":"2024-06-14T23:14:12.370589+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":333,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939057+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:14:11.939057+0000","last_peered":"2024-06-14T23:14:11.939057+0000","last_clean":"2024-06-14T23:14:11.939057+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:14:11.939057+0000","last_undegraded":"2024-06-14T23:14:11.939057+0000","last_fullsized":"2024-06-14T23:14:11.939057+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":337,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939569+0000","last_change":"2024-06-14T23:10:26.047451+0000","last_active":"2024-06-14T23:14:11.939569+0000","last_peered":"2024-06-14T23:14:11.939569+0000","last_clean":"2024-06-14T23:14:11.939569+0000","last_became_active":"2024-06-14T23:10:26.046781+0000","last_became_peered":"2024-06-14T23:10:26.046781+0000","last_unstale":"2024-06-14T23:14:11.939569+0000","last_undegraded":"2024-06-14T23:14:11.939569+0000","last_fullsized":"2024-06-14T23:14:11.939569+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:07:15.892864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":578,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.889924+0000","last_change":"2024-06-14T23:14:12.889878+0000","last_active":"2024-06-14T23:14:12.889924+0000","last_peered":"2024-06-14T23:14:12.889924+0000","last_clean":"2024-06-14T23:14:12.889924+0000","last_became_active":"2024-06-14T23:10:29.061386+0000","last_became_peered":"2024-06-14T23:10:29.061386+0000","last_unstale":"2024-06-14T23:14:12.889924+0000","last_undegraded":"2024-06-14T23:14:12.889924+0000","last_fullsized":"2024-06-14T23:14:12.889924+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:12.889819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:12.889819+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:04:46.976666+0000","scrub_duration":0.001439832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":310,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936427+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:14:11.936427+0000","last_peered":"2024-06-14T23:14:11.936427+0000","last_clean":"2024-06-14T23:14:11.936427+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:14:11.936427+0000","last_undegraded":"2024-06-14T23:14:11.936427+0000","last_fullsized":"2024-06-14T23:14:11.936427+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":322,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.938953+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:14:11.938953+0000","last_peered":"2024-06-14T23:14:11.938953+0000","last_clean":"2024-06-14T23:14:11.938953+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:14:11.938953+0000","last_undegraded":"2024-06-14T23:14:11.938953+0000","last_fullsized":"2024-06-14T23:14:11.938953+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":108,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.940280+0000","last_change":"2024-06-14T23:10:26.058423+0000","last_active":"2024-06-14T23:14:11.940280+0000","last_peered":"2024-06-14T23:14:11.940280+0000","last_clean":"2024-06-14T23:14:11.940280+0000","last_became_active":"2024-06-14T23:10:26.050244+0000","last_became_peered":"2024-06-14T23:10:26.050244+0000","last_unstale":"2024-06-14T23:14:11.940280+0000","last_undegraded":"2024-06-14T23:14:11.940280+0000","last_fullsized":"2024-06-14T23:14:11.940280+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:52.060325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":675,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.963700+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:14:11.963700+0000","last_peered":"2024-06-14T23:14:11.963700+0000","last_clean":"2024-06-14T23:14:11.963700+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:14:11.963700+0000","last_undegraded":"2024-06-14T23:14:11.963700+0000","last_fullsized":"2024-06-14T23:14:11.963700+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":264,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.959393+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:14:11.959393+0000","last_peered":"2024-06-14T23:14:11.959393+0000","last_clean":"2024-06-14T23:14:11.959393+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:14:11.959393+0000","last_undegraded":"2024-06-14T23:14:11.959393+0000","last_fullsized":"2024-06-14T23:14:11.959393+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":313,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.940561+0000","last_change":"2024-06-14T23:10:26.051450+0000","last_active":"2024-06-14T23:14:11.940561+0000","last_peered":"2024-06-14T23:14:11.940561+0000","last_clean":"2024-06-14T23:14:11.940561+0000","last_became_active":"2024-06-14T23:10:26.050830+0000","last_became_peered":"2024-06-14T23:10:26.050830+0000","last_unstale":"2024-06-14T23:14:11.940561+0000","last_undegraded":"2024-06-14T23:14:11.940561+0000","last_fullsized":"2024-06-14T23:14:11.940561+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:16:16.175014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":64,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370566+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:14:12.370566+0000","last_peered":"2024-06-14T23:14:12.370566+0000","last_clean":"2024-06-14T23:14:12.370566+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:14:12.370566+0000","last_undegraded":"2024-06-14T23:14:12.370566+0000","last_fullsized":"2024-06-14T23:14:12.370566+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":764,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939308+0000","last_change":"2024-06-14T23:11:23.401974+0000","last_active":"2024-06-14T23:14:11.939308+0000","last_peered":"2024-06-14T23:14:11.939308+0000","last_clean":"2024-06-14T23:14:11.939308+0000","last_became_active":"2024-06-14T23:11:23.401625+0000","last_became_peered":"2024-06-14T23:11:23.401625+0000","last_unstale":"2024-06-14T23:14:11.939308+0000","last_undegraded":"2024-06-14T23:14:11.939308+0000","last_fullsized":"2024-06-14T23:14:11.939308+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.214300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.214300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:05:54.552362+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":161,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370289+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:14:12.370289+0000","last_peered":"2024-06-14T23:14:12.370289+0000","last_clean":"2024-06-14T23:14:12.370289+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:14:12.370289+0000","last_undegraded":"2024-06-14T23:14:12.370289+0000","last_fullsized":"2024-06-14T23:14:12.370289+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":335,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.962428+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:14:11.962428+0000","last_peered":"2024-06-14T23:14:11.962428+0000","last_clean":"2024-06-14T23:14:11.962428+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:14:11.962428+0000","last_undegraded":"2024-06-14T23:14:11.962428+0000","last_fullsized":"2024-06-14T23:14:11.962428+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":287,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937242+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:14:11.937242+0000","last_peered":"2024-06-14T23:14:11.937242+0000","last_clean":"2024-06-14T23:14:11.937242+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:14:11.937242+0000","last_undegraded":"2024-06-14T23:14:11.937242+0000","last_fullsized":"2024-06-14T23:14:11.937242+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":649,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939376+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:14:11.939376+0000","last_peered":"2024-06-14T23:14:11.939376+0000","last_clean":"2024-06-14T23:14:11.939376+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:14:11.939376+0000","last_undegraded":"2024-06-14T23:14:11.939376+0000","last_fullsized":"2024-06-14T23:14:11.939376+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":126,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370068+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:14:12.370068+0000","last_peered":"2024-06-14T23:14:12.370068+0000","last_clean":"2024-06-14T23:14:12.370068+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:14:12.370068+0000","last_undegraded":"2024-06-14T23:14:12.370068+0000","last_fullsized":"2024-06-14T23:14:12.370068+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":71,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936943+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:14:11.936943+0000","last_peered":"2024-06-14T23:14:11.936943+0000","last_clean":"2024-06-14T23:14:11.936943+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:14:11.936943+0000","last_undegraded":"2024-06-14T23:14:11.936943+0000","last_fullsized":"2024-06-14T23:14:11.936943+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":365,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370462+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:14:12.370462+0000","last_peered":"2024-06-14T23:14:12.370462+0000","last_clean":"2024-06-14T23:14:12.370462+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:14:12.370462+0000","last_undegraded":"2024-06-14T23:14:12.370462+0000","last_fullsized":"2024-06-14T23:14:12.370462+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":629,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939019+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:14:11.939019+0000","last_peered":"2024-06-14T23:14:11.939019+0000","last_clean":"2024-06-14T23:14:11.939019+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:14:11.939019+0000","last_undegraded":"2024-06-14T23:14:11.939019+0000","last_fullsized":"2024-06-14T23:14:11.939019+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":302,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936707+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:14:11.936707+0000","last_peered":"2024-06-14T23:14:11.936707+0000","last_clean":"2024-06-14T23:14:11.936707+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:14:11.936707+0000","last_undegraded":"2024-06-14T23:14:11.936707+0000","last_fullsized":"2024-06-14T23:14:11.936707+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":348,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939708+0000","last_change":"2024-06-14T23:10:26.049033+0000","last_active":"2024-06-14T23:14:11.939708+0000","last_peered":"2024-06-14T23:14:11.939708+0000","last_clean":"2024-06-14T23:14:11.939708+0000","last_became_active":"2024-06-14T23:10:26.047775+0000","last_became_peered":"2024-06-14T23:10:26.047775+0000","last_unstale":"2024-06-14T23:14:11.939708+0000","last_undegraded":"2024-06-14T23:14:11.939708+0000","last_fullsized":"2024-06-14T23:14:11.939708+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:15:52.051858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":342,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370225+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:14:12.370225+0000","last_peered":"2024-06-14T23:14:12.370225+0000","last_clean":"2024-06-14T23:14:12.370225+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:14:12.370225+0000","last_undegraded":"2024-06-14T23:14:12.370225+0000","last_fullsized":"2024-06-14T23:14:12.370225+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":743,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.369899+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:14:12.369899+0000","last_peered":"2024-06-14T23:14:12.369899+0000","last_clean":"2024-06-14T23:14:12.369899+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:14:12.369899+0000","last_undegraded":"2024-06-14T23:14:12.369899+0000","last_fullsized":"2024-06-14T23:14:12.369899+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":310,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370665+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:14:12.370665+0000","last_peered":"2024-06-14T23:14:12.370665+0000","last_clean":"2024-06-14T23:14:12.370665+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:14:12.370665+0000","last_undegraded":"2024-06-14T23:14:12.370665+0000","last_fullsized":"2024-06-14T23:14:12.370665+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":347,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937518+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:14:11.937518+0000","last_peered":"2024-06-14T23:14:11.937518+0000","last_clean":"2024-06-14T23:14:11.937518+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:14:11.937518+0000","last_undegraded":"2024-06-14T23:14:11.937518+0000","last_fullsized":"2024-06-14T23:14:11.937518+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936339+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:14:11.936339+0000","last_peered":"2024-06-14T23:14:11.936339+0000","last_clean":"2024-06-14T23:14:11.936339+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:14:11.936339+0000","last_undegraded":"2024-06-14T23:14:11.936339+0000","last_fullsized":"2024-06-14T23:14:11.936339+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":637,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.938197+0000","last_change":"2024-06-14T23:11:23.413224+0000","last_active":"2024-06-14T23:14:11.938197+0000","last_peered":"2024-06-14T23:14:11.938197+0000","last_clean":"2024-06-14T23:14:11.938197+0000","last_became_active":"2024-06-14T23:11:23.412680+0000","last_became_peered":"2024-06-14T23:11:23.412680+0000","last_unstale":"2024-06-14T23:14:11.938197+0000","last_undegraded":"2024-06-14T23:14:11.938197+0000","last_fullsized":"2024-06-14T23:14:11.938197+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:48:10.967651+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":320,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937498+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:14:11.937498+0000","last_peered":"2024-06-14T23:14:11.937498+0000","last_clean":"2024-06-14T23:14:11.937498+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:14:11.937498+0000","last_undegraded":"2024-06-14T23:14:11.937498+0000","last_fullsized":"2024-06-14T23:14:11.937498+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":198,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936291+0000","last_change":"2024-06-14T23:09:41.991652+0000","last_active":"2024-06-14T23:14:11.936291+0000","last_peered":"2024-06-14T23:14:11.936291+0000","last_clean":"2024-06-14T23:14:11.936291+0000","last_became_active":"2024-06-14T23:09:41.991317+0000","last_became_peered":"2024-06-14T23:09:41.991317+0000","last_unstale":"2024-06-14T23:14:11.936291+0000","last_undegraded":"2024-06-14T23:14:11.936291+0000","last_fullsized":"2024-06-14T23:14:11.936291+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:16:18.209329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":315,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.958816+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:14:11.958816+0000","last_peered":"2024-06-14T23:14:11.958816+0000","last_clean":"2024-06-14T23:14:11.958816+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:14:11.958816+0000","last_undegraded":"2024-06-14T23:14:11.958816+0000","last_fullsized":"2024-06-14T23:14:11.958816+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":557,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937054+0000","last_change":"2024-06-14T23:10:29.063981+0000","last_active":"2024-06-14T23:14:11.937054+0000","last_peered":"2024-06-14T23:14:11.937054+0000","last_clean":"2024-06-14T23:14:11.937054+0000","last_became_active":"2024-06-14T23:10:29.063546+0000","last_became_peered":"2024-06-14T23:10:29.063546+0000","last_unstale":"2024-06-14T23:14:11.937054+0000","last_undegraded":"2024-06-14T23:14:11.937054+0000","last_fullsized":"2024-06-14T23:14:11.937054+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:26:23.411322+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":265,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937428+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:14:11.937428+0000","last_peered":"2024-06-14T23:14:11.937428+0000","last_clean":"2024-06-14T23:14:11.937428+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:14:11.937428+0000","last_undegraded":"2024-06-14T23:14:11.937428+0000","last_fullsized":"2024-06-14T23:14:11.937428+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":271,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370157+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:14:12.370157+0000","last_peered":"2024-06-14T23:14:12.370157+0000","last_clean":"2024-06-14T23:14:12.370157+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:14:12.370157+0000","last_undegraded":"2024-06-14T23:14:12.370157+0000","last_fullsized":"2024-06-14T23:14:12.370157+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":352,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370265+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:14:12.370265+0000","last_peered":"2024-06-14T23:14:12.370265+0000","last_clean":"2024-06-14T23:14:12.370265+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:14:12.370265+0000","last_undegraded":"2024-06-14T23:14:12.370265+0000","last_fullsized":"2024-06-14T23:14:12.370265+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":642,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-06-14T23:14:18.033022+0000","last_change":"2024-06-14T23:14:18.033116+0000","last_active":"2024-06-14T23:14:18.033022+0000","last_peered":"2024-06-14T23:14:18.033022+0000","last_clean":"2024-06-14T23:14:18.033022+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:14:18.033022+0000","last_undegraded":"2024-06-14T23:14:18.033022+0000","last_fullsized":"2024-06-14T23:14:18.033022+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:18.032997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:18.032997+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:57:13.313132+0000","scrub_duration":0.0019777449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":148,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.966307+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:14:11.966307+0000","last_peered":"2024-06-14T23:14:11.966307+0000","last_clean":"2024-06-14T23:14:11.966307+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:14:11.966307+0000","last_undegraded":"2024-06-14T23:14:11.966307+0000","last_fullsized":"2024-06-14T23:14:11.966307+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":320,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.959625+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:14:11.959625+0000","last_peered":"2024-06-14T23:14:11.959625+0000","last_clean":"2024-06-14T23:14:11.959625+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:14:11.959625+0000","last_undegraded":"2024-06-14T23:14:11.959625+0000","last_fullsized":"2024-06-14T23:14:11.959625+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":741,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.369931+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:14:12.369931+0000","last_peered":"2024-06-14T23:14:12.369931+0000","last_clean":"2024-06-14T23:14:12.369931+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:14:12.369931+0000","last_undegraded":"2024-06-14T23:14:12.369931+0000","last_fullsized":"2024-06-14T23:14:12.369931+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":337,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.940497+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:14:11.940497+0000","last_peered":"2024-06-14T23:14:11.940497+0000","last_clean":"2024-06-14T23:14:11.940497+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:14:11.940497+0000","last_undegraded":"2024-06-14T23:14:11.940497+0000","last_fullsized":"2024-06-14T23:14:11.940497+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":348,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936639+0000","last_change":"2024-06-14T23:11:23.427812+0000","last_active":"2024-06-14T23:14:11.936639+0000","last_peered":"2024-06-14T23:14:11.936639+0000","last_clean":"2024-06-14T23:14:11.936639+0000","last_became_active":"2024-06-14T23:11:23.427563+0000","last_became_peered":"2024-06-14T23:11:23.427563+0000","last_unstale":"2024-06-14T23:14:11.936639+0000","last_undegraded":"2024-06-14T23:14:11.936639+0000","last_fullsized":"2024-06-14T23:14:11.936639+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.941006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":709,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370171+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:14:12.370171+0000","last_peered":"2024-06-14T23:14:12.370171+0000","last_clean":"2024-06-14T23:14:12.370171+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:14:12.370171+0000","last_undegraded":"2024-06-14T23:14:12.370171+0000","last_fullsized":"2024-06-14T23:14:12.370171+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":367,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939517+0000","last_change":"2024-06-14T23:11:23.416264+0000","last_active":"2024-06-14T23:14:11.939517+0000","last_peered":"2024-06-14T23:14:11.939517+0000","last_clean":"2024-06-14T23:14:11.939517+0000","last_became_active":"2024-06-14T23:11:23.415946+0000","last_became_peered":"2024-06-14T23:11:23.415946+0000","last_unstale":"2024-06-14T23:14:11.939517+0000","last_undegraded":"2024-06-14T23:14:11.939517+0000","last_fullsized":"2024-06-14T23:14:11.939517+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:22.668569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":218,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.938763+0000","last_change":"2024-06-14T23:10:26.045363+0000","last_active":"2024-06-14T23:14:11.938763+0000","last_peered":"2024-06-14T23:14:11.938763+0000","last_clean":"2024-06-14T23:14:11.938763+0000","last_became_active":"2024-06-14T23:10:26.043844+0000","last_became_peered":"2024-06-14T23:10:26.043844+0000","last_unstale":"2024-06-14T23:14:11.938763+0000","last_undegraded":"2024-06-14T23:14:11.938763+0000","last_fullsized":"2024-06-14T23:14:11.938763+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:11:27.891184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":732,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.369865+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:14:12.369865+0000","last_peered":"2024-06-14T23:14:12.369865+0000","last_clean":"2024-06-14T23:14:12.369865+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:14:12.369865+0000","last_undegraded":"2024-06-14T23:14:12.369865+0000","last_fullsized":"2024-06-14T23:14:12.369865+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":327,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.938975+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:14:11.938975+0000","last_peered":"2024-06-14T23:14:11.938975+0000","last_clean":"2024-06-14T23:14:11.938975+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:14:11.938975+0000","last_undegraded":"2024-06-14T23:14:11.938975+0000","last_fullsized":"2024-06-14T23:14:11.938975+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":148,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937230+0000","last_change":"2024-06-14T23:10:26.056252+0000","last_active":"2024-06-14T23:14:11.937230+0000","last_peered":"2024-06-14T23:14:11.937230+0000","last_clean":"2024-06-14T23:14:11.937230+0000","last_became_active":"2024-06-14T23:10:26.046629+0000","last_became_peered":"2024-06-14T23:10:26.046629+0000","last_unstale":"2024-06-14T23:14:11.937230+0000","last_undegraded":"2024-06-14T23:14:11.937230+0000","last_fullsized":"2024-06-14T23:14:11.937230+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:51:19.277386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":633,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:14.052730+0000","last_change":"2024-06-14T23:14:14.052730+0000","last_active":"2024-06-14T23:14:14.052730+0000","last_peered":"2024-06-14T23:14:14.052730+0000","last_clean":"2024-06-14T23:14:14.052730+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:14:14.052730+0000","last_undegraded":"2024-06-14T23:14:14.052730+0000","last_fullsized":"2024-06-14T23:14:14.052730+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:14.052631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:14.052631+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:51:25.441426+0000","scrub_duration":0.0017584180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":147,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937138+0000","last_change":"2024-06-14T23:10:26.047111+0000","last_active":"2024-06-14T23:14:11.937138+0000","last_peered":"2024-06-14T23:14:11.937138+0000","last_clean":"2024-06-14T23:14:11.937138+0000","last_became_active":"2024-06-14T23:10:26.045540+0000","last_became_peered":"2024-06-14T23:10:26.045540+0000","last_unstale":"2024-06-14T23:14:11.937138+0000","last_undegraded":"2024-06-14T23:14:11.937138+0000","last_fullsized":"2024-06-14T23:14:11.937138+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:04.131400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":328,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370477+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:14:12.370477+0000","last_peered":"2024-06-14T23:14:12.370477+0000","last_clean":"2024-06-14T23:14:12.370477+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:14:12.370477+0000","last_undegraded":"2024-06-14T23:14:12.370477+0000","last_fullsized":"2024-06-14T23:14:12.370477+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":588,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.940179+0000","last_change":"2024-06-14T23:09:17.382466+0000","last_active":"2024-06-14T23:14:11.940179+0000","last_peered":"2024-06-14T23:14:11.940179+0000","last_clean":"2024-06-14T23:14:11.940179+0000","last_became_active":"2024-06-14T23:09:12.030205+0000","last_became_peered":"2024-06-14T23:09:12.030205+0000","last_unstale":"2024-06-14T23:14:11.940179+0000","last_undegraded":"2024-06-14T23:14:11.940179+0000","last_fullsized":"2024-06-14T23:14:11.940179+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:09:17.382389+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:33.121472+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":301,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.935406+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:14:11.935406+0000","last_peered":"2024-06-14T23:14:11.935406+0000","last_clean":"2024-06-14T23:14:11.935406+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:14:11.935406+0000","last_undegraded":"2024-06-14T23:14:11.935406+0000","last_fullsized":"2024-06-14T23:14:11.935406+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":245,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370356+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:14:12.370356+0000","last_peered":"2024-06-14T23:14:12.370356+0000","last_clean":"2024-06-14T23:14:12.370356+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:14:12.370356+0000","last_undegraded":"2024-06-14T23:14:12.370356+0000","last_fullsized":"2024-06-14T23:14:12.370356+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":629,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939082+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:14:11.939082+0000","last_peered":"2024-06-14T23:14:11.939082+0000","last_clean":"2024-06-14T23:14:11.939082+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:14:11.939082+0000","last_undegraded":"2024-06-14T23:14:11.939082+0000","last_fullsized":"2024-06-14T23:14:11.939082+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":337,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936579+0000","last_change":"2024-06-14T23:10:26.040201+0000","last_active":"2024-06-14T23:14:11.936579+0000","last_peered":"2024-06-14T23:14:11.936579+0000","last_clean":"2024-06-14T23:14:11.936579+0000","last_became_active":"2024-06-14T23:10:26.039865+0000","last_became_peered":"2024-06-14T23:10:26.039865+0000","last_unstale":"2024-06-14T23:14:11.936579+0000","last_undegraded":"2024-06-14T23:14:11.936579+0000","last_fullsized":"2024-06-14T23:14:11.936579+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:34.621202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":315,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.964066+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:14:11.964066+0000","last_peered":"2024-06-14T23:14:11.964066+0000","last_clean":"2024-06-14T23:14:11.964066+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:14:11.964066+0000","last_undegraded":"2024-06-14T23:14:11.964066+0000","last_fullsized":"2024-06-14T23:14:11.964066+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":758,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.963775+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:14:11.963775+0000","last_peered":"2024-06-14T23:14:11.963775+0000","last_clean":"2024-06-14T23:14:11.963775+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:14:11.963775+0000","last_undegraded":"2024-06-14T23:14:11.963775+0000","last_fullsized":"2024-06-14T23:14:11.963775+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":310,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370038+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:14:12.370038+0000","last_peered":"2024-06-14T23:14:12.370038+0000","last_clean":"2024-06-14T23:14:12.370038+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:14:12.370038+0000","last_undegraded":"2024-06-14T23:14:12.370038+0000","last_fullsized":"2024-06-14T23:14:12.370038+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":328,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.976191+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:14:11.976191+0000","last_peered":"2024-06-14T23:14:11.976191+0000","last_clean":"2024-06-14T23:14:11.976191+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:14:11.976191+0000","last_undegraded":"2024-06-14T23:14:11.976191+0000","last_fullsized":"2024-06-14T23:14:11.976191+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":783,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.939156+0000","last_change":"2024-06-14T23:11:23.418239+0000","last_active":"2024-06-14T23:14:11.939156+0000","last_peered":"2024-06-14T23:14:11.939156+0000","last_clean":"2024-06-14T23:14:11.939156+0000","last_became_active":"2024-06-14T23:11:23.417873+0000","last_became_peered":"2024-06-14T23:11:23.417873+0000","last_unstale":"2024-06-14T23:14:11.939156+0000","last_undegraded":"2024-06-14T23:14:11.939156+0000","last_fullsized":"2024-06-14T23:14:11.939156+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:04:50.740175+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":336,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.933960+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:14:11.933960+0000","last_peered":"2024-06-14T23:14:11.933960+0000","last_clean":"2024-06-14T23:14:11.933960+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:14:11.933960+0000","last_undegraded":"2024-06-14T23:14:11.933960+0000","last_fullsized":"2024-06-14T23:14:11.933960+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":333,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.966187+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:14:11.966187+0000","last_peered":"2024-06-14T23:14:11.966187+0000","last_clean":"2024-06-14T23:14:11.966187+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:14:11.966187+0000","last_undegraded":"2024-06-14T23:14:11.966187+0000","last_fullsized":"2024-06-14T23:14:11.966187+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":705,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370019+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:14:12.370019+0000","last_peered":"2024-06-14T23:14:12.370019+0000","last_clean":"2024-06-14T23:14:12.370019+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:14:12.370019+0000","last_undegraded":"2024-06-14T23:14:12.370019+0000","last_fullsized":"2024-06-14T23:14:12.370019+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":234,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.936364+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:14:11.936364+0000","last_peered":"2024-06-14T23:14:11.936364+0000","last_clean":"2024-06-14T23:14:11.936364+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:14:11.936364+0000","last_undegraded":"2024-06-14T23:14:11.936364+0000","last_fullsized":"2024-06-14T23:14:11.936364+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":214,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.966015+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:14:11.966015+0000","last_peered":"2024-06-14T23:14:11.966015+0000","last_clean":"2024-06-14T23:14:11.966015+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:14:11.966015+0000","last_undegraded":"2024-06-14T23:14:11.966015+0000","last_fullsized":"2024-06-14T23:14:11.966015+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":672,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.963560+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:14:11.963560+0000","last_peered":"2024-06-14T23:14:11.963560+0000","last_clean":"2024-06-14T23:14:11.963560+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:14:11.963560+0000","last_undegraded":"2024-06-14T23:14:11.963560+0000","last_fullsized":"2024-06-14T23:14:11.963560+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":52,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.938097+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:14:11.938097+0000","last_peered":"2024-06-14T23:14:11.938097+0000","last_clean":"2024-06-14T23:14:11.938097+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:14:11.938097+0000","last_undegraded":"2024-06-14T23:14:11.938097+0000","last_fullsized":"2024-06-14T23:14:11.938097+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":310,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.370386+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:14:12.370386+0000","last_peered":"2024-06-14T23:14:12.370386+0000","last_clean":"2024-06-14T23:14:12.370386+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:14:12.370386+0000","last_undegraded":"2024-06-14T23:14:12.370386+0000","last_fullsized":"2024-06-14T23:14:12.370386+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":669,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.369863+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:14:12.369863+0000","last_peered":"2024-06-14T23:14:12.369863+0000","last_clean":"2024-06-14T23:14:12.369863+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:14:12.369863+0000","last_undegraded":"2024-06-14T23:14:12.369863+0000","last_fullsized":"2024-06-14T23:14:12.369863+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":362,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.957159+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:14:11.957159+0000","last_peered":"2024-06-14T23:14:11.957159+0000","last_clean":"2024-06-14T23:14:11.957159+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:14:11.957159+0000","last_undegraded":"2024-06-14T23:14:11.957159+0000","last_fullsized":"2024-06-14T23:14:11.957159+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":52,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:11.937278+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:14:11.937278+0000","last_peered":"2024-06-14T23:14:11.937278+0000","last_clean":"2024-06-14T23:14:11.937278+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:14:11.937278+0000","last_undegraded":"2024-06-14T23:14:11.937278+0000","last_fullsized":"2024-06-14T23:14:11.937278+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":782,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-06-14T23:14:12.369901+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:14:12.369901+0000","last_peered":"2024-06-14T23:14:12.369901+0000","last_clean":"2024-06-14T23:14:12.369901+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:14:12.369901+0000","last_undegraded":"2024-06-14T23:14:12.369901+0000","last_fullsized":"2024-06-14T23:14:12.369901+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234483,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1515680,"kb_used_data":4692,"kb_used_omap":1,"kb_used_meta":1510974,"kb_avail":92229472,"statfs":{"total":95995035648,"available":94442979328,"internally_reserved":0,"allocated":4804608,"data_stored":2963067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1547237720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:34 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.0960000000000001,"15min":1.016},"min":{"1min":0.36599999999999999,"5min":0.24199999999999999,"15min":0.19800000000000001},"max":{"1min":5.8259999999999996,"5min":5.8259999999999996,"15min":6.9539999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.196,"15min":1.1930000000000001},"min":{"1min":0.30299999999999999,"5min":0.254,"15min":0.224},"max":{"1min":6.5149999999999997,"5min":6.5149999999999997,"15min":6.9550000000000001},"last":0.59399999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.179,"15min":1.2430000000000001},"min":{"1min":0.29599999999999999,"5min":0.26600000000000001,"15min":0.22600000000000001},"max":{"1min":6.0229999999999997,"5min":6.0229999999999997,"15min":12.010999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.248,"15min":1.294},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.22900000000000001},"max":{"1min":5.8490000000000002,"5min":5.8490000000000002,"15min":13.531000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.4970000000000001,"15min":1.4490000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":10.539,"5min":10.539,"15min":10.539},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.431,"15min":1.379},"min":{"1min":0.35899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":10.56,"5min":10.56,"15min":10.56},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Jun 14 23:13:34 2024","interfaces":[{"interface":"back","average":{"1min":2.7789999999999999,"5min":1.546,"15min":1.4410000000000001},"min":{"1min":0.40600000000000003,"5min":0.30599999999999999,"15min":0.28199999999999997},"max":{"1min":17.991,"5min":17.991,"15min":20.645},"last":0.64700000000000002},{"interface":"front","average":{"1min":2.5339999999999998,"5min":1.5640000000000001,"15min":1.532},"min":{"1min":0.443,"5min":0.29699999999999999,"15min":0.28299999999999997},"max":{"1min":17.824999999999999,"5min":17.824999999999999,"15min":22.042999999999999},"last":0.53600000000000003}]},{"osd":4,"last update":"Fri Jun 14 23:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.2749999999999999,"15min":1.107},"min":{"1min":0.42299999999999999,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":3.871,"5min":5.3319999999999999,"15min":11.342000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.776,"5min":1.452,"15min":1.1539999999999999},"min":{"1min":0.316,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":10.499000000000001,"5min":10.499000000000001,"15min":12.651},"last":0.60899999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:13:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.1200000000000001,"15min":1.1180000000000001},"min":{"1min":0.32800000000000001,"5min":0.23699999999999999,"15min":0.23100000000000001},"max":{"1min":4.0019999999999998,"5min":5.6849999999999996,"15min":6.9000000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.2849999999999999,"15min":1.135},"min":{"1min":0.45100000000000001,"5min":0.33100000000000002,"15min":0.24399999999999999},"max":{"1min":3.9020000000000001,"5min":5.3949999999999996,"15min":6.2809999999999997},"last":0.621}]},{"osd":6,"last update":"Fri Jun 14 23:14:03 2024","interfaces":[{"interface":"back","average":{"1min":2.2080000000000002,"5min":1.8979999999999999,"15min":1.508},"min":{"1min":0.39400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":7.1970000000000001,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.714,"15min":1.492},"min":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.32600000000000001},"max":{"1min":6.9569999999999999,"5min":10.087999999999999,"15min":10.087999999999999},"last":0.65800000000000003}]}]},{"osd":6,"up_from":272,"seq":1168231104590,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397712,"kb_used_data":4740,"kb_used_omap":7,"kb_used_meta":392952,"kb_avail":93347440,"statfs":{"total":95995035648,"available":95587778560,"internally_reserved":0,"allocated":4853760,"data_stored":3028160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":402383290},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.88700000000000001,"15min":1.5469999999999999},"min":{"1min":0.30599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.1850000000000001,"5min":4.6390000000000002,"15min":6.6440000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.996,"5min":0.95699999999999996,"15min":1.6020000000000001},"min":{"1min":0.39000000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.266,"5min":4.6150000000000002,"15min":6.6790000000000003},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0289999999999999,"15min":1.9930000000000001},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.806,"5min":5.4660000000000002,"15min":8.173},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95299999999999996,"15min":1.901},"min":{"1min":0.34499999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.234,"5min":4.6619999999999999,"15min":10.380000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Fri Jun 14 23:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.1970000000000001,"15min":1.143},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.476,"5min":7.476,"15min":7.476},"last":0.497},{"interface":"front","average":{"1min":1.782,"5min":1.304,"15min":1.224},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.7590000000000003,"5min":7.7590000000000003,"15min":7.7590000000000003},"last":0.54500000000000004}]},{"osd":3,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0269999999999999,"15min":1.6930000000000001},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.403,"5min":4.5819999999999999,"15min":6.7089999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.552,"5min":1.504,"15min":1.95},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.2300000000000004,"5min":8.4740000000000002,"15min":8.4740000000000002},"last":0.59699999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4079999999999999,"15min":1.444},"min":{"1min":0.24399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.3159999999999998,"5min":11.010999999999999,"15min":11.010999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.1200000000000001,"15min":1.252},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.2890000000000001,"5min":5.2309999999999999,"15min":5.2309999999999999},"last":0.71399999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.96699999999999997,"15min":1.3140000000000001},"min":{"1min":0.38700000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.1539999999999999,"5min":4.5579999999999998,"15min":7.7469999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.3839999999999999,"15min":1.5700000000000001},"min":{"1min":0.33300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.5700000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"last":0.64700000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.7549999999999999,"15min":1.7609999999999999},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":10.016999999999999,"5min":10.016999999999999,"15min":10.016999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":2.1619999999999999,"5min":1.587,"15min":1.591},"min":{"1min":0.35799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":8.1020000000000003,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":0.61099999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757759,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180108,"kb_used_data":4672,"kb_used_omap":1,"kb_used_meta":1175422,"kb_avail":92565044,"statfs":{"total":95995035648,"available":94786605056,"internally_reserved":0,"allocated":4784128,"data_stored":2958434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1203632472},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.74399999999999999,"15min":1.282},"min":{"1min":0.248,"5min":0.24099999999999999,"15min":0.20100000000000001},"max":{"1min":5.9199999999999999,"5min":5.9199999999999999,"15min":14.94},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.625,"15min":1.288},"min":{"1min":0.29999999999999999,"5min":0.26100000000000001,"15min":0.219},"max":{"1min":2.8900000000000001,"5min":4.181,"15min":9.3819999999999997},"last":0.51500000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.64800000000000002,"15min":0.91700000000000004},"min":{"1min":0.28499999999999998,"5min":0.245,"15min":0.23999999999999999},"max":{"1min":3.093,"5min":3.093,"15min":7.4370000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.71899999999999997,"15min":1.1160000000000001},"min":{"1min":0.318,"5min":0.27400000000000002,"15min":0.22900000000000001},"max":{"1min":5.952,"5min":5.952,"15min":9.5039999999999996},"last":0.60799999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.74099999999999999,"15min":0.67900000000000005},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.9800000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.272,"5min":0.76300000000000001,"15min":0.67800000000000005},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.0659999999999998,"5min":6.0659999999999998,"15min":6.0659999999999998},"last":0.63100000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.78100000000000003,"15min":1.363},"min":{"1min":0.29999999999999999,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":4.1600000000000001,"5min":4.3949999999999996,"15min":10.358000000000001},"last":0.622},{"interface":"front","average":{"1min":1.2290000000000001,"5min":0.93500000000000005,"15min":1.377},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.28899999999999998},"max":{"1min":6.0099999999999998,"5min":6.0099999999999998,"15min":14.914},"last":0.70999999999999996}]},{"osd":4,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.81899999999999995,"15min":0.90100000000000002},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":3.032,"5min":3.032,"15min":14.352},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.23,"5min":0.83299999999999996,"15min":1.0660000000000001},"min":{"1min":0.34999999999999998,"5min":0.30399999999999999,"15min":0.253},"max":{"1min":6.3630000000000004,"5min":6.3630000000000004,"15min":9.5980000000000008},"last":0.59399999999999997}]},{"osd":6,"last update":"Fri Jun 14 23:14:15 2024","interfaces":[{"interface":"back","average":{"1min":2.5659999999999998,"5min":1.163,"15min":1.417},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":30.934000000000001,"5min":30.934000000000001,"15min":30.934000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":2.5699999999999998,"5min":1.244,"15min":1.708},"min":{"1min":0.34100000000000003,"5min":0.27900000000000003,"15min":0.26100000000000001},"max":{"1min":31.059999999999999,"5min":31.059999999999999,"15min":31.059999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.4019999999999999,"15min":1.282},"min":{"1min":0.32500000000000001,"5min":0.29099999999999998,"15min":0.27900000000000003},"max":{"1min":6.8010000000000002,"5min":26.981999999999999,"15min":26.981999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.512,"15min":1.262},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.2629999999999999,"5min":23.326000000000001,"15min":23.326000000000001},"last":0.72299999999999998}]}]},{"osd":4,"up_from":30,"seq":128849019395,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1183828,"kb_used_data":4744,"kb_used_omap":1,"kb_used_meta":1179070,"kb_avail":92561324,"statfs":{"total":95995035648,"available":94782795776,"internally_reserved":0,"allocated":4857856,"data_stored":3027724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1207368024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.81000000000000005,"15min":1.1220000000000001},"min":{"1min":0.26000000000000001,"5min":0.223,"15min":0.17100000000000001},"max":{"1min":2.968,"5min":6.875,"15min":10.567},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.3080000000000001,"5min":0.873,"15min":1.1419999999999999},"min":{"1min":0.29199999999999998,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":6.7779999999999996,"5min":6.7779999999999996,"15min":8.3390000000000004},"last":0.54200000000000004}]},{"osd":1,"last update":"Fri Jun 14 23:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.85799999999999998,"15min":1.4330000000000001},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.23100000000000001},"max":{"1min":2.895,"5min":4.476,"15min":56.917000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.88500000000000001,"15min":1.4750000000000001},"min":{"1min":0.33600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":3.0419999999999998,"5min":6.9770000000000003,"15min":56.789999999999999},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Jun 14 23:13:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":0.88900000000000001,"15min":0.83799999999999997},"min":{"1min":0.29999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.468,"5min":3.468,"15min":3.468},"last":0.625},{"interface":"front","average":{"1min":1.1719999999999999,"5min":0.93899999999999995,"15min":0.90000000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.4209999999999998,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Jun 14 23:13:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.246,"15min":1.298},"min":{"1min":0.39900000000000002,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":3.173,"5min":44.389000000000003,"15min":44.389000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.325,"5min":1.452,"15min":1.492},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.27600000000000002},"max":{"1min":3.2109999999999999,"5min":46.101999999999997,"15min":46.101999999999997},"last":0.48899999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.89200000000000002,"15min":1.5389999999999999},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.20300000000000001},"max":{"1min":3.694,"5min":4.04,"15min":139.13200000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.94999999999999996,"15min":1.585},"min":{"1min":0.29099999999999998,"5min":0.28100000000000003,"15min":0.23300000000000001},"max":{"1min":5.077,"5min":6.9290000000000003,"15min":138.86000000000001},"last":0.40500000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:14:11 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":2.4049999999999998,"15min":1.5349999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.2999999999999998,"5min":122.88500000000001,"15min":122.88500000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":2.3620000000000001,"15min":1.887},"min":{"1min":0.30499999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.8200000000000003,"5min":122.21299999999999,"15min":122.21299999999999},"last":0.438}]},{"osd":7,"last update":"Fri Jun 14 23:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.194,"15min":1.075},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":2.5299999999999998,"5min":6.5910000000000002,"15min":6.5910000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.044,"5min":1.3160000000000001,"15min":1.1519999999999999},"min":{"1min":0.371,"5min":0.32400000000000001,"15min":0.27300000000000002},"max":{"1min":4.8659999999999997,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":0.47499999999999998}]}]},{"osd":3,"up_from":138,"seq":592705487093,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925188,"kb_used_data":4728,"kb_used_omap":4,"kb_used_meta":920443,"kb_avail":92819964,"statfs":{"total":95995035648,"available":95047643136,"internally_reserved":0,"allocated":4841472,"data_stored":3023177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":942534110},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.75700000000000001,"15min":1.2829999999999999},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.175,"5min":25.190000000000001,"15min":25.190000000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.044,"15min":1.329},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.17799999999999999},"max":{"1min":2.4020000000000001,"5min":25.652000000000001,"15min":25.652000000000001},"last":0.33100000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.218,"15min":1.393},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":2.431,"5min":25.754000000000001,"15min":25.754000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.274,"15min":1.5660000000000001},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.25},"max":{"1min":2.4649999999999999,"5min":27.992999999999999,"15min":27.992999999999999},"last":0.377}]},{"osd":2,"last update":"Fri Jun 14 23:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.877,"15min":0.89900000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.5230000000000001,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.72899999999999998,"15min":0.71099999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.8730000000000002,"5min":2.972,"15min":2.972},"last":0.35099999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":1.9019999999999999,"5min":1.6619999999999999,"15min":1.798},"min":{"1min":0.30399999999999999,"5min":0.28299999999999997,"15min":0.27900000000000003},"max":{"1min":17.018999999999998,"5min":25.094000000000001,"15min":73.263000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":2.1890000000000001,"5min":1.516,"15min":1.714},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.27300000000000002},"max":{"1min":17.074000000000002,"5min":25.5,"15min":73.5},"last":0.504}]},{"osd":5,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.3759999999999999,"15min":1.7270000000000001},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":3.4740000000000002,"5min":25.521000000000001,"15min":81.438999999999993},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.4950000000000001,"15min":1.7509999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.29299999999999998},"max":{"1min":2.3199999999999998,"5min":25.84,"15min":81.626999999999995},"last":0.46500000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:14:09 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.718,"15min":1.7130000000000001},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.34599999999999997},"max":{"1min":3.8010000000000002,"5min":25.869,"15min":25.869},"last":0.498},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.6100000000000001,"15min":1.5429999999999999},"min":{"1min":0.38400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.371,"5min":25.143000000000001,"15min":25.143000000000001},"last":0.55200000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":3.468,"5min":1.8080000000000001,"15min":1.571},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":44.887999999999998,"5min":44.887999999999998,"15min":44.887999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":3.6760000000000002,"5min":1.982,"15min":1.5309999999999999},"min":{"1min":0.39800000000000002,"5min":0.36099999999999999,"15min":0.32400000000000001},"max":{"1min":45.835000000000001,"5min":45.835000000000001,"15min":45.835000000000001},"last":0.48399999999999999}]}]},{"osd":2,"up_from":322,"seq":1382979469349,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":200904,"kb_used_data":4668,"kb_used_omap":10,"kb_used_meta":196213,"kb_avail":93544248,"statfs":{"total":95995035648,"available":95789309952,"internally_reserved":0,"allocated":4780032,"data_stored":2958399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":200922662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.41099999999999998,"15min":0.35099999999999998},"min":{"1min":0.19600000000000001,"5min":0.192,"15min":0.192},"max":{"1min":6.1319999999999997,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.45600000000000002,"15min":0.44400000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.34799999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.433,"15min":0.41299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.317},{"interface":"front","average":{"1min":0.63,"5min":0.47599999999999998,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.4089999999999998,"5min":3.4089999999999998,"15min":3.4089999999999998},"last":0.27700000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.67600000000000005,"15min":0.61899999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.7519999999999998,"5min":3.7519999999999998,"15min":3.7519999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.60799999999999998,"15min":0.54600000000000004},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.29499999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":0.96299999999999997,"15min":0.92300000000000004},"min":{"1min":0.34699999999999998,"5min":0.314,"15min":0.314},"max":{"1min":4.0250000000000004,"5min":4.0250000000000004,"15min":4.0250000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":0.76400000000000001,"15min":0.69799999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.3680000000000003,"5min":4.3680000000000003,"15min":4.3680000000000003},"last":0.44900000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.79300000000000004,"15min":0.78300000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.3519999999999999,"5min":3.3519999999999999,"15min":3.3519999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.86299999999999999,"15min":0.86799999999999999},"min":{"1min":0.377,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.4750000000000001,"5min":3.4750000000000001,"15min":3.4750000000000001},"last":0.48799999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":0.89300000000000002,"15min":0.81200000000000006},"min":{"1min":0.45500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":4.8099999999999996,"5min":4.8099999999999996,"15min":4.8099999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.98099999999999998,"15min":0.97499999999999998},"min":{"1min":0.35599999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.298,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":0.50800000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:13:29 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.94699999999999995,"15min":0.94299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":2.5089999999999999,"5min":3.008,"15min":3.008},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.8,"5min":1.24,"15min":1.147},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.47299999999999998}]}]},{"osd":1,"up_from":0,"seq":498216206637,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024292,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:14:26.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:26.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:26.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:26.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:26.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:26.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:26.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:25 smithi137 ceph-mon[90634]: from='client.36870 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:14:26.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:26.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:25 smithi086 ceph-mon[97913]: from='client.36870 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:14:26.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:25 smithi086 ceph-mon[102754]: from='client.36870 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:14:26.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:26.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:26.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:26.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:26.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:26.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:26.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:26.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:26.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:26.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:26.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-14T23:14:26.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:26.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:26.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:26.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:26.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:26.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:26.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:26.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:26.965 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:26.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:27.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:27.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:27.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:27.111 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:27.147 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:27.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:27.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:27.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:27.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:26 smithi086 ceph-mon[97913]: pgmap v1318: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:27.234 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:26 smithi086 ceph-mon[102754]: pgmap v1318: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:27.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:26 smithi137 ceph-mon[90634]: pgmap v1318: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:27.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:27.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:27.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:27.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:27.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:27.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:27.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:27.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:27.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:27.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:27.714 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:27.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:27.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:27.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:27.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:27.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:27.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:28.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:28.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:28.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:28.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:28.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:28.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:28.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:28.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:28.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:28.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:28.462 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:28.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:28.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:28.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:28.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:28.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:28 smithi086 ceph-mon[97913]: pgmap v1319: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:28.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:28.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:28.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:28 smithi086 ceph-mon[102754]: pgmap v1319: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:28.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:28.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:28.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:28.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:28.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:28.745 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:28 smithi137 ceph-mon[90634]: pgmap v1319: 103 pgs: 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:28.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:28.844 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:28.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:28.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:28.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:28.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:29.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:14:29.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:29.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:29.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:29.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:29.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:29.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:29.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:29.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:29.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:29.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:29.501 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:29.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:29.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:29.628 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:29.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:29.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:29.734 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_8' created 2024-06-14T23:14:29.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:29.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:29.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:29.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:29.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:29.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:29.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:29 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2949623450' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]: dispatch 2024-06-14T23:14:29.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:29 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]: dispatch 2024-06-14T23:14:30.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:30.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:30.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:29 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2949623450' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]: dispatch 2024-06-14T23:14:30.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:29 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]: dispatch 2024-06-14T23:14:30.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:29 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2949623450' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]: dispatch 2024-06-14T23:14:30.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:29 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]: dispatch 2024-06-14T23:14:30.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:30.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:30.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:30.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:30.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:30.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:30.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:30.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:30.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:30.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:30.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:30.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:30.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:30.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:30.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:30.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:30.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:30.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:30.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:30.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:30 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-06-14T23:14:30.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:30 smithi086 ceph-mon[97913]: osdmap e351: 8 total, 8 up, 6 in 2024-06-14T23:14:30.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:30 smithi086 ceph-mon[97913]: pgmap v1321: 119 pgs: 16 unknown, 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:30.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:30 smithi086 ceph-mon[97913]: osdmap e352: 8 total, 8 up, 6 in 2024-06-14T23:14:30.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:30 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-06-14T23:14:30.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:30 smithi086 ceph-mon[102754]: osdmap e351: 8 total, 8 up, 6 in 2024-06-14T23:14:30.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:30 smithi086 ceph-mon[102754]: pgmap v1321: 119 pgs: 16 unknown, 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:30.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:30 smithi086 ceph-mon[102754]: osdmap e352: 8 total, 8 up, 6 in 2024-06-14T23:14:30.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:30.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:30 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-06-14T23:14:30.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:30 smithi137 ceph-mon[90634]: osdmap e351: 8 total, 8 up, 6 in 2024-06-14T23:14:30.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:30 smithi137 ceph-mon[90634]: pgmap v1321: 119 pgs: 16 unknown, 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:30.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:30 smithi137 ceph-mon[90634]: osdmap e352: 8 total, 8 up, 6 in 2024-06-14T23:14:31.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:31.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:31.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:31.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:31.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:31.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:31.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:31.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:31.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:31.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:31.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:31.498 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:14:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:14:31.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:31.505 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:14:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:14:31.521 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:14:31.521 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:14:31.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:31.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:31.538 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:14:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:14:31.556 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:14:31.556 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:14:31.573 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:14:31.574 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:14:31.590 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:14:31.591 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:14:31.607 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:14:31.607 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:14:31.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:31.649 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:31.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:31.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:31.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:31.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:31.782 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:31 smithi086 ceph-mon[102754]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:14:31.782 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4208035287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:31.782 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:31 smithi086 ceph-mon[102754]: osdmap e353: 8 total, 8 up, 6 in 2024-06-14T23:14:31.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:31 smithi086 ceph-mon[97913]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:14:31.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4208035287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:31.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:31 smithi086 ceph-mon[97913]: osdmap e353: 8 total, 8 up, 6 in 2024-06-14T23:14:31.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:31.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:31.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:31.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:31.995 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:31 smithi137 ceph-mon[90634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:14:31.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4208035287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:31.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:31 smithi137 ceph-mon[90634]: osdmap e353: 8 total, 8 up, 6 in 2024-06-14T23:14:32.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:32.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:32.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:32.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:32.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:32.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:32.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:32.276 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:32.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:32.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:32.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:32.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:32.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:32.531 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:32.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:32.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:32.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:32.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:32.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:32.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:32.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:32.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:32.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:33.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:33.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:33.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:33.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:32 smithi086 ceph-mon[97913]: pgmap v1324: 119 pgs: 16 unknown, 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:33.059 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:32 smithi086 ceph-mon[102754]: pgmap v1324: 119 pgs: 16 unknown, 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:33.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:33.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:33.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:33.224 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:33.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:32 smithi137 ceph-mon[90634]: pgmap v1324: 119 pgs: 16 unknown, 103 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:33.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_8 rados --yes-i-really-mean-it || true 2024-06-14T23:14:33.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:33.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:33.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:33.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:33.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:33.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:33.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:33.537 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:33.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:33.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:33.661 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:33.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:33.758 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:14:33.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:33.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:33.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:33.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:33.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:33.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:34.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2547245218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:14:34.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:33 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:14:34.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2547245218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:14:34.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:33 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:14:34.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:34.031 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:34.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:34.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:34.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:34.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:34.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:34.243 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2547245218' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:14:34.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:33 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:14:34.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:34.283 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:34.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:34.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:34.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:34.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:34.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:34.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:34.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:34.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:34.658 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:34.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:34.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:14:34.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:34.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:34.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:34.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:34.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:34.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:34.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:34.937 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:14:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:14:34] "GET /metrics HTTP/1.1" 200 34259 "" "Prometheus/2.43.0" 2024-06-14T23:14:34.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:34 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:14:34.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:34 smithi086 ceph-mon[102754]: pgmap v1325: 119 pgs: 1 unknown, 118 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:34.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:34 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:14:34.937 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:34 smithi086 ceph-mon[102754]: osdmap e354: 8 total, 8 up, 6 in 2024-06-14T23:14:34.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:34 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:14:34.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:34 smithi086 ceph-mon[97913]: pgmap v1325: 119 pgs: 1 unknown, 118 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:34.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:34 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:14:34.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:34 smithi086 ceph-mon[97913]: osdmap e354: 8 total, 8 up, 6 in 2024-06-14T23:14:34.943 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:35.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:35.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:35.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:35.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:35.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:35.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:34 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:14:35.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:34 smithi137 ceph-mon[90634]: pgmap v1325: 119 pgs: 1 unknown, 118 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:35.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:34 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:14:35.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:34 smithi137 ceph-mon[90634]: osdmap e354: 8 total, 8 up, 6 in 2024-06-14T23:14:35.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:35.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:35.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:35.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:35.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:35.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:35.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:35.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:35.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:35.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:35.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:35.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:35.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:35.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:35.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:35.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:35.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:36.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:36.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:36.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:36.165 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:36.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:36.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:36.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:36.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:36.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:36.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:36.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:36.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:36.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:36.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:36.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:36.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:36.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:36.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:36.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:36.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:36.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:36.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:36.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:37.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:37.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:37.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:37.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:36 smithi086 ceph-mon[97913]: pgmap v1327: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:37.059 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:36 smithi086 ceph-mon[102754]: pgmap v1327: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:37.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:37.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:37.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:37.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:36 smithi137 ceph-mon[90634]: pgmap v1327: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:37.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:37.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:37.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:37.346 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:14:37.348 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:14:37.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:37.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:37.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:37.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:37.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:37.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:37.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:37.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:14:37.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:37.673 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:37.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:37.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:37.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:37.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:37.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:37.924 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:37.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:37.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:38.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:38.050 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:38.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:38.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:38.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:38.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:38.297 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:38.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:38.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:38.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:38.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:38.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:38.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:38.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:38.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:38.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:38.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:38.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:38.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:38.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:38.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:38.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:38.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:38.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:38.957 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:38 smithi086 ceph-mon[97913]: from='client.36894 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:14:38.957 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:38 smithi086 ceph-mon[97913]: pgmap v1328: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:38.958 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:38 smithi086 ceph-mon[102754]: from='client.36894 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:14:38.958 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:38 smithi086 ceph-mon[102754]: pgmap v1328: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:39.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:39.053 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:39.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:39.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:39.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:39.176 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:39.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:38 smithi137 ceph-mon[90634]: from='client.36894 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:14:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:38 smithi137 ceph-mon[90634]: pgmap v1328: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:39.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:39.282 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:14:39.282 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:14:39.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:39.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:39.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:39.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:39.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:39.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:39.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:39.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:39.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:39.679 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:39.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:39.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:39.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:39.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:39.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:39.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:39.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:40.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:40.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:39 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3097296705' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:14:40.045 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:39 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3097296705' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:14:40.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:40.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:40.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-14T23:14:40.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:40.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:40.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:40.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:39 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3097296705' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:14:40.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:40.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:40.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:40.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:40.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:40.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:40.462 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:40.485 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1327,"stamp":"2024-06-14T23:14:35.753205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5048,"ondisk_log_size":5048,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":346,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5403676,"kb_used_data":28472,"kb_used_omap":27,"kb_used_meta":5375076,"kb_avail":557067236,"statfs":{"total":575970213888,"available":570436849664,"internally_reserved":0,"allocated":29155328,"data_stored":18119841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":5504078278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001603"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":734,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062830+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:14:32.062830+0000","last_peered":"2024-06-14T23:14:32.062830+0000","last_clean":"2024-06-14T23:14:32.062830+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:14:32.062830+0000","last_undegraded":"2024-06-14T23:14:32.062830+0000","last_fullsized":"2024-06-14T23:14:32.062830+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":170,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737980+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:14:30.737980+0000","last_peered":"2024-06-14T23:14:30.737980+0000","last_clean":"2024-06-14T23:14:30.737980+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:14:30.737980+0000","last_undegraded":"2024-06-14T23:14:30.737980+0000","last_fullsized":"2024-06-14T23:14:30.737980+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":340,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735185+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:14:30.735185+0000","last_peered":"2024-06-14T23:14:30.735185+0000","last_clean":"2024-06-14T23:14:30.735185+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:14:30.735185+0000","last_undegraded":"2024-06-14T23:14:30.735185+0000","last_fullsized":"2024-06-14T23:14:30.735185+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":246,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735512+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:14:30.735512+0000","last_peered":"2024-06-14T23:14:30.735512+0000","last_clean":"2024-06-14T23:14:30.735512+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:14:30.735512+0000","last_undegraded":"2024-06-14T23:14:30.735512+0000","last_fullsized":"2024-06-14T23:14:30.735512+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":136,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735698+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:14:30.735698+0000","last_peered":"2024-06-14T23:14:30.735698+0000","last_clean":"2024-06-14T23:14:30.735698+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:14:30.735698+0000","last_undegraded":"2024-06-14T23:14:30.735698+0000","last_fullsized":"2024-06-14T23:14:30.735698+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":797,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.740864+0000","last_change":"2024-06-14T23:14:16.297931+0000","last_active":"2024-06-14T23:14:30.740864+0000","last_peered":"2024-06-14T23:14:30.740864+0000","last_clean":"2024-06-14T23:14:30.740864+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:14:30.740864+0000","last_undegraded":"2024-06-14T23:14:30.740864+0000","last_fullsized":"2024-06-14T23:14:30.740864+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:14:16.297860+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:14:16.297860+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:46.938506+0000","scrub_duration":0.0016077540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":316,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737266+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:14:30.737266+0000","last_peered":"2024-06-14T23:14:30.737266+0000","last_clean":"2024-06-14T23:14:30.737266+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:14:30.737266+0000","last_undegraded":"2024-06-14T23:14:30.737266+0000","last_fullsized":"2024-06-14T23:14:30.737266+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":357,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735666+0000","last_change":"2024-06-14T23:11:23.413009+0000","last_active":"2024-06-14T23:14:30.735666+0000","last_peered":"2024-06-14T23:14:30.735666+0000","last_clean":"2024-06-14T23:14:30.735666+0000","last_became_active":"2024-06-14T23:11:23.412491+0000","last_became_peered":"2024-06-14T23:11:23.412491+0000","last_unstale":"2024-06-14T23:14:30.735666+0000","last_undegraded":"2024-06-14T23:14:30.735666+0000","last_fullsized":"2024-06-14T23:14:30.735666+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:51:57.263547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":505,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.063045+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:14:32.063045+0000","last_peered":"2024-06-14T23:14:32.063045+0000","last_clean":"2024-06-14T23:14:32.063045+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:14:32.063045+0000","last_undegraded":"2024-06-14T23:14:32.063045+0000","last_fullsized":"2024-06-14T23:14:32.063045+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":330,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.741097+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:14:30.741097+0000","last_peered":"2024-06-14T23:14:30.741097+0000","last_clean":"2024-06-14T23:14:30.741097+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:14:30.741097+0000","last_undegraded":"2024-06-14T23:14:30.741097+0000","last_fullsized":"2024-06-14T23:14:30.741097+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":357,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738078+0000","last_change":"2024-06-14T23:11:23.427771+0000","last_active":"2024-06-14T23:14:30.738078+0000","last_peered":"2024-06-14T23:14:30.738078+0000","last_clean":"2024-06-14T23:14:30.738078+0000","last_became_active":"2024-06-14T23:11:23.427481+0000","last_became_peered":"2024-06-14T23:11:23.427481+0000","last_unstale":"2024-06-14T23:14:30.738078+0000","last_undegraded":"2024-06-14T23:14:30.738078+0000","last_fullsized":"2024-06-14T23:14:30.738078+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:01:04.162256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":146,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737939+0000","last_change":"2024-06-14T23:10:26.053647+0000","last_active":"2024-06-14T23:14:30.737939+0000","last_peered":"2024-06-14T23:14:30.737939+0000","last_clean":"2024-06-14T23:14:30.737939+0000","last_became_active":"2024-06-14T23:10:26.053318+0000","last_became_peered":"2024-06-14T23:10:26.053318+0000","last_unstale":"2024-06-14T23:14:30.737939+0000","last_undegraded":"2024-06-14T23:14:30.737939+0000","last_fullsized":"2024-06-14T23:14:30.737939+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:32.647107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":146,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735553+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:14:30.735553+0000","last_peered":"2024-06-14T23:14:30.735553+0000","last_clean":"2024-06-14T23:14:30.735553+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:14:30.735553+0000","last_undegraded":"2024-06-14T23:14:30.735553+0000","last_fullsized":"2024-06-14T23:14:30.735553+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":656,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.732712+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:14:30.732712+0000","last_peered":"2024-06-14T23:14:30.732712+0000","last_clean":"2024-06-14T23:14:30.732712+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:14:30.732712+0000","last_undegraded":"2024-06-14T23:14:30.732712+0000","last_fullsized":"2024-06-14T23:14:30.732712+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":379,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735752+0000","last_change":"2024-06-14T23:11:23.413241+0000","last_active":"2024-06-14T23:14:30.735752+0000","last_peered":"2024-06-14T23:14:30.735752+0000","last_clean":"2024-06-14T23:14:30.735752+0000","last_became_active":"2024-06-14T23:11:23.412936+0000","last_became_peered":"2024-06-14T23:11:23.412936+0000","last_unstale":"2024-06-14T23:14:30.735752+0000","last_undegraded":"2024-06-14T23:14:30.735752+0000","last_fullsized":"2024-06-14T23:14:30.735752+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:45:36.581716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":165,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737145+0000","last_change":"2024-06-14T23:10:26.038549+0000","last_active":"2024-06-14T23:14:30.737145+0000","last_peered":"2024-06-14T23:14:30.737145+0000","last_clean":"2024-06-14T23:14:30.737145+0000","last_became_active":"2024-06-14T23:10:26.038260+0000","last_became_peered":"2024-06-14T23:10:26.038260+0000","last_unstale":"2024-06-14T23:14:30.737145+0000","last_undegraded":"2024-06-14T23:14:30.737145+0000","last_fullsized":"2024-06-14T23:14:30.737145+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:59:44.020221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":759,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737974+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:14:30.737974+0000","last_peered":"2024-06-14T23:14:30.737974+0000","last_clean":"2024-06-14T23:14:30.737974+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:14:30.737974+0000","last_undegraded":"2024-06-14T23:14:30.737974+0000","last_fullsized":"2024-06-14T23:14:30.737974+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":151,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062493+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:14:32.062493+0000","last_peered":"2024-06-14T23:14:32.062493+0000","last_clean":"2024-06-14T23:14:32.062493+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:14:32.062493+0000","last_undegraded":"2024-06-14T23:14:32.062493+0000","last_fullsized":"2024-06-14T23:14:32.062493+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":214,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.063044+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:14:32.063044+0000","last_peered":"2024-06-14T23:14:32.063044+0000","last_clean":"2024-06-14T23:14:32.063044+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:14:32.063044+0000","last_undegraded":"2024-06-14T23:14:32.063044+0000","last_fullsized":"2024-06-14T23:14:32.063044+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":707,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.739403+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:14:30.739403+0000","last_peered":"2024-06-14T23:14:30.739403+0000","last_clean":"2024-06-14T23:14:30.739403+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:14:30.739403+0000","last_undegraded":"2024-06-14T23:14:30.739403+0000","last_fullsized":"2024-06-14T23:14:30.739403+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":74,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735317+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:14:30.735317+0000","last_peered":"2024-06-14T23:14:30.735317+0000","last_clean":"2024-06-14T23:14:30.735317+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:14:30.735317+0000","last_undegraded":"2024-06-14T23:14:30.735317+0000","last_fullsized":"2024-06-14T23:14:30.735317+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":304,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.733627+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:14:30.733627+0000","last_peered":"2024-06-14T23:14:30.733627+0000","last_clean":"2024-06-14T23:14:30.733627+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:14:30.733627+0000","last_undegraded":"2024-06-14T23:14:30.733627+0000","last_fullsized":"2024-06-14T23:14:30.733627+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":659,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735325+0000","last_change":"2024-06-14T23:10:29.062659+0000","last_active":"2024-06-14T23:14:30.735325+0000","last_peered":"2024-06-14T23:14:30.735325+0000","last_clean":"2024-06-14T23:14:30.735325+0000","last_became_active":"2024-06-14T23:10:29.062256+0000","last_became_peered":"2024-06-14T23:10:29.062256+0000","last_unstale":"2024-06-14T23:14:30.735325+0000","last_undegraded":"2024-06-14T23:14:30.735325+0000","last_fullsized":"2024-06-14T23:14:30.735325+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:38.814157+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":337,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062390+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:14:32.062390+0000","last_peered":"2024-06-14T23:14:32.062390+0000","last_clean":"2024-06-14T23:14:32.062390+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:14:32.062390+0000","last_undegraded":"2024-06-14T23:14:32.062390+0000","last_fullsized":"2024-06-14T23:14:32.062390+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":200,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738080+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:14:30.738080+0000","last_peered":"2024-06-14T23:14:30.738080+0000","last_clean":"2024-06-14T23:14:30.738080+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:14:30.738080+0000","last_undegraded":"2024-06-14T23:14:30.738080+0000","last_fullsized":"2024-06-14T23:14:30.738080+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":659,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737560+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:14:30.737560+0000","last_peered":"2024-06-14T23:14:30.737560+0000","last_clean":"2024-06-14T23:14:30.737560+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:14:30.737560+0000","last_undegraded":"2024-06-14T23:14:30.737560+0000","last_fullsized":"2024-06-14T23:14:30.737560+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":220,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062406+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:14:32.062406+0000","last_peered":"2024-06-14T23:14:32.062406+0000","last_clean":"2024-06-14T23:14:32.062406+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:14:32.062406+0000","last_undegraded":"2024-06-14T23:14:32.062406+0000","last_fullsized":"2024-06-14T23:14:32.062406+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":324,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.739342+0000","last_change":"2024-06-14T23:10:26.053008+0000","last_active":"2024-06-14T23:14:30.739342+0000","last_peered":"2024-06-14T23:14:30.739342+0000","last_clean":"2024-06-14T23:14:30.739342+0000","last_became_active":"2024-06-14T23:10:26.052701+0000","last_became_peered":"2024-06-14T23:10:26.052701+0000","last_unstale":"2024-06-14T23:14:30.739342+0000","last_undegraded":"2024-06-14T23:14:30.739342+0000","last_fullsized":"2024-06-14T23:14:30.739342+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:15:03.929062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":696,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735647+0000","last_change":"2024-06-14T23:14:17.053346+0000","last_active":"2024-06-14T23:14:30.735647+0000","last_peered":"2024-06-14T23:14:30.735647+0000","last_clean":"2024-06-14T23:14:30.735647+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:14:30.735647+0000","last_undegraded":"2024-06-14T23:14:30.735647+0000","last_fullsized":"2024-06-14T23:14:30.735647+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:17.053258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:17.053258+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:03.297247+0000","scrub_duration":0.0014699820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":238,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062981+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:14:32.062981+0000","last_peered":"2024-06-14T23:14:32.062981+0000","last_clean":"2024-06-14T23:14:32.062981+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:14:32.062981+0000","last_undegraded":"2024-06-14T23:14:32.062981+0000","last_fullsized":"2024-06-14T23:14:32.062981+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":366,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.732820+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:14:30.732820+0000","last_peered":"2024-06-14T23:14:30.732820+0000","last_clean":"2024-06-14T23:14:30.732820+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:14:30.732820+0000","last_undegraded":"2024-06-14T23:14:30.732820+0000","last_fullsized":"2024-06-14T23:14:30.732820+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":529,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737650+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:14:30.737650+0000","last_peered":"2024-06-14T23:14:30.737650+0000","last_clean":"2024-06-14T23:14:30.737650+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:14:30.737650+0000","last_undegraded":"2024-06-14T23:14:30.737650+0000","last_fullsized":"2024-06-14T23:14:30.737650+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":336,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.736283+0000","last_change":"2024-06-14T23:07:58.011638+0000","last_active":"2024-06-14T23:14:30.736283+0000","last_peered":"2024-06-14T23:14:30.736283+0000","last_clean":"2024-06-14T23:14:30.736283+0000","last_became_active":"2024-06-14T23:07:58.011126+0000","last_became_peered":"2024-06-14T23:07:58.011126+0000","last_unstale":"2024-06-14T23:14:30.736283+0000","last_undegraded":"2024-06-14T23:14:30.736283+0000","last_fullsized":"2024-06-14T23:14:30.736283+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:45:21.410365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":340,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738150+0000","last_change":"2024-06-14T23:10:26.047451+0000","last_active":"2024-06-14T23:14:30.738150+0000","last_peered":"2024-06-14T23:14:30.738150+0000","last_clean":"2024-06-14T23:14:30.738150+0000","last_became_active":"2024-06-14T23:10:26.046781+0000","last_became_peered":"2024-06-14T23:10:26.046781+0000","last_unstale":"2024-06-14T23:14:30.738150+0000","last_undegraded":"2024-06-14T23:14:30.738150+0000","last_fullsized":"2024-06-14T23:14:30.738150+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:07:15.892864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":581,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.736692+0000","last_change":"2024-06-14T23:14:12.889878+0000","last_active":"2024-06-14T23:14:30.736692+0000","last_peered":"2024-06-14T23:14:30.736692+0000","last_clean":"2024-06-14T23:14:30.736692+0000","last_became_active":"2024-06-14T23:10:29.061386+0000","last_became_peered":"2024-06-14T23:10:29.061386+0000","last_unstale":"2024-06-14T23:14:30.736692+0000","last_undegraded":"2024-06-14T23:14:30.736692+0000","last_fullsized":"2024-06-14T23:14:30.736692+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:12.889819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:12.889819+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:04:46.976666+0000","scrub_duration":0.001439832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:31.735592+0000","last_change":"2024-06-14T23:14:30.741601+0000","last_active":"2024-06-14T23:14:31.735592+0000","last_peered":"2024-06-14T23:14:31.735592+0000","last_clean":"2024-06-14T23:14:31.735592+0000","last_became_active":"2024-06-14T23:14:30.741403+0000","last_became_peered":"2024-06-14T23:14:30.741403+0000","last_unstale":"2024-06-14T23:14:31.735592+0000","last_undegraded":"2024-06-14T23:14:31.735592+0000","last_fullsized":"2024-06-14T23:14:31.735592+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:32.635229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":313,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735654+0000","last_change":"2024-06-14T23:07:58.017211+0000","last_active":"2024-06-14T23:14:30.735654+0000","last_peered":"2024-06-14T23:14:30.735654+0000","last_clean":"2024-06-14T23:14:30.735654+0000","last_became_active":"2024-06-14T23:07:58.016945+0000","last_became_peered":"2024-06-14T23:07:58.016945+0000","last_unstale":"2024-06-14T23:14:30.735654+0000","last_undegraded":"2024-06-14T23:14:30.735654+0000","last_fullsized":"2024-06-14T23:14:30.735654+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:10:43.530160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":325,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.733373+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:14:30.733373+0000","last_peered":"2024-06-14T23:14:30.733373+0000","last_clean":"2024-06-14T23:14:30.733373+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:14:30.733373+0000","last_undegraded":"2024-06-14T23:14:30.733373+0000","last_fullsized":"2024-06-14T23:14:30.733373+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":111,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.736549+0000","last_change":"2024-06-14T23:10:26.058423+0000","last_active":"2024-06-14T23:14:30.736549+0000","last_peered":"2024-06-14T23:14:30.736549+0000","last_clean":"2024-06-14T23:14:30.736549+0000","last_became_active":"2024-06-14T23:10:26.050244+0000","last_became_peered":"2024-06-14T23:10:26.050244+0000","last_unstale":"2024-06-14T23:14:30.736549+0000","last_undegraded":"2024-06-14T23:14:30.736549+0000","last_fullsized":"2024-06-14T23:14:30.736549+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:52.060325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":679,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062678+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:14:32.062678+0000","last_peered":"2024-06-14T23:14:32.062678+0000","last_clean":"2024-06-14T23:14:32.062678+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:14:32.062678+0000","last_undegraded":"2024-06-14T23:14:32.062678+0000","last_fullsized":"2024-06-14T23:14:32.062678+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.745525+0000","last_change":"2024-06-14T23:14:30.745071+0000","last_active":"2024-06-14T23:14:30.745525+0000","last_peered":"2024-06-14T23:14:30.745525+0000","last_clean":"2024-06-14T23:14:30.745525+0000","last_became_active":"2024-06-14T23:14:30.744589+0000","last_became_peered":"2024-06-14T23:14:30.744589+0000","last_unstale":"2024-06-14T23:14:30.745525+0000","last_undegraded":"2024-06-14T23:14:30.745525+0000","last_fullsized":"2024-06-14T23:14:30.745525+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:20:16.097928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":268,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062392+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:14:32.062392+0000","last_peered":"2024-06-14T23:14:32.062392+0000","last_clean":"2024-06-14T23:14:32.062392+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:14:32.062392+0000","last_undegraded":"2024-06-14T23:14:32.062392+0000","last_fullsized":"2024-06-14T23:14:32.062392+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":316,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.736830+0000","last_change":"2024-06-14T23:10:26.051450+0000","last_active":"2024-06-14T23:14:30.736830+0000","last_peered":"2024-06-14T23:14:30.736830+0000","last_clean":"2024-06-14T23:14:30.736830+0000","last_became_active":"2024-06-14T23:10:26.050830+0000","last_became_peered":"2024-06-14T23:10:26.050830+0000","last_unstale":"2024-06-14T23:14:30.736830+0000","last_undegraded":"2024-06-14T23:14:30.736830+0000","last_fullsized":"2024-06-14T23:14:30.736830+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:16:16.175014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":67,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737550+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:14:30.737550+0000","last_peered":"2024-06-14T23:14:30.737550+0000","last_clean":"2024-06-14T23:14:30.737550+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:14:30.737550+0000","last_undegraded":"2024-06-14T23:14:30.737550+0000","last_fullsized":"2024-06-14T23:14:30.737550+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":767,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737223+0000","last_change":"2024-06-14T23:11:23.401974+0000","last_active":"2024-06-14T23:14:30.737223+0000","last_peered":"2024-06-14T23:14:30.737223+0000","last_clean":"2024-06-14T23:14:30.737223+0000","last_became_active":"2024-06-14T23:11:23.401625+0000","last_became_peered":"2024-06-14T23:11:23.401625+0000","last_unstale":"2024-06-14T23:14:30.737223+0000","last_undegraded":"2024-06-14T23:14:30.737223+0000","last_fullsized":"2024-06-14T23:14:30.737223+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:54.214300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:54.214300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:05:54.552362+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:31.735732+0000","last_change":"2024-06-14T23:14:30.743581+0000","last_active":"2024-06-14T23:14:31.735732+0000","last_peered":"2024-06-14T23:14:31.735732+0000","last_clean":"2024-06-14T23:14:31.735732+0000","last_became_active":"2024-06-14T23:14:30.743244+0000","last_became_peered":"2024-06-14T23:14:30.743244+0000","last_unstale":"2024-06-14T23:14:31.735732+0000","last_undegraded":"2024-06-14T23:14:31.735732+0000","last_fullsized":"2024-06-14T23:14:31.735732+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:11:35.334274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":164,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.739605+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:14:30.739605+0000","last_peered":"2024-06-14T23:14:30.739605+0000","last_clean":"2024-06-14T23:14:30.739605+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:14:30.739605+0000","last_undegraded":"2024-06-14T23:14:30.739605+0000","last_fullsized":"2024-06-14T23:14:30.739605+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":338,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737651+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:14:30.737651+0000","last_peered":"2024-06-14T23:14:30.737651+0000","last_clean":"2024-06-14T23:14:30.737651+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:14:30.737651+0000","last_undegraded":"2024-06-14T23:14:30.737651+0000","last_fullsized":"2024-06-14T23:14:30.737651+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":290,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735649+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:14:30.735649+0000","last_peered":"2024-06-14T23:14:30.735649+0000","last_clean":"2024-06-14T23:14:30.735649+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:14:30.735649+0000","last_undegraded":"2024-06-14T23:14:30.735649+0000","last_fullsized":"2024-06-14T23:14:30.735649+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":652,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.733570+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:14:30.733570+0000","last_peered":"2024-06-14T23:14:30.733570+0000","last_clean":"2024-06-14T23:14:30.733570+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:14:30.733570+0000","last_undegraded":"2024-06-14T23:14:30.733570+0000","last_fullsized":"2024-06-14T23:14:30.733570+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:31.004751+0000","last_change":"2024-06-14T23:14:31.004195+0000","last_active":"2024-06-14T23:14:31.004751+0000","last_peered":"2024-06-14T23:14:31.004751+0000","last_clean":"2024-06-14T23:14:31.004751+0000","last_became_active":"2024-06-14T23:14:31.003779+0000","last_became_peered":"2024-06-14T23:14:31.003779+0000","last_unstale":"2024-06-14T23:14:31.004751+0000","last_undegraded":"2024-06-14T23:14:31.004751+0000","last_fullsized":"2024-06-14T23:14:31.004751+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:42:29.429100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":129,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738737+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:14:30.738737+0000","last_peered":"2024-06-14T23:14:30.738737+0000","last_clean":"2024-06-14T23:14:30.738737+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:14:30.738737+0000","last_undegraded":"2024-06-14T23:14:30.738737+0000","last_fullsized":"2024-06-14T23:14:30.738737+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735457+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:14:30.735457+0000","last_peered":"2024-06-14T23:14:30.735457+0000","last_clean":"2024-06-14T23:14:30.735457+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:14:30.735457+0000","last_undegraded":"2024-06-14T23:14:30.735457+0000","last_fullsized":"2024-06-14T23:14:30.735457+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":368,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737352+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:14:30.737352+0000","last_peered":"2024-06-14T23:14:30.737352+0000","last_clean":"2024-06-14T23:14:30.737352+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:14:30.737352+0000","last_undegraded":"2024-06-14T23:14:30.737352+0000","last_fullsized":"2024-06-14T23:14:30.737352+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":632,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.733458+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:14:30.733458+0000","last_peered":"2024-06-14T23:14:30.733458+0000","last_clean":"2024-06-14T23:14:30.733458+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:14:30.733458+0000","last_undegraded":"2024-06-14T23:14:30.733458+0000","last_fullsized":"2024-06-14T23:14:30.733458+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:31.005603+0000","last_change":"2024-06-14T23:14:31.005125+0000","last_active":"2024-06-14T23:14:31.005603+0000","last_peered":"2024-06-14T23:14:31.005603+0000","last_clean":"2024-06-14T23:14:31.005603+0000","last_became_active":"2024-06-14T23:14:31.004804+0000","last_became_peered":"2024-06-14T23:14:31.004804+0000","last_unstale":"2024-06-14T23:14:31.005603+0000","last_undegraded":"2024-06-14T23:14:31.005603+0000","last_fullsized":"2024-06-14T23:14:31.005603+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:59:19.674697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":305,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735393+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:14:30.735393+0000","last_peered":"2024-06-14T23:14:30.735393+0000","last_clean":"2024-06-14T23:14:30.735393+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:14:30.735393+0000","last_undegraded":"2024-06-14T23:14:30.735393+0000","last_fullsized":"2024-06-14T23:14:30.735393+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":351,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738293+0000","last_change":"2024-06-14T23:10:26.049033+0000","last_active":"2024-06-14T23:14:30.738293+0000","last_peered":"2024-06-14T23:14:30.738293+0000","last_clean":"2024-06-14T23:14:30.738293+0000","last_became_active":"2024-06-14T23:10:26.047775+0000","last_became_peered":"2024-06-14T23:10:26.047775+0000","last_unstale":"2024-06-14T23:14:30.738293+0000","last_undegraded":"2024-06-14T23:14:30.738293+0000","last_fullsized":"2024-06-14T23:14:30.738293+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:15:52.051858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":345,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738888+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:14:30.738888+0000","last_peered":"2024-06-14T23:14:30.738888+0000","last_clean":"2024-06-14T23:14:30.738888+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:14:30.738888+0000","last_undegraded":"2024-06-14T23:14:30.738888+0000","last_fullsized":"2024-06-14T23:14:30.738888+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":746,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.739428+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:14:30.739428+0000","last_peered":"2024-06-14T23:14:30.739428+0000","last_clean":"2024-06-14T23:14:30.739428+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:14:30.739428+0000","last_undegraded":"2024-06-14T23:14:30.739428+0000","last_fullsized":"2024-06-14T23:14:30.739428+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.743877+0000","last_change":"2024-06-14T23:14:30.743278+0000","last_active":"2024-06-14T23:14:30.743877+0000","last_peered":"2024-06-14T23:14:30.743877+0000","last_clean":"2024-06-14T23:14:30.743877+0000","last_became_active":"2024-06-14T23:14:30.743000+0000","last_became_peered":"2024-06-14T23:14:30.743000+0000","last_unstale":"2024-06-14T23:14:30.743877+0000","last_undegraded":"2024-06-14T23:14:30.743877+0000","last_fullsized":"2024-06-14T23:14:30.743877+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:22:35.618792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":313,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737744+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:14:30.737744+0000","last_peered":"2024-06-14T23:14:30.737744+0000","last_clean":"2024-06-14T23:14:30.737744+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:14:30.737744+0000","last_undegraded":"2024-06-14T23:14:30.737744+0000","last_fullsized":"2024-06-14T23:14:30.737744+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":350,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735154+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:14:30.735154+0000","last_peered":"2024-06-14T23:14:30.735154+0000","last_clean":"2024-06-14T23:14:30.735154+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:14:30.735154+0000","last_undegraded":"2024-06-14T23:14:30.735154+0000","last_fullsized":"2024-06-14T23:14:30.735154+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735614+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:14:30.735614+0000","last_peered":"2024-06-14T23:14:30.735614+0000","last_clean":"2024-06-14T23:14:30.735614+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:14:30.735614+0000","last_undegraded":"2024-06-14T23:14:30.735614+0000","last_fullsized":"2024-06-14T23:14:30.735614+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":640,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735412+0000","last_change":"2024-06-14T23:11:23.413224+0000","last_active":"2024-06-14T23:14:30.735412+0000","last_peered":"2024-06-14T23:14:30.735412+0000","last_clean":"2024-06-14T23:14:30.735412+0000","last_became_active":"2024-06-14T23:11:23.412680+0000","last_became_peered":"2024-06-14T23:11:23.412680+0000","last_unstale":"2024-06-14T23:14:30.735412+0000","last_undegraded":"2024-06-14T23:14:30.735412+0000","last_fullsized":"2024-06-14T23:14:30.735412+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:48:10.967651+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.742185+0000","last_change":"2024-06-14T23:14:30.740225+0000","last_active":"2024-06-14T23:14:30.742185+0000","last_peered":"2024-06-14T23:14:30.742185+0000","last_clean":"2024-06-14T23:14:30.742185+0000","last_became_active":"2024-06-14T23:14:30.739814+0000","last_became_peered":"2024-06-14T23:14:30.739814+0000","last_unstale":"2024-06-14T23:14:30.742185+0000","last_undegraded":"2024-06-14T23:14:30.742185+0000","last_fullsized":"2024-06-14T23:14:30.742185+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:27:30.267316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":323,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735594+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:14:30.735594+0000","last_peered":"2024-06-14T23:14:30.735594+0000","last_clean":"2024-06-14T23:14:30.735594+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:14:30.735594+0000","last_undegraded":"2024-06-14T23:14:30.735594+0000","last_fullsized":"2024-06-14T23:14:30.735594+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":201,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735500+0000","last_change":"2024-06-14T23:09:41.991652+0000","last_active":"2024-06-14T23:14:30.735500+0000","last_peered":"2024-06-14T23:14:30.735500+0000","last_clean":"2024-06-14T23:14:30.735500+0000","last_became_active":"2024-06-14T23:09:41.991317+0000","last_became_peered":"2024-06-14T23:09:41.991317+0000","last_unstale":"2024-06-14T23:14:30.735500+0000","last_undegraded":"2024-06-14T23:14:30.735500+0000","last_fullsized":"2024-06-14T23:14:30.735500+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:16:18.209329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":320,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062315+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:14:32.062315+0000","last_peered":"2024-06-14T23:14:32.062315+0000","last_clean":"2024-06-14T23:14:32.062315+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:14:32.062315+0000","last_undegraded":"2024-06-14T23:14:32.062315+0000","last_fullsized":"2024-06-14T23:14:32.062315+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":560,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737619+0000","last_change":"2024-06-14T23:10:29.063981+0000","last_active":"2024-06-14T23:14:30.737619+0000","last_peered":"2024-06-14T23:14:30.737619+0000","last_clean":"2024-06-14T23:14:30.737619+0000","last_became_active":"2024-06-14T23:10:29.063546+0000","last_became_peered":"2024-06-14T23:10:29.063546+0000","last_unstale":"2024-06-14T23:14:30.737619+0000","last_undegraded":"2024-06-14T23:14:30.737619+0000","last_fullsized":"2024-06-14T23:14:30.737619+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:26:23.411322+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.745685+0000","last_change":"2024-06-14T23:14:30.745351+0000","last_active":"2024-06-14T23:14:30.745685+0000","last_peered":"2024-06-14T23:14:30.745685+0000","last_clean":"2024-06-14T23:14:30.745685+0000","last_became_active":"2024-06-14T23:14:30.745017+0000","last_became_peered":"2024-06-14T23:14:30.745017+0000","last_unstale":"2024-06-14T23:14:30.745685+0000","last_undegraded":"2024-06-14T23:14:30.745685+0000","last_fullsized":"2024-06-14T23:14:30.745685+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:32.069491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":268,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.733381+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:14:30.733381+0000","last_peered":"2024-06-14T23:14:30.733381+0000","last_clean":"2024-06-14T23:14:30.733381+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:14:30.733381+0000","last_undegraded":"2024-06-14T23:14:30.733381+0000","last_fullsized":"2024-06-14T23:14:30.733381+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":274,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738893+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:14:30.738893+0000","last_peered":"2024-06-14T23:14:30.738893+0000","last_clean":"2024-06-14T23:14:30.738893+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:14:30.738893+0000","last_undegraded":"2024-06-14T23:14:30.738893+0000","last_fullsized":"2024-06-14T23:14:30.738893+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":355,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.739656+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:14:30.739656+0000","last_peered":"2024-06-14T23:14:30.739656+0000","last_clean":"2024-06-14T23:14:30.739656+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:14:30.739656+0000","last_undegraded":"2024-06-14T23:14:30.739656+0000","last_fullsized":"2024-06-14T23:14:30.739656+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":645,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735279+0000","last_change":"2024-06-14T23:14:18.033116+0000","last_active":"2024-06-14T23:14:30.735279+0000","last_peered":"2024-06-14T23:14:30.735279+0000","last_clean":"2024-06-14T23:14:30.735279+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:14:30.735279+0000","last_undegraded":"2024-06-14T23:14:30.735279+0000","last_fullsized":"2024-06-14T23:14:30.735279+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:18.032997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:18.032997+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:57:13.313132+0000","scrub_duration":0.0019777449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:31.005247+0000","last_change":"2024-06-14T23:14:31.004740+0000","last_active":"2024-06-14T23:14:31.005247+0000","last_peered":"2024-06-14T23:14:31.005247+0000","last_clean":"2024-06-14T23:14:31.005247+0000","last_became_active":"2024-06-14T23:14:31.003895+0000","last_became_peered":"2024-06-14T23:14:31.003895+0000","last_unstale":"2024-06-14T23:14:31.005247+0000","last_undegraded":"2024-06-14T23:14:31.005247+0000","last_fullsized":"2024-06-14T23:14:31.005247+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:15:16.776210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":152,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.063116+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:14:32.063116+0000","last_peered":"2024-06-14T23:14:32.063116+0000","last_clean":"2024-06-14T23:14:32.063116+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:14:32.063116+0000","last_undegraded":"2024-06-14T23:14:32.063116+0000","last_fullsized":"2024-06-14T23:14:32.063116+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":324,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062491+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:14:32.062491+0000","last_peered":"2024-06-14T23:14:32.062491+0000","last_clean":"2024-06-14T23:14:32.062491+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:14:32.062491+0000","last_undegraded":"2024-06-14T23:14:32.062491+0000","last_fullsized":"2024-06-14T23:14:32.062491+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":744,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738703+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:14:30.738703+0000","last_peered":"2024-06-14T23:14:30.738703+0000","last_clean":"2024-06-14T23:14:30.738703+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:14:30.738703+0000","last_undegraded":"2024-06-14T23:14:30.738703+0000","last_fullsized":"2024-06-14T23:14:30.738703+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.743987+0000","last_change":"2024-06-14T23:14:30.742252+0000","last_active":"2024-06-14T23:14:30.743987+0000","last_peered":"2024-06-14T23:14:30.743987+0000","last_clean":"2024-06-14T23:14:30.743987+0000","last_became_active":"2024-06-14T23:14:30.741955+0000","last_became_peered":"2024-06-14T23:14:30.741955+0000","last_unstale":"2024-06-14T23:14:30.743987+0000","last_undegraded":"2024-06-14T23:14:30.743987+0000","last_fullsized":"2024-06-14T23:14:30.743987+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.550055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":340,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.736764+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:14:30.736764+0000","last_peered":"2024-06-14T23:14:30.736764+0000","last_clean":"2024-06-14T23:14:30.736764+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:14:30.736764+0000","last_undegraded":"2024-06-14T23:14:30.736764+0000","last_fullsized":"2024-06-14T23:14:30.736764+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":351,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735842+0000","last_change":"2024-06-14T23:11:23.427812+0000","last_active":"2024-06-14T23:14:30.735842+0000","last_peered":"2024-06-14T23:14:30.735842+0000","last_clean":"2024-06-14T23:14:30.735842+0000","last_became_active":"2024-06-14T23:11:23.427563+0000","last_became_peered":"2024-06-14T23:11:23.427563+0000","last_unstale":"2024-06-14T23:14:30.735842+0000","last_undegraded":"2024-06-14T23:14:30.735842+0000","last_fullsized":"2024-06-14T23:14:30.735842+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.941006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":712,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.739568+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:14:30.739568+0000","last_peered":"2024-06-14T23:14:30.739568+0000","last_clean":"2024-06-14T23:14:30.739568+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:14:30.739568+0000","last_undegraded":"2024-06-14T23:14:30.739568+0000","last_fullsized":"2024-06-14T23:14:30.739568+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.744374+0000","last_change":"2024-06-14T23:14:30.742997+0000","last_active":"2024-06-14T23:14:30.744374+0000","last_peered":"2024-06-14T23:14:30.744374+0000","last_clean":"2024-06-14T23:14:30.744374+0000","last_became_active":"2024-06-14T23:14:30.742733+0000","last_became_peered":"2024-06-14T23:14:30.742733+0000","last_unstale":"2024-06-14T23:14:30.744374+0000","last_undegraded":"2024-06-14T23:14:30.744374+0000","last_fullsized":"2024-06-14T23:14:30.744374+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:48.248776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":370,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737432+0000","last_change":"2024-06-14T23:11:23.416264+0000","last_active":"2024-06-14T23:14:30.737432+0000","last_peered":"2024-06-14T23:14:30.737432+0000","last_clean":"2024-06-14T23:14:30.737432+0000","last_became_active":"2024-06-14T23:11:23.415946+0000","last_became_peered":"2024-06-14T23:11:23.415946+0000","last_unstale":"2024-06-14T23:14:30.737432+0000","last_undegraded":"2024-06-14T23:14:30.737432+0000","last_fullsized":"2024-06-14T23:14:30.737432+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:22.668569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":221,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735566+0000","last_change":"2024-06-14T23:10:26.045363+0000","last_active":"2024-06-14T23:14:30.735566+0000","last_peered":"2024-06-14T23:14:30.735566+0000","last_clean":"2024-06-14T23:14:30.735566+0000","last_became_active":"2024-06-14T23:10:26.043844+0000","last_became_peered":"2024-06-14T23:10:26.043844+0000","last_unstale":"2024-06-14T23:14:30.735566+0000","last_undegraded":"2024-06-14T23:14:30.735566+0000","last_fullsized":"2024-06-14T23:14:30.735566+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:11:27.891184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":735,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737809+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:14:30.737809+0000","last_peered":"2024-06-14T23:14:30.737809+0000","last_clean":"2024-06-14T23:14:30.737809+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:14:30.737809+0000","last_undegraded":"2024-06-14T23:14:30.737809+0000","last_fullsized":"2024-06-14T23:14:30.737809+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:31.005244+0000","last_change":"2024-06-14T23:14:31.004745+0000","last_active":"2024-06-14T23:14:31.005244+0000","last_peered":"2024-06-14T23:14:31.005244+0000","last_clean":"2024-06-14T23:14:31.005244+0000","last_became_active":"2024-06-14T23:14:31.004443+0000","last_became_peered":"2024-06-14T23:14:31.004443+0000","last_unstale":"2024-06-14T23:14:31.005244+0000","last_undegraded":"2024-06-14T23:14:31.005244+0000","last_fullsized":"2024-06-14T23:14:31.005244+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:56:14.724637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":330,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.736187+0000","last_change":"2024-06-14T23:07:58.024347+0000","last_active":"2024-06-14T23:14:30.736187+0000","last_peered":"2024-06-14T23:14:30.736187+0000","last_clean":"2024-06-14T23:14:30.736187+0000","last_became_active":"2024-06-14T23:07:58.024074+0000","last_became_peered":"2024-06-14T23:07:58.024074+0000","last_unstale":"2024-06-14T23:14:30.736187+0000","last_undegraded":"2024-06-14T23:14:30.736187+0000","last_fullsized":"2024-06-14T23:14:30.736187+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:26:45.849064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":151,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737785+0000","last_change":"2024-06-14T23:10:26.056252+0000","last_active":"2024-06-14T23:14:30.737785+0000","last_peered":"2024-06-14T23:14:30.737785+0000","last_clean":"2024-06-14T23:14:30.737785+0000","last_became_active":"2024-06-14T23:10:26.046629+0000","last_became_peered":"2024-06-14T23:10:26.046629+0000","last_unstale":"2024-06-14T23:14:30.737785+0000","last_undegraded":"2024-06-14T23:14:30.737785+0000","last_fullsized":"2024-06-14T23:14:30.737785+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:51:19.277386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":636,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735403+0000","last_change":"2024-06-14T23:14:14.052730+0000","last_active":"2024-06-14T23:14:30.735403+0000","last_peered":"2024-06-14T23:14:30.735403+0000","last_clean":"2024-06-14T23:14:30.735403+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:14:30.735403+0000","last_undegraded":"2024-06-14T23:14:30.735403+0000","last_fullsized":"2024-06-14T23:14:30.735403+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:14.052631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:14.052631+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:51:25.441426+0000","scrub_duration":0.0017584180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:31.004721+0000","last_change":"2024-06-14T23:14:31.003884+0000","last_active":"2024-06-14T23:14:31.004721+0000","last_peered":"2024-06-14T23:14:31.004721+0000","last_clean":"2024-06-14T23:14:31.004721+0000","last_became_active":"2024-06-14T23:14:31.003542+0000","last_became_peered":"2024-06-14T23:14:31.003542+0000","last_unstale":"2024-06-14T23:14:31.004721+0000","last_undegraded":"2024-06-14T23:14:31.004721+0000","last_fullsized":"2024-06-14T23:14:31.004721+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:27:38.082967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":150,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737698+0000","last_change":"2024-06-14T23:10:26.047111+0000","last_active":"2024-06-14T23:14:30.737698+0000","last_peered":"2024-06-14T23:14:30.737698+0000","last_clean":"2024-06-14T23:14:30.737698+0000","last_became_active":"2024-06-14T23:10:26.045540+0000","last_became_peered":"2024-06-14T23:10:26.045540+0000","last_unstale":"2024-06-14T23:14:30.737698+0000","last_undegraded":"2024-06-14T23:14:30.737698+0000","last_fullsized":"2024-06-14T23:14:30.737698+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:04.131400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":331,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.741182+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:14:30.741182+0000","last_peered":"2024-06-14T23:14:30.741182+0000","last_clean":"2024-06-14T23:14:30.741182+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:14:30.741182+0000","last_undegraded":"2024-06-14T23:14:30.741182+0000","last_fullsized":"2024-06-14T23:14:30.741182+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":591,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.736459+0000","last_change":"2024-06-14T23:09:17.382466+0000","last_active":"2024-06-14T23:14:30.736459+0000","last_peered":"2024-06-14T23:14:30.736459+0000","last_clean":"2024-06-14T23:14:30.736459+0000","last_became_active":"2024-06-14T23:09:12.030205+0000","last_became_peered":"2024-06-14T23:09:12.030205+0000","last_unstale":"2024-06-14T23:14:30.736459+0000","last_undegraded":"2024-06-14T23:14:30.736459+0000","last_fullsized":"2024-06-14T23:14:30.736459+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:09:17.382389+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:29:33.121472+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:31.004877+0000","last_change":"2024-06-14T23:14:31.004098+0000","last_active":"2024-06-14T23:14:31.004877+0000","last_peered":"2024-06-14T23:14:31.004877+0000","last_clean":"2024-06-14T23:14:31.004877+0000","last_became_active":"2024-06-14T23:14:31.003786+0000","last_became_peered":"2024-06-14T23:14:31.003786+0000","last_unstale":"2024-06-14T23:14:31.004877+0000","last_undegraded":"2024-06-14T23:14:31.004877+0000","last_fullsized":"2024-06-14T23:14:31.004877+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:08:03.948625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":304,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.732724+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:14:30.732724+0000","last_peered":"2024-06-14T23:14:30.732724+0000","last_clean":"2024-06-14T23:14:30.732724+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:14:30.732724+0000","last_undegraded":"2024-06-14T23:14:30.732724+0000","last_fullsized":"2024-06-14T23:14:30.732724+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":248,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737084+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:14:30.737084+0000","last_peered":"2024-06-14T23:14:30.737084+0000","last_clean":"2024-06-14T23:14:30.737084+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:14:30.737084+0000","last_undegraded":"2024-06-14T23:14:30.737084+0000","last_fullsized":"2024-06-14T23:14:30.737084+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":632,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.733545+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:14:30.733545+0000","last_peered":"2024-06-14T23:14:30.733545+0000","last_clean":"2024-06-14T23:14:30.733545+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:14:30.733545+0000","last_undegraded":"2024-06-14T23:14:30.733545+0000","last_fullsized":"2024-06-14T23:14:30.733545+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:31.735603+0000","last_change":"2024-06-14T23:14:30.738940+0000","last_active":"2024-06-14T23:14:31.735603+0000","last_peered":"2024-06-14T23:14:31.735603+0000","last_clean":"2024-06-14T23:14:31.735603+0000","last_became_active":"2024-06-14T23:14:30.738700+0000","last_became_peered":"2024-06-14T23:14:30.738700+0000","last_unstale":"2024-06-14T23:14:31.735603+0000","last_undegraded":"2024-06-14T23:14:31.735603+0000","last_fullsized":"2024-06-14T23:14:31.735603+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:29:01.585691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":340,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735754+0000","last_change":"2024-06-14T23:10:26.040201+0000","last_active":"2024-06-14T23:14:30.735754+0000","last_peered":"2024-06-14T23:14:30.735754+0000","last_clean":"2024-06-14T23:14:30.735754+0000","last_became_active":"2024-06-14T23:10:26.039865+0000","last_became_peered":"2024-06-14T23:10:26.039865+0000","last_unstale":"2024-06-14T23:14:30.735754+0000","last_undegraded":"2024-06-14T23:14:30.735754+0000","last_fullsized":"2024-06-14T23:14:30.735754+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:34.621202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":318,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737017+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:14:30.737017+0000","last_peered":"2024-06-14T23:14:30.737017+0000","last_clean":"2024-06-14T23:14:30.737017+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:14:30.737017+0000","last_undegraded":"2024-06-14T23:14:30.737017+0000","last_fullsized":"2024-06-14T23:14:30.737017+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":762,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062728+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:14:32.062728+0000","last_peered":"2024-06-14T23:14:32.062728+0000","last_clean":"2024-06-14T23:14:32.062728+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:14:32.062728+0000","last_undegraded":"2024-06-14T23:14:32.062728+0000","last_fullsized":"2024-06-14T23:14:32.062728+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738164+0000","last_change":"2024-06-14T23:14:30.735686+0000","last_active":"2024-06-14T23:14:30.738164+0000","last_peered":"2024-06-14T23:14:30.738164+0000","last_clean":"2024-06-14T23:14:30.738164+0000","last_became_active":"2024-06-14T23:14:30.735396+0000","last_became_peered":"2024-06-14T23:14:30.735396+0000","last_unstale":"2024-06-14T23:14:30.738164+0000","last_undegraded":"2024-06-14T23:14:30.738164+0000","last_fullsized":"2024-06-14T23:14:30.738164+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:25:23.806204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":313,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738053+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:14:30.738053+0000","last_peered":"2024-06-14T23:14:30.738053+0000","last_clean":"2024-06-14T23:14:30.738053+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:14:30.738053+0000","last_undegraded":"2024-06-14T23:14:30.738053+0000","last_fullsized":"2024-06-14T23:14:30.738053+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":331,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735021+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:14:30.735021+0000","last_peered":"2024-06-14T23:14:30.735021+0000","last_clean":"2024-06-14T23:14:30.735021+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:14:30.735021+0000","last_undegraded":"2024-06-14T23:14:30.735021+0000","last_fullsized":"2024-06-14T23:14:30.735021+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":786,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737065+0000","last_change":"2024-06-14T23:11:23.418239+0000","last_active":"2024-06-14T23:14:30.737065+0000","last_peered":"2024-06-14T23:14:30.737065+0000","last_clean":"2024-06-14T23:14:30.737065+0000","last_became_active":"2024-06-14T23:11:23.417873+0000","last_became_peered":"2024-06-14T23:11:23.417873+0000","last_unstale":"2024-06-14T23:14:30.737065+0000","last_undegraded":"2024-06-14T23:14:30.737065+0000","last_fullsized":"2024-06-14T23:14:30.737065+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:04:50.740175+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":339,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.731890+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:14:30.731890+0000","last_peered":"2024-06-14T23:14:30.731890+0000","last_clean":"2024-06-14T23:14:30.731890+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:14:30.731890+0000","last_undegraded":"2024-06-14T23:14:30.731890+0000","last_fullsized":"2024-06-14T23:14:30.731890+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":337,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.063036+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:14:32.063036+0000","last_peered":"2024-06-14T23:14:32.063036+0000","last_clean":"2024-06-14T23:14:32.063036+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:14:32.063036+0000","last_undegraded":"2024-06-14T23:14:32.063036+0000","last_fullsized":"2024-06-14T23:14:32.063036+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":708,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.738625+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:14:30.738625+0000","last_peered":"2024-06-14T23:14:30.738625+0000","last_clean":"2024-06-14T23:14:30.738625+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:14:30.738625+0000","last_undegraded":"2024-06-14T23:14:30.738625+0000","last_fullsized":"2024-06-14T23:14:30.738625+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":237,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735578+0000","last_change":"2024-06-14T23:07:58.023675+0000","last_active":"2024-06-14T23:14:30.735578+0000","last_peered":"2024-06-14T23:14:30.735578+0000","last_clean":"2024-06-14T23:14:30.735578+0000","last_became_active":"2024-06-14T23:07:58.023366+0000","last_became_peered":"2024-06-14T23:07:58.023366+0000","last_unstale":"2024-06-14T23:14:30.735578+0000","last_undegraded":"2024-06-14T23:14:30.735578+0000","last_fullsized":"2024-06-14T23:14:30.735578+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:11:14.587851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":218,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.063103+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:14:32.063103+0000","last_peered":"2024-06-14T23:14:32.063103+0000","last_clean":"2024-06-14T23:14:32.063103+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:14:32.063103+0000","last_undegraded":"2024-06-14T23:14:32.063103+0000","last_fullsized":"2024-06-14T23:14:32.063103+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":676,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-06-14T23:14:32.062692+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:14:32.062692+0000","last_peered":"2024-06-14T23:14:32.062692+0000","last_clean":"2024-06-14T23:14:32.062692+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:14:32.062692+0000","last_undegraded":"2024-06-14T23:14:32.062692+0000","last_fullsized":"2024-06-14T23:14:32.062692+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":55,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735603+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:14:30.735603+0000","last_peered":"2024-06-14T23:14:30.735603+0000","last_clean":"2024-06-14T23:14:30.735603+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:14:30.735603+0000","last_undegraded":"2024-06-14T23:14:30.735603+0000","last_fullsized":"2024-06-14T23:14:30.735603+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":313,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737171+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:14:30.737171+0000","last_peered":"2024-06-14T23:14:30.737171+0000","last_clean":"2024-06-14T23:14:30.737171+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:14:30.737171+0000","last_undegraded":"2024-06-14T23:14:30.737171+0000","last_fullsized":"2024-06-14T23:14:30.737171+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":672,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.737720+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:14:30.737720+0000","last_peered":"2024-06-14T23:14:30.737720+0000","last_clean":"2024-06-14T23:14:30.737720+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:14:30.737720+0000","last_undegraded":"2024-06-14T23:14:30.737720+0000","last_fullsized":"2024-06-14T23:14:30.737720+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":365,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.736422+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:14:30.736422+0000","last_peered":"2024-06-14T23:14:30.736422+0000","last_clean":"2024-06-14T23:14:30.736422+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:14:30.736422+0000","last_undegraded":"2024-06-14T23:14:30.736422+0000","last_fullsized":"2024-06-14T23:14:30.736422+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":55,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.735727+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:14:30.735727+0000","last_peered":"2024-06-14T23:14:30.735727+0000","last_clean":"2024-06-14T23:14:30.735727+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:14:30.735727+0000","last_undegraded":"2024-06-14T23:14:30.735727+0000","last_fullsized":"2024-06-14T23:14:30.735727+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":785,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-06-14T23:14:30.739336+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:14:30.739336+0000","last_peered":"2024-06-14T23:14:30.739336+0000","last_clean":"2024-06-14T23:14:30.739336+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:14:30.739336+0000","last_undegraded":"2024-06-14T23:14:30.739336+0000","last_fullsized":"2024-06-14T23:14:30.739336+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234486,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1515724,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":1510974,"kb_avail":92229428,"statfs":{"total":95995035648,"available":94442934272,"internally_reserved":0,"allocated":4841472,"data_stored":2988508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1547237720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:34 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.0960000000000001,"15min":1.016},"min":{"1min":0.36599999999999999,"5min":0.24199999999999999,"15min":0.19800000000000001},"max":{"1min":5.8259999999999996,"5min":5.8259999999999996,"15min":6.9539999999999997},"last":0.621},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.196,"15min":1.1930000000000001},"min":{"1min":0.30299999999999999,"5min":0.254,"15min":0.224},"max":{"1min":6.5149999999999997,"5min":6.5149999999999997,"15min":6.9550000000000001},"last":0.55100000000000005}]},{"osd":1,"last update":"Fri Jun 14 23:14:15 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.179,"15min":1.2430000000000001},"min":{"1min":0.29599999999999999,"5min":0.26600000000000001,"15min":0.22600000000000001},"max":{"1min":6.0229999999999997,"5min":6.0229999999999997,"15min":12.010999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.248,"15min":1.294},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.22900000000000001},"max":{"1min":5.8490000000000002,"5min":5.8490000000000002,"15min":13.531000000000001},"last":0.54000000000000004}]},{"osd":2,"last update":"Fri Jun 14 23:14:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":1.361,"15min":1.4039999999999999},"min":{"1min":0.30099999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.5510000000000002,"5min":10.539,"15min":10.539},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.3169999999999999,"15min":1.341},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.5960000000000001,"5min":10.56,"15min":10.56},"last":0.65300000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:13:34 2024","interfaces":[{"interface":"back","average":{"1min":2.7789999999999999,"5min":1.546,"15min":1.4410000000000001},"min":{"1min":0.40600000000000003,"5min":0.30599999999999999,"15min":0.28199999999999997},"max":{"1min":17.991,"5min":17.991,"15min":20.645},"last":0.60899999999999999},{"interface":"front","average":{"1min":2.5339999999999998,"5min":1.5640000000000001,"15min":1.532},"min":{"1min":0.443,"5min":0.29699999999999999,"15min":0.28299999999999997},"max":{"1min":17.824999999999999,"5min":17.824999999999999,"15min":22.042999999999999},"last":0.67900000000000005}]},{"osd":4,"last update":"Fri Jun 14 23:14:28 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.266,"15min":1.085},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.23100000000000001},"max":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":11.342000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.405,"15min":1.1299999999999999},"min":{"1min":0.27000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.5139999999999998,"5min":10.499000000000001,"15min":12.651},"last":0.44600000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:14:28 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.153,"15min":1.0449999999999999},"min":{"1min":0.371,"5min":0.245,"15min":0.23100000000000001},"max":{"1min":5.734,"5min":5.734,"15min":6.4320000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.252,"15min":1.1020000000000001},"min":{"1min":0.371,"5min":0.33500000000000002,"15min":0.24399999999999999},"max":{"1min":6.0899999999999999,"5min":6.0899999999999999,"15min":6.2809999999999997},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:14:03 2024","interfaces":[{"interface":"back","average":{"1min":2.2080000000000002,"5min":1.8979999999999999,"15min":1.508},"min":{"1min":0.39400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":7.1970000000000001,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.714,"15min":1.492},"min":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.32600000000000001},"max":{"1min":6.9569999999999999,"5min":10.087999999999999,"15min":10.087999999999999},"last":0.47499999999999998}]}]},{"osd":6,"up_from":272,"seq":1168231104592,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397748,"kb_used_data":4776,"kb_used_omap":7,"kb_used_meta":392952,"kb_avail":93347404,"statfs":{"total":95995035648,"available":95587741696,"internally_reserved":0,"allocated":4890624,"data_stored":3053601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":402383290},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.88700000000000001,"15min":1.5469999999999999},"min":{"1min":0.30599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.1850000000000001,"5min":4.6390000000000002,"15min":6.6440000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.996,"5min":0.95699999999999996,"15min":1.6020000000000001},"min":{"1min":0.39000000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.266,"5min":4.6150000000000002,"15min":6.6790000000000003},"last":0.50600000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0289999999999999,"15min":1.9930000000000001},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.806,"5min":5.4660000000000002,"15min":8.173},"last":0.442},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.95299999999999996,"15min":1.901},"min":{"1min":0.34499999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.234,"5min":4.6619999999999999,"15min":10.380000000000001},"last":0.41999999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:14:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":1.115,"15min":1.115},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.5019999999999998,"5min":7.476,"15min":7.476},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.751,"5min":1.2170000000000001,"15min":1.1950000000000001},"min":{"1min":0.31900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":3.5510000000000002,"5min":7.7590000000000003,"15min":7.7590000000000003},"last":0.60199999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0269999999999999,"15min":1.6930000000000001},"min":{"1min":0.34899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.403,"5min":4.5819999999999999,"15min":6.7089999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.552,"5min":1.504,"15min":1.95},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":7.2300000000000004,"5min":8.4740000000000002,"15min":8.4740000000000002},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.4079999999999999,"15min":1.444},"min":{"1min":0.24399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.3159999999999998,"5min":11.010999999999999,"15min":11.010999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.1200000000000001,"15min":1.252},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.2890000000000001,"5min":5.2309999999999999,"15min":5.2309999999999999},"last":0.54200000000000004}]},{"osd":5,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.96699999999999997,"15min":1.3140000000000001},"min":{"1min":0.38700000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.1539999999999999,"5min":4.5579999999999998,"15min":7.7469999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.3839999999999999,"15min":1.5700000000000001},"min":{"1min":0.33300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.5700000000000003,"5min":7.8680000000000003,"15min":7.8680000000000003},"last":0.55200000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:14:08 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.7549999999999999,"15min":1.7609999999999999},"min":{"1min":0.376,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":10.016999999999999,"5min":10.016999999999999,"15min":10.016999999999999},"last":0.624},{"interface":"front","average":{"1min":2.1619999999999999,"5min":1.587,"15min":1.591},"min":{"1min":0.35799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":8.1020000000000003,"5min":8.1020000000000003,"15min":8.1020000000000003},"last":0.48699999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757761,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180152,"kb_used_data":4708,"kb_used_omap":1,"kb_used_meta":1175422,"kb_avail":92565000,"statfs":{"total":95995035648,"available":94786560000,"internally_reserved":0,"allocated":4820992,"data_stored":2983875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1203632472},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:14:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.74399999999999999,"15min":1.282},"min":{"1min":0.248,"5min":0.24099999999999999,"15min":0.20100000000000001},"max":{"1min":5.9199999999999999,"5min":5.9199999999999999,"15min":14.94},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.625,"15min":1.288},"min":{"1min":0.29999999999999999,"5min":0.26100000000000001,"15min":0.219},"max":{"1min":2.8900000000000001,"5min":4.181,"15min":9.3819999999999997},"last":0.65500000000000003}]},{"osd":1,"last update":"Fri Jun 14 23:14:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.64800000000000002,"15min":0.91700000000000004},"min":{"1min":0.28499999999999998,"5min":0.245,"15min":0.23999999999999999},"max":{"1min":3.093,"5min":3.093,"15min":7.4370000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.71899999999999997,"15min":1.1160000000000001},"min":{"1min":0.318,"5min":0.27400000000000002,"15min":0.22900000000000001},"max":{"1min":5.952,"5min":5.952,"15min":9.5039999999999996},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:14:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.73699999999999999,"15min":0.67800000000000005},"min":{"1min":0.28699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.5019999999999998,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.76600000000000001,"15min":0.67900000000000005},"min":{"1min":0.42699999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.657,"5min":6.0659999999999998,"15min":6.0659999999999998},"last":0.71399999999999997}]},{"osd":3,"last update":"Fri Jun 14 23:13:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.78100000000000003,"15min":1.363},"min":{"1min":0.29999999999999999,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":4.1600000000000001,"5min":4.3949999999999996,"15min":10.358000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":0.93500000000000005,"15min":1.377},"min":{"1min":0.35099999999999998,"5min":0.32100000000000001,"15min":0.28899999999999998},"max":{"1min":6.0099999999999998,"5min":6.0099999999999998,"15min":14.914},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Jun 14 23:14:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81100000000000005,"15min":0.88100000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.2809999999999997,"5min":4.2809999999999997,"15min":14.352},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.77700000000000002,"15min":1.056},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.253},"max":{"1min":4.0880000000000001,"5min":6.3630000000000004,"15min":9.5980000000000008},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Jun 14 23:14:15 2024","interfaces":[{"interface":"back","average":{"1min":2.5659999999999998,"5min":1.163,"15min":1.417},"min":{"1min":0.313,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":30.934000000000001,"5min":30.934000000000001,"15min":30.934000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":1.244,"15min":1.708},"min":{"1min":0.34100000000000003,"5min":0.27900000000000003,"15min":0.26100000000000001},"max":{"1min":31.059999999999999,"5min":31.059999999999999,"15min":31.059999999999999},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:14:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.347,"15min":1.236},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":5.4420000000000002,"5min":26.981999999999999,"15min":26.981999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.526,"15min":1.2130000000000001},"min":{"1min":0.34399999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.5800000000000001,"5min":23.326000000000001,"15min":23.326000000000001},"last":0.53900000000000003}]}]},{"osd":4,"up_from":30,"seq":128849019397,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1183872,"kb_used_data":4780,"kb_used_omap":1,"kb_used_meta":1179070,"kb_avail":92561280,"statfs":{"total":95995035648,"available":94782750720,"internally_reserved":0,"allocated":4894720,"data_stored":3053165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1207368024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.81000000000000005,"15min":1.1220000000000001},"min":{"1min":0.26000000000000001,"5min":0.223,"15min":0.17100000000000001},"max":{"1min":2.968,"5min":6.875,"15min":10.567},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.3080000000000001,"5min":0.873,"15min":1.1419999999999999},"min":{"1min":0.29199999999999998,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":6.7779999999999996,"5min":6.7779999999999996,"15min":8.3390000000000004},"last":0.499}]},{"osd":1,"last update":"Fri Jun 14 23:14:04 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.85799999999999998,"15min":1.4330000000000001},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.23100000000000001},"max":{"1min":2.895,"5min":4.476,"15min":56.917000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.88500000000000001,"15min":1.4750000000000001},"min":{"1min":0.33600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":3.0419999999999998,"5min":6.9770000000000003,"15min":56.789999999999999},"last":0.53900000000000003}]},{"osd":2,"last update":"Fri Jun 14 23:14:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.84599999999999997,"15min":0.82399999999999995},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.462,"5min":3.468,"15min":3.468},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.90400000000000003,"15min":0.88800000000000001},"min":{"1min":0.34100000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.4980000000000002,"5min":3.4980000000000002,"15min":3.4980000000000002},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:13:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.246,"15min":1.298},"min":{"1min":0.39900000000000002,"5min":0.27800000000000002,"15min":0.25},"max":{"1min":3.173,"5min":44.389000000000003,"15min":44.389000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.325,"5min":1.452,"15min":1.492},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.27600000000000002},"max":{"1min":3.2109999999999999,"5min":46.101999999999997,"15min":46.101999999999997},"last":0.57599999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.89200000000000002,"15min":1.5389999999999999},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.20300000000000001},"max":{"1min":3.694,"5min":4.04,"15min":139.13200000000001},"last":0.377},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.94999999999999996,"15min":1.585},"min":{"1min":0.29099999999999998,"5min":0.28100000000000003,"15min":0.23300000000000001},"max":{"1min":5.077,"5min":6.9290000000000003,"15min":138.86000000000001},"last":0.41999999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:14:11 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":2.4049999999999998,"15min":1.5349999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.2999999999999998,"5min":122.88500000000001,"15min":122.88500000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.1839999999999999,"5min":2.3620000000000001,"15min":1.887},"min":{"1min":0.30499999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.8200000000000003,"5min":122.21299999999999,"15min":122.21299999999999},"last":0.44600000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.194,"15min":1.075},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":2.5299999999999998,"5min":6.5910000000000002,"15min":6.5910000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.044,"5min":1.3160000000000001,"15min":1.1519999999999999},"min":{"1min":0.371,"5min":0.32400000000000001,"15min":0.27300000000000002},"max":{"1min":4.8659999999999997,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":0.59699999999999998}]}]},{"osd":3,"up_from":138,"seq":592705487096,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925236,"kb_used_data":4776,"kb_used_omap":4,"kb_used_meta":920443,"kb_avail":92819916,"statfs":{"total":95995035648,"available":95047593984,"internally_reserved":0,"allocated":4890624,"data_stored":3056852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":942534110},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.75700000000000001,"15min":1.2829999999999999},"min":{"1min":0.191,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.175,"5min":25.190000000000001,"15min":25.190000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.044,"15min":1.329},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.17799999999999999},"max":{"1min":2.4020000000000001,"5min":25.652000000000001,"15min":25.652000000000001},"last":0.41999999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.218,"15min":1.393},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.20300000000000001},"max":{"1min":2.431,"5min":25.754000000000001,"15min":25.754000000000001},"last":0.38},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.274,"15min":1.5660000000000001},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.25},"max":{"1min":2.4649999999999999,"5min":27.992999999999999,"15min":27.992999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:14:28 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.81999999999999995,"15min":0.88},"min":{"1min":0.29499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.3519999999999999,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.434},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.74299999999999999,"15min":0.71499999999999997},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.5550000000000002,"5min":3.5550000000000002,"15min":3.5550000000000002},"last":0.48899999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":1.9019999999999999,"5min":1.6619999999999999,"15min":1.798},"min":{"1min":0.30399999999999999,"5min":0.28299999999999997,"15min":0.27900000000000003},"max":{"1min":17.018999999999998,"5min":25.094000000000001,"15min":73.263000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":2.1890000000000001,"5min":1.516,"15min":1.714},"min":{"1min":0.36899999999999999,"5min":0.27600000000000002,"15min":0.27300000000000002},"max":{"1min":17.074000000000002,"5min":25.5,"15min":73.5},"last":0.58599999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.3759999999999999,"15min":1.7270000000000001},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.28699999999999998},"max":{"1min":3.4740000000000002,"5min":25.521000000000001,"15min":81.438999999999993},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.4950000000000001,"15min":1.7509999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.29299999999999998},"max":{"1min":2.3199999999999998,"5min":25.84,"15min":81.626999999999995},"last":0.68700000000000006}]},{"osd":6,"last update":"Fri Jun 14 23:14:09 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.718,"15min":1.7130000000000001},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.34599999999999997},"max":{"1min":3.8010000000000002,"5min":25.869,"15min":25.869},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.6100000000000001,"15min":1.5429999999999999},"min":{"1min":0.38400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":3.371,"5min":25.143000000000001,"15min":25.143000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Fri Jun 14 23:13:43 2024","interfaces":[{"interface":"back","average":{"1min":3.468,"5min":1.8080000000000001,"15min":1.571},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":44.887999999999998,"5min":44.887999999999998,"15min":44.887999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":3.6760000000000002,"5min":1.982,"15min":1.5309999999999999},"min":{"1min":0.39800000000000002,"5min":0.36099999999999999,"15min":0.32400000000000001},"max":{"1min":45.835000000000001,"5min":45.835000000000001,"15min":45.835000000000001},"last":0.69999999999999996}]}]},{"osd":2,"up_from":322,"seq":1382979469351,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":200944,"kb_used_data":4704,"kb_used_omap":10,"kb_used_meta":196213,"kb_avail":93544208,"statfs":{"total":95995035648,"available":95789268992,"internally_reserved":0,"allocated":4816896,"data_stored":2983840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":200922662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44500000000000001,"15min":0.36299999999999999},"min":{"1min":0.20499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.359,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":0.376},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.50700000000000001,"15min":0.46100000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.0699999999999998,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.39300000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.46200000000000002,"15min":0.42299999999999999},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.4340000000000002,"5min":3.4340000000000002,"15min":3.4340000000000002},"last":0.31},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.54800000000000004,"15min":0.47399999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.0389999999999997,"5min":5.0389999999999997,"15min":5.0389999999999997},"last":0.27200000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69099999999999995,"15min":0.624},"min":{"1min":0.23899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.694,"5min":3.7519999999999998,"15min":3.7519999999999998},"last":0.441},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.64900000000000002,"15min":0.55900000000000005},"min":{"1min":0.26300000000000001,"5min":0.223,"15min":0.223},"max":{"1min":5.1980000000000004,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.47199999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:14:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.024,"15min":0.94299999999999995},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.1779999999999999,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79100000000000004,"15min":0.70699999999999996},"min":{"1min":0.377,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.8069999999999999,"5min":4.3680000000000003,"15min":4.3680000000000003},"last":0.64900000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:14:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.76900000000000002,"15min":0.77500000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.2690000000000001,"5min":3.3519999999999999,"15min":3.3519999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.83199999999999996,"15min":0.85799999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.1480000000000001,"5min":3.4750000000000001,"15min":3.4750000000000001},"last":0.746}]},{"osd":6,"last update":"Fri Jun 14 23:14:31 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":0.97299999999999998,"15min":0.83899999999999997},"min":{"1min":0.41799999999999998,"5min":0.38,"15min":0.38},"max":{"1min":5.0880000000000001,"5min":5.0880000000000001,"15min":5.0880000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.0429999999999999,"5min":0.995,"15min":0.97999999999999998},"min":{"1min":0.44400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.9359999999999999,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":0.76100000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:14:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.97099999999999997,"15min":0.95099999999999996},"min":{"1min":0.39400000000000002,"5min":0.372,"15min":0.372},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":4.0460000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.2470000000000001,"15min":1.149},"min":{"1min":0.38100000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.2130000000000001,"5min":5.2130000000000001,"15min":5.2130000000000001},"last":0.56699999999999995}]}]},{"osd":1,"up_from":0,"seq":498216206640,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024294,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:14:40.491 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-14T23:14:40.491 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:14:40.492 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T23:14:40.492 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-06-14T23:14:40.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 2 1 2024-06-14T23:14:40.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:40.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:40.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:40.590 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_8' 2024-06-14T23:14:40.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:40.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:40.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:40.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:40.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:40.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:40.833 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:40.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:40.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:40.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:41.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:41.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:40 smithi086 ceph-mon[97913]: pgmap v1329: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:41.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3097296705' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:14:41.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:40 smithi086 ceph-mon[97913]: osdmap e355: 8 total, 8 up, 6 in 2024-06-14T23:14:41.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:40 smithi086 ceph-mon[102754]: pgmap v1329: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:41.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3097296705' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:14:41.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:40 smithi086 ceph-mon[102754]: osdmap e355: 8 total, 8 up, 6 in 2024-06-14T23:14:41.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:41.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:41.070 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:41.147 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:40 smithi137 ceph-mon[90634]: pgmap v1329: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:41.147 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3097296705' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:14:41.147 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:40 smithi137 ceph-mon[90634]: osdmap e355: 8 total, 8 up, 6 in 2024-06-14T23:14:41.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:41.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:41.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:41.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:41.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:41.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:41.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:41.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:41.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:41.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:41.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:41.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:41.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:41.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:41.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:41.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:41.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:41.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:41.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:41.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:41.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:41.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:42.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:42.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:42.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:42.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:41 smithi086 ceph-mon[97913]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:14:42.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:41 smithi086 ceph-mon[102754]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:14:42.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:42.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:42.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:42.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:41 smithi137 ceph-mon[90634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:14:42.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:42.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:42.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:42.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:42.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:42.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:42.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:42.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:42.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:42.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:42.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:42.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:42.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:42.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:42.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:42.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:42.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:42.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:42.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:43.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:43.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:43.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:43.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:42 smithi086 ceph-mon[97913]: pgmap v1331: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:43.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:43.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:42 smithi086 ceph-mon[102754]: pgmap v1331: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:43.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:43.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:43.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:43.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:43.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:42 smithi137 ceph-mon[90634]: pgmap v1331: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:43.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:43.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:43.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:43.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:43.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:43.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:43.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:43.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:43.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:43.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:43.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:43.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:43.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:43.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:43.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:43.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:43.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:43.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:43.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:44.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:44.033 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:44.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:44.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:44.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:44.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]: dispatch 2024-06-14T23:14:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]}]: dispatch 2024-06-14T23:14:44.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:44.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]: dispatch 2024-06-14T23:14:44.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]}]: dispatch 2024-06-14T23:14:44.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:14:44.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]: dispatch 2024-06-14T23:14:44.248 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]}]: dispatch 2024-06-14T23:14:44.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:44.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:44.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:44.290 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_8 bench 90 write' 2024-06-14T23:14:44.292 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T23:14:44.293 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T23:14:44.337 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T23:14:44.337 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T23:14:44.337 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_299538 2024-06-14T23:14:44.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:14:44.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T23:14:44.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:44.420 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:44.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:44.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:44.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:44.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:44.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:44.691 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:44.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:44.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:44.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:44.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:44.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:44.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:44.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:45.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:45.108 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:45.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:45.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:45.186 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:44 smithi086 ceph-mon[102754]: pgmap v1332: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:45.186 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:44 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]': finished 2024-06-14T23:14:45.186 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:44 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]}]': finished 2024-06-14T23:14:45.186 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:44 smithi086 ceph-mon[102754]: osdmap e356: 8 total, 8 up, 6 in 2024-06-14T23:14:45.186 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:44 smithi086 ceph-mon[97913]: pgmap v1332: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:45.186 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:44 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]': finished 2024-06-14T23:14:45.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:44 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]}]': finished 2024-06-14T23:14:45.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:44 smithi086 ceph-mon[97913]: osdmap e356: 8 total, 8 up, 6 in 2024-06-14T23:14:45.187 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:14:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:14:44] "GET /metrics HTTP/1.1" 200 36608 "" "Prometheus/2.43.0" 2024-06-14T23:14:45.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:45.220 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:45.222 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:14:45.223 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:14:45.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:44 smithi137 ceph-mon[90634]: pgmap v1332: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:44 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.16", "id": [5, 2]}]': finished 2024-06-14T23:14:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:44 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]}]': finished 2024-06-14T23:14:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:44 smithi137 ceph-mon[90634]: osdmap e356: 8 total, 8 up, 6 in 2024-06-14T23:14:45.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:45.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:45.260 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:14:45.260 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:14:45.286 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:14:45.287 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:14:45.310 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:14:45.311 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:14:45.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 3534 3518 219.863 219.875 0.00281161 0.00453388 2024-06-14T23:14:45.338 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:14:45.339 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:14:45.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:45.368 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:14:45.369 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:14:45.378 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:45.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:45.396 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:14:45.396 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:14:45.421 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:14:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:14:45.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:45.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:45.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:45.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:45.650 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:45.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:45.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:45.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:45.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:45.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:45.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:45.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:46.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:46.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:46.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:46.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:45 smithi086 ceph-mon[97913]: osdmap e357: 8 total, 8 up, 6 in 2024-06-14T23:14:46.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3564916799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:46.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:45 smithi086 ceph-mon[102754]: osdmap e357: 8 total, 8 up, 6 in 2024-06-14T23:14:46.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3564916799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:46.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:46.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:46.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:46.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:45 smithi137 ceph-mon[90634]: osdmap e357: 8 total, 8 up, 6 in 2024-06-14T23:14:46.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3564916799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:14:46.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:46.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:46.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:46.337 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 15 6890 6875 214.829 209.812 0.00561652 0.00464116 2024-06-14T23:14:46.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:46.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:46.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:46.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:46.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:46.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:46.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:46.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:46.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:46.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:46.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:46.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:46.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:46.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:47.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:46 smithi086 ceph-mon[97913]: pgmap v1335: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:47.013 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:46 smithi086 ceph-mon[102754]: pgmap v1335: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:47.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:47.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:47.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:47.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:47.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:47.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:46 smithi137 ceph-mon[90634]: pgmap v1335: 119 pgs: 119 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail 2024-06-14T23:14:47.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:47.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:47.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:47.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 15 9872 9857 205.339 186.375 0.00484918 0.00486242 2024-06-14T23:14:47.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:47.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:47.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:47.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:47.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:47.632 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:14:47.632 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:14:47.633 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-06-14T23:14:47.633 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:14:47.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:47.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:47.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:47.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:47.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:47.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:47.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:47.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:48.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:48.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:48.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:48.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:48.230 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:48.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:48.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:48.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:48.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 15 12839 12824 200.36 185.438 0.00459039 0.0049854 2024-06-14T23:14:48.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:48.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:48.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:48.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-14T23:14:48.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:48.504 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:48.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:48.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:48.636 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:48.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:48.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:48.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:48.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:48.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:48.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:48.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:48.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:49.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:49.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:49.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:49.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:49.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:49.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:48 smithi137 ceph-mon[90634]: pgmap v1336: 119 pgs: 119 active+clean; 215 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 27 MiB/s wr, 429 op/s 2024-06-14T23:14:49.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:49.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:49.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:48 smithi086 ceph-mon[97913]: pgmap v1336: 119 pgs: 119 active+clean; 215 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 27 MiB/s wr, 429 op/s 2024-06-14T23:14:49.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:48 smithi086 ceph-mon[102754]: pgmap v1336: 119 pgs: 119 active+clean; 215 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 27 MiB/s wr, 429 op/s 2024-06-14T23:14:49.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 16 15591 15575 194.672 171.938 0.00512736 0.00513146 2024-06-14T23:14:49.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:49.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:49.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:49.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:49.598 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:49.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:49.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:49.739 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:49.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:49.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:49.848 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:49.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:49.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:49.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:50.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:50.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:50.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:50.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:50.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:50.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:50.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:50.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:50.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 18184 18168 189.234 162.062 0.00566981 0.00527867 2024-06-14T23:14:50.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:50.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:50.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:50.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:50.544 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:50.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:50.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:50.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:50.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:50.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:50.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:50.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:50.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:50.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:50.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:51.045 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:50 smithi086 ceph-mon[102754]: pgmap v1337: 119 pgs: 119 active+clean; 724 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-06-14T23:14:51.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:50 smithi086 ceph-mon[97913]: pgmap v1337: 119 pgs: 119 active+clean; 724 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-06-14T23:14:51.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:51.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:51.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:51.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:51.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:51.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:51.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:50 smithi137 ceph-mon[90634]: pgmap v1337: 119 pgs: 119 active+clean; 724 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-06-14T23:14:51.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 16 20663 20647 184.332 154.938 0.00744396 0.00541971 2024-06-14T23:14:51.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:51.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:51.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:51.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:51.507 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:51.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:51.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:14:51.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:51.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:51.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:51.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:51.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:51.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:51.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:51.925 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:51.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:51.968 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-06-14T23:14:51.969 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:14:52.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:52.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:52.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:52.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:52.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:52.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1097984065' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:14:52.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1821663466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:52.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:52.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:52.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:52.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:52.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1097984065' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:14:52.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1821663466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:52.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:52.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1097984065' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:14:52.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1821663466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:52.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:14:52.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 22899 22883 178.758 139.75 0.00618698 0.00558735 2024-06-14T23:14:52.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:52.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:52.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:52.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:52.582 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:52.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:52.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:52.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:52.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:52.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:52.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:52.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:52.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:52.983 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:52.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:53.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:53.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:53.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:53.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:52 smithi137 ceph-mon[90634]: pgmap v1338: 119 pgs: 119 active+clean; 724 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-06-14T23:14:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:52 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:14:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1097984065' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:14:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:52 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:14:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:52 smithi137 ceph-mon[90634]: osdmap e358: 8 total, 8 up, 6 in 2024-06-14T23:14:53.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:53.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[97913]: pgmap v1338: 119 pgs: 119 active+clean; 724 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-06-14T23:14:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:14:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1097984065' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:14:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:14:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[97913]: osdmap e358: 8 total, 8 up, 6 in 2024-06-14T23:14:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[102754]: pgmap v1338: 119 pgs: 119 active+clean; 724 MiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-06-14T23:14:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:14:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1097984065' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:14:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:14:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:52 smithi086 ceph-mon[102754]: osdmap e358: 8 total, 8 up, 6 in 2024-06-14T23:14:53.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 25049 25033 173.826 134.375 0.0066932 0.00574651 2024-06-14T23:14:53.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:53.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:53.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:53.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:53.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:53.517 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:53.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:53.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:53.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:53.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:53.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:53.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:53.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:53.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:53.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:53.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:54.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:54.051 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:54.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:54.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:54.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:54.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:54.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:54.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:54.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:54.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 16 27120 27104 169.386 129.438 0.0105273 0.00589745 2024-06-14T23:14:54.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:54.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:54.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:54.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:54.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:54.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:54.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:54.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:54.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:54.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:54.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:54.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:54.966 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:14:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:14:54] "GET /metrics HTTP/1.1" 200 36608 "" "Prometheus/2.43.0" 2024-06-14T23:14:54.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:54.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:55.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:55.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:55.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:55.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:55.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:54 smithi137 ceph-mon[90634]: pgmap v1340: 119 pgs: 119 active+clean; 1.4 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-06-14T23:14:55.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:54 smithi086 ceph-mon[102754]: pgmap v1340: 119 pgs: 119 active+clean; 1.4 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-06-14T23:14:55.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:54 smithi086 ceph-mon[97913]: pgmap v1340: 119 pgs: 119 active+clean; 1.4 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-06-14T23:14:55.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:55.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:55.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:55.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 29219 29203 165.913 131.188 0.00988166 0.00602304 2024-06-14T23:14:55.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:55.398 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:55.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:55.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:55.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:55.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:55.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:55.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:55.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:55.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:55.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:55.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:55.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:55.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:55.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:56.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:56.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:56.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:56.188 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:14:56.188 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:14:56.188 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:14:56.188 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:14:56.188 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:14:56.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:56.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:56.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:14:56.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 16 31217 31201 162.492 124.875 0.00541608 0.00614951 2024-06-14T23:14:56.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:56.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:14:56.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:56.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:56.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:56.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:56.630 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:56.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:56.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:14:56.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:56.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:56.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:56.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:56.910 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:56.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:57.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-14T23:14:57.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:57.077 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:57.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:57.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:57.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:57.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:57.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:56 smithi137 ceph-mon[90634]: pgmap v1341: 119 pgs: 119 active+clean; 1.5 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-06-14T23:14:57.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:56 smithi086 ceph-mon[97913]: pgmap v1341: 119 pgs: 119 active+clean; 1.5 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-06-14T23:14:57.305 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:56 smithi086 ceph-mon[102754]: pgmap v1341: 119 pgs: 119 active+clean; 1.5 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-06-14T23:14:57.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:57.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 33175 33159 159.405 122.375 0.00557385 0.00626829 2024-06-14T23:14:57.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:57.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:57.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:57.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:57.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:57.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:57.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:57.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:57.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:57.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:57.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:57.850 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:57.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:57.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:57.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:58.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:58.103 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:58.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:58.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:58.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:58.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:58.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:58.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:58.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:14:58.338 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 35065 35049 156.456 118.125 0.00418411 0.00638668 2024-06-14T23:14:58.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:58.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:58.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:58.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:14:58.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:58.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:14:58.543 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:14:58.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:58.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:58.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:58.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:14:58.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:58.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:14:58.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:58.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:59.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:14:59.061 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:14:58 smithi086 ceph-mon[97913]: pgmap v1342: 119 pgs: 119 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-06-14T23:14:59.061 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:14:58 smithi086 ceph-mon[102754]: pgmap v1342: 119 pgs: 119 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-06-14T23:14:59.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:59.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:14:59.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:59.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:59.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:59.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:14:58 smithi137 ceph-mon[90634]: pgmap v1342: 119 pgs: 119 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-06-14T23:14:59.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:59.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 36802 36786 153.262 108.562 0.0105184 0.00651874 2024-06-14T23:14:59.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:59.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:59.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:14:59.477 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:59.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:14:59.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:14:59.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:59.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:14:59.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:14:59.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:14:59.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:59.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:14:59.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:14:59.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:14:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:14:59.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:14:59.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:00.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:00.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:00.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:00.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:00.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:00.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:00.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:00.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:00.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:00.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:00.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 38515 38499 150.374 107.062 0.006316 0.00664464 2024-06-14T23:15:00.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:00.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:00.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:00.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:00.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:00.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:00.590 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 2] out_osds: [0, 1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:15:00.590 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:15:00.590 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T23:15:00.590 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-06-14T23:15:00.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 4 0 2024-06-14T23:15:00.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:00.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:00.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:00.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:00.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:00.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:00.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:00.955 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:00.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:01.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:01.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:01.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:01.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:01.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:01.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:00 smithi137 ceph-mon[90634]: pgmap v1343: 119 pgs: 119 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-06-14T23:15:01.295 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:00 smithi086 ceph-mon[97913]: pgmap v1343: 119 pgs: 119 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-06-14T23:15:01.295 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:00 smithi086 ceph-mon[102754]: pgmap v1343: 119 pgs: 119 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-06-14T23:15:01.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:01.331 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:15:01.332 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:15:01.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 16 40286 40270 148.039 110.688 0.00936403 0.00674931 2024-06-14T23:15:01.353 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:15:01.361 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:15:01.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:01.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:01.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:15:01.390 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:15:01.412 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:15:01.412 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:15:01.435 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:15:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:15:01.457 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:15:01.457 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:15:01.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:01.485 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:15:01.485 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:15:01.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:01.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:01.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:15:01.510 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:15:01.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:01.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:01.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:01.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:01.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:01.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:01.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:01.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:01.899 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:01.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:02.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:02.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:02.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:02.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:02.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:02.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:02.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3775517669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:02.187 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3775517669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:02.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3775517669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:02.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:02.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:02.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:02.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 41970 41954 145.662 105.25 0.00711136 0.00686048 2024-06-14T23:15:02.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:02.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:02.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:02.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:02.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:02.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:02.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:02.705 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:02.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:02.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:02.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:02.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:02.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:02.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:02.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:02.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:03.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:03.127 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:03.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:03.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:03.258 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:02 smithi086 ceph-mon[97913]: pgmap v1344: 119 pgs: 119 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-06-14T23:15:03.259 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:02 smithi086 ceph-mon[102754]: pgmap v1344: 119 pgs: 119 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-06-14T23:15:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:03.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:15:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:15:03.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 43632 43616 143.462 103.875 0.00888404 0.00696589 2024-06-14T23:15:03.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:03.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:03.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:03.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:02 smithi137 ceph-mon[90634]: pgmap v1344: 119 pgs: 119 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-06-14T23:15:03.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:03.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:03.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:03.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:03.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:03.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:03.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:03.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:03.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:03.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:03.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:03.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:03.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:04.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:04.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:04.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:04.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:04.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:04.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:04.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:04.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-14T23:15:04.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:04.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:04.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:04.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:15:04.337759+0000 min lat: 0.00100209 max lat: 0.0822968 avg lat: 0.00706946 2024-06-14T23:15:04.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:15:04.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 45255 45239 141.36 101.438 0.00726301 0.00706946 2024-06-14T23:15:04.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:04.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:04.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:04.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:04.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:04.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:04.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:04.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:04.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:04.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:04.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:04.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:04.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:04.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:04.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:04.998 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:15:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:15:04] "GET /metrics HTTP/1.1" 200 36671 "" "Prometheus/2.43.0" 2024-06-14T23:15:05.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:05.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:05.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:05.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:05.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:05.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:04 smithi086 ceph-mon[97913]: pgmap v1345: 119 pgs: 119 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-06-14T23:15:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:04 smithi086 ceph-mon[102754]: pgmap v1345: 119 pgs: 119 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-06-14T23:15:05.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 16 46875 46859 139.45 101.25 0.00692866 0.00716692 2024-06-14T23:15:05.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:05.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:05.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:05.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:15:05.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:04 smithi137 ceph-mon[90634]: pgmap v1345: 119 pgs: 119 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-06-14T23:15:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:05.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:05.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:05.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:05.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:05.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:05.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:05.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:05.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:05.915 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:05.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:06.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:06.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:06.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:06.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:06.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:06.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:06.214 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:06.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:06.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:06.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:06.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 48344 48328 137.284 91.8125 0.013321 0.00727945 2024-06-14T23:15:06.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:06.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:06.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:06.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:06.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:06.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:06.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:06.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:06.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:06.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:06.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:06.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:06.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:06.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:06.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:06.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:06.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:07.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:07.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:07.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:07.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:07.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:07.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:07.278 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:07 smithi086 ceph-mon[97913]: pgmap v1346: 119 pgs: 119 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-06-14T23:15:07.278 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:07 smithi086 ceph-mon[102754]: pgmap v1346: 119 pgs: 119 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-06-14T23:15:07.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 49881 49865 135.492 96.0625 0.00798183 0.00737555 2024-06-14T23:15:07.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:07.396 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:07.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:07.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:07.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:07 smithi137 ceph-mon[90634]: pgmap v1346: 119 pgs: 119 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-06-14T23:15:07.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:07.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:07.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:07.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:07.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:07.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:07.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:07.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:07.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:07.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:07.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:07.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:08.018 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.4 primary-affinity to 0 (802) 2024-06-14T23:15:08.022 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:15:08.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:08.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:08.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:08.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:08.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:08.195 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:08.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:08.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:08.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:08.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:08.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2465745703' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:15:08.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:08 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:15:08.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3478387090' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-06-14T23:15:08.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2465745703' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:15:08.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:08 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:15:08.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3478387090' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-06-14T23:15:08.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 50916 50900 132.541 64.6875 0.00210101 0.00744879 2024-06-14T23:15:08.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:08.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:08.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:08.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2465745703' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:15:08.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:08 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:15:08.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3478387090' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-06-14T23:15:08.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:08.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:08.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:08.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:08.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:08.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:08.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:08.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:08.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:08.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:08.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:08.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:08.975 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:09.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:09.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:09.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:09.128 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[97913]: pgmap v1347: 119 pgs: 119 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-06-14T23:15:09.129 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:15:09.129 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:15:09.129 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3478387090' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-06-14T23:15:09.129 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[97913]: osdmap e359: 8 total, 8 up, 6 in 2024-06-14T23:15:09.129 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[102754]: pgmap v1347: 119 pgs: 119 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-06-14T23:15:09.129 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:15:09.129 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:15:09.129 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3478387090' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-06-14T23:15:09.129 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:09 smithi086 ceph-mon[102754]: osdmap e359: 8 total, 8 up, 6 in 2024-06-14T23:15:09.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:09.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:09.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:09.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:09.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 51366 51350 128.365 28.125 0.0251462 0.00778508 2024-06-14T23:15:09.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:09.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:09.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:09.490 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:09.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:09.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:09.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:09 smithi137 ceph-mon[90634]: pgmap v1347: 119 pgs: 119 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-06-14T23:15:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:09 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:15:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:09 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:15:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3478387090' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-06-14T23:15:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:09 smithi137 ceph-mon[90634]: osdmap e359: 8 total, 8 up, 6 in 2024-06-14T23:15:09.542 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:15:09.543 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:15:09.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:09.620 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:09.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:09.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:09.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:09.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:09.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:09.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:09.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:10.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:10.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:10.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:10 smithi086 ceph-mon[97913]: osdmap e360: 8 total, 8 up, 6 in 2024-06-14T23:15:10.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:10 smithi086 ceph-mon[97913]: 1.16 scrub starts 2024-06-14T23:15:10.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:10 smithi086 ceph-mon[97913]: 1.16 scrub ok 2024-06-14T23:15:10.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:10 smithi086 ceph-mon[102754]: osdmap e360: 8 total, 8 up, 6 in 2024-06-14T23:15:10.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:10 smithi086 ceph-mon[102754]: 1.16 scrub starts 2024-06-14T23:15:10.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:10 smithi086 ceph-mon[102754]: 1.16 scrub ok 2024-06-14T23:15:10.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:10.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:10.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:10.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:10.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:10.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:10.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 52833 52817 126.954 91.6875 0.0108181 0.00787223 2024-06-14T23:15:10.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:10.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:10.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:10.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:10 smithi137 ceph-mon[90634]: osdmap e360: 8 total, 8 up, 6 in 2024-06-14T23:15:10.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:10 smithi137 ceph-mon[90634]: 1.16 scrub starts 2024-06-14T23:15:10.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:10 smithi137 ceph-mon[90634]: 1.16 scrub ok 2024-06-14T23:15:10.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:10.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:10.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:10.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:10.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:10.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:10.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:10.828 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:10.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:10.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:10.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:10.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:11.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:11.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:11.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:11.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:11.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:11.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:11.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:11 smithi086 ceph-mon[97913]: pgmap v1350: 119 pgs: 11 peering, 108 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 5.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:15:11.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:11 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 7 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-06-14T23:15:11.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:11 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:15:11.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:11 smithi086 ceph-mon[97913]: 1.5 scrub ok 2024-06-14T23:15:11.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:11 smithi086 ceph-mon[102754]: pgmap v1350: 119 pgs: 11 peering, 108 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 5.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:15:11.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:11 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 7 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-06-14T23:15:11.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:11 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:15:11.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:11 smithi086 ceph-mon[102754]: 1.5 scrub ok 2024-06-14T23:15:11.339 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 54255 54239 125.543 88.875 0.0129242 0.00796016 2024-06-14T23:15:11.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:11.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:11.397 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:11 smithi137 ceph-mon[90634]: pgmap v1350: 119 pgs: 11 peering, 108 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 5.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:15:11.397 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:11 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 7 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-06-14T23:15:11.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:11 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:15:11.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:11 smithi137 ceph-mon[90634]: 1.5 scrub ok 2024-06-14T23:15:11.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:11.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:11.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:11.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:11.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:11.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:11.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:11.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:11.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:11.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:11.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:11.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:11.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:12.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:12.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:12.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:12.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:12.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:12.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:12.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:12.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:12.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 55546 55530 123.941 80.6875 0.00773524 0.00806401 2024-06-14T23:15:12.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:12.407 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:12.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:12.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-14T23:15:12.510 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:12.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:12.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:12.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:15:12.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:12.703 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:12.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:12.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:12.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:12.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:12.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:12.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:12.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:13.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:13 smithi086 ceph-mon[97913]: pgmap v1351: 119 pgs: 11 peering, 108 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 5.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:15:13.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:13.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:13.101 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:13.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:13.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:13.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:13.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:13.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:13.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:13 smithi086 ceph-mon[102754]: pgmap v1351: 119 pgs: 11 peering, 108 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 5.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:15:13.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:13.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:13.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 56916 56900 122.619 85.625 0.0121758 0.00815074 2024-06-14T23:15:13.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:13.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:13.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:13.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:13.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:13 smithi137 ceph-mon[90634]: pgmap v1351: 119 pgs: 11 peering, 108 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 5.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:15:13.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:13.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:13.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:13.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:13.603 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:13.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:13.654 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:13.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:13.664 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:13.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:13.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:13.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:13.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:13.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:13.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:14.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:14.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:14.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:14.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:14.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:14.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:14.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:14.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:14.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:14.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:14 smithi086 ceph-mon[97913]: 1.e scrub starts 2024-06-14T23:15:14.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:14 smithi086 ceph-mon[97913]: 1.e scrub ok 2024-06-14T23:15:14.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:14 smithi086 ceph-mon[102754]: 1.e scrub starts 2024-06-14T23:15:14.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:14 smithi086 ceph-mon[102754]: 1.e scrub ok 2024-06-14T23:15:14.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 58300 58284 121.415 86.5 0.0102262 0.00823111 2024-06-14T23:15:14.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:14.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:14.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:14.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:14.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:14 smithi137 ceph-mon[90634]: 1.e scrub starts 2024-06-14T23:15:14.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:14 smithi137 ceph-mon[90634]: 1.e scrub ok 2024-06-14T23:15:14.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:14.582 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:14.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:14.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:14.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:14.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:14.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:14.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:14.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:14.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:14.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:14.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:14.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:15.037 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:15:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:15:14] "GET /metrics HTTP/1.1" 200 36681 "" "Prometheus/2.43.0" 2024-06-14T23:15:15.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:15.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:15.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:15.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:15.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:15.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:15.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:15 smithi086 ceph-mon[97913]: pgmap v1352: 119 pgs: 5 peering, 114 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 2.6 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 4.9 KiB/s, 0 objects/s recovering 2024-06-14T23:15:15.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:15 smithi086 ceph-mon[102754]: pgmap v1352: 119 pgs: 5 peering, 114 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 2.6 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 4.9 KiB/s, 0 objects/s recovering 2024-06-14T23:15:15.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 59647 59631 120.214 84.1875 0.0079903 0.00831364 2024-06-14T23:15:15.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:15.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:15.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:15.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:15 smithi137 ceph-mon[90634]: pgmap v1352: 119 pgs: 5 peering, 114 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 2.6 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 4.9 KiB/s, 0 objects/s recovering 2024-06-14T23:15:15.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:15.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:15.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:15.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:15.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:15.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:15.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:15.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:15.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:15.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:15.908 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 2] out_osds: [0, 1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:15:15.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 3 1 2024-06-14T23:15:15.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:15.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:16.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:16.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:16.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:16.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:16.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:16.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:16.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:16.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 60986 60970 119.072 83.6875 0.00819734 0.00839362 2024-06-14T23:15:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:16.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:16.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:16.428 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:16 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-06-14T23:15:16.429 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:16 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-06-14T23:15:16.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:16.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:16.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:16.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:16 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-06-14T23:15:16.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:16.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:16.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:16.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:16.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:16.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:16.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:16.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:16.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:16.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:17.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:17.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:17.099 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:15:17.100 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:15:17.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:17.129 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:15:17.129 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:15:17.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:17.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:17.153 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:15:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:15:17.175 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:15:17.175 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:15:17.204 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:15:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:15:17.230 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:15:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:15:17.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:17 smithi086 ceph-mon[97913]: pgmap v1353: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T23:15:17.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:17 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:15:17.231 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2927185069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:17.231 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:17 smithi086 ceph-mon[102754]: pgmap v1353: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T23:15:17.231 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:17 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:15:17.231 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2927185069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:17.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:17.258 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:15:17.259 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:15:17.287 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:15:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:15:17.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:17.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:17.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 16 62231 62215 117.822 77.8125 0.0105479 0.00848225 2024-06-14T23:15:17.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:17.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:17.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:17.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:17 smithi137 ceph-mon[90634]: pgmap v1353: 119 pgs: 119 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-06-14T23:15:17.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:17 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:15:17.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2927185069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:17.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:17.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:17.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:17.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:17.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:17.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:17.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:17.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:17.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:17.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:17.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:17.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:17.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:18.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:18.065 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:18.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:18.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:18.193 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:18.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:18.246 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:15:18.246 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:15:18.247 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:15:18.247 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:15:18.247 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:15:18.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:18.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 63424 63408 116.549 74.5625 0.0189309 0.00857486 2024-06-14T23:15:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:18.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:18.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:18.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:18.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:18.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:18.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:18.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:18.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:18.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:18.758 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:18.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:18.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:18.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:18.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:18.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:19.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:19.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:19.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-14T23:15:19.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:19.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:19.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:19.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:19.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:19.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:19.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:19 smithi086 ceph-mon[102754]: pgmap v1354: 119 pgs: 119 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-06-14T23:15:19.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:19 smithi086 ceph-mon[97913]: pgmap v1354: 119 pgs: 119 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-06-14T23:15:19.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 64609 64593 115.335 74.0625 0.00835227 0.00866476 2024-06-14T23:15:19.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:19.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:19.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:19.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:19 smithi137 ceph-mon[90634]: pgmap v1354: 119 pgs: 119 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-06-14T23:15:19.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:19.585 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:19.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:19.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:19.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:19.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:19.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:19.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:19.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:19.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:19.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:20.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:20.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:20.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:20.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:20.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:20.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:20.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:20.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:20.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:20.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 65747 65731 114.107 71.125 0.0129996 0.00875826 2024-06-14T23:15:20.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:20.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:20.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:20.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:20.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:20.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:20.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:20.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:20.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:20.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:20.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:20.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:20.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:20.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:20.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:20.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:20.966 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:21.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:21.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:21.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:21.104 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:15:21.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:21.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:21.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:21.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 66886 66870 112.947 71.1875 0.0143719 0.00884861 2024-06-14T23:15:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:21.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:21.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:21 smithi086 ceph-mon[102754]: pgmap v1355: 119 pgs: 119 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-14T23:15:21.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2278063687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:15:21.387 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:21 smithi086 ceph-mon[97913]: pgmap v1355: 119 pgs: 119 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-14T23:15:21.387 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2278063687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:15:21.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:21.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:21.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:21 smithi137 ceph-mon[90634]: pgmap v1355: 119 pgs: 119 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-14T23:15:21.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2278063687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:15:21.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:21.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:21.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:21.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:21.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:21.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:21.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:21.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:21.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:21.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:21.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:21.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:21.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:22.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:15:22.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:22.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:22.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:22.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:22.248 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:22.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:22.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:15:22.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:15:22.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2278063687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:15:22.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[97913]: osdmap e361: 8 total, 8 up, 6 in 2024-06-14T23:15:22.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[97913]: pgmap v1357: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-14T23:15:22.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:15:22.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:15:22.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2278063687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:15:22.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[102754]: osdmap e361: 8 total, 8 up, 6 in 2024-06-14T23:15:22.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:22 smithi086 ceph-mon[102754]: pgmap v1357: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-14T23:15:22.340 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 68030 68014 111.856 71.5 0.0130783 0.00893539 2024-06-14T23:15:22.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:22.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:22.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:22.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:22.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:22 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:15:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:22 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:15:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2278063687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:15:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:22 smithi137 ceph-mon[90634]: osdmap e361: 8 total, 8 up, 6 in 2024-06-14T23:15:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:22 smithi137 ceph-mon[90634]: pgmap v1357: 119 pgs: 119 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-14T23:15:22.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:22.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:22.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:22.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:22.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:22.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:22.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:22.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:22.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:22.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:22.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:22.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:23.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:23.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:23.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:23.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:23.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:23.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:23.303 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:23.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:23.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 16 69193 69177 110.851 72.6875 0.0116047 0.00901689 2024-06-14T23:15:23.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:23.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:23.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:23.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:23.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:23.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:23.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:23.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:23.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:23.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:23.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:23.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:24.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:24.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:24.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:24.133 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:24.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:24.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:24.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:24.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:24.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:24.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:24.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:24.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:24.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:24.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:24.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:24.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:15:24.339428+0000 min lat: 0.00100209 max lat: 1.03127 avg lat: 0.00910608 2024-06-14T23:15:24.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:15:24.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 70268 70252 109.76 67.1875 0.0130606 0.00910608 2024-06-14T23:15:24.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:24.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:24.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:24.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:24.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:24.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:24.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:24.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:24.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:24.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:24.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:24.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:24.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:24.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:24.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:24.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:24.942 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:15:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:15:24] "GET /metrics HTTP/1.1" 200 36681 "" "Prometheus/2.43.0" 2024-06-14T23:15:25.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:25.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:25.065 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:25.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:25.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:15:25.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:25.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:25.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:25.215 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:24 smithi086 ceph-mon[97913]: pgmap v1358: 119 pgs: 119 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T23:15:25.215 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:24 smithi086 ceph-mon[102754]: pgmap v1358: 119 pgs: 119 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T23:15:25.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:24 smithi137 ceph-mon[90634]: pgmap v1358: 119 pgs: 119 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-14T23:15:25.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:25.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:25.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 71455 71439 108.892 74.1875 0.013552 0.00917879 2024-06-14T23:15:25.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:25.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:25.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:25.501 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:15:25.501 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:15:25.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:15:25.501 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:15:25.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.501 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82010 11.b 11:d04237fd:::benchmark_data_smithi086_226760_object82009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.501 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.910681+0000", 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.57896383100001, 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.018046529, 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82010 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910681+0000", 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910681+0000", 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910679+0000", 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910712+0000", 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3068000000000003e-05 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910724+0000", 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1460999999999999e-05 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910732+0000", 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0180000000000001e-06 2024-06-14T23:15:25.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910986+0000", 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025449499999999998 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.911005+0000", 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8603000000000001e-05 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.917578+0000", 2024-06-14T23:15:25.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0065735409999999996 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.917587+0000", 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6200000000000005e-06 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.927350+0000", 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.009763497 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.928727+0000", 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001376866 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82017 11.9 11:99bbe177:::benchmark_data_smithi086_226760_object82016:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.910750+0000", 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.57889425099995, 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0179228979999999, 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.507 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82017 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910750+0000", 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910750+0000", 2024-06-14T23:15:25.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910750+0000", 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910776+0000", 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5797999999999999e-05 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910785+0000", 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7059999999999999e-06 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.910792+0000", 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3080000000000004e-06 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.911194+0000", 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00040272099999999998 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.911212+0000", 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7467999999999999e-05 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.918135+0000", 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069234769999999999 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.918143+0000", 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9929999999999995e-06 2024-06-14T23:15:25.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.927647+0000", 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0095035109999999 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.928673+0000", 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001026487 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82032 11.9 11:911528d0:::benchmark_data_smithi086_226760_object82031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.923203+0000", 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.56644194299997, 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.014084118, 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82032 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.923203+0000", 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.923203+0000", 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.923201+0000", 2024-06-14T23:15:25.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.923226+0000", 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4961e-05 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.923241+0000", 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4980999999999999e-05 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.923250+0000", 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7280000000000001e-06 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924546+0000", 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001295893 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924556+0000", 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0669999999999999e-05 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.929008+0000", 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0044517150000001 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.929398+0000", 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000389529 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.929506+0000", 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000107977 2024-06-14T23:15:25.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937109+0000", 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0076029569999999996 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937144+0000", 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5008000000000001e-05 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937200+0000", 2024-06-14T23:15:25.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6614999999999999e-05 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937253+0000", 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2808e-05 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937287+0000", 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3763999999999997e-05 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82034 11.9 11:9fc3c925:::benchmark_data_smithi086_226760_object82033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.924453+0000", 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.56519114900004, 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0186964569999999, 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82034 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924453+0000", 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924453+0000", 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924453+0000", 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924483+0000", 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0360000000000001e-05 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924495+0000", 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1789e-05 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924502+0000", 2024-06-14T23:15:25.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4839999999999998e-06 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924579+0000", 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7124000000000002e-05 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.924586+0000", 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0949999999999998e-06 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.931575+0000", 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.006988969 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.931690+0000", 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114758 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.931829+0000", 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013888400000000001 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937148+0000", 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005319341 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937322+0000", 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000174119 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943046+0000", 2024-06-14T23:15:25.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057238130000000003 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943105+0000", 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9157999999999998e-05 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943150+0000", 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4540999999999999e-05 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82041 11.9 11:9eb7e8d0:::benchmark_data_smithi086_226760_object82040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.930281+0000", 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.55936327699999, 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0129785570000001, 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82041 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.527 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930281+0000", 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930281+0000", 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930280+0000", 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930308+0000", 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8563000000000001e-05 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930334+0000", 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5967999999999999e-05 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930346+0000", 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1616e-05 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930415+0000", 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9152000000000005e-05 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930423+0000", 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5630000000000003e-06 2024-06-14T23:15:25.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.934343+0000", 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.003920025 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.934462+0000", 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119107 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.934625+0000", 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016265799999999999 2024-06-14T23:15:25.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937329+0000", 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002704087 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937638+0000", 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00030918100000000001 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943207+0000", 2024-06-14T23:15:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055689850000000003 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943241+0000", 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4632000000000003e-05 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943260+0000", 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8533e-05 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82045 11.9 11:93276e74:::benchmark_data_smithi086_226760_object82044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.930668+0000", 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.55897652700003, 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.015518575, 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82045 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.534 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930668+0000", 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930668+0000", 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930667+0000", 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930690+0000", 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2823000000000002e-05 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930704+0000", 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3995000000000001e-05 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930712+0000", 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6850000000000001e-06 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930758+0000", 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5825999999999998e-05 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.930769+0000", 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1571e-05 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937702+0000", 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0069331699999999 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937831+0000", 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012814099999999999 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937977+0000", 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000146849 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943297+0000", 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053192960000000003 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943325+0000", 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7991e-05 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946085+0000", 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00275991 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946135+0000", 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0000000000000002e-05 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946187+0000", 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2088e-05 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.540 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82049 11.b 11:d4f9bbad:::benchmark_data_smithi086_226760_object82048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.934788+0000", 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.55485675399996, 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0030785769999999, 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82049 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.934788+0000", 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.934788+0000", 2024-06-14T23:15:25.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.934786+0000", 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.934811+0000", 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5211999999999998e-05 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.934831+0000", 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9276999999999999e-05 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.934840+0000", 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3270000000000007e-06 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.934944+0000", 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010443299999999999 2024-06-14T23:15:25.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.934960+0000", 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5549999999999999e-05 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.929072+0000", 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99411225800000003 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.929471+0000", 2024-06-14T23:15:25.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00039842000000000002 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.929580+0000", 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108924 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937611+0000", 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080310980000000004 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937643+0000", 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2780999999999997e-05 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937689+0000", 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5643999999999999e-05 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937797+0000", 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010807299999999999 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937866+0000", 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9261e-05 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82050 11.9 11:96a6f45b:::benchmark_data_smithi086_226760_object82049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.936009+0000", 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.55363563200001, 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.012956492, 2024-06-14T23:15:25.548 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82050 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.936009+0000", 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.936009+0000", 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.936008+0000", 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.936030+0000", 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2291000000000001e-05 2024-06-14T23:15:25.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.936044+0000", 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3723000000000001e-05 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.936051+0000", 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.576e-06 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.936190+0000", 2024-06-14T23:15:25.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000138875 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.936207+0000", 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7266999999999999e-05 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.940514+0000", 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0043066430000001 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.940636+0000", 2024-06-14T23:15:25.552 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000122118 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.940803+0000", 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016677700000000001 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946253+0000", 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054503700000000004 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.553 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.948829+0000", 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002575726 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.948864+0000", 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4897999999999997e-05 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.948927+0000", 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3202999999999998e-05 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.948965+0000", 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8300000000000003e-05 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82061 11.b 11:d1a66fef:::benchmark_data_smithi086_226760_object82060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.944617+0000", 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.54502746799994, 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99943677900000005, 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.555 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82061 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.944617+0000", 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.944617+0000", 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.944616+0000", 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.944666+0000", 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0033e-05 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.944685+0000", 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9335999999999999e-05 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.944694+0000", 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6489999999999997e-06 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.944860+0000", 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016613500000000001 2024-06-14T23:15:25.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.944871+0000", 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0832000000000001e-05 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.932113+0000", 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98724226800000003 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.932295+0000", 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018175 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.932413+0000", 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011815399999999999 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937874+0000", 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054611060000000003 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937910+0000", 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6103000000000003e-05 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943932+0000", 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006021818 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.944019+0000", 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7119000000000004e-05 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.944054+0000", 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5080999999999998e-05 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82067 11.9 11:9dca8c01:::benchmark_data_smithi086_226760_object82066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.951490+0000", 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.53815456999996, 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0032526319999999, 2024-06-14T23:15:25.562 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82067 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.951490+0000", 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.951490+0000", 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.951488+0000", 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.951510+0000", 2024-06-14T23:15:25.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2337e-05 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.951524+0000", 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3633e-05 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.951532+0000", 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9130000000000003e-06 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.951595+0000", 2024-06-14T23:15:25.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3402e-05 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.951605+0000", 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5060000000000001e-06 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943347+0000", 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99174175799999997 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943463+0000", 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011592 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.943585+0000", 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000122006 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.948975+0000", 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053907429999999999 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.949031+0000", 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5302e-05 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.954659+0000", 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005628096 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.954709+0000", 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0414999999999998e-05 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.954743+0000", 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3612999999999998e-05 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82070 11.b 11:d3aace23:::benchmark_data_smithi086_226760_object82069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.956031+0000", 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.53361403400004, 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98815257199999995, 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.569 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.570 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.570 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82070 2024-06-14T23:15:25.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.570 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.570 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:25.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.956031+0000", 2024-06-14T23:15:25.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.956031+0000", 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.956029+0000", 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.956055+0000", 2024-06-14T23:15:25.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5789999999999999e-05 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.956075+0000", 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.05e-05 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.956083+0000", 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.537e-06 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.956154+0000", 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1626999999999997e-05 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.956165+0000", 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1062e-05 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.934870+0000", 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97870442099999999 2024-06-14T23:15:25.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.935075+0000", 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020542499999999999 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.935192+0000", 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011712 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.940970+0000", 2024-06-14T23:15:25.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057771949999999997 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.944086+0000", 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031169510000000002 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.944114+0000", 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7866999999999999e-05 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.944154+0000", 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9558999999999999e-05 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.944183+0000", 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9258999999999999e-05 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.577 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82078 11.9 11:96a75542:::benchmark_data_smithi086_226760_object82077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.965349+0000", 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.52429595800004, 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99471942099999999, 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82078 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.965349+0000", 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.965349+0000", 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.965347+0000", 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.965373+0000", 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5525e-05 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.965389+0000", 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6833999999999998e-05 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.965397+0000", 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6459999999999993e-06 2024-06-14T23:15:25.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.965464+0000", 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6950000000000001e-05 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.965514+0000", 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0173000000000002e-05 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946271+0000", 2024-06-14T23:15:25.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98075640500000005 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946379+0000", 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010866000000000001 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946507+0000", 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012744500000000001 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.954598+0000", 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080910979999999997 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.954785+0000", 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018673 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.959991+0000", 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052059630000000001 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.960030+0000", 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9047000000000001e-05 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.960068+0000", 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8501000000000003e-05 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.584 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82083 11.9 11:95d2932b:::benchmark_data_smithi086_226760_object82082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:03.969864+0000", 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.51978047399996, 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99298238299999997, 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82083 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.969864+0000", 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.969864+0000", 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.969862+0000", 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.586 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.969891+0000", 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8717e-05 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.969911+0000", 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0222000000000001e-05 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.969926+0000", 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.483e-05 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.970006+0000", 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9495999999999999e-05 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:03.970018+0000", 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2431e-05 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.949051+0000", 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97903342299999996 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.949155+0000", 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010328 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.949270+0000", 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011499500000000001 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.960108+0000", 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010838191 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.960172+0000", 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4148999999999999e-05 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.962760+0000", 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025885190000000001 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.962810+0000", 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9020999999999997e-05 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.962847+0000", 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7039000000000002e-05 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82117 11.b 11:d15565ff:::benchmark_data_smithi086_226760_object82116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:04.019318+0000", 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.47032698299995, 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92725679299999997, 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.591 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82117 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.019318+0000", 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.019318+0000", 2024-06-14T23:15:25.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.019316+0000", 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.019372+0000", 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6425000000000003e-05 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.593 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.019389+0000", 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6903999999999999e-05 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.019398+0000", 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9050000000000003e-06 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.019459+0000", 2024-06-14T23:15:25.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1216999999999993e-05 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.019471+0000", 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.19e-05 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.937935+0000", 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.91846433400000005 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.938039+0000", 2024-06-14T23:15:25.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010388 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.938180+0000", 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014079899999999999 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.944204+0000", 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060234310000000001 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.944640+0000", 2024-06-14T23:15:25.596 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00043616599999999998 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946419+0000", 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001779341 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946538+0000", 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119166 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946574+0000", 2024-06-14T23:15:25.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6248999999999998e-05 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82157 11.9 11:99798677:::benchmark_data_smithi086_226760_object82156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:04.089754+0000", 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.39989088300001, 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.87475122699999996, 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.598 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82157 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.089754+0000", 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.089754+0000", 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.089753+0000", 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.089793+0000", 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0365999999999999e-05 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.089806+0000", 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3039e-05 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.089811+0000", 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4709999999999998e-06 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.089856+0000", 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4554000000000001e-05 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.089864+0000", 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7689999999999996e-06 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.951929+0000", 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.86206494099999997 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.952016+0000", 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7427000000000002e-05 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.952119+0000", 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102916 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.960142+0000", 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080234659999999999 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.962752+0000", 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026099610000000001 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.962973+0000", 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000220158 2024-06-14T23:15:25.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.964455+0000", 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014822399999999999 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.964505+0000", 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0136000000000002e-05 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:82163 11.b 11:da45505f:::benchmark_data_smithi086_226760_object82162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:04.108641+0000", 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 561.38100320900003, 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.83811172599999995, 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 82163 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.605 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.108641+0000", 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.108641+0000", 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.108640+0000", 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.108660+0000", 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9789999999999999e-05 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.108671+0000", 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0971999999999999e-05 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.108676+0000", 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.356e-06 2024-06-14T23:15:25.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.108718+0000", 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1931999999999998e-05 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.108726+0000", 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1869999999999997e-06 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.940994+0000", 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.83226814100000002 2024-06-14T23:15:25.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.941104+0000", 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010979700000000001 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.941247+0000", 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014273700000000001 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.944208+0000", 2024-06-14T23:15:25.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029613719999999999 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946389+0000", 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021807599999999999 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946608+0000", 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021867999999999999 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946714+0000", 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106345 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:04.946753+0000", 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9125999999999998e-05 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:136530 11.1 11:8a041a7d:::benchmark_data_smithi086_226760_object9209:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:15:25.611 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:59.945417+0000", 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 505.54422763899998, 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.011934302, 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136530 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.945417+0000", 2024-06-14T23:15:25.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.945417+0000", 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.945415+0000", 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999981 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.945424+0000", 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.003e-06 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.945455+0000", 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1222e-05 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.945467+0000", 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1419e-05 2024-06-14T23:15:25.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.945529+0000", 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2168000000000005e-05 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.945536+0000", 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.075e-06 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.961271+0000", 2024-06-14T23:15:25.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015735361 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.961280+0000", 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1989999999999995e-06 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.955418+0000", 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99413812499999998 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.957351+0000", 2024-06-14T23:15:25.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001932816 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:136553 11.1 11:8ddb6caf:::benchmark_data_smithi086_226760_object9232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:59.956902+0000", 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 505.53274274299997, 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.009430968, 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.617 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136553 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.956902+0000", 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.956902+0000", 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.956900+0000", 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.956907+0000", 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8260000000000004e-06 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.619 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.958750+0000", 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001842619 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.958762+0000", 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1904e-05 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.961302+0000", 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002539917 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.961307+0000", 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1039999999999998e-06 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.957853+0000", 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99654618800000005 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.958535+0000", 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068226199999999999 2024-06-14T23:15:25.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.958606+0000", 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0939999999999995e-05 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.961394+0000", 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027876749999999999 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.966164+0000", 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047702810000000003 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.966249+0000", 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5309000000000006e-05 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.966333+0000", 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3604999999999997e-05 2024-06-14T23:15:25.623 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:136556 11.1 11:84582f12:::benchmark_data_smithi086_226760_object9235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:06:59.959692+0000", 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 505.52995288300002, 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0068310009999999, 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.624 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 136556 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.959692+0000", 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.959692+0000", 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.959690+0000", 2024-06-14T23:15:25.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.959697+0000", 2024-06-14T23:15:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7920000000000004e-06 2024-06-14T23:15:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.959710+0000", 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2670000000000001e-05 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.959717+0000", 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5170000000000002e-06 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.961324+0000", 2024-06-14T23:15:25.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016063659999999999 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:06:59.961328+0000", 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7509999999999999e-06 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.961439+0000", 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.000110359 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.961555+0000", 2024-06-14T23:15:25.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116463 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.961667+0000", 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111906 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.966346+0000", 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046783789999999999 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.966375+0000", 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9601999999999999e-05 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.966448+0000", 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2516999999999997e-05 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:07:00.966523+0000", 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5130999999999996e-05 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:211945 11.19 11:99c0c8fa:::benchmark_data_smithi086_226760_object84624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:08:29.068567+0000", 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 416.421077733, 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.17197489499999999, 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 211945 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.632 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.068567+0000", 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.068567+0000", 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:15:25.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.068566+0000", 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.068571+0000", 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5949999999999998e-06 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.068582+0000", 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0263e-05 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.068587+0000", 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2159999999999997e-06 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.068628+0000", 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1300000000000001e-05 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.068747+0000", 2024-06-14T23:15:25.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011860499999999999 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,6", 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.069016+0000", 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026917500000000002 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.070054+0000", 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001038214 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.070458+0000", 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000403738 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.240321+0000", 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.169863653 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.240374+0000", 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2911999999999998e-05 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:15:25.637 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:15:25.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:29.240542+0000", 2024-06-14T23:15:25.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016740400000000001 2024-06-14T23:15:25.638 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.638 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.638 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.638 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:15:25.638 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:15:25.638 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:15:25.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:25.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:25.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:25.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:25.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:25.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:25.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:25.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:25.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:25.952 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.3 to 1 (10000) 2024-06-14T23:15:26.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:26.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:26.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:26.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:26.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:26.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:26.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:26.209 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3471932851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:15:26.209 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3471932851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:15:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3471932851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:15:26.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:26.320 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:26.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:26.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:26.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 15 72627 72612 108.045 73.3125 0.0195705 0.00925109 2024-06-14T23:15:26.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:26.448 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:26.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:26.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:26.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:26.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:26.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:26.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:26.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:26.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:26.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:26.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:27.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:26 smithi086 ceph-mon[102754]: pgmap v1359: 119 pgs: 119 active+clean; 4.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-14T23:15:27.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3471932851' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:15:27.002 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:26 smithi086 ceph-mon[102754]: osdmap e362: 8 total, 8 up, 6 in 2024-06-14T23:15:27.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:26 smithi086 ceph-mon[97913]: pgmap v1359: 119 pgs: 119 active+clean; 4.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-14T23:15:27.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3471932851' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:15:27.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:26 smithi086 ceph-mon[97913]: osdmap e362: 8 total, 8 up, 6 in 2024-06-14T23:15:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:27.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:27.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:27.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:27.133 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:27.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:27.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:26 smithi137 ceph-mon[90634]: pgmap v1359: 119 pgs: 119 active+clean; 4.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-14T23:15:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3471932851' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:15:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:26 smithi137 ceph-mon[90634]: osdmap e362: 8 total, 8 up, 6 in 2024-06-14T23:15:27.261 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:27.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:27.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 73735 73719 107.141 69.1875 0.00486294 0.00932821 2024-06-14T23:15:27.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:27.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:27.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:27.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:27.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 4 1 2024-06-14T23:15:27.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:27.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:27.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:27.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:27.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:27.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:27.780 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:27.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:27.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:27.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:27.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:27.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:28.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:28.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:28.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:28.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:28.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:28.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:28.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:28.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:28.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:28.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 16 74822 74806 106.25 67.9375 0.00957388 0.00940706 2024-06-14T23:15:28.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:28.492 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:28.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:28.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:28.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:28.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:28.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:28 smithi137 ceph-mon[90634]: pgmap v1361: 119 pgs: 119 active+clean; 4.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-14T23:15:28.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:28.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:28.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:28.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:28.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:28 smithi086 ceph-mon[97913]: pgmap v1361: 119 pgs: 119 active+clean; 4.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-14T23:15:28.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:28.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:28.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:28 smithi086 ceph-mon[102754]: pgmap v1361: 119 pgs: 119 active+clean; 4.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-14T23:15:28.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:28.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:28.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:28.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-14T23:15:28.990 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:29.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:29.034 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:29.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:29.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:29.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:29.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:29.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:29.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:29.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:29.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 75904 75888 105.391 67.625 0.0148043 0.0094829 2024-06-14T23:15:29.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:29.446 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:29.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:29.455 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:15:29.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:29.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:29.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:29.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:29.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:29.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:29.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:29.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:29.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:29.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:29.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:30.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:30.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:30.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:30.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:30.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:30.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:30.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 16 76981 76965 104.563 67.3125 0.01041 0.00955885 2024-06-14T23:15:30.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:30.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:30.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:30.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:30.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:30.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:30.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:30.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:30.772 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:15:30.773 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:15:30.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:30.795 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:15:30.795 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:15:30.817 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:15:30.817 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:15:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:30.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:30.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:30.840 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:15:30.841 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:15:30.869 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:15:30.869 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:15:30.890 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:15:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:15:30.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:30.920 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:15:30.921 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:15:30.945 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:15:30.945 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:15:30.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:30.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:31.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:31.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:31.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:30 smithi086 ceph-mon[97913]: pgmap v1362: 119 pgs: 119 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-14T23:15:31.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2235203753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:31.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:30 smithi086 ceph-mon[102754]: pgmap v1362: 119 pgs: 119 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-14T23:15:31.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2235203753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:31.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:31.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:31.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:31.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:30 smithi137 ceph-mon[90634]: pgmap v1362: 119 pgs: 119 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-14T23:15:31.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2235203753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:31.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:31.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:31.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 78061 78045 103.775 67.5 0.0187033 0.00963148 2024-06-14T23:15:31.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:31.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:31.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:31.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:31.625 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:31.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:31.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:31.766 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:31.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:31.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:31.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:31.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:31.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:32.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:32.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:32.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:32.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:32.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:32.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:32.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:32.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:32.342 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 79136 79120 103.012 67.1875 0.0108049 0.00970266 2024-06-14T23:15:32.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:32.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:32.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:32.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:32.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:32.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:32.564 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:32.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:32.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:32.697 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:15:32.698 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:15:32.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:32.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:32.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:32.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:32.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:33.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:33.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:32 smithi086 ceph-mon[97913]: pgmap v1363: 119 pgs: 119 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-14T23:15:33.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:32 smithi086 ceph-mon[102754]: pgmap v1363: 119 pgs: 119 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-14T23:15:33.116 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:33.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:33.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:33.241 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:33.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:33.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:33.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:32 smithi137 ceph-mon[90634]: pgmap v1363: 119 pgs: 119 active+clean; 4.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-14T23:15:33.342 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 80185 80169 102.248 65.5625 0.00885802 0.00977538 2024-06-14T23:15:33.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:33.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:33.451 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-14T23:15:33.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:33.496 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:33.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:33.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:33.651 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:33.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:33.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:33.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:33.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:33.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:33.921 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:15:33.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:33.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:34.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:34.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:34.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:34.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:34.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:34.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:34.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:34.336 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:34.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:34.341 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 81175 81159 101.44 61.875 0.0078147 0.00985252 2024-06-14T23:15:34.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:34.462 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:34.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:34.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:34.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:34.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:34.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:34.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:34.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:34.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:34.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:34.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:34.838 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:15:34.852 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:34.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:34.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:34.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:34.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:35.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:34 smithi086 ceph-mon[97913]: pgmap v1364: 119 pgs: 119 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:15:35.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2503360203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:15:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:34 smithi086 ceph-mon[102754]: pgmap v1364: 119 pgs: 119 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:15:35.085 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2503360203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:15:35.085 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:15:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:15:34] "GET /metrics HTTP/1.1" 200 36673 "" "Prometheus/2.43.0" 2024-06-14T23:15:35.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:35.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:35.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:35.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:15:35.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:35.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:34 smithi137 ceph-mon[90634]: pgmap v1364: 119 pgs: 119 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:15:35.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2503360203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:15:35.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:35.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:35.342 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 82193 82177 100.699 63.625 0.00554311 0.00992549 2024-06-14T23:15:35.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:35.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:35.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:35.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:35.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:35.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:35.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:35.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:35.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:35.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:35.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:35.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:35.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:35.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:35.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:36.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:36.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:36.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:35 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:15:36.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2503360203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:15:36.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:35 smithi086 ceph-mon[97913]: osdmap e363: 8 total, 8 up, 6 in 2024-06-14T23:15:36.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:35 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:15:36.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2503360203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:15:36.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:35 smithi086 ceph-mon[102754]: osdmap e363: 8 total, 8 up, 6 in 2024-06-14T23:15:36.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:36.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:36.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:36.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:35 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:15:36.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2503360203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:15:36.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:35 smithi137 ceph-mon[90634]: osdmap e363: 8 total, 8 up, 6 in 2024-06-14T23:15:36.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:36.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:36.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:36.342 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 16 83229 83213 100.007 64.75 0.0167241 0.00999405 2024-06-14T23:15:36.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:36.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:36.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:36.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:36.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:36.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:36.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:36.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:36.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:36.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:36.853 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.4 to 1 (10000) 2024-06-14T23:15:36.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:36.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:36.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:37.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:37.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:37.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:36 smithi086 ceph-mon[97913]: pgmap v1366: 119 pgs: 119 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:15:37.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/352385337' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:15:37.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:36 smithi086 ceph-mon[102754]: pgmap v1366: 119 pgs: 119 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:15:37.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/352385337' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:15:37.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:37.132 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:37.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:37.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:36 smithi137 ceph-mon[90634]: pgmap v1366: 119 pgs: 119 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-14T23:15:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/352385337' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:15:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:37.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:37.342 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 16 84191 84175 99.2547 60.125 0.0163065 0.0100703 2024-06-14T23:15:37.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:37.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:37.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:37.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:37.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:37.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:37.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:37.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:37.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:37.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:37.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:37.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:37.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:37.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:37.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:38.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:15:38.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:38.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:38.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:38.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:38.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:38.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:38.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/352385337' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:15:38.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:37 smithi137 ceph-mon[90634]: osdmap e364: 8 total, 8 up, 6 in 2024-06-14T23:15:38.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:38.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/352385337' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:15:38.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:37 smithi086 ceph-mon[97913]: osdmap e364: 8 total, 8 up, 6 in 2024-06-14T23:15:38.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/352385337' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:15:38.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:37 smithi086 ceph-mon[102754]: osdmap e364: 8 total, 8 up, 6 in 2024-06-14T23:15:38.342 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 85152 85136 98.5288 60.0625 0.0149005 0.0101445 2024-06-14T23:15:38.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:38.398 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:38.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:38.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:38.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:38.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:38.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:38.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:38.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:38.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:38.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:38.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:38.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:38.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:38.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:38.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:39.029 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:15:39.029 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:15:39.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:39.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:39.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:39.104 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:38 smithi137 ceph-mon[90634]: pgmap v1368: 119 pgs: 119 active+clean; 5.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 949 op/s 2024-06-14T23:15:39.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:39.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:39.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:39.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:39.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:38 smithi086 ceph-mon[97913]: pgmap v1368: 119 pgs: 119 active+clean; 5.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 949 op/s 2024-06-14T23:15:39.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:38 smithi086 ceph-mon[102754]: pgmap v1368: 119 pgs: 119 active+clean; 5.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 59 MiB/s wr, 949 op/s 2024-06-14T23:15:39.342 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 86164 86148 97.8873 63.25 0.00764221 0.0102112 2024-06-14T23:15:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:39.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:39.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:39.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:39.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:39.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:39.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:39.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:39.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:39.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:39.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:39.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:39.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:39.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-14T23:15:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:39.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:39.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:39.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 5 1 2024-06-14T23:15:39.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:40.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:40.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:40.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:40.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:40.158 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:40.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:40.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:40.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:40.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:40.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 87107 87091 97.1908 58.9375 0.0219245 0.0102844 2024-06-14T23:15:40.367 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:15:40.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:40.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:40.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:40.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:40.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:40.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:40.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:40.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:40.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:40.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:40.713 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:40.760 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:40.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:40.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:40.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:40.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:40.979 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:40.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:41.042 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:40 smithi137 ceph-mon[90634]: pgmap v1369: 119 pgs: 119 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-14T23:15:41.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:41.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:41.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:41.201 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:40 smithi086 ceph-mon[97913]: pgmap v1369: 119 pgs: 119 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-14T23:15:41.201 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:40 smithi086 ceph-mon[102754]: pgmap v1369: 119 pgs: 119 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-14T23:15:41.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:41.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:41.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:41.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:41.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 88161 88145 96.6413 65.875 0.0104106 0.0103416 2024-06-14T23:15:41.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:41.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:41.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:41.492 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:41.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:41.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:41.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:41.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:41.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:41.773 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:41.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:41.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:41.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:41.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:42.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:42.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:42.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:42.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:42.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:42.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:42.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:42.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:42.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:42.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 16 89200 89184 96.0946 64.9375 0.0122627 0.0104019 2024-06-14T23:15:42.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:42.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:42.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:42.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:42.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:42.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:42.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:42.691 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:42.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:42.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:42.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:42.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:42.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:42.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:42.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:43.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:43.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:43.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:43.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:43.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:42 smithi137 ceph-mon[90634]: pgmap v1370: 119 pgs: 119 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 47 MiB/s wr, 751 op/s 2024-06-14T23:15:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:43.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:43.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:43.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:42 smithi086 ceph-mon[102754]: pgmap v1370: 119 pgs: 119 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 47 MiB/s wr, 751 op/s 2024-06-14T23:15:43.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:43.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:42 smithi086 ceph-mon[97913]: pgmap v1370: 119 pgs: 119 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 47 MiB/s wr, 751 op/s 2024-06-14T23:15:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:43.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:43.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 90185 90169 95.5093 61.5625 0.013846 0.0104657 2024-06-14T23:15:43.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:43.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:43.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:43.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:43.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:43.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:43.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:43.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:43.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:43.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:43.770 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:15:43.770 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:15:43.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:43.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:43.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:43.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:43.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:43.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:44.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:44.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:44.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:44.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:44.118 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:44.118 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:44.118 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:15:44.118 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:15:44.118 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:44.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:44.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:44.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:44.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:15:44.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:15:44.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:44.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:44.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:44.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:15:44.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:15:44.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:15:44.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:44.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:44.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:44.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:15:44.341607+0000 min lat: 0.00100209 max lat: 1.03127 avg lat: 0.0105314 2024-06-14T23:15:44.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:15:44.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 91139 91123 94.9111 59.625 0.0149646 0.0105314 2024-06-14T23:15:44.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:44.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:44.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:44.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:44.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:44.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:44.583 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:15:44.583 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:15:44.583 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:15:44.583 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:15:44.583 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:15:44.583 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:15:44.583 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:15:44.583 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:15:44.584 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:15:44.584 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:15:44.584 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:15:44.584 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:15:44.584 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:15:44.584 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:15:44.584 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:15:44.584 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:15:44.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:44.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:44.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:44.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T23:15:44.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:44.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:44.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:44.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:44.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:44.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:44.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:45.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:45.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:45.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:45.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:45.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:45.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:45.186 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:15:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:44 smithi137 ceph-mon[90634]: pgmap v1371: 119 pgs: 119 active+clean; 5.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-14T23:15:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2009032081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:45.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:45.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:45.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:45.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:44 smithi086 ceph-mon[97913]: pgmap v1371: 119 pgs: 119 active+clean; 5.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-14T23:15:45.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2009032081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:45.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:15:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:15:44] "GET /metrics HTTP/1.1" 200 36674 "" "Prometheus/2.43.0" 2024-06-14T23:15:45.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:44 smithi086 ceph-mon[102754]: pgmap v1371: 119 pgs: 119 active+clean; 5.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-14T23:15:45.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2009032081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:45.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 91976 91960 94.2127 52.3125 0.0167688 0.0106091 2024-06-14T23:15:45.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:45.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:45.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:45.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:45.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:45.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:45.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:45.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:45.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:45.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:45.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:45.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:45.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:45.979 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:45.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:46.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:46.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:46.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:46.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:46.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:46.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:46.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:46.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 93015 92999 93.7404 64.9375 0.0232489 0.010663 2024-06-14T23:15:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:46.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:46.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:46.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:46.503 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:46.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:46.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:46.630 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:46.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:46.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:46.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:46.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:46.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:46.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:46.959 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:15:47.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:47.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:47.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:47.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:47.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:47.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:47.180 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:46 smithi137 ceph-mon[90634]: pgmap v1372: 119 pgs: 119 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-14T23:15:47.180 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/933323264' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:15:47.180 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:46 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:15:47.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:47.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:46 smithi086 ceph-mon[97913]: pgmap v1372: 119 pgs: 119 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-14T23:15:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/933323264' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:15:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:46 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:15:47.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:46 smithi086 ceph-mon[102754]: pgmap v1372: 119 pgs: 119 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-14T23:15:47.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/933323264' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:15:47.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:46 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:15:47.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 94048 94032 93.2772 64.5625 0.00902939 0.0107151 2024-06-14T23:15:47.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:47.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:47.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:47.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:47.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:47.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:47.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:47.666 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:15:47.666 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:15:47.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.666 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:94347 13.5 13:af97acf7:::benchmark_data_smithi086_299538_object94346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-14T23:15:47.666 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.649936+0000", 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.011734453000000001, 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011769553, 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 94347 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.649936+0000", 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.649936+0000", 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.649934+0000", 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.649968+0000", 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3639000000000001e-05 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.649982+0000", 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4316e-05 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.649988+0000", 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6969999999999998e-06 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.650125+0000", 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013673400000000001 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.650243+0000", 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117997 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.650383+0000", 2024-06-14T23:15:47.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014012200000000001 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.653635+0000", 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032522580000000001 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655132+0000", 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014970039999999999 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46844.0:94349 13.8 e365/351 13:1d32950b:::benchmark_data_smithi086_299538_object94348:head v 365'5863, mlcod=365'5861)", 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.650866+0000", 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.010804233, 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010926312000000001, 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.650866+0000", 2024-06-14T23:15:47.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.650866+0000", 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.650866+0000", 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.650902+0000", 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6535000000000002e-05 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.650913+0000", 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0944000000000001e-05 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.650929+0000", 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5934999999999999e-05 2024-06-14T23:15:47.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.653337+0000", 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024077539999999998 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.653356+0000", 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8791999999999999e-05 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:94351 13.9 13:9d132412:::benchmark_data_smithi086_299538_object94350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.651248+0000", 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.01042241, 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010608088999999999, 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 94351 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.676 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651248+0000", 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651248+0000", 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651247+0000", 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651276+0000", 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8989e-05 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651288+0000", 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2371000000000001e-05 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651293+0000", 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4929999999999998e-06 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651629+0000", 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033553000000000001 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651736+0000", 2024-06-14T23:15:47.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106591 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651861+0000", 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012529500000000001 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.654803+0000", 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029423090000000002 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:47.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:15:47.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655156+0000", 2024-06-14T23:15:47.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000353013 2024-06-14T23:15:47.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.682 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.682 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:94352 13.6 13:64ddeb62:::benchmark_data_smithi086_299538_object94351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.651309+0000", 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.010361462, 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010636092, 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 94352 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651309+0000", 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651309+0000", 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651308+0000", 2024-06-14T23:15:47.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651335+0000", 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6954999999999999e-05 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651342+0000", 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8419999999999999e-06 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.651346+0000", 2024-06-14T23:15:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7620000000000001e-06 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655243+0000", 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038968689999999999 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655370+0000", 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126938 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:15:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655524+0000", 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015451000000000001 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46844.0:94353 13.8 e365/351 13:142c9f08:::benchmark_data_smithi086_299538_object94352:head v 365'5864, mlcod=365'5861)", 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.653569+0000", 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0081010070000000003, 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008444577, 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.653569+0000", 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.653569+0000", 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.653569+0000", 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.653653+0000", 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4253999999999997e-05 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.653660+0000", 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2799999999999998e-06 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.653664+0000", 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8789999999999996e-06 2024-06-14T23:15:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656347+0000", 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026824349999999999 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656373+0000", 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.641e-05 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:94356 13.2 13:4343c18e:::benchmark_data_smithi086_299538_object94355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.655132+0000", 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.006537724, 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0069410390000000004, 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 94356 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655132+0000", 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655132+0000", 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655131+0000", 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655164+0000", 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2907000000000002e-05 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655177+0000", 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.27e-05 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655183+0000", 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.993e-06 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655233+0000", 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0380000000000002e-05 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655371+0000", 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000138174 2024-06-14T23:15:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.655524+0000", 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000152931 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.658315+0000", 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027909039999999999 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:94357 13.c 13:3414127c:::benchmark_data_smithi086_299538_object94356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.656217+0000", 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0054529950000000004, 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059305260000000002, 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 94357 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656217+0000", 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656217+0000", 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656216+0000", 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656250+0000", 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3841999999999999e-05 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656266+0000", 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6518e-05 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656273+0000", 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9380000000000003e-06 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656621+0000", 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000347549 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656728+0000", 2024-06-14T23:15:47.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107376 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656854+0000", 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125413 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.660080+0000", 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032259979999999999 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:94359 13.e 13:7b05323e:::benchmark_data_smithi086_299538_object94358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.656428+0000", 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.005242203, 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057925700000000004, 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 94359 2024-06-14T23:15:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656428+0000", 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656428+0000", 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656427+0000", 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656481+0000", 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4007e-05 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656499+0000", 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7815000000000001e-05 2024-06-14T23:15:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.656505+0000", 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9009999999999999e-06 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.658092+0000", 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0015871539999999999 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.658198+0000", 2024-06-14T23:15:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106221 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.658334+0000", 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000135716 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46844.0:94354 13.8 e365/351)", 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.660188+0000", 2024-06-14T23:15:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0014819989999999999, 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020917370000000002, 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.660188+0000", 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.660188+0000", 2024-06-14T23:15:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.660187+0000", 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.660222+0000", 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4604999999999998e-05 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.660234+0000", 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1609e-05 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.660239+0000", 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8849999999999998e-06 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46844.0:94362 13.3 e365/359 13:c535f7a8:::benchmark_data_smithi086_299538_object94361:head v 365'5948, mlcod=365'5946)", 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.662076+0000", 2024-06-14T23:15:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4294967295.9995942, 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024721299999999998, 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.662076+0000", 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.662076+0000", 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.662075+0000", 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.662108+0000", 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2971e-05 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.662131+0000", 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2682999999999999e-05 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.662138+0000", 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7150000000000001e-06 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.662202+0000", 2024-06-14T23:15:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4702000000000005e-05 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.662220+0000", 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7992e-05 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:94344 13.5 13:a0f5088d:::benchmark_data_smithi086_299538_object94343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:15:47.642817+0000", 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.018853590999999999, 2024-06-14T23:15:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019563616999999998, 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 94344 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.642817+0000", 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.642817+0000", 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.642816+0000", 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:15:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.642899+0000", 2024-06-14T23:15:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3220000000000006e-05 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.642912+0000", 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3016999999999999e-05 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.642927+0000", 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4687e-05 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.642971+0000", 2024-06-14T23:15:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4515999999999999e-05 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.643092+0000", 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120579 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.643228+0000", 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013640599999999999 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.647198+0000", 2024-06-14T23:15:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039693430000000002 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:15:47.648829+0000", 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00163142 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 11 2024-06-14T23:15:47.718 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:15:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:47.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:47.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:47.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:47.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:47.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:47.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:48.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:48.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:48.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:48.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:48.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:48.224 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:47 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:15:48.224 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:47 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:15:48.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:47 smithi086 ceph-mon[97913]: osdmap e365: 8 total, 8 up, 6 in 2024-06-14T23:15:48.225 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:47 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:15:48.225 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:47 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:15:48.225 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:47 smithi086 ceph-mon[102754]: osdmap e365: 8 total, 8 up, 6 in 2024-06-14T23:15:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:47 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:15:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:47 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:15:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:47 smithi137 ceph-mon[90634]: osdmap e365: 8 total, 8 up, 6 in 2024-06-14T23:15:48.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:48.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:48.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:48.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 95054 95038 92.8021 62.875 0.0144798 0.0107714 2024-06-14T23:15:48.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:48.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:48.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:48.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:48.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:48.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:48.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:48.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:48.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:48.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:48.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:48.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:48.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:48.964 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.5 to 1 (10000) 2024-06-14T23:15:49.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:49.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:49.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:49.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:49.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:49.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:49.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:48 smithi137 ceph-mon[90634]: pgmap v1374: 119 pgs: 119 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 62 MiB/s wr, 993 op/s 2024-06-14T23:15:49.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1576580853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:15:49.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:48 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:15:49.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:49.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:49.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:48 smithi086 ceph-mon[97913]: pgmap v1374: 119 pgs: 119 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 62 MiB/s wr, 993 op/s 2024-06-14T23:15:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1576580853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:15:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:48 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:15:49.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:48 smithi086 ceph-mon[102754]: pgmap v1374: 119 pgs: 119 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 62 MiB/s wr, 993 op/s 2024-06-14T23:15:49.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1576580853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:15:49.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:48 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:15:49.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 96056 96040 92.3378 62.625 0.0127087 0.0108248 2024-06-14T23:15:49.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:49.472 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:49.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:49.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:15:49.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:49.626 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:49.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:49.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:49.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:49.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:49.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:49.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:49.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:49.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:50.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-14T23:15:50.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:15:50.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:50.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:50.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 6 1 2024-06-14T23:15:50.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:50.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:50.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:50.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:50.226 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:50.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:49 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:15:50.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:49 smithi137 ceph-mon[90634]: osdmap e366: 8 total, 8 up, 6 in 2024-06-14T23:15:50.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:50.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:49 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:15:50.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:49 smithi086 ceph-mon[97913]: osdmap e366: 8 total, 8 up, 6 in 2024-06-14T23:15:50.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:49 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:15:50.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:49 smithi086 ceph-mon[102754]: osdmap e366: 8 total, 8 up, 6 in 2024-06-14T23:15:50.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:50.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:50.343 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 97070 97054 91.8989 63.375 0.0307268 0.0108768 2024-06-14T23:15:50.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:50.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:50.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:50.529 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:15:50.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:50.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:50.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:50.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:50.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:50.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:50.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:50.864 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:50.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:50.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:50.990 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:50.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:51.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:50 smithi086 ceph-mon[97913]: pgmap v1376: 119 pgs: 119 active+clean; 5.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-14T23:15:51.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:50 smithi086 ceph-mon[102754]: pgmap v1376: 119 pgs: 119 active+clean; 5.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-14T23:15:51.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:51.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:51.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:50 smithi137 ceph-mon[90634]: pgmap v1376: 119 pgs: 119 active+clean; 5.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-14T23:15:51.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:51.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:51.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 98067 98051 91.4572 62.3125 0.0197218 0.0109291 2024-06-14T23:15:51.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:51.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:51.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:51.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:51.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:51.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:51.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:51.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:51.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:51.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:51.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:51.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:51.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:51.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:51.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:52.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:52.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:52.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:52.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:52.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:52.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 99076 99060 91.0396 63.0625 0.0154983 0.0109797 2024-06-14T23:15:52.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:52.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:52.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:52.416 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:15:52.417 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:15:52.438 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:15:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:15:52.459 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:15:52.459 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:15:52.480 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:15:52.480 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:15:52.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:52.510 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:15:52.510 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:15:52.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:52.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:52.539 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:15:52.539 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:15:52.566 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:15:52.566 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:15:52.594 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:15:52.594 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:15:52.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:52.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:52.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:52.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:52.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:52.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:52.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:52.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:52.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:52.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:53.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:53.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:53.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:53.084 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:52 smithi137 ceph-mon[90634]: pgmap v1377: 119 pgs: 119 active+clean; 5.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 45 MiB/s wr, 726 op/s 2024-06-14T23:15:53.084 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:52 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3797597393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:53.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:53.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:53.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:53.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:53.296 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:52 smithi086 ceph-mon[97913]: pgmap v1377: 119 pgs: 119 active+clean; 5.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 45 MiB/s wr, 726 op/s 2024-06-14T23:15:53.296 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:52 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3797597393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:53.296 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:52 smithi086 ceph-mon[102754]: pgmap v1377: 119 pgs: 119 active+clean; 5.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 45 MiB/s wr, 726 op/s 2024-06-14T23:15:53.296 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:52 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3797597393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:15:53.299 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:15:53.299 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:15:53.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:53.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:53.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 100078 100062 90.6277 62.625 0.0113761 0.0110297 2024-06-14T23:15:53.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:53.443 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:53.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:53.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:53.569 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:53.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:53.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:53.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:53.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:53.812 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:53.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:53.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:53.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:53.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:53.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:53.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:53.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:54.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-14T23:15:54.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:54.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:54.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:54.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:54.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:54.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:54.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 101115 101099 90.2589 64.8125 0.0242571 0.0110746 2024-06-14T23:15:54.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:54.471 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:54.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:54.556 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:15:54.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:54.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:54.599 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:54.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:54.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:54.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:54.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:54.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:54.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:54.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:54.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:54.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:55.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:55.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:55.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:55.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:55.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:55.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:55.201 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:54 smithi137 ceph-mon[90634]: pgmap v1378: 119 pgs: 119 active+clean; 6.0 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:15:55.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:55.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:55.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:54 smithi086 ceph-mon[97913]: pgmap v1378: 119 pgs: 119 active+clean; 6.0 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:15:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:54 smithi086 ceph-mon[102754]: pgmap v1378: 119 pgs: 119 active+clean; 6.0 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:15:55.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:15:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:15:54] "GET /metrics HTTP/1.1" 200 36674 "" "Prometheus/2.43.0" 2024-06-14T23:15:55.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 102112 102096 89.8652 62.3125 0.00453017 0.011123 2024-06-14T23:15:55.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:55.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:55.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:55.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:55.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:55.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:55.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:55.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:55.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:55.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:55.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:55.870 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:55.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:55.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:55.995 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:55.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:56.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:56.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:56.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:56.162 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.6 to 1 (10000) 2024-06-14T23:15:56.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:56.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2561686110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:15:56.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:55 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:15:56.246 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:56.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:56.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:15:56.312 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:55 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2561686110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:15:56.312 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:55 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:15:56.312 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:55 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2561686110' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:15:56.312 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:55 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:15:56.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 103088 103072 89.4642 61 0.0105551 0.0111731 2024-06-14T23:15:56.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:56.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:56.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:56.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:56.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:56.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:56.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:56.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:56.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:56.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:56.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:56.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:56.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:56.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:56.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:56.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:57.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:57.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:57.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:57.087 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:56 smithi137 ceph-mon[90634]: pgmap v1379: 119 pgs: 119 active+clean; 6.1 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-14T23:15:57.087 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:56 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:15:57.087 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:56 smithi137 ceph-mon[90634]: osdmap e367: 8 total, 8 up, 6 in 2024-06-14T23:15:57.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:57.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:57.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:57.255 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:15:57.255 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:15:57.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:57.326 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:57.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:57.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:56 smithi086 ceph-mon[97913]: pgmap v1379: 119 pgs: 119 active+clean; 6.1 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-14T23:15:57.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:56 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:15:57.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:56 smithi086 ceph-mon[97913]: osdmap e367: 8 total, 8 up, 6 in 2024-06-14T23:15:57.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:56 smithi086 ceph-mon[102754]: pgmap v1379: 119 pgs: 119 active+clean; 6.1 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-14T23:15:57.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:56 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:15:57.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:56 smithi086 ceph-mon[102754]: osdmap e367: 8 total, 8 up, 6 in 2024-06-14T23:15:57.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 104080 104064 89.0879 62 0.0115471 0.011221 2024-06-14T23:15:57.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:57.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:57.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:15:57.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 7 1 2024-06-14T23:15:57.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:57.632 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:57.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:57.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:57.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:57.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:57.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:57.896 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:57.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:57.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:15:58.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-14T23:15:58.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:58.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:15:58.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:58.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:58.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:58.228 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:58.243 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:15:58.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:58.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:58.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:58.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:58.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:15:58.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 16 105093 105077 88.7395 63.3125 0.021226 0.0112644 2024-06-14T23:15:58.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:58.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:58.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:58.503 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:15:58.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:58.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:58.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:58.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:58.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:58.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:58.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:58.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:58.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:58.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:58.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:58.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:59.014 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:15:59.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:15:59.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:59.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:15:59.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:15:59.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:59.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:15:59.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:58 smithi086 ceph-mon[97913]: pgmap v1381: 119 pgs: 119 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 51 MiB/s wr, 821 op/s 2024-06-14T23:15:59.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:15:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2927313994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:15:59.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:58 smithi086 ceph-mon[102754]: pgmap v1381: 119 pgs: 119 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 51 MiB/s wr, 821 op/s 2024-06-14T23:15:59.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:15:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2927313994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:15:59.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 106087 106071 88.3846 62.125 0.0186226 0.0113097 2024-06-14T23:15:59.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:15:59.420 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:59.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:15:59.480 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:58 smithi137 ceph-mon[90634]: pgmap v1381: 119 pgs: 119 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 51 MiB/s wr, 821 op/s 2024-06-14T23:15:59.480 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:15:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2927313994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:15:59.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:15:59.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:59.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:15:59.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:15:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:59.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:15:59.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:15:59.816 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:15:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:59.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:15:59.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:15:59.958 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:15:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:15:59.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:00.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:00.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:00.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:00.186 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:00 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:00.186 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2927313994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:16:00.186 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:00 smithi137 ceph-mon[90634]: osdmap e368: 8 total, 8 up, 6 in 2024-06-14T23:16:00.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:00.212 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:00.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:00.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:00.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:00 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:00.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2927313994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:16:00.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:00 smithi086 ceph-mon[102754]: osdmap e368: 8 total, 8 up, 6 in 2024-06-14T23:16:00.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:00 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:00.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2927313994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:16:00.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:00 smithi086 ceph-mon[97913]: osdmap e368: 8 total, 8 up, 6 in 2024-06-14T23:16:00.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:00.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:00.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 107124 107108 88.0744 64.8125 0.00761126 0.01135 2024-06-14T23:16:00.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:00.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:00.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:00.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:00.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:00.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:00.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:00.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:00.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:00.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:00.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:00.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:00.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:00.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:00.948 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:16:00.948 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:16:00.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:00.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:01.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:01.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:01.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:01.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:01.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:01.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:01.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:01 smithi137 ceph-mon[90634]: pgmap v1383: 119 pgs: 119 active+clean; 6.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T23:16:01.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:01 smithi086 ceph-mon[97913]: pgmap v1383: 119 pgs: 119 active+clean; 6.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T23:16:01.247 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:01 smithi086 ceph-mon[102754]: pgmap v1383: 119 pgs: 119 active+clean; 6.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-14T23:16:01.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:01.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 16 108067 108051 87.696 58.9375 0.012135 0.0113986 2024-06-14T23:16:01.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:01.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:01.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:01.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:01.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:01.539 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:16:01.539 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:16:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:16:01.562 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:16:01.582 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:16:01.582 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:16:01.605 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:16:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:16:01.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:01.634 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:16:01.634 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:16:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:01.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:01.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:16:01.660 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:16:01.686 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:16:01.686 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:16:01.718 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:16:01.719 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:16:01.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:01.769 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:01.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:01.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-14T23:16:01.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:01.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:01.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:01.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:02.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:02.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:02.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:02.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:02.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:02.249 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:02 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3413973659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:02.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:02.327 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:02.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:02.331 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:16:02.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:02 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3413973659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:02.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:02 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3413973659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:02.344 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 109057 109041 87.3649 61.875 0.0211521 0.0114418 2024-06-14T23:16:02.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:02.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:02.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:02.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:02.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:02.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:02.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:02.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:02.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:02.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:02.869 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:02.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:02.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:02.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:02.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:16:03.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:03.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:03.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:03.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:03.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:03.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:03.201 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:03 smithi086 ceph-mon[97913]: pgmap v1384: 119 pgs: 119 active+clean; 6.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 47 MiB/s wr, 757 op/s 2024-06-14T23:16:03.201 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:03 smithi086 ceph-mon[102754]: pgmap v1384: 119 pgs: 119 active+clean; 6.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 47 MiB/s wr, 757 op/s 2024-06-14T23:16:03.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:03.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:03 smithi137 ceph-mon[90634]: pgmap v1384: 119 pgs: 119 active+clean; 6.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 47 MiB/s wr, 757 op/s 2024-06-14T23:16:03.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:03.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:03.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 110084 110068 87.0714 64.1875 0.00820835 0.0114807 2024-06-14T23:16:03.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:03.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:03.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:03.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:03.515 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:03.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:03.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:03.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:03.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:03.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:03.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:03.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:03.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:03.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:03.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:03.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:04.038 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.7 to 1 (10000) 2024-06-14T23:16:04.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:04.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:04 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2322352395' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:16:04.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:04 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2322352395' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:16:04.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:04.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:04.199 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:04 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2322352395' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:16:04.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:04.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:04.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:04.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:16:04.343169+0000 min lat: 0.00100209 max lat: 1.03127 avg lat: 0.0115214 2024-06-14T23:16:04.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:16:04.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 111077 111061 86.7588 62.0625 0.0184094 0.0115214 2024-06-14T23:16:04.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:04.363 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:16:04.364 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:16:04.364 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:16:04.364 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:16:04.364 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:16:04.376 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:04.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:04.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:04.504 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:04.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:04.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:04.643 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:04.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:04.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:04.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:04.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:04.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:04.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:04.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:04.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:16:04.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:05.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:05.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:05.038 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:16:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:16:04] "GET /metrics HTTP/1.1" 200 36674 "" "Prometheus/2.43.0" 2024-06-14T23:16:05.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 2 1 2024-06-14T23:16:05.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:05.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:05.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:05.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:05.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:05.335 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:05 smithi137 ceph-mon[90634]: pgmap v1385: 119 pgs: 119 active+clean; 6.6 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-14T23:16:05.335 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2322352395' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:16:05.335 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:05 smithi137 ceph-mon[90634]: osdmap e369: 8 total, 8 up, 6 in 2024-06-14T23:16:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:05 smithi086 ceph-mon[97913]: pgmap v1385: 119 pgs: 119 active+clean; 6.6 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-14T23:16:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2322352395' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:16:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:05 smithi086 ceph-mon[97913]: osdmap e369: 8 total, 8 up, 6 in 2024-06-14T23:16:05.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:05 smithi086 ceph-mon[102754]: pgmap v1385: 119 pgs: 119 active+clean; 6.6 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-14T23:16:05.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2322352395' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:16:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:05 smithi086 ceph-mon[102754]: osdmap e369: 8 total, 8 up, 6 in 2024-06-14T23:16:05.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 112109 112093 86.4839 64.5 0.00524109 0.0115582 2024-06-14T23:16:05.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:05.451 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:05.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:05.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:05.579 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:05.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:05.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:05.721 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:05.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:05.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:05.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:05.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:05.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:05.992 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:05.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:06.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-14T23:16:06.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:06.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:06.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:06.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:06.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:06.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:06.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:06.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 112979 112963 86.0923 54.375 0.00392433 0.0116105 2024-06-14T23:16:06.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:06.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:06.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:06.556 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:16:06.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:06.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:06.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:06.716 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:06.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:06.763 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:06.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:06.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:06.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:06.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:06.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:07.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:07.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:07.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:07.059 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:16:07.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:07.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:07.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:07 smithi137 ceph-mon[90634]: pgmap v1387: 119 pgs: 119 active+clean; 6.7 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-14T23:16:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1597896935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:07 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:07.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:07.298 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:07 smithi086 ceph-mon[97913]: pgmap v1387: 119 pgs: 119 active+clean; 6.7 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-14T23:16:07.298 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1597896935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:07.299 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:07 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:07.299 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:07 smithi086 ceph-mon[102754]: pgmap v1387: 119 pgs: 119 active+clean; 6.7 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-14T23:16:07.299 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1597896935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:07.299 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:07 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:07.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:07.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:07.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 16 113953 113937 85.7884 60.875 0.0083445 0.0116523 2024-06-14T23:16:07.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:07.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:07.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:07.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:07.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:07.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:07.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:07.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:07.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:07.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:07.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:07.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:07.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:07.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:07.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:07.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:08.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:08.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:08.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:08.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:08.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:08 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:16:08.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:08 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:16:08.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:08 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:16:08.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:08 smithi137 ceph-mon[90634]: osdmap e370: 8 total, 8 up, 6 in 2024-06-14T23:16:08.248 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:08.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:08.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:08.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:08 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:16:08.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:08 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:16:08.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:08 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:16:08.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:08 smithi086 ceph-mon[97913]: osdmap e370: 8 total, 8 up, 6 in 2024-06-14T23:16:08.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:08 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:16:08.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:08 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:16:08.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:08 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:16:08.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:08 smithi086 ceph-mon[102754]: osdmap e370: 8 total, 8 up, 6 in 2024-06-14T23:16:08.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 115012 114996 85.555 66.1875 0.01769 0.0116839 2024-06-14T23:16:08.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:08.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:08.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:08.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:08.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:08.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:08.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:08.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:08.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:08.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:08.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:08.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:08.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:08.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:08.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:09.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:09.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:09.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:09.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:09.103 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:09 smithi086 ceph-mon[97913]: pgmap v1389: 119 pgs: 119 active+clean; 6.8 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 55 MiB/s wr, 887 op/s 2024-06-14T23:16:09.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:09 smithi086 ceph-mon[102754]: pgmap v1389: 119 pgs: 119 active+clean; 6.8 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 55 MiB/s wr, 887 op/s 2024-06-14T23:16:09.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:09.202 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:09.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:09.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:09.332 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:09 smithi137 ceph-mon[90634]: pgmap v1389: 119 pgs: 119 active+clean; 6.8 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 55 MiB/s wr, 887 op/s 2024-06-14T23:16:09.333 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:16:09.333 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:16:09.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:16:09.333 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:16:09.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.333 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136543 11.4 11:20fdfdc0:::benchmark_data_smithi086_226760_object9222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-06-14T23:16:09.333 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.932504+0000", 2024-06-14T23:16:09.333 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 549.39521630900003, 2024-06-14T23:16:09.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0301282709999999, 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136543 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932504+0000", 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932504+0000", 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932502+0000", 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932509+0000", 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9290000000000001e-06 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932524+0000", 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4260000000000001e-05 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932531+0000", 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0319999999999998e-06 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932570+0000", 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9038999999999997e-05 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932685+0000", 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115599 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932804+0000", 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118542 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958388+0000", 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025584134000000001 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958395+0000", 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.559e-06 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962564+0000", 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0041688070000001 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962632+0000", 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8134999999999997e-05 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136544 11.4 11:2b4506cd:::benchmark_data_smithi086_226760_object9223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.932542+0000", 2024-06-14T23:16:09.339 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 549.39517800099998, 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0454412900000001, 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136544 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932542+0000", 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932542+0000", 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932542+0000", 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932546+0000", 2024-06-14T23:16:09.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5669999999999999e-06 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932552+0000", 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4899999999999997e-06 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.932556+0000", 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2259999999999999e-06 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.936405+0000", 2024-06-14T23:16:09.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038487030000000002 2024-06-14T23:16:09.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.936523+0000", 2024-06-14T23:16:09.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117587 2024-06-14T23:16:09.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:16:09.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.936636+0000", 2024-06-14T23:16:09.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011364100000000001 2024-06-14T23:16:09.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958423+0000", 2024-06-14T23:16:09.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021786934000000001 2024-06-14T23:16:09.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958429+0000", 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5520000000000004e-06 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962848+0000", 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.004419715 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.962996+0000", 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014767099999999999 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.977890+0000", 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014894220999999999 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.977984+0000", 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3461000000000005e-05 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.347 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136549 11.3 11:c6ade1fb:::benchmark_data_smithi086_226760_object9228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.945488+0000", 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 549.38223255399998, 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.012765999, 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136549 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.349 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 116011 115995 85.283 62.4375 0.0291123 0.0117211 2024-06-14T23:16:09.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945488+0000", 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945488+0000", 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945486+0000", 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945494+0000", 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5989999999999996e-06 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945506+0000", 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1997999999999999e-05 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945512+0000", 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9959999999999999e-06 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945896+0000", 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038445299999999999 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.945903+0000", 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6089999999999997e-06 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.949574+0000", 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0036716459999999 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.951889+0000", 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023141390000000002 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1", 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.951987+0000", 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8564e-05 2024-06-14T23:16:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.958120+0000", 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061325629999999997 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.958132+0000", 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2276000000000001e-05 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.958175+0000", 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2803999999999999e-05 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.958254+0000", 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9128999999999994e-05 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136555 11.4 11:26a47b0e:::benchmark_data_smithi086_226760_object9234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.958695+0000", 2024-06-14T23:16:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 549.36902565399998, 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0183652350000001, 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136555 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958695+0000", 2024-06-14T23:16:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958695+0000", 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958694+0000", 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958698+0000", 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1710000000000004e-06 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958708+0000", 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0266e-05 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958714+0000", 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6110000000000001e-06 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958763+0000", 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9339000000000003e-05 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.958770+0000", 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9199999999999998e-06 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963038+0000", 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.004267749 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963138+0000", 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100512 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.963255+0000", 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011699199999999999 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.969271+0000", 2024-06-14T23:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060157680000000003 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.976977+0000", 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077061819999999998 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.977010+0000", 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2836999999999997e-05 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.977060+0000", 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0037000000000002e-05 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:136560 11.4 11:227da07d:::benchmark_data_smithi086_226760_object9239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:06:59.966742+0000", 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 549.36097798499998, 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0114153640000001, 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136560 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966742+0000", 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966742+0000", 2024-06-14T23:16:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966740+0000", 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966752+0000", 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1289e-05 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966766+0000", 2024-06-14T23:16:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4632e-05 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966773+0000", 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2979999999999997e-06 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966824+0000", 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1702000000000001e-05 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:06:59.966834+0000", 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3659999999999998e-06 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.966441+0000", 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99960689800000002 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.966541+0000", 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100348 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.966655+0000", 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113701 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.969390+0000", 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027356390000000002 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.978031+0000", 2024-06-14T23:16:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086405820000000008 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.978078+0000", 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6854999999999998e-05 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:00.978158+0000", 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9998000000000002e-05 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211922 11.4 11:2352494a:::benchmark_data_smithi086_226760_object84601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.120211+0000", 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 461.207509577, 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.924341317, 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211922 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120211+0000", 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120211+0000", 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120209+0000", 2024-06-14T23:16:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120217+0000", 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0220000000000004e-06 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120237+0000", 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9423999999999999e-05 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120245+0000", 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4309999999999994e-06 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120307+0000", 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1558999999999998e-05 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.120315+0000", 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5199999999999997e-06 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.034034+0000", 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91371895000000003 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.034193+0000", 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015935299999999999 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.034585+0000", 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039136599999999998 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.038389+0000", 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038037409999999998 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.038424+0000", 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5942999999999998e-05 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044266+0000", 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058418389999999997 2024-06-14T23:16:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044320+0000", 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3764999999999998e-05 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.044552+0000", 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000232072 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211928 11.4 11:21e85ab3:::benchmark_data_smithi086_226760_object84607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.139112+0000", 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 461.18860802500001, 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90937327199999995, 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211928 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139112+0000", 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139112+0000", 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139111+0000", 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139118+0000", 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1829999999999997e-06 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139132+0000", 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3835999999999999e-05 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139138+0000", 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9959999999999999e-06 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139190+0000", 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1583999999999999e-05 2024-06-14T23:16:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.139201+0000", 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1466e-05 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.036511+0000", 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89731032300000002 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.036651+0000", 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000139321 2024-06-14T23:16:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,6", 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.036980+0000", 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032952300000000002 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.038457+0000", 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014771299999999999 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.038565+0000", 2024-06-14T23:16:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010767100000000001 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048205+0000", 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009639613 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048262+0000", 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7645999999999997e-05 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.048486+0000", 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022344300000000001 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:116503 12.d 12:b8615e71:::benchmark_last_metadata:head [writefull 0~24] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:23.514456+0000", 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 285.81326466799999, 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.160583887, 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 116503 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:23.514456+0000", 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:23.514456+0000", 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:23.514454+0000", 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:23.514463+0000", 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6559999999999998e-06 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:23.514480+0000", 2024-06-14T23:16:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7816e-05 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:23.514486+0000", 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7799999999999997e-06 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:23.514581+0000", 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4812e-05 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:23.514594+0000", 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3363999999999999e-05 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:24.662189+0000", 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.147594282 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:24.662339+0000", 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015052899999999999 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:24.662500+0000", 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016089699999999999 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:24.670459+0000", 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079590609999999999 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:24.670499+0000", 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9852999999999999e-05 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:24.670508+0000", 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9970000000000004e-06 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:24.670564+0000", 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6242999999999997e-05 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:24.675040+0000", 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044754920000000002 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:117594 12.7 12:ec475cd4:::benchmark_data_smithi086_261492_object1089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:26.387555+0000", 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 282.940165901, 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.30337988900000001, 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117594 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.387555+0000", 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.387555+0000", 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:09.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:09.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.387553+0000", 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.387560+0000", 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3119999999999996e-06 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.387577+0000", 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6903000000000001e-05 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.387583+0000", 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9429999999999997e-06 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.398003+0000", 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010419652 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.398062+0000", 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9839000000000001e-05 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.609968+0000", 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.21190585300000001 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.610047+0000", 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8282999999999996e-05 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.610179+0000", 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000132266 2024-06-14T23:16:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.644050+0000", 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033870829999999998 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.685411+0000", 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041360900999999999 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.685456+0000", 2024-06-14T23:16:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5778000000000001e-05 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.685560+0000", 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103329 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.690934+0000", 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053747780000000002 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:117700 12.7 12:ef8cb9b9:::benchmark_data_smithi086_261492_object1195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:26.601266+0000", 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 282.726454418, 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23133727200000001, 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117700 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.601266+0000", 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.601266+0000", 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.601265+0000", 2024-06-14T23:16:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.601271+0000", 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2639999999999997e-06 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.601283+0000", 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2011000000000001e-05 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.601289+0000", 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9449999999999998e-06 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.715028+0000", 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.113738692 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.715103+0000", 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5498000000000001e-05 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.736758+0000", 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021654690000000001 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.736848+0000", 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9791999999999997e-05 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.736989+0000", 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141765 2024-06-14T23:16:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.780110+0000", 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.043120607999999998 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.827135+0000", 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047024679 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.827158+0000", 2024-06-14T23:16:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3776000000000001e-05 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.827193+0000", 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4394999999999998e-05 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:26.832603+0000", 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054104920000000003 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:128905 12.d 12:bd8abaa4:::benchmark_data_smithi086_261492_object12400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:45.492300+0000", 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 263.83542008400002, 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.24720653200000001, 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 128905 2024-06-14T23:16:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.492300+0000", 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.492300+0000", 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.492299+0000", 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.492306+0000", 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8590000000000005e-06 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.492323+0000", 2024-06-14T23:16:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6398000000000001e-05 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.492330+0000", 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6919999999999996e-06 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.603785+0000", 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.111455212 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.604617+0000", 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00083227699999999997 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.641040+0000", 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.036422698000000003 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.641153+0000", 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113503 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.641310+0000", 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015720799999999999 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.657750+0000", 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016439562000000001 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.734306+0000", 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.076555508999999994 2024-06-14T23:16:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.734362+0000", 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6091000000000002e-05 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.734420+0000", 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8828e-05 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.739507+0000", 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005086424 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:128912 12.d 12:b3c349e6:::benchmark_data_smithi086_261492_object12407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:45.502698+0000", 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 263.82502209199998, 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.242293387, 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 128912 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.502698+0000", 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.502698+0000", 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.502697+0000", 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.502704+0000", 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.263e-06 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.502713+0000", 2024-06-14T23:16:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6500000000000008e-06 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.502719+0000", 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2519999999999999e-06 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.604658+0000", 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.101939239 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.605214+0000", 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00055623599999999999 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.654971+0000", 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.049756829000000002 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.655050+0000", 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8727999999999996e-05 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.655167+0000", 2024-06-14T23:16:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011791800000000001 2024-06-14T23:16:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.731425+0000", 2024-06-14T23:16:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.076257770000000002 2024-06-14T23:16:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.739614+0000", 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081885069999999994 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.739659+0000", 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5717000000000003e-05 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.739753+0000", 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3843000000000003e-05 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:45.744992+0000", 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052383910000000002 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:132315 12.d 12:bf445e35:::benchmark_data_smithi086_261492_object15810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:51.358409+0000", 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.96931124100001, 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26116741100000002, 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 132315 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.358409+0000", 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.358409+0000", 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.358407+0000", 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.358414+0000", 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.764e-06 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.358429+0000", 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4986e-05 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.358435+0000", 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6439999999999997e-06 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.479119+0000", 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12068440599999999 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.482845+0000", 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037256989999999999 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.509733+0000", 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026888081000000001 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.509829+0000", 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5565999999999999e-05 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.509978+0000", 2024-06-14T23:16:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014903 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.556976+0000", 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046998608999999997 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.593663+0000", 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.036686972999999998 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.615112+0000", 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021448775 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.615147+0000", 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4715999999999998e-05 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.619577+0000", 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044298860000000001 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:132316 12.d 12:b615e68a:::benchmark_data_smithi086_261492_object15811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:51.361642+0000", 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.966078788, 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.264461116, 2024-06-14T23:16:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 132316 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.361642+0000", 2024-06-14T23:16:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.361642+0000", 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.361641+0000", 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:16:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.361645+0000", 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8949999999999997e-06 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.361656+0000", 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0458999999999999e-05 2024-06-14T23:16:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.361661+0000", 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6500000000000004e-06 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.482883+0000", 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.121222229 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.485321+0000", 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024377140000000001 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.522972+0000", 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037651466000000001 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.523050+0000", 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8027000000000004e-05 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.523167+0000", 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011653499999999999 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.557051+0000", 2024-06-14T23:16:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033884097000000002 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.619644+0000", 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.062593030999999993 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.619709+0000", 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5667000000000007e-05 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.619766+0000", 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6548000000000001e-05 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.626103+0000", 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063368610000000001 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:132329 12.d 12:b18e28b7:::benchmark_data_smithi086_261492_object15824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:51.379072+0000", 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.94864841399999, 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.252070144, 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 132329 2024-06-14T23:16:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.379072+0000", 2024-06-14T23:16:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.379072+0000", 2024-06-14T23:16:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.379072+0000", 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.379074+0000", 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.39e-06 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.379083+0000", 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8319999999999995e-06 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.379086+0000", 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2339999999999999e-06 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.485355+0000", 2024-06-14T23:16:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.106268591 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.486056+0000", 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00070175900000000004 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.536771+0000", 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050714594000000002 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.536865+0000", 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4080999999999996e-05 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.537010+0000", 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000144888 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.557057+0000", 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020047105999999999 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.626162+0000", 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.069105318999999998 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.626214+0000", 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1368e-05 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.626265+0000", 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0865999999999997e-05 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.631142+0000", 2024-06-14T23:16:09.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048774999999999999 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:132342 12.d 12:be8ad113:::benchmark_data_smithi086_261492_object15837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:51.398177+0000", 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.92954371500002, 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23840234199999999, 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.447 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 132342 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.398177+0000", 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.398177+0000", 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.398175+0000", 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.398182+0000", 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5270000000000004e-06 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.398195+0000", 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3171e-05 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.398201+0000", 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8379999999999998e-06 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.486097+0000", 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.087896066999999994 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.486752+0000", 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065539299999999999 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.553854+0000", 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.067101971999999996 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.553958+0000", 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103992 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.554097+0000", 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000139099 2024-06-14T23:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.593612+0000", 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.039514487000000001 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.631190+0000", 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037578113000000003 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.631220+0000", 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0294000000000002e-05 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.631306+0000", 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5729000000000005e-05 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.636579+0000", 2024-06-14T23:16:09.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052730800000000003 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:132347 12.d 12:b6aff093:::benchmark_data_smithi086_261492_object15842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:51.408069+0000", 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.91965105100002, 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23401456300000001, 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.455 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 132347 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.408069+0000", 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.408069+0000", 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.408068+0000", 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.408075+0000", 2024-06-14T23:16:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6890000000000004e-06 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.408089+0000", 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3893000000000001e-05 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.408095+0000", 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9850000000000003e-06 2024-06-14T23:16:09.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.489539+0000", 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081444062999999997 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.491100+0000", 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001560437 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.580594+0000", 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.089494689000000002 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.580669+0000", 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4674999999999999e-05 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.580801+0000", 2024-06-14T23:16:09.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013226199999999999 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.601989+0000", 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021187918 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.636634+0000", 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034644403999999997 2024-06-14T23:16:09.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.636670+0000", 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6091999999999999e-05 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.636723+0000", 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3699000000000002e-05 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.642084+0000", 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053605249999999997 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:132406 12.d 12:b70b9d85:::benchmark_data_smithi086_261492_object15901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:51.499847+0000", 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.82787385900002, 2024-06-14T23:16:09.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.254497525, 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 132406 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.499847+0000", 2024-06-14T23:16:09.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.499847+0000", 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.499845+0000", 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.499850+0000", 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3600000000000004e-06 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.499864+0000", 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3567000000000001e-05 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.499869+0000", 2024-06-14T23:16:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0819999999999998e-06 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.655853+0000", 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15598379900000001 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.656621+0000", 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00076844200000000004 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.681090+0000", 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024468230000000001 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.681177+0000", 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7682000000000005e-05 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.681306+0000", 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000128892 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.697135+0000", 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015828616 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.697178+0000", 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2970999999999999e-05 2024-06-14T23:16:09.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.746730+0000", 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.049552303999999998 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.746784+0000", 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3579000000000003e-05 2024-06-14T23:16:09.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:51.754344+0000", 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075604890000000001 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:132853 12.d 12:b099b6bf:::benchmark_data_smithi086_261492_object16348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:52.141235+0000", 2024-06-14T23:16:09.471 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.18648495799999, 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.24130648599999999, 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 132853 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.141235+0000", 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.141235+0000", 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.141235+0000", 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.141238+0000", 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5000000000000002e-06 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.141243+0000", 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.305e-06 2024-06-14T23:16:09.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.141246+0000", 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9289999999999998e-06 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.264059+0000", 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.122813687 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.264775+0000", 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00071526700000000001 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.287193+0000", 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022418067 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.287278+0000", 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4872999999999999e-05 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.287402+0000", 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012457399999999999 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.302736+0000", 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015333368 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.302776+0000", 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0729999999999998e-05 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.378719+0000", 2024-06-14T23:16:09.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.075942168000000004 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.378753+0000", 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4675999999999998e-05 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:52.382542+0000", 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037887419999999999 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.479 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:147525 12.d 12:b599a6fb:::benchmark_data_smithi086_261492_object31020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:12:13.789218+0000", 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 235.53850261400001, 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.24379415900000001, 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:09.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 147525 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:13.789218+0000", 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:13.789218+0000", 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:09.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:13.789216+0000", 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:13.789223+0000", 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1330000000000004e-06 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:13.789234+0000", 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1885999999999999e-05 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:13.789240+0000", 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4990000000000002e-06 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:13.875035+0000", 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.085795469999999999 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:13.876260+0000", 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012249299999999999 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:14.004739+0000", 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.128479116 2024-06-14T23:16:09.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:14.004832+0000", 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2690999999999997e-05 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:14.004973+0000", 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014065799999999999 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:09.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:14.009355+0000", 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043819330000000002 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:14.013028+0000", 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003673745 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:14.021236+0000", 2024-06-14T23:16:09.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082070709999999998 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:14.021285+0000", 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9098000000000003e-05 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:14.033012+0000", 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011727326999999999 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.487 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.488 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.488 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:09.488 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:09.488 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:16:09.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:09.519 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:09.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:09.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:09.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:09.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:09.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:09.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:09.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:09.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:09.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:09.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:10.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:10.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:10.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:10.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:10.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:10.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:10.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:10.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 117023 117007 85.0267 63.25 0.0080044 0.0117561 2024-06-14T23:16:10.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:10.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:10.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T23:16:10.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:10.546 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:10.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:10.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:10.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:10.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:10.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:10.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:10.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:10.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:10.946 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:16:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:10.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:10.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:11.067 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.2 to 1 (10000) 2024-06-14T23:16:11.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:11.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:11 smithi086 ceph-mon[97913]: pgmap v1390: 119 pgs: 119 active+clean; 7.0 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-14T23:16:11.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1330328954' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:16:11.118 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:11 smithi086 ceph-mon[102754]: pgmap v1390: 119 pgs: 119 active+clean; 7.0 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-14T23:16:11.118 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1330328954' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:16:11.121 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:16:11.121 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:16:11.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:11.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:11.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:11.143 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:16:11.144 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:16:11.164 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:16:11.164 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:16:11.187 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:16:11.188 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:16:11.207 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:16:11.208 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:16:11.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:16:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:16:11.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:11.263 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:16:11.263 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:16:11.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:11.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:11.285 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:16:11.286 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:16:11.335 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:11 smithi137 ceph-mon[90634]: pgmap v1390: 119 pgs: 119 active+clean; 7.0 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-14T23:16:11.336 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1330328954' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:16:11.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 117969 117953 84.729 59.125 0.0237808 0.0117978 2024-06-14T23:16:11.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:11.398 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:11.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:11.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:11.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:11.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:11.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:16:11.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:11.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:11.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:11.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:11.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:11.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:11.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:11.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:11.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:12.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:12.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:12.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:12.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:12.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:12.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:12.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:12.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:12.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:12.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1330328954' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:16:12.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:12 smithi086 ceph-mon[97913]: osdmap e371: 8 total, 8 up, 6 in 2024-06-14T23:16:12.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3173062804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:12.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1330328954' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:16:12.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:12 smithi086 ceph-mon[102754]: osdmap e371: 8 total, 8 up, 6 in 2024-06-14T23:16:12.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3173062804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:12.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 119004 118988 84.5012 64.6875 0.0141083 0.0118301 2024-06-14T23:16:12.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:12.436 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:12.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:12.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1330328954' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:16:12.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:12 smithi137 ceph-mon[90634]: osdmap e371: 8 total, 8 up, 6 in 2024-06-14T23:16:12.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3173062804' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:12.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:12.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:12.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:12.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:12.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:12.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:12.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:12.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:12.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:12.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:12.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:12.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:13.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:13.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:13.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:13.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:13.184 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:13 smithi137 ceph-mon[90634]: pgmap v1392: 119 pgs: 119 active+clean; 7.0 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 42 MiB/s wr, 675 op/s 2024-06-14T23:16:13.184 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:13.184 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:13.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:13.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:13.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:13.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:13 smithi086 ceph-mon[97913]: pgmap v1392: 119 pgs: 119 active+clean; 7.0 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 42 MiB/s wr, 675 op/s 2024-06-14T23:16:13.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:13.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:13.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:13 smithi086 ceph-mon[102754]: pgmap v1392: 119 pgs: 119 active+clean; 7.0 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 42 MiB/s wr, 675 op/s 2024-06-14T23:16:13.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:13.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:13.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:13.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:13.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 119846 119830 84.143 52.625 0.0120049 0.0118802 2024-06-14T23:16:13.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:13.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:13.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:13.570 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-14T23:16:13.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:16:13.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:13.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:13.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:13.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:13.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:13.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:13.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:13.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:13.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:13.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:13.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:13.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:14.021 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:14.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:14.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:14.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:14.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:14.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:14.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:14.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:14.306 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:14.345 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0066 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 120835 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 83.9071 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 36.6681 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 219.875 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 28.125 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1342 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 586.689 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 3518 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 450 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0119144 2024-06-14T23:16:14.346 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.0136923 2024-06-14T23:16:14.347 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 1.03127 2024-06-14T23:16:14.347 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00100209 2024-06-14T23:16:14.352 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T23:16:14.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:14.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:14.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:14.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:14.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:14.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:14.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:14.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:14.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:14.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:14.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:14.809 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:14.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:14.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:14.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:14.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:15.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:15.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:15.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:15.110 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:15.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:15.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:15.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:15.226 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:15 smithi086 ceph-mon[97913]: pgmap v1393: 119 pgs: 119 active+clean; 7.2 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:16:15.226 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:15 smithi086 ceph-mon[102754]: pgmap v1393: 119 pgs: 119 active+clean; 7.2 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:16:15.226 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:16:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:16:14] "GET /metrics HTTP/1.1" 200 36675 "" "Prometheus/2.43.0" 2024-06-14T23:16:15.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:15.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:15.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:15.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:15.380 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:15 smithi137 ceph-mon[90634]: pgmap v1393: 119 pgs: 119 active+clean; 7.2 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-06-14T23:16:15.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:15.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:15.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:15.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:15.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:15.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:15.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:15.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:15.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:15.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:15.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:15.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:15.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:16.028 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:16:16.028 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:16:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:16.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:16.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:16.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:16.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:16.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:16.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:16.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:16.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:16.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:16:16.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:16.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:16.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:16.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:16.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:16.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:16.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:16.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:16.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:16.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:16.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:16.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:16.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-14T23:16:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:16.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:16.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:16.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:17.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:17.094 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:16:17.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:17.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:17.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:17.256 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:17.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:17.282 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:16:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:17 smithi086 ceph-mon[97913]: pgmap v1394: 119 pgs: 119 active+clean; 7.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 63 MiB/s wr, 1.03k op/s 2024-06-14T23:16:17.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:17 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3086544686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:16:17.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:17 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:16:17.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:17 smithi086 ceph-mon[102754]: pgmap v1394: 119 pgs: 119 active+clean; 7.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 63 MiB/s wr, 1.03k op/s 2024-06-14T23:16:17.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:17 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3086544686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:16:17.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:17 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:16:17.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:17.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:17.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:17.432 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:17 smithi137 ceph-mon[90634]: pgmap v1394: 119 pgs: 119 active+clean; 7.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 63 MiB/s wr, 1.03k op/s 2024-06-14T23:16:17.433 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:17 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3086544686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:16:17.433 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:17 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:16:17.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:17.507 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:17.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:17.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:17.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:17.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:17.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:17.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:17.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:17.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:17.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:17.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:17.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:18.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:18.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:18.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:18.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:18.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:18.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:18.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:18.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:18.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:18.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:18.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:18.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:18 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:18 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:16:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:18 smithi137 ceph-mon[90634]: osdmap e372: 8 total, 8 up, 6 in 2024-06-14T23:16:18.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:18.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:18 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:18.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:18 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:16:18.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:18 smithi086 ceph-mon[102754]: osdmap e372: 8 total, 8 up, 6 in 2024-06-14T23:16:18.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:18 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:18.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:18 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:16:18.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:18 smithi086 ceph-mon[97913]: osdmap e372: 8 total, 8 up, 6 in 2024-06-14T23:16:18.598 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:18.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:18.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:18.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:18.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:18.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:18.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:18.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:18.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:19.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:19.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:19.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:19.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:19.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:19.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:19.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:19.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:19.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:19 smithi086 ceph-mon[97913]: pgmap v1396: 119 pgs: 119 active+clean; 7.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 46 MiB/s wr, 884 op/s 2024-06-14T23:16:19.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:19 smithi086 ceph-mon[102754]: pgmap v1396: 119 pgs: 119 active+clean; 7.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 46 MiB/s wr, 884 op/s 2024-06-14T23:16:19.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:19.446 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:19 smithi137 ceph-mon[90634]: pgmap v1396: 119 pgs: 119 active+clean; 7.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 46 MiB/s wr, 884 op/s 2024-06-14T23:16:19.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:19.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:19.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:19.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:19.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:19.605 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:16:19.605 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:16:19.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:19.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:19.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:19.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:19.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:20.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:20.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:20.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:20.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:20.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:20.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:20.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:20.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-14T23:16:20.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:20.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:20.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:20.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:20.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:20.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:20.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:20.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:20.679 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:16:20.681 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:16:20.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:20.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:20.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:20.793 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:16:20.833 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:20.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:20.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:20.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:20.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:21.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:21.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:21.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:21.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:21.241 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:21 smithi086 ceph-mon[102754]: pgmap v1397: 119 pgs: 119 active+clean; 7.2 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 117 B/s rd, 43 MiB/s wr, 1.10k op/s 2024-06-14T23:16:21.241 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:21 smithi086 ceph-mon[97913]: pgmap v1397: 119 pgs: 119 active+clean; 7.2 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 117 B/s rd, 43 MiB/s wr, 1.10k op/s 2024-06-14T23:16:21.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:21 smithi137 ceph-mon[90634]: pgmap v1397: 119 pgs: 119 active+clean; 7.2 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 117 B/s rd, 43 MiB/s wr, 1.10k op/s 2024-06-14T23:16:21.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:21.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:21.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:21.372 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:21.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:21.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:21.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:21.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:21.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:21.660 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:21.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:21.672 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1397,"stamp":"2024-06-14T23:16:19.779107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7686723550,"num_objects":117306,"num_object_clones":0,"num_object_copies":351918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117306,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124451,"num_write_kb":7733470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86438,"ondisk_log_size":86438,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":29096808,"kb_used_data":22682924,"kb_used_omap":27,"kb_used_meta":6413732,"kb_avail":533374104,"statfs":{"total":575970213888,"available":546175082496,"internally_reserved":0,"allocated":23227314176,"data_stored":23215830748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":6567662022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":164364312,"num_objects":2509,"num_object_clones":0,"num_object_copies":7527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9617,"num_write_kb":387969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.716800"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":751,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060804+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:16:07.060804+0000","last_peered":"2024-06-14T23:16:07.060804+0000","last_clean":"2024-06-14T23:16:07.060804+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:16:07.060804+0000","last_undegraded":"2024-06-14T23:16:07.060804+0000","last_fullsized":"2024-06-14T23:16:07.060804+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":189,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073478+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:16:11.073478+0000","last_peered":"2024-06-14T23:16:11.073478+0000","last_clean":"2024-06-14T23:16:11.073478+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:16:11.073478+0000","last_undegraded":"2024-06-14T23:16:11.073478+0000","last_fullsized":"2024-06-14T23:16:11.073478+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":359,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.066301+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:16:11.066301+0000","last_peered":"2024-06-14T23:16:11.066301+0000","last_clean":"2024-06-14T23:16:11.066301+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:16:11.066301+0000","last_undegraded":"2024-06-14T23:16:11.066301+0000","last_fullsized":"2024-06-14T23:16:11.066301+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":266,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075504+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:16:11.075504+0000","last_peered":"2024-06-14T23:16:11.075504+0000","last_clean":"2024-06-14T23:16:11.075504+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:16:11.075504+0000","last_undegraded":"2024-06-14T23:16:11.075504+0000","last_fullsized":"2024-06-14T23:16:11.075504+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":156,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:12.431488+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:16:12.431488+0000","last_peered":"2024-06-14T23:16:12.431488+0000","last_clean":"2024-06-14T23:16:12.431488+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:16:12.431488+0000","last_undegraded":"2024-06-14T23:16:12.431488+0000","last_fullsized":"2024-06-14T23:16:12.431488+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":816,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.078609+0000","last_change":"2024-06-14T23:14:16.297931+0000","last_active":"2024-06-14T23:16:11.078609+0000","last_peered":"2024-06-14T23:16:11.078609+0000","last_clean":"2024-06-14T23:16:11.078609+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:16:11.078609+0000","last_undegraded":"2024-06-14T23:16:11.078609+0000","last_fullsized":"2024-06-14T23:16:11.078609+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:14:16.297860+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:14:16.297860+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:46.938506+0000","scrub_duration":0.0016077540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":335,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074610+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:16:11.074610+0000","last_peered":"2024-06-14T23:16:11.074610+0000","last_clean":"2024-06-14T23:16:11.074610+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:16:11.074610+0000","last_undegraded":"2024-06-14T23:16:11.074610+0000","last_fullsized":"2024-06-14T23:16:11.074610+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":342,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075774+0000","last_change":"2024-06-14T23:15:09.033723+0000","last_active":"2024-06-14T23:16:11.075774+0000","last_peered":"2024-06-14T23:16:11.075774+0000","last_clean":"2024-06-14T23:16:11.075774+0000","last_became_active":"2024-06-14T23:15:09.033433+0000","last_became_peered":"2024-06-14T23:15:09.033433+0000","last_unstale":"2024-06-14T23:16:11.075774+0000","last_undegraded":"2024-06-14T23:16:11.075774+0000","last_fullsized":"2024-06-14T23:16:11.075774+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:39.865328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":523,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.061418+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:16:07.061418+0000","last_peered":"2024-06-14T23:16:07.061418+0000","last_clean":"2024-06-14T23:16:07.061418+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:16:07.061418+0000","last_undegraded":"2024-06-14T23:16:07.061418+0000","last_fullsized":"2024-06-14T23:16:07.061418+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":349,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.078803+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:16:11.078803+0000","last_peered":"2024-06-14T23:16:11.078803+0000","last_clean":"2024-06-14T23:16:11.078803+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:16:11.078803+0000","last_undegraded":"2024-06-14T23:16:11.078803+0000","last_fullsized":"2024-06-14T23:16:11.078803+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":341,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060793+0000","last_change":"2024-06-14T23:15:09.029798+0000","last_active":"2024-06-14T23:16:07.060793+0000","last_peered":"2024-06-14T23:16:07.060793+0000","last_clean":"2024-06-14T23:16:07.060793+0000","last_became_active":"2024-06-14T23:15:09.028148+0000","last_became_peered":"2024-06-14T23:15:09.028148+0000","last_unstale":"2024-06-14T23:16:07.060793+0000","last_undegraded":"2024-06-14T23:16:07.060793+0000","last_fullsized":"2024-06-14T23:16:07.060793+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:07:26.557680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":117,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060612+0000","last_change":"2024-06-14T23:15:09.030207+0000","last_active":"2024-06-14T23:16:07.060612+0000","last_peered":"2024-06-14T23:16:07.060612+0000","last_clean":"2024-06-14T23:16:07.060612+0000","last_became_active":"2024-06-14T23:15:09.029856+0000","last_became_peered":"2024-06-14T23:15:09.029856+0000","last_unstale":"2024-06-14T23:16:07.060612+0000","last_undegraded":"2024-06-14T23:16:07.060612+0000","last_fullsized":"2024-06-14T23:16:07.060612+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:50:21.877412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":165,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076047+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:16:11.076047+0000","last_peered":"2024-06-14T23:16:11.076047+0000","last_clean":"2024-06-14T23:16:11.076047+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:16:11.076047+0000","last_undegraded":"2024-06-14T23:16:11.076047+0000","last_fullsized":"2024-06-14T23:16:11.076047+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":674,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.067592+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:16:07.067592+0000","last_peered":"2024-06-14T23:16:07.067592+0000","last_clean":"2024-06-14T23:16:07.067592+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:16:07.067592+0000","last_undegraded":"2024-06-14T23:16:07.067592+0000","last_fullsized":"2024-06-14T23:16:07.067592+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":244,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075607+0000","last_change":"2024-06-14T23:15:09.033745+0000","last_active":"2024-06-14T23:16:11.075607+0000","last_peered":"2024-06-14T23:16:11.075607+0000","last_clean":"2024-06-14T23:16:11.075607+0000","last_became_active":"2024-06-14T23:15:09.033449+0000","last_became_peered":"2024-06-14T23:15:09.033449+0000","last_unstale":"2024-06-14T23:16:11.075607+0000","last_undegraded":"2024-06-14T23:16:11.075607+0000","last_fullsized":"2024-06-14T23:16:11.075607+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:33.546853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":162,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060835+0000","last_change":"2024-06-14T23:15:09.028225+0000","last_active":"2024-06-14T23:16:07.060835+0000","last_peered":"2024-06-14T23:16:07.060835+0000","last_clean":"2024-06-14T23:16:07.060835+0000","last_became_active":"2024-06-14T23:15:09.027806+0000","last_became_peered":"2024-06-14T23:15:09.027806+0000","last_unstale":"2024-06-14T23:16:07.060835+0000","last_undegraded":"2024-06-14T23:16:07.060835+0000","last_fullsized":"2024-06-14T23:16:07.060835+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:43.248153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":778,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073517+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:16:11.073517+0000","last_peered":"2024-06-14T23:16:11.073517+0000","last_clean":"2024-06-14T23:16:11.073517+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:16:11.073517+0000","last_undegraded":"2024-06-14T23:16:11.073517+0000","last_fullsized":"2024-06-14T23:16:11.073517+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":169,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060437+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:16:07.060437+0000","last_peered":"2024-06-14T23:16:07.060437+0000","last_clean":"2024-06-14T23:16:07.060437+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:16:07.060437+0000","last_undegraded":"2024-06-14T23:16:07.060437+0000","last_fullsized":"2024-06-14T23:16:07.060437+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":231,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.061391+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:16:07.061391+0000","last_peered":"2024-06-14T23:16:07.061391+0000","last_clean":"2024-06-14T23:16:07.061391+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:16:07.061391+0000","last_undegraded":"2024-06-14T23:16:07.061391+0000","last_fullsized":"2024-06-14T23:16:07.061391+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":726,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072154+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:16:11.072154+0000","last_peered":"2024-06-14T23:16:11.072154+0000","last_clean":"2024-06-14T23:16:11.072154+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:16:11.072154+0000","last_undegraded":"2024-06-14T23:16:11.072154+0000","last_fullsized":"2024-06-14T23:16:11.072154+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":94,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:12.431448+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:16:12.431448+0000","last_peered":"2024-06-14T23:16:12.431448+0000","last_clean":"2024-06-14T23:16:12.431448+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:16:12.431448+0000","last_undegraded":"2024-06-14T23:16:12.431448+0000","last_fullsized":"2024-06-14T23:16:12.431448+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":322,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.071987+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:16:07.071987+0000","last_peered":"2024-06-14T23:16:07.071987+0000","last_clean":"2024-06-14T23:16:07.071987+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:16:07.071987+0000","last_undegraded":"2024-06-14T23:16:07.071987+0000","last_fullsized":"2024-06-14T23:16:07.071987+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":625,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073904+0000","last_change":"2024-06-14T23:15:09.027462+0000","last_active":"2024-06-14T23:16:11.073904+0000","last_peered":"2024-06-14T23:16:11.073904+0000","last_clean":"2024-06-14T23:16:11.073904+0000","last_became_active":"2024-06-14T23:15:09.027093+0000","last_became_peered":"2024-06-14T23:15:09.027093+0000","last_unstale":"2024-06-14T23:16:11.073904+0000","last_undegraded":"2024-06-14T23:16:11.073904+0000","last_fullsized":"2024-06-14T23:16:11.073904+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:41:49.451565+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":355,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060265+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:16:07.060265+0000","last_peered":"2024-06-14T23:16:07.060265+0000","last_clean":"2024-06-14T23:16:07.060265+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:16:07.060265+0000","last_undegraded":"2024-06-14T23:16:07.060265+0000","last_fullsized":"2024-06-14T23:16:07.060265+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":219,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073532+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:16:11.073532+0000","last_peered":"2024-06-14T23:16:11.073532+0000","last_clean":"2024-06-14T23:16:11.073532+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:16:11.073532+0000","last_undegraded":"2024-06-14T23:16:11.073532+0000","last_fullsized":"2024-06-14T23:16:11.073532+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":678,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073315+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:16:11.073315+0000","last_peered":"2024-06-14T23:16:11.073315+0000","last_clean":"2024-06-14T23:16:11.073315+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:16:11.073315+0000","last_undegraded":"2024-06-14T23:16:11.073315+0000","last_fullsized":"2024-06-14T23:16:11.073315+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":238,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060256+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:16:07.060256+0000","last_peered":"2024-06-14T23:16:07.060256+0000","last_clean":"2024-06-14T23:16:07.060256+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:16:07.060256+0000","last_undegraded":"2024-06-14T23:16:07.060256+0000","last_fullsized":"2024-06-14T23:16:07.060256+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075419+0000","last_change":"2024-06-14T23:14:44.938003+0000","last_active":"2024-06-14T23:16:11.075419+0000","last_peered":"2024-06-14T23:16:11.075419+0000","last_clean":"2024-06-14T23:16:11.075419+0000","last_became_active":"2024-06-14T23:14:44.937736+0000","last_became_peered":"2024-06-14T23:14:44.937736+0000","last_unstale":"2024-06-14T23:16:11.075419+0000","last_undegraded":"2024-06-14T23:16:11.075419+0000","last_fullsized":"2024-06-14T23:16:11.075419+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:13:31.172524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":716,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075970+0000","last_change":"2024-06-14T23:14:17.053346+0000","last_active":"2024-06-14T23:16:11.075970+0000","last_peered":"2024-06-14T23:16:11.075970+0000","last_clean":"2024-06-14T23:16:11.075970+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:16:11.075970+0000","last_undegraded":"2024-06-14T23:16:11.075970+0000","last_fullsized":"2024-06-14T23:16:11.075970+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:17.053258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:17.053258+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:03.297247+0000","scrub_duration":0.0014699820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":256,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.061419+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:16:07.061419+0000","last_peered":"2024-06-14T23:16:07.061419+0000","last_clean":"2024-06-14T23:16:07.061419+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:16:07.061419+0000","last_undegraded":"2024-06-14T23:16:07.061419+0000","last_fullsized":"2024-06-14T23:16:07.061419+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":384,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.067411+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:16:07.067411+0000","last_peered":"2024-06-14T23:16:07.067411+0000","last_clean":"2024-06-14T23:16:07.067411+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:16:07.067411+0000","last_undegraded":"2024-06-14T23:16:07.067411+0000","last_fullsized":"2024-06-14T23:16:07.067411+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":548,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073208+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:16:11.073208+0000","last_peered":"2024-06-14T23:16:11.073208+0000","last_clean":"2024-06-14T23:16:11.073208+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:16:11.073208+0000","last_undegraded":"2024-06-14T23:16:11.073208+0000","last_fullsized":"2024-06-14T23:16:11.073208+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":71,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072504+0000","last_change":"2024-06-14T23:15:09.030189+0000","last_active":"2024-06-14T23:16:11.072504+0000","last_peered":"2024-06-14T23:16:11.072504+0000","last_clean":"2024-06-14T23:16:11.072504+0000","last_became_active":"2024-06-14T23:15:09.029757+0000","last_became_peered":"2024-06-14T23:15:09.029757+0000","last_unstale":"2024-06-14T23:16:11.072504+0000","last_undegraded":"2024-06-14T23:16:11.072504+0000","last_fullsized":"2024-06-14T23:16:11.072504+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:58:48.305434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":312,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.068165+0000","last_change":"2024-06-14T23:15:09.044391+0000","last_active":"2024-06-14T23:16:07.068165+0000","last_peered":"2024-06-14T23:16:07.068165+0000","last_clean":"2024-06-14T23:16:07.068165+0000","last_became_active":"2024-06-14T23:15:09.043763+0000","last_became_peered":"2024-06-14T23:15:09.043763+0000","last_unstale":"2024-06-14T23:16:07.068165+0000","last_undegraded":"2024-06-14T23:16:07.068165+0000","last_fullsized":"2024-06-14T23:16:07.068165+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:46:25.597564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":538,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074150+0000","last_change":"2024-06-14T23:15:09.438359+0000","last_active":"2024-06-14T23:16:11.074150+0000","last_peered":"2024-06-14T23:16:11.074150+0000","last_clean":"2024-06-14T23:16:11.074150+0000","last_became_active":"2024-06-14T23:15:09.026760+0000","last_became_peered":"2024-06-14T23:15:09.026760+0000","last_unstale":"2024-06-14T23:16:11.074150+0000","last_undegraded":"2024-06-14T23:16:11.074150+0000","last_fullsized":"2024-06-14T23:16:11.074150+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:09.438276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:15:09.438276+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:07.051915+0000","scrub_duration":0.011767011000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"372'7701","reported_seq":7731,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:17.418187+0000","last_change":"2024-06-14T23:14:30.741601+0000","last_active":"2024-06-14T23:16:17.418187+0000","last_peered":"2024-06-14T23:16:17.418187+0000","last_clean":"2024-06-14T23:16:17.418187+0000","last_became_active":"2024-06-14T23:14:30.741403+0000","last_became_peered":"2024-06-14T23:14:30.741403+0000","last_unstale":"2024-06-14T23:16:17.418187+0000","last_undegraded":"2024-06-14T23:16:17.418187+0000","last_fullsized":"2024-06-14T23:16:17.418187+0000","mapping_epoch":351,"log_start":"358'1900","ondisk_log_start":"358'1900","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5801,"log_dups_size":0,"ondisk_log_size":5801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:32.635229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479789056,"num_objects":7321,"num_object_clones":0,"num_object_copies":21963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7701,"num_write_kb":480704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":249,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072078+0000","last_change":"2024-06-14T23:15:09.039701+0000","last_active":"2024-06-14T23:16:11.072078+0000","last_peered":"2024-06-14T23:16:11.072078+0000","last_clean":"2024-06-14T23:16:11.072078+0000","last_became_active":"2024-06-14T23:15:09.039273+0000","last_became_peered":"2024-06-14T23:15:09.039273+0000","last_unstale":"2024-06-14T23:16:11.072078+0000","last_undegraded":"2024-06-14T23:16:11.072078+0000","last_fullsized":"2024-06-14T23:16:11.072078+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:51:34.296727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":343,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.071788+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:16:07.071788+0000","last_peered":"2024-06-14T23:16:07.071788+0000","last_clean":"2024-06-14T23:16:07.071788+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:16:07.071788+0000","last_undegraded":"2024-06-14T23:16:07.071788+0000","last_fullsized":"2024-06-14T23:16:07.071788+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":56,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072808+0000","last_change":"2024-06-14T23:15:09.032054+0000","last_active":"2024-06-14T23:16:11.072808+0000","last_peered":"2024-06-14T23:16:11.072808+0000","last_clean":"2024-06-14T23:16:11.072808+0000","last_became_active":"2024-06-14T23:15:09.031613+0000","last_became_peered":"2024-06-14T23:15:09.031613+0000","last_unstale":"2024-06-14T23:16:11.072808+0000","last_undegraded":"2024-06-14T23:16:11.072808+0000","last_fullsized":"2024-06-14T23:16:11.072808+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:14.012846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":697,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060509+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:16:07.060509+0000","last_peered":"2024-06-14T23:16:07.060509+0000","last_clean":"2024-06-14T23:16:07.060509+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:16:07.060509+0000","last_undegraded":"2024-06-14T23:16:07.060509+0000","last_fullsized":"2024-06-14T23:16:07.060509+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"372'7888","reported_seq":7917,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:18.384216+0000","last_change":"2024-06-14T23:14:30.745071+0000","last_active":"2024-06-14T23:16:18.384216+0000","last_peered":"2024-06-14T23:16:18.384216+0000","last_clean":"2024-06-14T23:16:18.384216+0000","last_became_active":"2024-06-14T23:14:30.744589+0000","last_became_peered":"2024-06-14T23:14:30.744589+0000","last_unstale":"2024-06-14T23:16:18.384216+0000","last_undegraded":"2024-06-14T23:16:18.384216+0000","last_fullsized":"2024-06-14T23:16:18.384216+0000","mapping_epoch":351,"log_start":"358'3100","ondisk_log_start":"358'3100","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4788,"log_dups_size":0,"ondisk_log_size":4788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:20:16.097928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480772096,"num_objects":7336,"num_object_clones":0,"num_object_copies":22008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":487168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":286,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060271+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:16:07.060271+0000","last_peered":"2024-06-14T23:16:07.060271+0000","last_clean":"2024-06-14T23:16:07.060271+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:16:07.060271+0000","last_undegraded":"2024-06-14T23:16:07.060271+0000","last_fullsized":"2024-06-14T23:16:07.060271+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":287,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.061091+0000","last_change":"2024-06-14T23:15:09.029369+0000","last_active":"2024-06-14T23:16:07.061091+0000","last_peered":"2024-06-14T23:16:07.061091+0000","last_clean":"2024-06-14T23:16:07.061091+0000","last_became_active":"2024-06-14T23:15:09.029112+0000","last_became_peered":"2024-06-14T23:15:09.029112+0000","last_unstale":"2024-06-14T23:16:07.061091+0000","last_undegraded":"2024-06-14T23:16:07.061091+0000","last_fullsized":"2024-06-14T23:16:07.061091+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:39:40.009379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":86,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073108+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:16:11.073108+0000","last_peered":"2024-06-14T23:16:11.073108+0000","last_clean":"2024-06-14T23:16:11.073108+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:16:11.073108+0000","last_undegraded":"2024-06-14T23:16:11.073108+0000","last_fullsized":"2024-06-14T23:16:11.073108+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":587,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075442+0000","last_change":"2024-06-14T23:15:10.023278+0000","last_active":"2024-06-14T23:16:11.075442+0000","last_peered":"2024-06-14T23:16:11.075442+0000","last_clean":"2024-06-14T23:16:11.075442+0000","last_became_active":"2024-06-14T23:15:09.027691+0000","last_became_peered":"2024-06-14T23:15:09.027691+0000","last_unstale":"2024-06-14T23:16:11.075442+0000","last_undegraded":"2024-06-14T23:16:11.075442+0000","last_fullsized":"2024-06-14T23:16:11.075442+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:10.023159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:15:10.023159+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:28:13.357778+0000","scrub_duration":0.0066373669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.a","version":"372'7624","reported_seq":7654,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:17.432121+0000","last_change":"2024-06-14T23:14:30.743581+0000","last_active":"2024-06-14T23:16:17.432121+0000","last_peered":"2024-06-14T23:16:17.432121+0000","last_clean":"2024-06-14T23:16:17.432121+0000","last_became_active":"2024-06-14T23:14:30.743244+0000","last_became_peered":"2024-06-14T23:14:30.743244+0000","last_unstale":"2024-06-14T23:16:17.432121+0000","last_undegraded":"2024-06-14T23:16:17.432121+0000","last_fullsized":"2024-06-14T23:16:17.432121+0000","mapping_epoch":351,"log_start":"358'1800","ondisk_log_start":"358'1800","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:11:35.334274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475922432,"num_objects":7262,"num_object_clones":0,"num_object_copies":21786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7624,"num_write_kb":476352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":183,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072340+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:16:11.072340+0000","last_peered":"2024-06-14T23:16:11.072340+0000","last_clean":"2024-06-14T23:16:11.072340+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:16:11.072340+0000","last_undegraded":"2024-06-14T23:16:11.072340+0000","last_fullsized":"2024-06-14T23:16:11.072340+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":357,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.069925+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:16:11.069925+0000","last_peered":"2024-06-14T23:16:11.069925+0000","last_clean":"2024-06-14T23:16:11.069925+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:16:11.069925+0000","last_undegraded":"2024-06-14T23:16:11.069925+0000","last_fullsized":"2024-06-14T23:16:11.069925+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":310,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076124+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:16:11.076124+0000","last_peered":"2024-06-14T23:16:11.076124+0000","last_clean":"2024-06-14T23:16:11.076124+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:16:11.076124+0000","last_undegraded":"2024-06-14T23:16:11.076124+0000","last_fullsized":"2024-06-14T23:16:11.076124+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":670,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.068033+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:16:07.068033+0000","last_peered":"2024-06-14T23:16:07.068033+0000","last_clean":"2024-06-14T23:16:07.068033+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:16:07.068033+0000","last_undegraded":"2024-06-14T23:16:07.068033+0000","last_fullsized":"2024-06-14T23:16:07.068033+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"372'7907","reported_seq":7936,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:18.374329+0000","last_change":"2024-06-14T23:14:31.004195+0000","last_active":"2024-06-14T23:16:18.374329+0000","last_peered":"2024-06-14T23:16:18.374329+0000","last_clean":"2024-06-14T23:16:18.374329+0000","last_became_active":"2024-06-14T23:14:31.003779+0000","last_became_peered":"2024-06-14T23:14:31.003779+0000","last_unstale":"2024-06-14T23:16:18.374329+0000","last_undegraded":"2024-06-14T23:16:18.374329+0000","last_fullsized":"2024-06-14T23:16:18.374329+0000","mapping_epoch":351,"log_start":"358'3100","ondisk_log_start":"358'3100","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:42:29.429100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485031936,"num_objects":7401,"num_object_clones":0,"num_object_copies":22203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":148,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073835+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:16:11.073835+0000","last_peered":"2024-06-14T23:16:11.073835+0000","last_clean":"2024-06-14T23:16:11.073835+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:16:11.073835+0000","last_undegraded":"2024-06-14T23:16:11.073835+0000","last_fullsized":"2024-06-14T23:16:11.073835+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075821+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:16:11.075821+0000","last_peered":"2024-06-14T23:16:11.075821+0000","last_clean":"2024-06-14T23:16:11.075821+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:16:11.075821+0000","last_undegraded":"2024-06-14T23:16:11.075821+0000","last_fullsized":"2024-06-14T23:16:11.075821+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":387,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074689+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:16:11.074689+0000","last_peered":"2024-06-14T23:16:11.074689+0000","last_clean":"2024-06-14T23:16:11.074689+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:16:11.074689+0000","last_undegraded":"2024-06-14T23:16:11.074689+0000","last_fullsized":"2024-06-14T23:16:11.074689+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":650,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.071868+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:16:07.071868+0000","last_peered":"2024-06-14T23:16:07.071868+0000","last_clean":"2024-06-14T23:16:07.071868+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:16:07.071868+0000","last_undegraded":"2024-06-14T23:16:07.071868+0000","last_fullsized":"2024-06-14T23:16:07.071868+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"372'7900","reported_seq":7938,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:18.943268+0000","last_change":"2024-06-14T23:15:09.042072+0000","last_active":"2024-06-14T23:16:18.943268+0000","last_peered":"2024-06-14T23:16:18.943268+0000","last_clean":"2024-06-14T23:16:18.943268+0000","last_became_active":"2024-06-14T23:15:09.034036+0000","last_became_peered":"2024-06-14T23:15:09.034036+0000","last_unstale":"2024-06-14T23:16:18.943268+0000","last_undegraded":"2024-06-14T23:16:18.943268+0000","last_fullsized":"2024-06-14T23:16:18.943268+0000","mapping_epoch":359,"log_start":"358'2800","ondisk_log_start":"358'2800","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:49:17.304835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480313368,"num_objects":7330,"num_object_clones":0,"num_object_copies":21990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7330,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7900,"num_write_kb":487297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":325,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075176+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:16:11.075176+0000","last_peered":"2024-06-14T23:16:11.075176+0000","last_clean":"2024-06-14T23:16:11.075176+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:16:11.075176+0000","last_undegraded":"2024-06-14T23:16:11.075176+0000","last_fullsized":"2024-06-14T23:16:11.075176+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":323,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.067240+0000","last_change":"2024-06-14T23:15:09.044549+0000","last_active":"2024-06-14T23:16:07.067240+0000","last_peered":"2024-06-14T23:16:07.067240+0000","last_clean":"2024-06-14T23:16:07.067240+0000","last_became_active":"2024-06-14T23:15:09.044122+0000","last_became_peered":"2024-06-14T23:15:09.044122+0000","last_unstale":"2024-06-14T23:16:07.067240+0000","last_undegraded":"2024-06-14T23:16:07.067240+0000","last_fullsized":"2024-06-14T23:16:07.067240+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:11:38.624888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":364,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074044+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:16:11.074044+0000","last_peered":"2024-06-14T23:16:11.074044+0000","last_clean":"2024-06-14T23:16:11.074044+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:16:11.074044+0000","last_undegraded":"2024-06-14T23:16:11.074044+0000","last_fullsized":"2024-06-14T23:16:11.074044+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":765,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072012+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:16:11.072012+0000","last_peered":"2024-06-14T23:16:11.072012+0000","last_clean":"2024-06-14T23:16:11.072012+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:16:11.072012+0000","last_undegraded":"2024-06-14T23:16:11.072012+0000","last_fullsized":"2024-06-14T23:16:11.072012+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"372'7846","reported_seq":7874,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:18.377604+0000","last_change":"2024-06-14T23:14:30.743278+0000","last_active":"2024-06-14T23:16:18.377604+0000","last_peered":"2024-06-14T23:16:18.377604+0000","last_clean":"2024-06-14T23:16:18.377604+0000","last_became_active":"2024-06-14T23:14:30.743000+0000","last_became_peered":"2024-06-14T23:14:30.743000+0000","last_unstale":"2024-06-14T23:16:18.377604+0000","last_undegraded":"2024-06-14T23:16:18.377604+0000","last_fullsized":"2024-06-14T23:16:18.377604+0000","mapping_epoch":351,"log_start":"358'3000","ondisk_log_start":"358'3000","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:22:35.618792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479854592,"num_objects":7322,"num_object_clones":0,"num_object_copies":21966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":332,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073312+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:16:11.073312+0000","last_peered":"2024-06-14T23:16:11.073312+0000","last_clean":"2024-06-14T23:16:11.073312+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:16:11.073312+0000","last_undegraded":"2024-06-14T23:16:11.073312+0000","last_fullsized":"2024-06-14T23:16:11.073312+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":370,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074543+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:16:11.074543+0000","last_peered":"2024-06-14T23:16:11.074543+0000","last_clean":"2024-06-14T23:16:11.074543+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:16:11.074543+0000","last_undegraded":"2024-06-14T23:16:11.074543+0000","last_fullsized":"2024-06-14T23:16:11.074543+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":175,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076263+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:16:11.076263+0000","last_peered":"2024-06-14T23:16:11.076263+0000","last_clean":"2024-06-14T23:16:11.076263+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:16:11.076263+0000","last_undegraded":"2024-06-14T23:16:11.076263+0000","last_fullsized":"2024-06-14T23:16:11.076263+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":578,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074485+0000","last_change":"2024-06-14T23:15:09.027736+0000","last_active":"2024-06-14T23:16:11.074485+0000","last_peered":"2024-06-14T23:16:11.074485+0000","last_clean":"2024-06-14T23:16:11.074485+0000","last_became_active":"2024-06-14T23:15:09.027333+0000","last_became_peered":"2024-06-14T23:15:09.027333+0000","last_unstale":"2024-06-14T23:16:11.074485+0000","last_undegraded":"2024-06-14T23:16:11.074485+0000","last_fullsized":"2024-06-14T23:16:11.074485+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:25:29.986791+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.f","version":"371'7664","reported_seq":7692,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:16.861610+0000","last_change":"2024-06-14T23:14:30.740225+0000","last_active":"2024-06-14T23:16:16.861610+0000","last_peered":"2024-06-14T23:16:16.861610+0000","last_clean":"2024-06-14T23:16:16.861610+0000","last_became_active":"2024-06-14T23:14:30.739814+0000","last_became_peered":"2024-06-14T23:14:30.739814+0000","last_unstale":"2024-06-14T23:16:16.861610+0000","last_undegraded":"2024-06-14T23:16:16.861610+0000","last_fullsized":"2024-06-14T23:16:16.861610+0000","mapping_epoch":351,"log_start":"358'2500","ondisk_log_start":"358'2500","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5164,"log_dups_size":0,"ondisk_log_size":5164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:27:30.267316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483917824,"num_objects":7384,"num_object_clones":0,"num_object_copies":22152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7664,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":343,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075639+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:16:11.075639+0000","last_peered":"2024-06-14T23:16:11.075639+0000","last_clean":"2024-06-14T23:16:11.075639+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:16:11.075639+0000","last_undegraded":"2024-06-14T23:16:11.075639+0000","last_fullsized":"2024-06-14T23:16:11.075639+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":162,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060359+0000","last_change":"2024-06-14T23:15:09.025369+0000","last_active":"2024-06-14T23:16:07.060359+0000","last_peered":"2024-06-14T23:16:07.060359+0000","last_clean":"2024-06-14T23:16:07.060359+0000","last_became_active":"2024-06-14T23:15:09.025108+0000","last_became_peered":"2024-06-14T23:15:09.025108+0000","last_unstale":"2024-06-14T23:16:07.060359+0000","last_undegraded":"2024-06-14T23:16:07.060359+0000","last_fullsized":"2024-06-14T23:16:07.060359+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:04:22.310031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":337,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060193+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:16:07.060193+0000","last_peered":"2024-06-14T23:16:07.060193+0000","last_clean":"2024-06-14T23:16:07.060193+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:16:07.060193+0000","last_undegraded":"2024-06-14T23:16:07.060193+0000","last_fullsized":"2024-06-14T23:16:07.060193+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":526,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.071977+0000","last_change":"2024-06-14T23:15:09.035281+0000","last_active":"2024-06-14T23:16:11.071977+0000","last_peered":"2024-06-14T23:16:11.071977+0000","last_clean":"2024-06-14T23:16:11.071977+0000","last_became_active":"2024-06-14T23:15:09.034141+0000","last_became_peered":"2024-06-14T23:15:09.034141+0000","last_unstale":"2024-06-14T23:16:11.071977+0000","last_undegraded":"2024-06-14T23:16:11.071977+0000","last_fullsized":"2024-06-14T23:16:11.071977+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:36.329142+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"372'7733","reported_seq":7762,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:18.385811+0000","last_change":"2024-06-14T23:14:30.745351+0000","last_active":"2024-06-14T23:16:18.385811+0000","last_peered":"2024-06-14T23:16:18.385811+0000","last_clean":"2024-06-14T23:16:18.385811+0000","last_became_active":"2024-06-14T23:14:30.745017+0000","last_became_peered":"2024-06-14T23:14:30.745017+0000","last_unstale":"2024-06-14T23:16:18.385811+0000","last_undegraded":"2024-06-14T23:16:18.385811+0000","last_fullsized":"2024-06-14T23:16:18.385811+0000","mapping_epoch":351,"log_start":"358'2900","ondisk_log_start":"358'2900","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4833,"log_dups_size":0,"ondisk_log_size":4833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:32.069491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471531520,"num_objects":7195,"num_object_clones":0,"num_object_copies":21585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7733,"num_write_kb":477696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":286,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.067798+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:16:07.067798+0000","last_peered":"2024-06-14T23:16:07.067798+0000","last_clean":"2024-06-14T23:16:07.067798+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:16:07.067798+0000","last_undegraded":"2024-06-14T23:16:07.067798+0000","last_fullsized":"2024-06-14T23:16:07.067798+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":293,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074014+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:16:11.074014+0000","last_peered":"2024-06-14T23:16:11.074014+0000","last_clean":"2024-06-14T23:16:11.074014+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:16:11.074014+0000","last_undegraded":"2024-06-14T23:16:11.074014+0000","last_fullsized":"2024-06-14T23:16:11.074014+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":374,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073680+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:16:11.073680+0000","last_peered":"2024-06-14T23:16:11.073680+0000","last_clean":"2024-06-14T23:16:11.073680+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:16:11.073680+0000","last_undegraded":"2024-06-14T23:16:11.073680+0000","last_fullsized":"2024-06-14T23:16:11.073680+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":665,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074467+0000","last_change":"2024-06-14T23:14:18.033116+0000","last_active":"2024-06-14T23:16:11.074467+0000","last_peered":"2024-06-14T23:16:11.074467+0000","last_clean":"2024-06-14T23:16:11.074467+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:16:11.074467+0000","last_undegraded":"2024-06-14T23:16:11.074467+0000","last_fullsized":"2024-06-14T23:16:11.074467+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:18.032997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:18.032997+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:57:13.313132+0000","scrub_duration":0.0019777449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.6","version":"372'7811","reported_seq":7848,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:18.361746+0000","last_change":"2024-06-14T23:15:09.037260+0000","last_active":"2024-06-14T23:16:18.361746+0000","last_peered":"2024-06-14T23:16:18.361746+0000","last_clean":"2024-06-14T23:16:18.361746+0000","last_became_active":"2024-06-14T23:15:09.033102+0000","last_became_peered":"2024-06-14T23:15:09.033102+0000","last_unstale":"2024-06-14T23:16:18.361746+0000","last_undegraded":"2024-06-14T23:16:18.361746+0000","last_fullsized":"2024-06-14T23:16:18.361746+0000","mapping_epoch":359,"log_start":"358'3000","ondisk_log_start":"358'3000","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4811,"log_dups_size":0,"ondisk_log_size":4811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:35:04.940982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477954048,"num_objects":7293,"num_object_clones":0,"num_object_copies":21879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7811,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":170,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.061154+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:16:07.061154+0000","last_peered":"2024-06-14T23:16:07.061154+0000","last_clean":"2024-06-14T23:16:07.061154+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:16:07.061154+0000","last_undegraded":"2024-06-14T23:16:07.061154+0000","last_fullsized":"2024-06-14T23:16:07.061154+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":342,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060388+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:16:07.060388+0000","last_peered":"2024-06-14T23:16:07.060388+0000","last_clean":"2024-06-14T23:16:07.060388+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:16:07.060388+0000","last_undegraded":"2024-06-14T23:16:07.060388+0000","last_fullsized":"2024-06-14T23:16:07.060388+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":763,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073693+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:16:11.073693+0000","last_peered":"2024-06-14T23:16:11.073693+0000","last_clean":"2024-06-14T23:16:11.073693+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:16:11.073693+0000","last_undegraded":"2024-06-14T23:16:11.073693+0000","last_fullsized":"2024-06-14T23:16:11.073693+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"372'7941","reported_seq":7970,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:18.927462+0000","last_change":"2024-06-14T23:14:30.742252+0000","last_active":"2024-06-14T23:16:18.927462+0000","last_peered":"2024-06-14T23:16:18.927462+0000","last_clean":"2024-06-14T23:16:18.927462+0000","last_became_active":"2024-06-14T23:14:30.741955+0000","last_became_peered":"2024-06-14T23:14:30.741955+0000","last_unstale":"2024-06-14T23:16:18.927462+0000","last_undegraded":"2024-06-14T23:16:18.927462+0000","last_fullsized":"2024-06-14T23:16:18.927462+0000","mapping_epoch":351,"log_start":"358'2900","ondisk_log_start":"358'2900","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5041,"log_dups_size":0,"ondisk_log_size":5041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.550055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478740480,"num_objects":7305,"num_object_clones":0,"num_object_copies":21915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":487872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":359,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.066675+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:16:11.066675+0000","last_peered":"2024-06-14T23:16:11.066675+0000","last_clean":"2024-06-14T23:16:11.066675+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:16:11.066675+0000","last_undegraded":"2024-06-14T23:16:11.066675+0000","last_fullsized":"2024-06-14T23:16:11.066675+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":336,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076132+0000","last_change":"2024-06-14T23:15:09.029095+0000","last_active":"2024-06-14T23:16:11.076132+0000","last_peered":"2024-06-14T23:16:11.076132+0000","last_clean":"2024-06-14T23:16:11.076132+0000","last_became_active":"2024-06-14T23:15:09.028817+0000","last_became_peered":"2024-06-14T23:15:09.028817+0000","last_unstale":"2024-06-14T23:16:11.076132+0000","last_undegraded":"2024-06-14T23:16:11.076132+0000","last_fullsized":"2024-06-14T23:16:11.076132+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:43:20.264367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":731,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072274+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:16:11.072274+0000","last_peered":"2024-06-14T23:16:11.072274+0000","last_clean":"2024-06-14T23:16:11.072274+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:16:11.072274+0000","last_undegraded":"2024-06-14T23:16:11.072274+0000","last_fullsized":"2024-06-14T23:16:11.072274+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"371'7689","reported_seq":7717,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:16.847065+0000","last_change":"2024-06-14T23:14:30.742997+0000","last_active":"2024-06-14T23:16:16.847065+0000","last_peered":"2024-06-14T23:16:16.847065+0000","last_clean":"2024-06-14T23:16:16.847065+0000","last_became_active":"2024-06-14T23:14:30.742733+0000","last_became_peered":"2024-06-14T23:14:30.742733+0000","last_unstale":"2024-06-14T23:16:16.847065+0000","last_undegraded":"2024-06-14T23:16:16.847065+0000","last_fullsized":"2024-06-14T23:16:16.847065+0000","mapping_epoch":351,"log_start":"358'2600","ondisk_log_start":"358'2600","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:48.248776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482541568,"num_objects":7363,"num_object_clones":0,"num_object_copies":22089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7689,"num_write_kb":481664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":355,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074507+0000","last_change":"2024-06-14T23:15:09.028336+0000","last_active":"2024-06-14T23:16:11.074507+0000","last_peered":"2024-06-14T23:16:11.074507+0000","last_clean":"2024-06-14T23:16:11.074507+0000","last_became_active":"2024-06-14T23:15:09.028078+0000","last_became_peered":"2024-06-14T23:15:09.028078+0000","last_unstale":"2024-06-14T23:16:11.074507+0000","last_undegraded":"2024-06-14T23:16:11.074507+0000","last_fullsized":"2024-06-14T23:16:11.074507+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:36:57.434975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":223,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.067476+0000","last_change":"2024-06-14T23:15:09.037746+0000","last_active":"2024-06-14T23:16:07.067476+0000","last_peered":"2024-06-14T23:16:07.067476+0000","last_clean":"2024-06-14T23:16:07.067476+0000","last_became_active":"2024-06-14T23:15:09.037427+0000","last_became_peered":"2024-06-14T23:15:09.037427+0000","last_unstale":"2024-06-14T23:16:07.067476+0000","last_undegraded":"2024-06-14T23:16:07.067476+0000","last_fullsized":"2024-06-14T23:16:07.067476+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:40:34.511973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":754,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073362+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:16:11.073362+0000","last_peered":"2024-06-14T23:16:11.073362+0000","last_clean":"2024-06-14T23:16:11.073362+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:16:11.073362+0000","last_undegraded":"2024-06-14T23:16:11.073362+0000","last_fullsized":"2024-06-14T23:16:11.073362+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.5","version":"372'7868","reported_seq":7906,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:18.388611+0000","last_change":"2024-06-14T23:15:09.032574+0000","last_active":"2024-06-14T23:16:18.388611+0000","last_peered":"2024-06-14T23:16:18.388611+0000","last_clean":"2024-06-14T23:16:18.388611+0000","last_became_active":"2024-06-14T23:15:09.027289+0000","last_became_peered":"2024-06-14T23:15:09.027289+0000","last_unstale":"2024-06-14T23:16:18.388611+0000","last_undegraded":"2024-06-14T23:16:18.388611+0000","last_fullsized":"2024-06-14T23:16:18.388611+0000","mapping_epoch":359,"log_start":"358'3100","ondisk_log_start":"358'3100","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:31:01.504384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485490688,"num_objects":7408,"num_object_clones":0,"num_object_copies":22224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7868,"num_write_kb":488832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":265,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.069165+0000","last_change":"2024-06-14T23:15:09.038060+0000","last_active":"2024-06-14T23:16:07.069165+0000","last_peered":"2024-06-14T23:16:07.069165+0000","last_clean":"2024-06-14T23:16:07.069165+0000","last_became_active":"2024-06-14T23:15:09.037814+0000","last_became_peered":"2024-06-14T23:15:09.037814+0000","last_unstale":"2024-06-14T23:16:07.069165+0000","last_undegraded":"2024-06-14T23:16:07.069165+0000","last_fullsized":"2024-06-14T23:16:07.069165+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:48.977466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":120,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.069339+0000","last_change":"2024-06-14T23:15:09.024949+0000","last_active":"2024-06-14T23:16:11.069339+0000","last_peered":"2024-06-14T23:16:11.069339+0000","last_clean":"2024-06-14T23:16:11.069339+0000","last_became_active":"2024-06-14T23:15:09.023224+0000","last_became_peered":"2024-06-14T23:15:09.023224+0000","last_unstale":"2024-06-14T23:16:11.069339+0000","last_undegraded":"2024-06-14T23:16:11.069339+0000","last_fullsized":"2024-06-14T23:16:11.069339+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:12.547802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":656,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076053+0000","last_change":"2024-06-14T23:14:14.052730+0000","last_active":"2024-06-14T23:16:11.076053+0000","last_peered":"2024-06-14T23:16:11.076053+0000","last_clean":"2024-06-14T23:16:11.076053+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:16:11.076053+0000","last_undegraded":"2024-06-14T23:16:11.076053+0000","last_fullsized":"2024-06-14T23:16:11.076053+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:14.052631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:14.052631+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:51:25.441426+0000","scrub_duration":0.0017584180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"372'7671","reported_seq":7699,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:18.383465+0000","last_change":"2024-06-14T23:14:31.003884+0000","last_active":"2024-06-14T23:16:18.383465+0000","last_peered":"2024-06-14T23:16:18.383465+0000","last_clean":"2024-06-14T23:16:18.383465+0000","last_became_active":"2024-06-14T23:14:31.003542+0000","last_became_peered":"2024-06-14T23:14:31.003542+0000","last_unstale":"2024-06-14T23:16:18.383465+0000","last_undegraded":"2024-06-14T23:16:18.383465+0000","last_fullsized":"2024-06-14T23:16:18.383465+0000","mapping_epoch":351,"log_start":"358'2900","ondisk_log_start":"358'2900","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4771,"log_dups_size":0,"ondisk_log_size":4771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:27:38.082967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469172224,"num_objects":7159,"num_object_clones":0,"num_object_copies":21477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":474560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":179,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074436+0000","last_change":"2024-06-14T23:15:09.022932+0000","last_active":"2024-06-14T23:16:11.074436+0000","last_peered":"2024-06-14T23:16:11.074436+0000","last_clean":"2024-06-14T23:16:11.074436+0000","last_became_active":"2024-06-14T23:15:09.022603+0000","last_became_peered":"2024-06-14T23:15:09.022603+0000","last_unstale":"2024-06-14T23:16:11.074436+0000","last_undegraded":"2024-06-14T23:16:11.074436+0000","last_fullsized":"2024-06-14T23:16:11.074436+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:03:53.567467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":350,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.078814+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:16:11.078814+0000","last_peered":"2024-06-14T23:16:11.078814+0000","last_clean":"2024-06-14T23:16:11.078814+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:16:11.078814+0000","last_undegraded":"2024-06-14T23:16:11.078814+0000","last_fullsized":"2024-06-14T23:16:11.078814+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":493,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072722+0000","last_change":"2024-06-14T23:15:12.943908+0000","last_active":"2024-06-14T23:16:11.072722+0000","last_peered":"2024-06-14T23:16:11.072722+0000","last_clean":"2024-06-14T23:16:11.072722+0000","last_became_active":"2024-06-14T23:15:09.031629+0000","last_became_peered":"2024-06-14T23:15:09.031629+0000","last_unstale":"2024-06-14T23:16:11.072722+0000","last_undegraded":"2024-06-14T23:16:11.072722+0000","last_fullsized":"2024-06-14T23:16:11.072722+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:12.943827+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:15:12.943827+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:48:02.155745+0000","scrub_duration":0.0029576450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3","version":"371'7670","reported_seq":7706,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:15.662616+0000","last_change":"2024-06-14T23:15:09.025883+0000","last_active":"2024-06-14T23:16:15.662616+0000","last_peered":"2024-06-14T23:16:15.662616+0000","last_clean":"2024-06-14T23:16:15.662616+0000","last_became_active":"2024-06-14T23:15:09.022582+0000","last_became_peered":"2024-06-14T23:15:09.022582+0000","last_unstale":"2024-06-14T23:16:15.662616+0000","last_undegraded":"2024-06-14T23:16:15.662616+0000","last_fullsized":"2024-06-14T23:16:15.662616+0000","mapping_epoch":359,"log_start":"358'2900","ondisk_log_start":"358'2900","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:20:48.026157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489553920,"num_objects":7470,"num_object_clones":0,"num_object_copies":22410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7670,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":322,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.067286+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:16:07.067286+0000","last_peered":"2024-06-14T23:16:07.067286+0000","last_clean":"2024-06-14T23:16:07.067286+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:16:07.067286+0000","last_undegraded":"2024-06-14T23:16:07.067286+0000","last_fullsized":"2024-06-14T23:16:07.067286+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":267,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074311+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:16:11.074311+0000","last_peered":"2024-06-14T23:16:11.074311+0000","last_clean":"2024-06-14T23:16:11.074311+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:16:11.074311+0000","last_undegraded":"2024-06-14T23:16:11.074311+0000","last_fullsized":"2024-06-14T23:16:11.074311+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":650,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.066888+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:16:07.066888+0000","last_peered":"2024-06-14T23:16:07.066888+0000","last_clean":"2024-06-14T23:16:07.066888+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:16:07.066888+0000","last_undegraded":"2024-06-14T23:16:07.066888+0000","last_fullsized":"2024-06-14T23:16:07.066888+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"372'7812","reported_seq":7842,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:17.400855+0000","last_change":"2024-06-14T23:14:30.738940+0000","last_active":"2024-06-14T23:16:17.400855+0000","last_peered":"2024-06-14T23:16:17.400855+0000","last_clean":"2024-06-14T23:16:17.400855+0000","last_became_active":"2024-06-14T23:14:30.738700+0000","last_became_peered":"2024-06-14T23:14:30.738700+0000","last_unstale":"2024-06-14T23:16:17.400855+0000","last_undegraded":"2024-06-14T23:16:17.400855+0000","last_fullsized":"2024-06-14T23:16:17.400855+0000","mapping_epoch":351,"log_start":"358'2000","ondisk_log_start":"358'2000","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:29:01.585691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484966400,"num_objects":7400,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7812,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":311,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060354+0000","last_change":"2024-06-14T23:15:09.030332+0000","last_active":"2024-06-14T23:16:07.060354+0000","last_peered":"2024-06-14T23:16:07.060354+0000","last_clean":"2024-06-14T23:16:07.060354+0000","last_became_active":"2024-06-14T23:15:09.030035+0000","last_became_peered":"2024-06-14T23:15:09.030035+0000","last_unstale":"2024-06-14T23:16:07.060354+0000","last_undegraded":"2024-06-14T23:16:07.060354+0000","last_fullsized":"2024-06-14T23:16:07.060354+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:14:00.226413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":337,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073073+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:16:11.073073+0000","last_peered":"2024-06-14T23:16:11.073073+0000","last_clean":"2024-06-14T23:16:11.073073+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:16:11.073073+0000","last_undegraded":"2024-06-14T23:16:11.073073+0000","last_fullsized":"2024-06-14T23:16:11.073073+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":780,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.061088+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:16:07.061088+0000","last_peered":"2024-06-14T23:16:07.061088+0000","last_clean":"2024-06-14T23:16:07.061088+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:16:07.061088+0000","last_undegraded":"2024-06-14T23:16:07.061088+0000","last_fullsized":"2024-06-14T23:16:07.061088+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"371'7665","reported_seq":7693,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:16.780146+0000","last_change":"2024-06-14T23:14:30.735686+0000","last_active":"2024-06-14T23:16:16.780146+0000","last_peered":"2024-06-14T23:16:16.780146+0000","last_clean":"2024-06-14T23:16:16.780146+0000","last_became_active":"2024-06-14T23:14:30.735396+0000","last_became_peered":"2024-06-14T23:14:30.735396+0000","last_unstale":"2024-06-14T23:16:16.780146+0000","last_undegraded":"2024-06-14T23:16:16.780146+0000","last_fullsized":"2024-06-14T23:16:16.780146+0000","mapping_epoch":351,"log_start":"358'2500","ondisk_log_start":"358'2500","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:25:23.806204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480575488,"num_objects":7333,"num_object_clones":0,"num_object_copies":21999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":479936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":332,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073594+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:16:11.073594+0000","last_peered":"2024-06-14T23:16:11.073594+0000","last_clean":"2024-06-14T23:16:11.073594+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:16:11.073594+0000","last_undegraded":"2024-06-14T23:16:11.073594+0000","last_fullsized":"2024-06-14T23:16:11.073594+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":350,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074856+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:16:11.074856+0000","last_peered":"2024-06-14T23:16:11.074856+0000","last_clean":"2024-06-14T23:16:11.074856+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:16:11.074856+0000","last_undegraded":"2024-06-14T23:16:11.074856+0000","last_fullsized":"2024-06-14T23:16:11.074856+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":734,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074432+0000","last_change":"2024-06-14T23:15:09.028234+0000","last_active":"2024-06-14T23:16:11.074432+0000","last_peered":"2024-06-14T23:16:11.074432+0000","last_clean":"2024-06-14T23:16:11.074432+0000","last_became_active":"2024-06-14T23:15:09.027710+0000","last_became_peered":"2024-06-14T23:15:09.027710+0000","last_unstale":"2024-06-14T23:16:11.074432+0000","last_undegraded":"2024-06-14T23:16:11.074432+0000","last_fullsized":"2024-06-14T23:16:11.074432+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:46:47.401663+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":357,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.067691+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:16:07.067691+0000","last_peered":"2024-06-14T23:16:07.067691+0000","last_clean":"2024-06-14T23:16:07.067691+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:16:07.067691+0000","last_undegraded":"2024-06-14T23:16:07.067691+0000","last_fullsized":"2024-06-14T23:16:07.067691+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":355,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060984+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:16:07.060984+0000","last_peered":"2024-06-14T23:16:07.060984+0000","last_clean":"2024-06-14T23:16:07.060984+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:16:07.060984+0000","last_undegraded":"2024-06-14T23:16:07.060984+0000","last_fullsized":"2024-06-14T23:16:07.060984+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":727,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073765+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:16:11.073765+0000","last_peered":"2024-06-14T23:16:11.073765+0000","last_clean":"2024-06-14T23:16:11.073765+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:16:11.073765+0000","last_undegraded":"2024-06-14T23:16:11.073765+0000","last_fullsized":"2024-06-14T23:16:11.073765+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":172,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.067705+0000","last_change":"2024-06-14T23:15:09.039681+0000","last_active":"2024-06-14T23:16:07.067705+0000","last_peered":"2024-06-14T23:16:07.067705+0000","last_clean":"2024-06-14T23:16:07.067705+0000","last_became_active":"2024-06-14T23:15:09.039180+0000","last_became_peered":"2024-06-14T23:15:09.039180+0000","last_unstale":"2024-06-14T23:16:07.067705+0000","last_undegraded":"2024-06-14T23:16:07.067705+0000","last_fullsized":"2024-06-14T23:16:07.067705+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:54.312788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":236,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.061468+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:16:07.061468+0000","last_peered":"2024-06-14T23:16:07.061468+0000","last_clean":"2024-06-14T23:16:07.061468+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:16:07.061468+0000","last_undegraded":"2024-06-14T23:16:07.061468+0000","last_fullsized":"2024-06-14T23:16:07.061468+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":693,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-06-14T23:16:07.060702+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:16:07.060702+0000","last_peered":"2024-06-14T23:16:07.060702+0000","last_clean":"2024-06-14T23:16:07.060702+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:16:07.060702+0000","last_undegraded":"2024-06-14T23:16:07.060702+0000","last_fullsized":"2024-06-14T23:16:07.060702+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":75,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075699+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:16:11.075699+0000","last_peered":"2024-06-14T23:16:11.075699+0000","last_clean":"2024-06-14T23:16:11.075699+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:16:11.075699+0000","last_undegraded":"2024-06-14T23:16:11.075699+0000","last_fullsized":"2024-06-14T23:16:11.075699+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":332,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074384+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:16:11.074384+0000","last_peered":"2024-06-14T23:16:11.074384+0000","last_clean":"2024-06-14T23:16:11.074384+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:16:11.074384+0000","last_undegraded":"2024-06-14T23:16:11.074384+0000","last_fullsized":"2024-06-14T23:16:11.074384+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":691,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073272+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:16:11.073272+0000","last_peered":"2024-06-14T23:16:11.073272+0000","last_clean":"2024-06-14T23:16:11.073272+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:16:11.073272+0000","last_undegraded":"2024-06-14T23:16:11.073272+0000","last_fullsized":"2024-06-14T23:16:11.073272+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":384,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.069517+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:16:11.069517+0000","last_peered":"2024-06-14T23:16:11.069517+0000","last_clean":"2024-06-14T23:16:11.069517+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:16:11.069517+0000","last_undegraded":"2024-06-14T23:16:11.069517+0000","last_fullsized":"2024-06-14T23:16:11.069517+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":75,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076191+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:16:11.076191+0000","last_peered":"2024-06-14T23:16:11.076191+0000","last_clean":"2024-06-14T23:16:11.076191+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:16:11.076191+0000","last_undegraded":"2024-06-14T23:16:11.076191+0000","last_fullsized":"2024-06-14T23:16:11.076191+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":804,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.071934+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:16:11.071934+0000","last_peered":"2024-06-14T23:16:11.071934+0000","last_clean":"2024-06-14T23:16:11.071934+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:16:11.071934+0000","last_undegraded":"2024-06-14T23:16:11.071934+0000","last_fullsized":"2024-06-14T23:16:11.071934+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":7686127640,"num_objects":117282,"num_object_clones":0,"num_object_copies":351846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124390,"num_write_kb":7733441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23065604096,"data_stored":23065591880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81390,"ondisk_log_size":81390,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234507,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4795308,"kb_used_data":3280224,"kb_used_omap":1,"kb_used_meta":1515070,"kb_avail":88949844,"statfs":{"total":95995035648,"available":91084640256,"internally_reserved":0,"allocated":3358949376,"data_stored":3357017052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1551432024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.1399999999999999,"15min":0.93300000000000005},"min":{"1min":0.25900000000000001,"5min":0.24199999999999999,"15min":0.19800000000000001},"max":{"1min":2.3809999999999998,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":1.673},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.171,"15min":1.1120000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.224},"max":{"1min":3.4089999999999998,"5min":6.5149999999999997,"15min":6.9550000000000001},"last":0.40999999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:16:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1599999999999999,"15min":1.23},"min":{"1min":0.29699999999999999,"5min":0.25600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1719999999999997,"5min":6.0229999999999997,"15min":12.010999999999999},"last":0.376},{"interface":"front","average":{"1min":2.3740000000000001,"5min":1.512,"15min":1.331},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.22900000000000001},"max":{"1min":12.295,"5min":12.295,"15min":13.531000000000001},"last":1.6359999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.242,"15min":1.3640000000000001},"min":{"1min":0.309,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.9220000000000002,"5min":10.539,"15min":10.539},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.2230000000000001,"15min":1.3100000000000001},"min":{"1min":0.39900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.9849999999999999,"5min":10.56,"15min":10.56},"last":0.90500000000000003}]},{"osd":3,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.3979999999999999,"15min":1.3859999999999999},"min":{"1min":0.34999999999999998,"5min":0.30599999999999999,"15min":0.28299999999999997},"max":{"1min":2.956,"5min":17.991,"15min":20.645},"last":0.88},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.498,"15min":1.466},"min":{"1min":0.318,"5min":0.318,"15min":0.28299999999999997},"max":{"1min":3.3780000000000001,"5min":17.824999999999999,"15min":22.042999999999999},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.2689999999999999,"15min":1.079},"min":{"1min":0.311,"5min":0.26300000000000001,"15min":0.23100000000000001},"max":{"1min":2.6200000000000001,"5min":6.0490000000000004,"15min":11.342000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.5229999999999999,"15min":1.1479999999999999},"min":{"1min":0.33400000000000002,"5min":0.27000000000000002,"15min":0.22700000000000001},"max":{"1min":3.0070000000000001,"5min":10.499000000000001,"15min":12.651},"last":0.92900000000000005}]},{"osd":5,"last update":"Fri Jun 14 23:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.3149999999999999,"15min":1.048},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.23100000000000001},"max":{"1min":4.5330000000000004,"5min":5.734,"15min":6.4320000000000004},"last":2.1150000000000002},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.3640000000000001,"15min":1.1160000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.24399999999999999},"max":{"1min":5.3410000000000002,"5min":6.0899999999999999,"15min":6.2809999999999997},"last":0.438}]},{"osd":6,"last update":"Fri Jun 14 23:16:06 2024","interfaces":[{"interface":"back","average":{"1min":3.4940000000000002,"5min":2.2930000000000001,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":12.317,"5min":12.317,"15min":12.317},"last":1.7090000000000001},{"interface":"front","average":{"1min":2.2919999999999998,"5min":1.8720000000000001,"15min":1.5369999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.32600000000000001},"max":{"1min":4.843,"5min":10.087999999999999,"15min":10.087999999999999},"last":0.505}]}]},{"osd":6,"up_from":272,"seq":1168231104613,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4014920,"kb_used_data":3337664,"kb_used_omap":7,"kb_used_meta":677240,"kb_avail":89730232,"statfs":{"total":95995035648,"available":91883757568,"internally_reserved":0,"allocated":3417767936,"data_stored":3415859608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":693494202},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.95099999999999996,"15min":1.4099999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.5709999999999997,"5min":4.5709999999999997,"15min":6.6440000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.028,"15min":1.4710000000000001},"min":{"1min":0.32200000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.6420000000000003,"5min":4.6420000000000003,"15min":6.6790000000000003},"last":0.32200000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.113,"15min":1.8149999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.25800000000000001},"max":{"1min":5.6509999999999998,"5min":5.6509999999999998,"15min":8.173},"last":1.343},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.97299999999999998,"15min":1.704},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":10.380000000000001},"last":0.248}]},{"osd":2,"last update":"Fri Jun 14 23:15:32 2024","interfaces":[{"interface":"back","average":{"1min":2.5310000000000001,"5min":1.3979999999999999,"15min":1.21},"min":{"1min":0.32100000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":29.940999999999999,"5min":29.940999999999999,"15min":29.940999999999999},"last":1.2809999999999999},{"interface":"front","average":{"1min":2.5939999999999999,"5min":1.4990000000000001,"15min":1.2889999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":30.135999999999999,"5min":30.135999999999999,"15min":30.135999999999999},"last":1.3600000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.0349999999999999,"15min":1.548},"min":{"1min":0.36299999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.5330000000000004,"5min":4.5330000000000004,"15min":6.7089999999999996},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.635,"5min":1.3899999999999999,"15min":1.8149999999999999},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.1449999999999996,"5min":7.4969999999999999,"15min":8.4740000000000002},"last":1.2390000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":2.2349999999999999,"5min":1.484,"15min":1.45},"min":{"1min":0.51900000000000002,"5min":0.24399999999999999,"15min":0.22900000000000001},"max":{"1min":5.758,"5min":7.3159999999999998,"15min":11.010999999999999},"last":1.292},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.325,"15min":1.27},"min":{"1min":0.28799999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":6.4269999999999996,"5min":6.4269999999999996,"15min":6.4269999999999996},"last":0.372}]},{"osd":5,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":1.4490000000000001,"15min":1.379},"min":{"1min":0.33000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.0670000000000002,"5min":7.0670000000000002,"15min":7.7469999999999999},"last":2.778},{"interface":"front","average":{"1min":2.7429999999999999,"5min":1.597,"15min":1.593},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":6.7889999999999997,"5min":7.4359999999999999,"15min":7.8680000000000003},"last":5.4939999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":2.4489999999999998,"5min":1.7549999999999999,"15min":1.738},"min":{"1min":0.40000000000000002,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":6.827,"5min":10.016999999999999,"15min":10.016999999999999},"last":1.2130000000000001},{"interface":"front","average":{"1min":2.9350000000000001,"5min":1.839,"15min":1.625},"min":{"1min":0.36599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.906000000000001,"5min":10.906000000000001,"15min":10.906000000000001},"last":5.3719999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757782,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6193428,"kb_used_data":4705480,"kb_used_omap":1,"kb_used_meta":1487870,"kb_avail":87551724,"statfs":{"total":95995035648,"available":89652965376,"internally_reserved":0,"allocated":4818411520,"data_stored":4816499115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1523579224},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":1.1910000000000001,"15min":1.3979999999999999},"min":{"1min":0.26500000000000001,"5min":0.23999999999999999,"15min":0.20100000000000001},"max":{"1min":8.3770000000000007,"5min":8.3770000000000007,"15min":14.94},"last":0.78300000000000003},{"interface":"front","average":{"1min":2.7250000000000001,"5min":1.0940000000000001,"15min":1.383},"min":{"1min":0.245,"5min":0.245,"15min":0.219},"max":{"1min":13.398999999999999,"5min":13.398999999999999,"15min":13.398999999999999},"last":0.66200000000000003}]},{"osd":1,"last update":"Fri Jun 14 23:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.71499999999999997,"15min":0.88200000000000001},"min":{"1min":0.29899999999999999,"5min":0.249,"15min":0.23999999999999999},"max":{"1min":2.2010000000000001,"5min":3.093,"15min":7.4370000000000003},"last":2.8580000000000001},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.76100000000000001,"15min":1.077},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":5.657,"5min":5.952,"15min":9.5039999999999996},"last":0.82199999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":0.96099999999999997,"15min":0.753},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.7709999999999999,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.798,"5min":0.999,"15min":0.75700000000000001},"min":{"1min":0.29699999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.718,"5min":6.0659999999999998,"15min":6.0659999999999998},"last":0.91200000000000003}]},{"osd":3,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":3.044,"5min":1.462,"15min":1.444},"min":{"1min":0.34000000000000002,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":13.352,"5min":21.815000000000001,"15min":21.815000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":2.9729999999999999,"5min":1.5509999999999999,"15min":1.4750000000000001},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":8.4269999999999996,"5min":21.797000000000001,"15min":21.797000000000001},"last":0.88}]},{"osd":4,"last update":"Fri Jun 14 23:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":0.91700000000000004,"15min":0.92600000000000005},"min":{"1min":0.29299999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.0839999999999996,"5min":5.0839999999999996,"15min":14.352},"last":2.8290000000000002},{"interface":"front","average":{"1min":2.5009999999999999,"5min":1.151,"15min":1.1539999999999999},"min":{"1min":0.36499999999999999,"5min":0.27000000000000002,"15min":0.253},"max":{"1min":8.0350000000000001,"5min":8.0350000000000001,"15min":9.5980000000000008},"last":7.9950000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.214,"15min":1.373},"min":{"1min":0.374,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.3460000000000001,"5min":30.934000000000001,"15min":30.934000000000001},"last":2.9089999999999998},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.3360000000000001,"15min":1.6779999999999999},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.26100000000000001},"max":{"1min":4.1689999999999996,"5min":31.059999999999999,"15min":31.059999999999999},"last":4.4299999999999997}]},{"osd":7,"last update":"Fri Jun 14 23:15:29 2024","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":1.738,"15min":1.347},"min":{"1min":0.38800000000000001,"5min":0.25,"15min":0.25},"max":{"1min":8.9659999999999993,"5min":26.981999999999999,"15min":26.981999999999999},"last":4.0010000000000003},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.7030000000000001,"15min":1.2849999999999999},"min":{"1min":0.40500000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":8.407,"5min":23.326000000000001,"15min":23.326000000000001},"last":2.8860000000000001}]}]},{"osd":4,"up_from":30,"seq":128849019418,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5711200,"kb_used_data":4210772,"kb_used_omap":1,"kb_used_meta":1500414,"kb_avail":88033952,"statfs":{"total":95995035648,"available":90146766848,"internally_reserved":0,"allocated":4311830528,"data_stored":4309909613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1536424280},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":0.86599999999999999,"15min":1.1100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.17100000000000001},"max":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":10.567},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.93600000000000005,"15min":1.0880000000000001},"min":{"1min":0.32900000000000001,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3920000000000003,"5min":6.7779999999999996,"15min":7.4900000000000002},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Jun 14 23:16:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.88600000000000001,"15min":1.3660000000000001},"min":{"1min":0.33600000000000002,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":3.944,"5min":4.476,"15min":56.917000000000002},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.052,"5min":0.92800000000000005,"15min":1.431},"min":{"1min":0.34799999999999998,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":3.7890000000000001,"5min":4.6600000000000001,"15min":56.789999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.85299999999999998,"15min":0.82599999999999996},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.5899999999999999,"5min":3.468,"15min":3.468},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.90500000000000003,"15min":0.88900000000000001},"min":{"1min":0.32700000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.5129999999999999,"5min":3.4980000000000002,"15min":3.4980000000000002},"last":0.35699999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:16:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.91500000000000004,"15min":1.274},"min":{"1min":0.38300000000000001,"5min":0.30599999999999999,"15min":0.25},"max":{"1min":5.3540000000000001,"5min":5.3540000000000001,"15min":44.389000000000003},"last":0.626},{"interface":"front","average":{"1min":1.222,"5min":1.016,"15min":1.4530000000000001},"min":{"1min":0.33500000000000002,"5min":0.29599999999999999,"15min":0.27600000000000002},"max":{"1min":4.6120000000000001,"5min":4.6120000000000001,"15min":46.101999999999997},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:15:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.0529999999999999,"15min":1.0109999999999999},"min":{"1min":0.61699999999999999,"5min":0.24399999999999999,"15min":0.20300000000000001},"max":{"1min":5.4160000000000004,"5min":5.4160000000000004,"15min":12.315},"last":1.179},{"interface":"front","average":{"1min":1.972,"5min":1.1399999999999999,"15min":1.038},"min":{"1min":0.51500000000000001,"5min":0.28799999999999998,"15min":0.23300000000000001},"max":{"1min":6.5289999999999999,"5min":6.9290000000000003,"15min":10.959},"last":0.39800000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":2.548,"15min":1.5680000000000001},"min":{"1min":0.28399999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.5780000000000003,"5min":122.88500000000001,"15min":122.88500000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":2.1269999999999998,"5min":2.6640000000000001,"15min":1.863},"min":{"1min":0.65100000000000002,"5min":0.30499999999999999,"15min":0.25700000000000001},"max":{"1min":7.2439999999999998,"5min":122.21299999999999,"15min":122.21299999999999},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:15:27 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.2949999999999999,"15min":1.0629999999999999},"min":{"1min":0.40799999999999997,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":5.4699999999999998,"5min":6.5910000000000002,"15min":6.5910000000000002},"last":0.996},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.3779999999999999,"15min":1.0860000000000001},"min":{"1min":0.46899999999999997,"5min":0.33300000000000002,"15min":0.27300000000000002},"max":{"1min":2.028,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":1.03}]}]},{"osd":3,"up_from":138,"seq":592705487116,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3855488,"kb_used_data":2898936,"kb_used_omap":4,"kb_used_meta":956539,"kb_avail":89889664,"statfs":{"total":95995035648,"available":92047015936,"internally_reserved":0,"allocated":2968510464,"data_stored":2966605369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":979496414},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35799999999999998,"15min":1.1339999999999999},"min":{"1min":0.17199999999999999,"5min":0.16900000000000001,"15min":0.16800000000000001},"max":{"1min":0.84399999999999997,"5min":1.175,"15min":25.190000000000001},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.53400000000000003,"15min":1.23},"min":{"1min":0.246,"5min":0.20699999999999999,"15min":0.17799999999999999},"max":{"1min":0.93400000000000005,"5min":4.6369999999999996,"15min":25.652000000000001},"last":0.27300000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.56100000000000005,"15min":1.2090000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20300000000000001},"max":{"1min":0.998,"5min":4.6840000000000002,"15min":25.754000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.55400000000000005,"15min":1.3560000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.0189999999999999,"5min":2.6899999999999999,"15min":27.992999999999999},"last":0.29499999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:15:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.72899999999999998,"15min":0.84899999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.1659999999999999,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.27000000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.2709999999999999,"5min":3.5550000000000002,"15min":3.5550000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.1519999999999999,"15min":1.762},"min":{"1min":0.29699999999999999,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":2.419,"5min":17.018999999999998,"15min":73.263000000000005},"last":0.315},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.2609999999999999,"15min":1.651},"min":{"1min":0.30199999999999999,"5min":0.29499999999999998,"15min":0.27300000000000002},"max":{"1min":3.6699999999999999,"5min":17.074000000000002,"15min":73.5},"last":0.32900000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":0.92000000000000004,"15min":1.643},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.28699999999999998},"max":{"1min":3.339,"5min":3.4740000000000002,"15min":81.438999999999993},"last":12.718999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":0.93400000000000005,"15min":1.6930000000000001},"min":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.29299999999999998},"max":{"1min":2.8530000000000002,"5min":4.5990000000000002,"15min":81.626999999999995},"last":12.706}]},{"osd":6,"last update":"Fri Jun 14 23:16:12 2024","interfaces":[{"interface":"back","average":{"1min":1.867,"5min":1.5089999999999999,"15min":1.6579999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.34599999999999997},"max":{"1min":15.458,"5min":17.141999999999999,"15min":25.869},"last":15.458},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.599,"15min":1.5489999999999999},"min":{"1min":0.38600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":15.699,"5min":20.056999999999999,"15min":25.143000000000001},"last":15.699}]},{"osd":7,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.4850000000000001,"15min":1.4810000000000001},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":3.4060000000000001,"5min":44.887999999999998,"15min":44.887999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.635,"15min":1.427},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":3.5259999999999998,"5min":45.835000000000001,"15min":45.835000000000001},"last":0.38600000000000001}]}]},{"osd":2,"up_from":322,"seq":1382979469372,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4526464,"kb_used_data":4249848,"kb_used_omap":10,"kb_used_meta":276597,"kb_avail":89218688,"statfs":{"total":95995035648,"available":91359936512,"internally_reserved":0,"allocated":4351844352,"data_stored":4349939991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":283235878},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.44600000000000001,"15min":0.36299999999999999},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":0.76600000000000001,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.498,"15min":0.45800000000000002},"min":{"1min":0.27000000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.90700000000000003,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.32900000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.45300000000000001,"15min":0.41999999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.64800000000000002,"5min":3.4340000000000002,"15min":3.4340000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.54500000000000004,"15min":0.47299999999999998},"min":{"1min":0.26300000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.1799999999999999,"5min":5.0389999999999997,"15min":5.0389999999999997},"last":0.27700000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.65000000000000002,"15min":0.60999999999999999},"min":{"1min":0.24299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.86799999999999999,"5min":3.7519999999999998,"15min":3.7519999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.63600000000000001,"15min":0.55500000000000005},"min":{"1min":0.245,"5min":0.223,"15min":0.223},"max":{"1min":1.1439999999999999,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.29999999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.0860000000000001,"15min":0.96399999999999997},"min":{"1min":0.47299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.7859999999999996,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":3.8079999999999998},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.89100000000000001,"15min":0.73999999999999999},"min":{"1min":0.35599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.8220000000000001,"5min":4.3680000000000003,"15min":4.3680000000000003},"last":1.151}]},{"osd":5,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":0.86199999999999999,"15min":0.80600000000000005},"min":{"1min":0.33200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":5.9100000000000001,"5min":5.9100000000000001,"15min":5.9100000000000001},"last":0.379},{"interface":"front","average":{"1min":1.375,"5min":0.93300000000000005,"15min":0.89100000000000001},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":3.7810000000000001},"last":0.59199999999999997}]},{"osd":6,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.0629999999999999,"15min":0.86899999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.6989999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0049999999999999,"15min":0.98299999999999998},"min":{"1min":0.48699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.9209999999999998,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":0.39500000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.0509999999999999,"15min":0.97699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":4.4480000000000004,"5min":4.4480000000000004,"15min":4.4480000000000004},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.2529999999999999,"15min":1.151},"min":{"1min":0.45700000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.879,"5min":5.2130000000000001,"15min":5.2130000000000001},"last":0.49099999999999999}]}]},{"osd":1,"up_from":0,"seq":498216206660,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024315,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4330160128,"data_stored":4330160128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2944536576,"data_stored":2944532504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4292874240,"data_stored":4292870168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4778819584,"data_stored":4778819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3392405504,"data_stored":3392405504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3326808064,"data_stored":3326803992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:16:21.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:16:21.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:21.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:21.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:21.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:21.924 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:21.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:22.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:22.062 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:22.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:22.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:22.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:22.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:22.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:22.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:22.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:22.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:22.401 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:22 smithi086 ceph-mon[97913]: from='client.37059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:16:22.401 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:22 smithi086 ceph-mon[102754]: from='client.37059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:16:22.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:22.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:22.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:22 smithi137 ceph-mon[90634]: from='client.37059 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:16:22.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:22.582 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:22.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:22.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:22.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:22.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:22.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:22.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:22.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:22.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:22.980 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:22.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:23.054 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:16:23.054 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:16:23.079 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:16:23.079 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:16:23.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:23.107 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:16:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:16:23.116 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:23 smithi137 ceph-mon[90634]: pgmap v1398: 119 pgs: 119 active+clean; 7.1 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 102 B/s rd, 38 MiB/s wr, 994 op/s 2024-06-14T23:16:23.116 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3248235608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:23.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:23.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:23.134 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:16:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:16:23.155 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:16:23.155 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:16:23.181 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:16:23.182 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:16:23.210 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:16:23.210 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:16:23.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:23.241 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:16:23.241 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:16:23.251 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:23.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:23.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:23 smithi086 ceph-mon[97913]: pgmap v1398: 119 pgs: 119 active+clean; 7.1 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 102 B/s rd, 38 MiB/s wr, 994 op/s 2024-06-14T23:16:23.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3248235608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:23.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:23 smithi086 ceph-mon[102754]: pgmap v1398: 119 pgs: 119 active+clean; 7.1 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 102 B/s rd, 38 MiB/s wr, 994 op/s 2024-06-14T23:16:23.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3248235608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:23.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:23.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:23.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:23.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:23.457 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:16:23.457 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:16:23.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:16:23.457 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:16:23.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.457 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:190005 11.8 11.7dcd6c68 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e272)", 2024-06-14T23:16:23.457 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:57.018688+0000", 2024-06-14T23:16:23.457 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 506.433088338, 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98166048699999997, 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 190005 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018688+0000", 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018688+0000", 2024-06-14T23:16:23.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018686+0000", 2024-06-14T23:16:23.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:23.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018693+0000", 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4169999999999997e-06 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018704+0000", 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1572e-05 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018711+0000", 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.131e-06 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018882+0000", 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017030600000000001 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018889+0000", 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5290000000000003e-06 2024-06-14T23:16:23.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.000337+0000", 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98144818700000003 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.000348+0000", 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1017e-05 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:190018 11.8 11.11b9ebc8 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e272)", 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:57.018852+0000", 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 506.43292415899998, 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.981612916, 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 190018 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018852+0000", 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018852+0000", 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018851+0000", 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:23.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018856+0000", 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4580000000000003e-06 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018865+0000", 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6399999999999992e-06 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018872+0000", 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1970000000000001e-06 2024-06-14T23:16:23.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018912+0000", 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0429000000000002e-05 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018925+0000", 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2816000000000001e-05 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.000381+0000", 2024-06-14T23:16:23.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98145597500000004 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.000465+0000", 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4031999999999998e-05 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:190020 11.8 11.b1fa44f8 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e272)", 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:57.018878+0000", 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 506.432898292, 2024-06-14T23:16:23.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98237043599999996, 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 190020 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018878+0000", 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018878+0000", 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018877+0000", 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018880+0000", 2024-06-14T23:16:23.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0580000000000002e-06 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018887+0000", 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2879999999999998e-06 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018891+0000", 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8600000000000003e-06 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018953+0000", 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2670999999999997e-05 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.018958+0000", 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2300000000000002e-06 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.001232+0000", 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98227410900000001 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.001248+0000", 2024-06-14T23:16:23.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6648000000000001e-05 2024-06-14T23:16:23.474 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:190032 11.1 11.ddb5a631 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:57.023495+0000", 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 506.42828101600003, 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.976982448, 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.475 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 190032 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.023495+0000", 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.023495+0000", 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.023494+0000", 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.023500+0000", 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.849e-06 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.023511+0000", 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0202e-05 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.023517+0000", 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5610000000000004e-06 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.023573+0000", 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5368999999999998e-05 2024-06-14T23:16:23.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.023582+0000", 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1360000000000003e-06 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.000468+0000", 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97688614200000001 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.000478+0000", 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.747e-06 2024-06-14T23:16:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:190037 11.8 11.588b3ee8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:57.034391+0000", 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 506.41738516999999, 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96689680099999997, 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 190037 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.034391+0000", 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.034391+0000", 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.034390+0000", 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.034396+0000", 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0299999999999999e-06 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.034408+0000", 2024-06-14T23:16:23.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2006e-05 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.034414+0000", 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4420000000000003e-06 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.034462+0000", 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7447999999999997e-05 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.034471+0000", 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8320000000000001e-06 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.001274+0000", 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96680256499999995 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.001288+0000", 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3747000000000001e-05 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.484 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:190042 11.8 11.cb729068 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:57.039857+0000", 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 506.411919582, 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96391727900000002, 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 190042 2024-06-14T23:16:23.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.039857+0000", 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.039857+0000", 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.039855+0000", 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.039865+0000", 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9660000000000008e-06 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.039888+0000", 2024-06-14T23:16:23.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3065999999999999e-05 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.039898+0000", 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0356999999999999e-05 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.040025+0000", 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012664100000000001 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.040047+0000", 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2657e-05 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.003764+0000", 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96371669500000001 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.003774+0000", 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.747e-06 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.489 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:190047 11.1 11.8dc3f11 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:07:57.053607+0000", 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 506.39816873400002, 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94759226299999999, 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 190047 2024-06-14T23:16:23.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.053607+0000", 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.053607+0000", 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.053607+0000", 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.053611+0000", 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6759999999999998e-06 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.053622+0000", 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0679000000000001e-05 2024-06-14T23:16:23.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.053628+0000", 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5820000000000001e-06 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.053693+0000", 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5191999999999995e-05 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:57.053699+0000", 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.195e-06 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.001174+0000", 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94747471800000005 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:07:58.001200+0000", 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5928e-05 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.494 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211897 11.1 11:81a1cef4:::benchmark_data_smithi086_226760_object84576:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.006978+0000", 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 475.44479816199998, 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.250541216, 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211897 2024-06-14T23:16:23.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.006978+0000", 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.006978+0000", 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.006977+0000", 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.006982+0000", 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5659999999999997e-06 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.006993+0000", 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0892e-05 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007000+0000", 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1119999999999998e-06 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.011205+0000", 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042044719999999999 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.011211+0000", 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7020000000000004e-06 2024-06-14T23:16:23.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.257278+0000", 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2460667700000001 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.257519+0000", 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024101999999999999 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.499 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211898 11.3 11:c6ecd256:::benchmark_data_smithi086_226760_object84577:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.007006+0000", 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 475.44477001000001, 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0681876749999999, 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211898 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007006+0000", 2024-06-14T23:16:23.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007006+0000", 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007006+0000", 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007008+0000", 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.266e-06 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007015+0000", 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2529999999999998e-06 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007019+0000", 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0289999999999999e-06 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.010614+0000", 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035947840000000002 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.010622+0000", 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1990000000000006e-06 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.074904+0000", 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0642816470000001 2024-06-14T23:16:23.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.075194+0000", 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028989700000000001 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211900 11.1 11:87bb7fd2:::benchmark_data_smithi086_226760_object84579:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.007842+0000", 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 475.44393446700002, 2024-06-14T23:16:23.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2504614469999999, 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211900 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007842+0000", 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007842+0000", 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007841+0000", 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007846+0000", 2024-06-14T23:16:23.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8350000000000003e-06 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007855+0000", 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4199999999999996e-06 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007861+0000", 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1639999999999998e-06 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.011231+0000", 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033700819999999999 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.011237+0000", 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6149999999999996e-06 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.258065+0000", 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2468281450000001 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.258303+0000", 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000238032 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211906 11.3 11:c5a16678:::benchmark_data_smithi086_226760_object84585:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.007866+0000", 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 475.44390996700002, 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0761865559999999, 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.509 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211906 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007866+0000", 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007866+0000", 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007866+0000", 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.007870+0000", 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1649999999999999e-06 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008237+0000", 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036718100000000001 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008245+0000", 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7020000000000001e-06 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.010652+0000", 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024070039999999999 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.010658+0000", 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5939999999999999e-06 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.075251+0000", 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0645927829999999 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.075665+0000", 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00041383099999999998 2024-06-14T23:16:23.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.076420+0000", 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00075491200000000005 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.082994+0000", 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065736850000000001 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.083116+0000", 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122378 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.083380+0000", 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026442399999999997 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.083547+0000", 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016669300000000001 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.084053+0000", 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050561499999999995 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211909 11.3 11:c6a15f4b:::benchmark_data_smithi086_226760_object84588:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.008417+0000", 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 475.44335921599998, 2024-06-14T23:16:23.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.079672747, 2024-06-14T23:16:23.518 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.518 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.518 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.518 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.518 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.518 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211909 2024-06-14T23:16:23.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.519 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.519 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.519 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:23.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008417+0000", 2024-06-14T23:16:23.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008417+0000", 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008416+0000", 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008421+0000", 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7190000000000001e-06 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008429+0000", 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6770000000000001e-06 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008434+0000", 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9470000000000002e-06 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.010681+0000", 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022465129999999999 2024-06-14T23:16:23.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.010687+0000", 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3690000000000003e-06 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.078117+0000", 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0674303059999999 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.078519+0000", 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040188099999999998 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.080906+0000", 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002386885 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.084186+0000", 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003280271 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.084288+0000", 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101628 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.087410+0000", 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003122143 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.087609+0000", 2024-06-14T23:16:23.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019917700000000001 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.088090+0000", 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048029300000000002 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211911 11.8 11:173a28c9:::benchmark_data_smithi086_226760_object84590:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.008438+0000", 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 475.44333782699999, 2024-06-14T23:16:23.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.063088856, 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211911 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008438+0000", 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008438+0000", 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008438+0000", 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008440+0000", 2024-06-14T23:16:23.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1289999999999999e-06 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008446+0000", 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5389999999999998e-06 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.008449+0000", 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.613e-06 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.009489+0000", 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010395949999999999 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.009499+0000", 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0479000000000001e-05 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.066539+0000", 2024-06-14T23:16:23.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0570399029999999 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.067249+0000", 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00071030500000000005 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.068066+0000", 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00081680499999999998 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.070692+0000", 2024-06-14T23:16:23.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026260509999999999 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.070730+0000", 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7190999999999997e-05 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.070849+0000", 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011943900000000001 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.071020+0000", 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000171404 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.071527+0000", 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050678300000000002 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211915 11.11 11:8c098787:::benchmark_data_smithi086_226760_object84594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:23.533 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.092497+0000", 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 475.35927937500003, 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.164451795, 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211915 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.092497+0000", 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.092497+0000", 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.092496+0000", 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.092503+0000", 2024-06-14T23:16:23.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6140000000000003e-06 2024-06-14T23:16:23.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.092515+0000", 2024-06-14T23:16:23.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2326e-05 2024-06-14T23:16:23.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.537 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.537 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.092524+0000", 2024-06-14T23:16:23.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2039999999999997e-06 2024-06-14T23:16:23.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.537 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.537 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.537 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.092555+0000", 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1146000000000003e-05 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.092562+0000", 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8109999999999997e-06 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.249444+0000", 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.156882524 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.249834+0000", 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000390269 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.250744+0000", 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00090983100000000005 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.256071+0000", 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053269679999999996 2024-06-14T23:16:23.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.256214+0000", 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014326399999999999 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.256249+0000", 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5018000000000003e-05 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.256419+0000", 2024-06-14T23:16:23.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016925999999999999 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.256949+0000", 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052979999999999998 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211920 11.1 11:801c9c59:::benchmark_data_smithi086_226760_object84599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.117988+0000", 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 475.33378855400002, 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152486809, 2024-06-14T23:16:23.541 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211920 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.117988+0000", 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.542 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.117988+0000", 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.117986+0000", 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.117993+0000", 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0029999999999997e-06 2024-06-14T23:16:23.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.118005+0000", 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1865000000000001e-05 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.118010+0000", 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0239999999999997e-06 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.118048+0000", 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8146999999999999e-05 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.118057+0000", 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2010000000000005e-06 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.258347+0000", 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1402895049999999 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.258733+0000", 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038585300000000002 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.259681+0000", 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000948003 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.266397+0000", 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067162769999999997 2024-06-14T23:16:23.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.266507+0000", 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110362 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.269710+0000", 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032024890000000002 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.269945+0000", 2024-06-14T23:16:23.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023531 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.270474+0000", 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000529299 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36237.0:211937 11.11 11:8dcf2a6a:::benchmark_data_smithi086_226760_object84616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:08:28.166502+0000", 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 475.28527444600002, 2024-06-14T23:16:23.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.091490815, 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36237", 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211937 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.166502+0000", 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.166502+0000", 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.166499+0000", 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999976 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.166512+0000", 2024-06-14T23:16:23.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3154e-05 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.166542+0000", 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9745000000000001e-05 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.166559+0000", 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7119999999999999e-05 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.166654+0000", 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4548000000000003e-05 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:28.166661+0000", 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2250000000000003e-06 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.252445+0000", 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0857842689999999 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.252900+0000", 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00045424899999999999 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.253810+0000", 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000910413 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.257127+0000", 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033165619999999999 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.257274+0000", 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014751099999999999 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.257308+0000", 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4150000000000003e-05 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.257486+0000", 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017771099999999999 2024-06-14T23:16:23.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:08:29.257992+0000", 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050658500000000004 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:129230 12.5 12:a066fbee:::benchmark_data_smithi086_261492_object12725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:46.019841+0000", 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 277.43193488999998, 2024-06-14T23:16:23.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.31967097700000002, 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 129230 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019841+0000", 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.556 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019841+0000", 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019840+0000", 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019845+0000", 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4879999999999998e-06 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.557 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019857+0000", 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1355999999999999e-05 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019862+0000", 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0019999999999997e-06 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.212112+0000", 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.192250002 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.214077+0000", 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019654690000000001 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.242751+0000", 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028673700999999999 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.242841+0000", 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0150999999999998e-05 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.242981+0000", 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014017500000000001 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.288886+0000", 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045905051000000002 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.333634+0000", 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.044748084 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.333679+0000", 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4746999999999997e-05 2024-06-14T23:16:23.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.333735+0000", 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5760999999999997e-05 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.339512+0000", 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057774380000000002 2024-06-14T23:16:23.562 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:129255 12.5 12:aa1ad356:::benchmark_data_smithi086_261492_object12750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:46.061829+0000", 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 277.389946775, 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28357850400000001, 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:23.563 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 129255 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061829+0000", 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061829+0000", 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.564 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061828+0000", 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061834+0000", 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2260000000000002e-06 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061846+0000", 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2514000000000001e-05 2024-06-14T23:16:23.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061852+0000", 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.169e-06 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.214111+0000", 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.152259175 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.214693+0000", 2024-06-14T23:16:23.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058183400000000004 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.262221+0000", 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047528473000000002 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.262306+0000", 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5204000000000006e-05 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:23.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.262424+0000", 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118159 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.288893+0000", 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026468814 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.339567+0000", 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050673608000000002 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.339786+0000", 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021960999999999999 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.339882+0000", 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5200999999999999e-05 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.345408+0000", 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055261850000000003 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:129271 12.5 12:ab5a4303:::benchmark_data_smithi086_261492_object12766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:46.081893+0000", 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 277.369883034, 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26981149900000001, 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 129271 2024-06-14T23:16:23.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081893+0000", 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081893+0000", 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081892+0000", 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081897+0000", 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3229999999999999e-06 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081909+0000", 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2832e-05 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081923+0000", 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3682e-05 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.214723+0000", 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.132799429 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.215997+0000", 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012744060000000001 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.285772+0000", 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.069775167999999999 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.285855+0000", 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2842000000000004e-05 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.285986+0000", 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000130615 2024-06-14T23:16:23.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.328223+0000", 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.042237784 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.345483+0000", 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017259535999999999 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.345522+0000", 2024-06-14T23:16:23.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8757000000000002e-05 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.345572+0000", 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0362000000000003e-05 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.351705+0000", 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061325850000000003 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.576 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:152612 12.5 12:a222237b:::benchmark_data_smithi086_261492_object36107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-06-14T23:16:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:12:20.906053+0000", 2024-06-14T23:16:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 242.545723455, 2024-06-14T23:16:23.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.24355275700000001, 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 152612 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906053+0000", 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906053+0000", 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906052+0000", 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906057+0000", 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8000000000000004e-06 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906069+0000", 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1222e-05 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906074+0000", 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5509999999999999e-06 2024-06-14T23:16:23.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.029226+0000", 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12315158800000001 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.032720+0000", 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034940399999999999 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:16:23.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.067028+0000", 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034307799 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.067125+0000", 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7015000000000005e-05 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.067265+0000", 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014023400000000001 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.093412+0000", 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026147169000000001 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.145382+0000", 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.051969862999999998 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.145450+0000", 2024-06-14T23:16:23.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8207000000000006e-05 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.145516+0000", 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5505000000000004e-05 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.149605+0000", 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040898130000000003 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.584 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:16:23.585 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:16:23.585 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:16:23.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:23.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:23.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:23.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:23.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:23.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:23.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:23.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:23.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:23.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:24.008 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:24.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:24.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:24.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:24.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:24 smithi137 ceph-mon[90634]: pgmap v1399: 119 pgs: 119 active+clean; 6.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 102 B/s rd, 13 MiB/s wr, 1.02k op/s 2024-06-14T23:16:24.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:24.273 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:24.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:24.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:24.414 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:24.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:24.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T23:16:24.505 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:24 smithi086 ceph-mon[97913]: pgmap v1399: 119 pgs: 119 active+clean; 6.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 102 B/s rd, 13 MiB/s wr, 1.02k op/s 2024-06-14T23:16:24.506 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:24 smithi086 ceph-mon[102754]: pgmap v1399: 119 pgs: 119 active+clean; 6.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 102 B/s rd, 13 MiB/s wr, 1.02k op/s 2024-06-14T23:16:24.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:24.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:24.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:24.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:24.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:24.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:24.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:16:24.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:24.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:24.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:24.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:24.960 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:24.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:25.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:25.064 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:16:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:16:24] "GET /metrics HTTP/1.1" 200 36675 "" "Prometheus/2.43.0" 2024-06-14T23:16:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:25.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:25.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:25.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:25.226 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:25.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:25.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:25.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:25.352 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:25.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:25.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:25.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:25.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:25.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:25.624 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:25.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:25.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:25.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:25.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:25.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:25.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:25.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:16:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:16:26.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:26.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:26.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:26.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:26.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:26.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:26.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:26.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:26.308 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:26.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:26.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:26.433 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:26.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:26.528 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:26.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:26.571 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:26.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:26.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:26.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:26.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:26.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:26.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:26.839 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:26 smithi137 ceph-mon[90634]: pgmap v1400: 119 pgs: 119 active+clean; 6.8 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 102 B/s rd, 7.7 MiB/s wr, 1.03k op/s 2024-06-14T23:16:26.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:26.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:26.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:26.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:26.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:27.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:26 smithi086 ceph-mon[97913]: pgmap v1400: 119 pgs: 119 active+clean; 6.8 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 102 B/s rd, 7.7 MiB/s wr, 1.03k op/s 2024-06-14T23:16:27.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:26 smithi086 ceph-mon[97913]: from='client.47000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:16:27.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:26 smithi086 ceph-mon[102754]: pgmap v1400: 119 pgs: 119 active+clean; 6.8 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 102 B/s rd, 7.7 MiB/s wr, 1.03k op/s 2024-06-14T23:16:27.001 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:26 smithi086 ceph-mon[102754]: from='client.47000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:16:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:27.039 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1400,"stamp":"2024-06-14T23:16:25.780770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7317755870,"num_objects":111676,"num_object_clones":0,"num_object_copies":335028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111676,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130081,"num_write_kb":7733470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86668,"ondisk_log_size":86668,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":28213888,"kb_used_data":21536528,"kb_used_omap":27,"kb_used_meta":6676900,"kb_avail":534257024,"statfs":{"total":575970213888,"available":547079192576,"internally_reserved":0,"allocated":22053404672,"data_stored":22041909295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":6837146054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-514850792,"num_objects":-7855,"num_object_clones":0,"num_object_copies":-23565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7855,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10313,"num_write_kb":78593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002606"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":752,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076139+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:16:11.076139+0000","last_peered":"2024-06-14T23:16:11.076139+0000","last_clean":"2024-06-14T23:16:11.076139+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:16:11.076139+0000","last_undegraded":"2024-06-14T23:16:11.076139+0000","last_fullsized":"2024-06-14T23:16:11.076139+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":189,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073478+0000","last_change":"2024-06-14T23:11:23.439289+0000","last_active":"2024-06-14T23:16:11.073478+0000","last_peered":"2024-06-14T23:16:11.073478+0000","last_clean":"2024-06-14T23:16:11.073478+0000","last_became_active":"2024-06-14T23:11:23.438856+0000","last_became_peered":"2024-06-14T23:11:23.438856+0000","last_unstale":"2024-06-14T23:16:11.073478+0000","last_undegraded":"2024-06-14T23:16:11.073478+0000","last_fullsized":"2024-06-14T23:16:11.073478+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:49:06.932709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":359,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.066301+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:16:11.066301+0000","last_peered":"2024-06-14T23:16:11.066301+0000","last_clean":"2024-06-14T23:16:11.066301+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:16:11.066301+0000","last_undegraded":"2024-06-14T23:16:11.066301+0000","last_fullsized":"2024-06-14T23:16:11.066301+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":266,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075504+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:16:11.075504+0000","last_peered":"2024-06-14T23:16:11.075504+0000","last_clean":"2024-06-14T23:16:11.075504+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:16:11.075504+0000","last_undegraded":"2024-06-14T23:16:11.075504+0000","last_fullsized":"2024-06-14T23:16:11.075504+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":156,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:12.431488+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:16:12.431488+0000","last_peered":"2024-06-14T23:16:12.431488+0000","last_clean":"2024-06-14T23:16:12.431488+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:16:12.431488+0000","last_undegraded":"2024-06-14T23:16:12.431488+0000","last_fullsized":"2024-06-14T23:16:12.431488+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":816,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.078609+0000","last_change":"2024-06-14T23:14:16.297931+0000","last_active":"2024-06-14T23:16:11.078609+0000","last_peered":"2024-06-14T23:16:11.078609+0000","last_clean":"2024-06-14T23:16:11.078609+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:16:11.078609+0000","last_undegraded":"2024-06-14T23:16:11.078609+0000","last_fullsized":"2024-06-14T23:16:11.078609+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:14:16.297860+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:14:16.297860+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:46.938506+0000","scrub_duration":0.0016077540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":335,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074610+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:16:11.074610+0000","last_peered":"2024-06-14T23:16:11.074610+0000","last_clean":"2024-06-14T23:16:11.074610+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:16:11.074610+0000","last_undegraded":"2024-06-14T23:16:11.074610+0000","last_fullsized":"2024-06-14T23:16:11.074610+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":342,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075774+0000","last_change":"2024-06-14T23:15:09.033723+0000","last_active":"2024-06-14T23:16:11.075774+0000","last_peered":"2024-06-14T23:16:11.075774+0000","last_clean":"2024-06-14T23:16:11.075774+0000","last_became_active":"2024-06-14T23:15:09.033433+0000","last_became_peered":"2024-06-14T23:15:09.033433+0000","last_unstale":"2024-06-14T23:16:11.075774+0000","last_undegraded":"2024-06-14T23:16:11.075774+0000","last_fullsized":"2024-06-14T23:16:11.075774+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:39.865328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":524,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.077294+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:16:11.077294+0000","last_peered":"2024-06-14T23:16:11.077294+0000","last_clean":"2024-06-14T23:16:11.077294+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:16:11.077294+0000","last_undegraded":"2024-06-14T23:16:11.077294+0000","last_fullsized":"2024-06-14T23:16:11.077294+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":349,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.078803+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:16:11.078803+0000","last_peered":"2024-06-14T23:16:11.078803+0000","last_clean":"2024-06-14T23:16:11.078803+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:16:11.078803+0000","last_undegraded":"2024-06-14T23:16:11.078803+0000","last_fullsized":"2024-06-14T23:16:11.078803+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":342,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076027+0000","last_change":"2024-06-14T23:15:09.029798+0000","last_active":"2024-06-14T23:16:11.076027+0000","last_peered":"2024-06-14T23:16:11.076027+0000","last_clean":"2024-06-14T23:16:11.076027+0000","last_became_active":"2024-06-14T23:15:09.028148+0000","last_became_peered":"2024-06-14T23:15:09.028148+0000","last_unstale":"2024-06-14T23:16:11.076027+0000","last_undegraded":"2024-06-14T23:16:11.076027+0000","last_fullsized":"2024-06-14T23:16:11.076027+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:07:26.557680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":118,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075849+0000","last_change":"2024-06-14T23:15:09.030207+0000","last_active":"2024-06-14T23:16:11.075849+0000","last_peered":"2024-06-14T23:16:11.075849+0000","last_clean":"2024-06-14T23:16:11.075849+0000","last_became_active":"2024-06-14T23:15:09.029856+0000","last_became_peered":"2024-06-14T23:15:09.029856+0000","last_unstale":"2024-06-14T23:16:11.075849+0000","last_undegraded":"2024-06-14T23:16:11.075849+0000","last_fullsized":"2024-06-14T23:16:11.075849+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:50:21.877412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":166,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:17.103686+0000","last_change":"2024-06-14T23:11:23.416412+0000","last_active":"2024-06-14T23:16:17.103686+0000","last_peered":"2024-06-14T23:16:17.103686+0000","last_clean":"2024-06-14T23:16:17.103686+0000","last_became_active":"2024-06-14T23:11:23.404452+0000","last_became_peered":"2024-06-14T23:11:23.404452+0000","last_unstale":"2024-06-14T23:16:17.103686+0000","last_undegraded":"2024-06-14T23:16:17.103686+0000","last_fullsized":"2024-06-14T23:16:17.103686+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:54.949784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":675,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.081886+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:16:11.081886+0000","last_peered":"2024-06-14T23:16:11.081886+0000","last_clean":"2024-06-14T23:16:11.081886+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:16:11.081886+0000","last_undegraded":"2024-06-14T23:16:11.081886+0000","last_fullsized":"2024-06-14T23:16:11.081886+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":244,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075607+0000","last_change":"2024-06-14T23:15:09.033745+0000","last_active":"2024-06-14T23:16:11.075607+0000","last_peered":"2024-06-14T23:16:11.075607+0000","last_clean":"2024-06-14T23:16:11.075607+0000","last_became_active":"2024-06-14T23:15:09.033449+0000","last_became_peered":"2024-06-14T23:15:09.033449+0000","last_unstale":"2024-06-14T23:16:11.075607+0000","last_undegraded":"2024-06-14T23:16:11.075607+0000","last_fullsized":"2024-06-14T23:16:11.075607+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:33.546853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":163,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076217+0000","last_change":"2024-06-14T23:15:09.028225+0000","last_active":"2024-06-14T23:16:11.076217+0000","last_peered":"2024-06-14T23:16:11.076217+0000","last_clean":"2024-06-14T23:16:11.076217+0000","last_became_active":"2024-06-14T23:15:09.027806+0000","last_became_peered":"2024-06-14T23:15:09.027806+0000","last_unstale":"2024-06-14T23:16:11.076217+0000","last_undegraded":"2024-06-14T23:16:11.076217+0000","last_fullsized":"2024-06-14T23:16:11.076217+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:43.248153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":778,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073517+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:16:11.073517+0000","last_peered":"2024-06-14T23:16:11.073517+0000","last_clean":"2024-06-14T23:16:11.073517+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:16:11.073517+0000","last_undegraded":"2024-06-14T23:16:11.073517+0000","last_fullsized":"2024-06-14T23:16:11.073517+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":170,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075531+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:16:11.075531+0000","last_peered":"2024-06-14T23:16:11.075531+0000","last_clean":"2024-06-14T23:16:11.075531+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:16:11.075531+0000","last_undegraded":"2024-06-14T23:16:11.075531+0000","last_fullsized":"2024-06-14T23:16:11.075531+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":233,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:17.098768+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:16:17.098768+0000","last_peered":"2024-06-14T23:16:17.098768+0000","last_clean":"2024-06-14T23:16:17.098768+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:16:17.098768+0000","last_undegraded":"2024-06-14T23:16:17.098768+0000","last_fullsized":"2024-06-14T23:16:17.098768+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":726,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072154+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:16:11.072154+0000","last_peered":"2024-06-14T23:16:11.072154+0000","last_clean":"2024-06-14T23:16:11.072154+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:16:11.072154+0000","last_undegraded":"2024-06-14T23:16:11.072154+0000","last_fullsized":"2024-06-14T23:16:11.072154+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":94,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:12.431448+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:16:12.431448+0000","last_peered":"2024-06-14T23:16:12.431448+0000","last_clean":"2024-06-14T23:16:12.431448+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:16:12.431448+0000","last_undegraded":"2024-06-14T23:16:12.431448+0000","last_fullsized":"2024-06-14T23:16:12.431448+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":323,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.085891+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:16:11.085891+0000","last_peered":"2024-06-14T23:16:11.085891+0000","last_clean":"2024-06-14T23:16:11.085891+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:16:11.085891+0000","last_undegraded":"2024-06-14T23:16:11.085891+0000","last_fullsized":"2024-06-14T23:16:11.085891+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":625,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073904+0000","last_change":"2024-06-14T23:15:09.027462+0000","last_active":"2024-06-14T23:16:11.073904+0000","last_peered":"2024-06-14T23:16:11.073904+0000","last_clean":"2024-06-14T23:16:11.073904+0000","last_became_active":"2024-06-14T23:15:09.027093+0000","last_became_peered":"2024-06-14T23:15:09.027093+0000","last_unstale":"2024-06-14T23:16:11.073904+0000","last_undegraded":"2024-06-14T23:16:11.073904+0000","last_fullsized":"2024-06-14T23:16:11.073904+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:41:49.451565+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":356,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075311+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:16:11.075311+0000","last_peered":"2024-06-14T23:16:11.075311+0000","last_clean":"2024-06-14T23:16:11.075311+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:16:11.075311+0000","last_undegraded":"2024-06-14T23:16:11.075311+0000","last_fullsized":"2024-06-14T23:16:11.075311+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":219,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073532+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:16:11.073532+0000","last_peered":"2024-06-14T23:16:11.073532+0000","last_clean":"2024-06-14T23:16:11.073532+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:16:11.073532+0000","last_undegraded":"2024-06-14T23:16:11.073532+0000","last_fullsized":"2024-06-14T23:16:11.073532+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":678,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073315+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:16:11.073315+0000","last_peered":"2024-06-14T23:16:11.073315+0000","last_clean":"2024-06-14T23:16:11.073315+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:16:11.073315+0000","last_undegraded":"2024-06-14T23:16:11.073315+0000","last_fullsized":"2024-06-14T23:16:11.073315+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":239,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075299+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:16:11.075299+0000","last_peered":"2024-06-14T23:16:11.075299+0000","last_clean":"2024-06-14T23:16:11.075299+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:16:11.075299+0000","last_undegraded":"2024-06-14T23:16:11.075299+0000","last_fullsized":"2024-06-14T23:16:11.075299+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075419+0000","last_change":"2024-06-14T23:14:44.938003+0000","last_active":"2024-06-14T23:16:11.075419+0000","last_peered":"2024-06-14T23:16:11.075419+0000","last_clean":"2024-06-14T23:16:11.075419+0000","last_became_active":"2024-06-14T23:14:44.937736+0000","last_became_peered":"2024-06-14T23:14:44.937736+0000","last_unstale":"2024-06-14T23:16:11.075419+0000","last_undegraded":"2024-06-14T23:16:11.075419+0000","last_fullsized":"2024-06-14T23:16:11.075419+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:13:31.172524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":716,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075970+0000","last_change":"2024-06-14T23:14:17.053346+0000","last_active":"2024-06-14T23:16:11.075970+0000","last_peered":"2024-06-14T23:16:11.075970+0000","last_clean":"2024-06-14T23:16:11.075970+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:16:11.075970+0000","last_undegraded":"2024-06-14T23:16:11.075970+0000","last_fullsized":"2024-06-14T23:16:11.075970+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:17.053258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:17.053258+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:03.297247+0000","scrub_duration":0.0014699820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":257,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076356+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:16:11.076356+0000","last_peered":"2024-06-14T23:16:11.076356+0000","last_clean":"2024-06-14T23:16:11.076356+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:16:11.076356+0000","last_undegraded":"2024-06-14T23:16:11.076356+0000","last_fullsized":"2024-06-14T23:16:11.076356+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":385,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.081694+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:16:11.081694+0000","last_peered":"2024-06-14T23:16:11.081694+0000","last_clean":"2024-06-14T23:16:11.081694+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:16:11.081694+0000","last_undegraded":"2024-06-14T23:16:11.081694+0000","last_fullsized":"2024-06-14T23:16:11.081694+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":548,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073208+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:16:11.073208+0000","last_peered":"2024-06-14T23:16:11.073208+0000","last_clean":"2024-06-14T23:16:11.073208+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:16:11.073208+0000","last_undegraded":"2024-06-14T23:16:11.073208+0000","last_fullsized":"2024-06-14T23:16:11.073208+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":71,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072504+0000","last_change":"2024-06-14T23:15:09.030189+0000","last_active":"2024-06-14T23:16:11.072504+0000","last_peered":"2024-06-14T23:16:11.072504+0000","last_clean":"2024-06-14T23:16:11.072504+0000","last_became_active":"2024-06-14T23:15:09.029757+0000","last_became_peered":"2024-06-14T23:15:09.029757+0000","last_unstale":"2024-06-14T23:16:11.072504+0000","last_undegraded":"2024-06-14T23:16:11.072504+0000","last_fullsized":"2024-06-14T23:16:11.072504+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:58:48.305434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":313,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.082076+0000","last_change":"2024-06-14T23:15:09.044391+0000","last_active":"2024-06-14T23:16:11.082076+0000","last_peered":"2024-06-14T23:16:11.082076+0000","last_clean":"2024-06-14T23:16:11.082076+0000","last_became_active":"2024-06-14T23:15:09.043763+0000","last_became_peered":"2024-06-14T23:15:09.043763+0000","last_unstale":"2024-06-14T23:16:11.082076+0000","last_undegraded":"2024-06-14T23:16:11.082076+0000","last_fullsized":"2024-06-14T23:16:11.082076+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:46:25.597564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":538,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074150+0000","last_change":"2024-06-14T23:15:09.438359+0000","last_active":"2024-06-14T23:16:11.074150+0000","last_peered":"2024-06-14T23:16:11.074150+0000","last_clean":"2024-06-14T23:16:11.074150+0000","last_became_active":"2024-06-14T23:15:09.026760+0000","last_became_peered":"2024-06-14T23:15:09.026760+0000","last_unstale":"2024-06-14T23:16:11.074150+0000","last_undegraded":"2024-06-14T23:16:11.074150+0000","last_fullsized":"2024-06-14T23:16:11.074150+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:09.438276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:15:09.438276+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:07.051915+0000","scrub_duration":0.011767011000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"372'8040","reported_seq":8070,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:22.406735+0000","last_change":"2024-06-14T23:14:30.741601+0000","last_active":"2024-06-14T23:16:22.406735+0000","last_peered":"2024-06-14T23:16:22.406735+0000","last_clean":"2024-06-14T23:16:22.406735+0000","last_became_active":"2024-06-14T23:14:30.741403+0000","last_became_peered":"2024-06-14T23:14:30.741403+0000","last_unstale":"2024-06-14T23:16:22.406735+0000","last_undegraded":"2024-06-14T23:16:22.406735+0000","last_fullsized":"2024-06-14T23:16:22.406735+0000","mapping_epoch":351,"log_start":"358'2200","ondisk_log_start":"358'2200","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5840,"log_dups_size":0,"ondisk_log_size":5840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:32.635229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457572352,"num_objects":6982,"num_object_clones":0,"num_object_copies":20946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8040,"num_write_kb":480704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":249,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072078+0000","last_change":"2024-06-14T23:15:09.039701+0000","last_active":"2024-06-14T23:16:11.072078+0000","last_peered":"2024-06-14T23:16:11.072078+0000","last_clean":"2024-06-14T23:16:11.072078+0000","last_became_active":"2024-06-14T23:15:09.039273+0000","last_became_peered":"2024-06-14T23:15:09.039273+0000","last_unstale":"2024-06-14T23:16:11.072078+0000","last_undegraded":"2024-06-14T23:16:11.072078+0000","last_fullsized":"2024-06-14T23:16:11.072078+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:51:34.296727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":344,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.085677+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:16:11.085677+0000","last_peered":"2024-06-14T23:16:11.085677+0000","last_clean":"2024-06-14T23:16:11.085677+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:16:11.085677+0000","last_undegraded":"2024-06-14T23:16:11.085677+0000","last_fullsized":"2024-06-14T23:16:11.085677+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":56,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072808+0000","last_change":"2024-06-14T23:15:09.032054+0000","last_active":"2024-06-14T23:16:11.072808+0000","last_peered":"2024-06-14T23:16:11.072808+0000","last_clean":"2024-06-14T23:16:11.072808+0000","last_became_active":"2024-06-14T23:15:09.031613+0000","last_became_peered":"2024-06-14T23:15:09.031613+0000","last_unstale":"2024-06-14T23:16:11.072808+0000","last_undegraded":"2024-06-14T23:16:11.072808+0000","last_fullsized":"2024-06-14T23:16:11.072808+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:14.012846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":698,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076031+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:16:11.076031+0000","last_peered":"2024-06-14T23:16:11.076031+0000","last_clean":"2024-06-14T23:16:11.076031+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:16:11.076031+0000","last_undegraded":"2024-06-14T23:16:11.076031+0000","last_fullsized":"2024-06-14T23:16:11.076031+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"372'8228","reported_seq":8256,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:23.369512+0000","last_change":"2024-06-14T23:14:30.745071+0000","last_active":"2024-06-14T23:16:23.369512+0000","last_peered":"2024-06-14T23:16:23.369512+0000","last_clean":"2024-06-14T23:16:23.369512+0000","last_became_active":"2024-06-14T23:14:30.744589+0000","last_became_peered":"2024-06-14T23:14:30.744589+0000","last_unstale":"2024-06-14T23:16:23.369512+0000","last_undegraded":"2024-06-14T23:16:23.369512+0000","last_fullsized":"2024-06-14T23:16:23.369512+0000","mapping_epoch":351,"log_start":"360'3400","ondisk_log_start":"360'3400","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4828,"log_dups_size":0,"ondisk_log_size":4828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:20:16.097928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458489856,"num_objects":6996,"num_object_clones":0,"num_object_copies":20988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8228,"num_write_kb":487168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":287,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075383+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:16:11.075383+0000","last_peered":"2024-06-14T23:16:11.075383+0000","last_clean":"2024-06-14T23:16:11.075383+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:16:11.075383+0000","last_undegraded":"2024-06-14T23:16:11.075383+0000","last_fullsized":"2024-06-14T23:16:11.075383+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":288,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.077122+0000","last_change":"2024-06-14T23:15:09.029369+0000","last_active":"2024-06-14T23:16:11.077122+0000","last_peered":"2024-06-14T23:16:11.077122+0000","last_clean":"2024-06-14T23:16:11.077122+0000","last_became_active":"2024-06-14T23:15:09.029112+0000","last_became_peered":"2024-06-14T23:15:09.029112+0000","last_unstale":"2024-06-14T23:16:11.077122+0000","last_undegraded":"2024-06-14T23:16:11.077122+0000","last_fullsized":"2024-06-14T23:16:11.077122+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:39:40.009379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":86,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073108+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:16:11.073108+0000","last_peered":"2024-06-14T23:16:11.073108+0000","last_clean":"2024-06-14T23:16:11.073108+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:16:11.073108+0000","last_undegraded":"2024-06-14T23:16:11.073108+0000","last_fullsized":"2024-06-14T23:16:11.073108+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":587,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075442+0000","last_change":"2024-06-14T23:15:10.023278+0000","last_active":"2024-06-14T23:16:11.075442+0000","last_peered":"2024-06-14T23:16:11.075442+0000","last_clean":"2024-06-14T23:16:11.075442+0000","last_became_active":"2024-06-14T23:15:09.027691+0000","last_became_peered":"2024-06-14T23:15:09.027691+0000","last_unstale":"2024-06-14T23:16:11.075442+0000","last_undegraded":"2024-06-14T23:16:11.075442+0000","last_fullsized":"2024-06-14T23:16:11.075442+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:10.023159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:15:10.023159+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:28:13.357778+0000","scrub_duration":0.0066373669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.a","version":"372'7969","reported_seq":7999,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:22.429229+0000","last_change":"2024-06-14T23:14:30.743581+0000","last_active":"2024-06-14T23:16:22.429229+0000","last_peered":"2024-06-14T23:16:22.429229+0000","last_clean":"2024-06-14T23:16:22.429229+0000","last_became_active":"2024-06-14T23:14:30.743244+0000","last_became_peered":"2024-06-14T23:14:30.743244+0000","last_unstale":"2024-06-14T23:16:22.429229+0000","last_undegraded":"2024-06-14T23:16:22.429229+0000","last_fullsized":"2024-06-14T23:16:22.429229+0000","mapping_epoch":351,"log_start":"358'2100","ondisk_log_start":"358'2100","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:11:35.334274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453312512,"num_objects":6917,"num_object_clones":0,"num_object_copies":20751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7969,"num_write_kb":476352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":183,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072340+0000","last_change":"2024-06-14T23:11:23.429130+0000","last_active":"2024-06-14T23:16:11.072340+0000","last_peered":"2024-06-14T23:16:11.072340+0000","last_clean":"2024-06-14T23:16:11.072340+0000","last_became_active":"2024-06-14T23:11:23.428106+0000","last_became_peered":"2024-06-14T23:11:23.428106+0000","last_unstale":"2024-06-14T23:16:11.072340+0000","last_undegraded":"2024-06-14T23:16:11.072340+0000","last_fullsized":"2024-06-14T23:16:11.072340+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:51.011414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":357,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.069925+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:16:11.069925+0000","last_peered":"2024-06-14T23:16:11.069925+0000","last_clean":"2024-06-14T23:16:11.069925+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:16:11.069925+0000","last_undegraded":"2024-06-14T23:16:11.069925+0000","last_fullsized":"2024-06-14T23:16:11.069925+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":310,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076124+0000","last_change":"2024-06-14T23:11:23.418586+0000","last_active":"2024-06-14T23:16:11.076124+0000","last_peered":"2024-06-14T23:16:11.076124+0000","last_clean":"2024-06-14T23:16:11.076124+0000","last_became_active":"2024-06-14T23:11:23.406987+0000","last_became_peered":"2024-06-14T23:11:23.406987+0000","last_unstale":"2024-06-14T23:16:11.076124+0000","last_undegraded":"2024-06-14T23:16:11.076124+0000","last_fullsized":"2024-06-14T23:16:11.076124+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:20:09.974982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":671,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.082967+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:16:11.082967+0000","last_peered":"2024-06-14T23:16:11.082967+0000","last_clean":"2024-06-14T23:16:11.082967+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:16:11.082967+0000","last_undegraded":"2024-06-14T23:16:11.082967+0000","last_fullsized":"2024-06-14T23:16:11.082967+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"372'8246","reported_seq":8275,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:23.390295+0000","last_change":"2024-06-14T23:14:31.004195+0000","last_active":"2024-06-14T23:16:23.390295+0000","last_peered":"2024-06-14T23:16:23.390295+0000","last_clean":"2024-06-14T23:16:23.390295+0000","last_became_active":"2024-06-14T23:14:31.003779+0000","last_became_peered":"2024-06-14T23:14:31.003779+0000","last_unstale":"2024-06-14T23:16:23.390295+0000","last_undegraded":"2024-06-14T23:16:23.390295+0000","last_fullsized":"2024-06-14T23:16:23.390295+0000","mapping_epoch":351,"log_start":"360'3400","ondisk_log_start":"360'3400","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:42:29.429100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462815232,"num_objects":7062,"num_object_clones":0,"num_object_copies":21186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":148,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073835+0000","last_change":"2024-06-14T23:11:23.385942+0000","last_active":"2024-06-14T23:16:11.073835+0000","last_peered":"2024-06-14T23:16:11.073835+0000","last_clean":"2024-06-14T23:16:11.073835+0000","last_became_active":"2024-06-14T23:11:23.385673+0000","last_became_peered":"2024-06-14T23:11:23.385673+0000","last_unstale":"2024-06-14T23:16:11.073835+0000","last_undegraded":"2024-06-14T23:16:11.073835+0000","last_fullsized":"2024-06-14T23:16:11.073835+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:21.539058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075821+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:16:11.075821+0000","last_peered":"2024-06-14T23:16:11.075821+0000","last_clean":"2024-06-14T23:16:11.075821+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:16:11.075821+0000","last_undegraded":"2024-06-14T23:16:11.075821+0000","last_fullsized":"2024-06-14T23:16:11.075821+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":387,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074689+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:16:11.074689+0000","last_peered":"2024-06-14T23:16:11.074689+0000","last_clean":"2024-06-14T23:16:11.074689+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:16:11.074689+0000","last_undegraded":"2024-06-14T23:16:11.074689+0000","last_fullsized":"2024-06-14T23:16:11.074689+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":651,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.085756+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:16:11.085756+0000","last_peered":"2024-06-14T23:16:11.085756+0000","last_clean":"2024-06-14T23:16:11.085756+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:16:11.085756+0000","last_undegraded":"2024-06-14T23:16:11.085756+0000","last_fullsized":"2024-06-14T23:16:11.085756+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"372'8240","reported_seq":8278,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:23.900722+0000","last_change":"2024-06-14T23:15:09.042072+0000","last_active":"2024-06-14T23:16:23.900722+0000","last_peered":"2024-06-14T23:16:23.900722+0000","last_clean":"2024-06-14T23:16:23.900722+0000","last_became_active":"2024-06-14T23:15:09.034036+0000","last_became_peered":"2024-06-14T23:15:09.034036+0000","last_unstale":"2024-06-14T23:16:23.900722+0000","last_undegraded":"2024-06-14T23:16:23.900722+0000","last_fullsized":"2024-06-14T23:16:23.900722+0000","mapping_epoch":359,"log_start":"360'3200","ondisk_log_start":"360'3200","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:49:17.304835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458031128,"num_objects":6990,"num_object_clones":0,"num_object_copies":20970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6990,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8240,"num_write_kb":487297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":325,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075176+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:16:11.075176+0000","last_peered":"2024-06-14T23:16:11.075176+0000","last_clean":"2024-06-14T23:16:11.075176+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:16:11.075176+0000","last_undegraded":"2024-06-14T23:16:11.075176+0000","last_fullsized":"2024-06-14T23:16:11.075176+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":324,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.081936+0000","last_change":"2024-06-14T23:15:09.044549+0000","last_active":"2024-06-14T23:16:11.081936+0000","last_peered":"2024-06-14T23:16:11.081936+0000","last_clean":"2024-06-14T23:16:11.081936+0000","last_became_active":"2024-06-14T23:15:09.044122+0000","last_became_peered":"2024-06-14T23:15:09.044122+0000","last_unstale":"2024-06-14T23:16:11.081936+0000","last_undegraded":"2024-06-14T23:16:11.081936+0000","last_fullsized":"2024-06-14T23:16:11.081936+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:11:38.624888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":364,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074044+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:16:11.074044+0000","last_peered":"2024-06-14T23:16:11.074044+0000","last_clean":"2024-06-14T23:16:11.074044+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:16:11.074044+0000","last_undegraded":"2024-06-14T23:16:11.074044+0000","last_fullsized":"2024-06-14T23:16:11.074044+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":765,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072012+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:16:11.072012+0000","last_peered":"2024-06-14T23:16:11.072012+0000","last_clean":"2024-06-14T23:16:11.072012+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:16:11.072012+0000","last_undegraded":"2024-06-14T23:16:11.072012+0000","last_fullsized":"2024-06-14T23:16:11.072012+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"372'8210","reported_seq":8239,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:23.383266+0000","last_change":"2024-06-14T23:14:30.743278+0000","last_active":"2024-06-14T23:16:23.383266+0000","last_peered":"2024-06-14T23:16:23.383266+0000","last_clean":"2024-06-14T23:16:23.383266+0000","last_became_active":"2024-06-14T23:14:30.743000+0000","last_became_peered":"2024-06-14T23:14:30.743000+0000","last_unstale":"2024-06-14T23:16:23.383266+0000","last_undegraded":"2024-06-14T23:16:23.383266+0000","last_fullsized":"2024-06-14T23:16:23.383266+0000","mapping_epoch":351,"log_start":"360'3400","ondisk_log_start":"360'3400","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:22:35.618792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455999488,"num_objects":6958,"num_object_clones":0,"num_object_copies":20874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8210,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":332,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073312+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:16:11.073312+0000","last_peered":"2024-06-14T23:16:11.073312+0000","last_clean":"2024-06-14T23:16:11.073312+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:16:11.073312+0000","last_undegraded":"2024-06-14T23:16:11.073312+0000","last_fullsized":"2024-06-14T23:16:11.073312+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":370,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074543+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:16:11.074543+0000","last_peered":"2024-06-14T23:16:11.074543+0000","last_clean":"2024-06-14T23:16:11.074543+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:16:11.074543+0000","last_undegraded":"2024-06-14T23:16:11.074543+0000","last_fullsized":"2024-06-14T23:16:11.074543+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":175,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076263+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:16:11.076263+0000","last_peered":"2024-06-14T23:16:11.076263+0000","last_clean":"2024-06-14T23:16:11.076263+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:16:11.076263+0000","last_undegraded":"2024-06-14T23:16:11.076263+0000","last_fullsized":"2024-06-14T23:16:11.076263+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":578,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074485+0000","last_change":"2024-06-14T23:15:09.027736+0000","last_active":"2024-06-14T23:16:11.074485+0000","last_peered":"2024-06-14T23:16:11.074485+0000","last_clean":"2024-06-14T23:16:11.074485+0000","last_became_active":"2024-06-14T23:15:09.027333+0000","last_became_peered":"2024-06-14T23:15:09.027333+0000","last_unstale":"2024-06-14T23:16:11.074485+0000","last_undegraded":"2024-06-14T23:16:11.074485+0000","last_fullsized":"2024-06-14T23:16:11.074485+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:25:29.986791+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.f","version":"372'7982","reported_seq":8011,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:21.880326+0000","last_change":"2024-06-14T23:14:30.740225+0000","last_active":"2024-06-14T23:16:21.880326+0000","last_peered":"2024-06-14T23:16:21.880326+0000","last_clean":"2024-06-14T23:16:21.880326+0000","last_became_active":"2024-06-14T23:14:30.739814+0000","last_became_peered":"2024-06-14T23:14:30.739814+0000","last_unstale":"2024-06-14T23:16:21.880326+0000","last_undegraded":"2024-06-14T23:16:21.880326+0000","last_fullsized":"2024-06-14T23:16:21.880326+0000","mapping_epoch":351,"log_start":"358'2800","ondisk_log_start":"358'2800","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5182,"log_dups_size":0,"ondisk_log_size":5182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:27:30.267316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463077376,"num_objects":7066,"num_object_clones":0,"num_object_copies":21198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7982,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":343,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075639+0000","last_change":"2024-06-14T23:11:23.402821+0000","last_active":"2024-06-14T23:16:11.075639+0000","last_peered":"2024-06-14T23:16:11.075639+0000","last_clean":"2024-06-14T23:16:11.075639+0000","last_became_active":"2024-06-14T23:11:23.398164+0000","last_became_peered":"2024-06-14T23:11:23.398164+0000","last_unstale":"2024-06-14T23:16:11.075639+0000","last_undegraded":"2024-06-14T23:16:11.075639+0000","last_fullsized":"2024-06-14T23:16:11.075639+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:24.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":163,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075379+0000","last_change":"2024-06-14T23:15:09.025369+0000","last_active":"2024-06-14T23:16:11.075379+0000","last_peered":"2024-06-14T23:16:11.075379+0000","last_clean":"2024-06-14T23:16:11.075379+0000","last_became_active":"2024-06-14T23:15:09.025108+0000","last_became_peered":"2024-06-14T23:15:09.025108+0000","last_unstale":"2024-06-14T23:16:11.075379+0000","last_undegraded":"2024-06-14T23:16:11.075379+0000","last_fullsized":"2024-06-14T23:16:11.075379+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:04:22.310031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":339,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:17.097846+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:16:17.097846+0000","last_peered":"2024-06-14T23:16:17.097846+0000","last_clean":"2024-06-14T23:16:17.097846+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:16:17.097846+0000","last_undegraded":"2024-06-14T23:16:17.097846+0000","last_fullsized":"2024-06-14T23:16:17.097846+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":526,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.071977+0000","last_change":"2024-06-14T23:15:09.035281+0000","last_active":"2024-06-14T23:16:11.071977+0000","last_peered":"2024-06-14T23:16:11.071977+0000","last_clean":"2024-06-14T23:16:11.071977+0000","last_became_active":"2024-06-14T23:15:09.034141+0000","last_became_peered":"2024-06-14T23:15:09.034141+0000","last_unstale":"2024-06-14T23:16:11.071977+0000","last_undegraded":"2024-06-14T23:16:11.071977+0000","last_fullsized":"2024-06-14T23:16:11.071977+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:36.329142+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"372'8049","reported_seq":8078,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:23.400524+0000","last_change":"2024-06-14T23:14:30.745351+0000","last_active":"2024-06-14T23:16:23.400524+0000","last_peered":"2024-06-14T23:16:23.400524+0000","last_clean":"2024-06-14T23:16:23.400524+0000","last_became_active":"2024-06-14T23:14:30.745017+0000","last_became_peered":"2024-06-14T23:14:30.745017+0000","last_unstale":"2024-06-14T23:16:23.400524+0000","last_undegraded":"2024-06-14T23:16:23.400524+0000","last_fullsized":"2024-06-14T23:16:23.400524+0000","mapping_epoch":351,"log_start":"360'3200","ondisk_log_start":"360'3200","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:32.069491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450822144,"num_objects":6879,"num_object_clones":0,"num_object_copies":20637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8049,"num_write_kb":477696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":287,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.082562+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:16:11.082562+0000","last_peered":"2024-06-14T23:16:11.082562+0000","last_clean":"2024-06-14T23:16:11.082562+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:16:11.082562+0000","last_undegraded":"2024-06-14T23:16:11.082562+0000","last_fullsized":"2024-06-14T23:16:11.082562+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":293,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074014+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:16:11.074014+0000","last_peered":"2024-06-14T23:16:11.074014+0000","last_clean":"2024-06-14T23:16:11.074014+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:16:11.074014+0000","last_undegraded":"2024-06-14T23:16:11.074014+0000","last_fullsized":"2024-06-14T23:16:11.074014+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":374,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073680+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:16:11.073680+0000","last_peered":"2024-06-14T23:16:11.073680+0000","last_clean":"2024-06-14T23:16:11.073680+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:16:11.073680+0000","last_undegraded":"2024-06-14T23:16:11.073680+0000","last_fullsized":"2024-06-14T23:16:11.073680+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":665,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074467+0000","last_change":"2024-06-14T23:14:18.033116+0000","last_active":"2024-06-14T23:16:11.074467+0000","last_peered":"2024-06-14T23:16:11.074467+0000","last_clean":"2024-06-14T23:16:11.074467+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:16:11.074467+0000","last_undegraded":"2024-06-14T23:16:11.074467+0000","last_fullsized":"2024-06-14T23:16:11.074467+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:18.032997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:18.032997+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:57:13.313132+0000","scrub_duration":0.0019777449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.6","version":"372'8133","reported_seq":8170,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:23.347192+0000","last_change":"2024-06-14T23:15:09.037260+0000","last_active":"2024-06-14T23:16:23.347192+0000","last_peered":"2024-06-14T23:16:23.347192+0000","last_clean":"2024-06-14T23:16:23.347192+0000","last_became_active":"2024-06-14T23:15:09.033102+0000","last_became_peered":"2024-06-14T23:15:09.033102+0000","last_unstale":"2024-06-14T23:16:23.347192+0000","last_undegraded":"2024-06-14T23:16:23.347192+0000","last_fullsized":"2024-06-14T23:16:23.347192+0000","mapping_epoch":359,"log_start":"360'3300","ondisk_log_start":"360'3300","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4833,"log_dups_size":0,"ondisk_log_size":4833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:35:04.940982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456851456,"num_objects":6971,"num_object_clones":0,"num_object_copies":20913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":171,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076850+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:16:11.076850+0000","last_peered":"2024-06-14T23:16:11.076850+0000","last_clean":"2024-06-14T23:16:11.076850+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:16:11.076850+0000","last_undegraded":"2024-06-14T23:16:11.076850+0000","last_fullsized":"2024-06-14T23:16:11.076850+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":343,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075511+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:16:11.075511+0000","last_peered":"2024-06-14T23:16:11.075511+0000","last_clean":"2024-06-14T23:16:11.075511+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:16:11.075511+0000","last_undegraded":"2024-06-14T23:16:11.075511+0000","last_fullsized":"2024-06-14T23:16:11.075511+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":763,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073693+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:16:11.073693+0000","last_peered":"2024-06-14T23:16:11.073693+0000","last_clean":"2024-06-14T23:16:11.073693+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:16:11.073693+0000","last_undegraded":"2024-06-14T23:16:11.073693+0000","last_fullsized":"2024-06-14T23:16:11.073693+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"372'8266","reported_seq":8295,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:23.941991+0000","last_change":"2024-06-14T23:14:30.742252+0000","last_active":"2024-06-14T23:16:23.941991+0000","last_peered":"2024-06-14T23:16:23.941991+0000","last_clean":"2024-06-14T23:16:23.941991+0000","last_became_active":"2024-06-14T23:14:30.741955+0000","last_became_peered":"2024-06-14T23:14:30.741955+0000","last_unstale":"2024-06-14T23:16:23.941991+0000","last_undegraded":"2024-06-14T23:16:23.941991+0000","last_fullsized":"2024-06-14T23:16:23.941991+0000","mapping_epoch":351,"log_start":"360'3200","ondisk_log_start":"360'3200","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.550055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457441280,"num_objects":6980,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8266,"num_write_kb":487872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":359,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.066675+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:16:11.066675+0000","last_peered":"2024-06-14T23:16:11.066675+0000","last_clean":"2024-06-14T23:16:11.066675+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:16:11.066675+0000","last_undegraded":"2024-06-14T23:16:11.066675+0000","last_fullsized":"2024-06-14T23:16:11.066675+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":336,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076132+0000","last_change":"2024-06-14T23:15:09.029095+0000","last_active":"2024-06-14T23:16:11.076132+0000","last_peered":"2024-06-14T23:16:11.076132+0000","last_clean":"2024-06-14T23:16:11.076132+0000","last_became_active":"2024-06-14T23:15:09.028817+0000","last_became_peered":"2024-06-14T23:15:09.028817+0000","last_unstale":"2024-06-14T23:16:11.076132+0000","last_undegraded":"2024-06-14T23:16:11.076132+0000","last_fullsized":"2024-06-14T23:16:11.076132+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:43:20.264367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":731,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072274+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:16:11.072274+0000","last_peered":"2024-06-14T23:16:11.072274+0000","last_clean":"2024-06-14T23:16:11.072274+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:16:11.072274+0000","last_undegraded":"2024-06-14T23:16:11.072274+0000","last_fullsized":"2024-06-14T23:16:11.072274+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"372'8017","reported_seq":8046,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:21.860897+0000","last_change":"2024-06-14T23:14:30.742997+0000","last_active":"2024-06-14T23:16:21.860897+0000","last_peered":"2024-06-14T23:16:21.860897+0000","last_clean":"2024-06-14T23:16:21.860897+0000","last_became_active":"2024-06-14T23:14:30.742733+0000","last_became_peered":"2024-06-14T23:14:30.742733+0000","last_unstale":"2024-06-14T23:16:21.860897+0000","last_undegraded":"2024-06-14T23:16:21.860897+0000","last_fullsized":"2024-06-14T23:16:21.860897+0000","mapping_epoch":351,"log_start":"358'2900","ondisk_log_start":"358'2900","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5117,"log_dups_size":0,"ondisk_log_size":5117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:48.248776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461045760,"num_objects":7035,"num_object_clones":0,"num_object_copies":21105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":481664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":355,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074507+0000","last_change":"2024-06-14T23:15:09.028336+0000","last_active":"2024-06-14T23:16:11.074507+0000","last_peered":"2024-06-14T23:16:11.074507+0000","last_clean":"2024-06-14T23:16:11.074507+0000","last_became_active":"2024-06-14T23:15:09.028078+0000","last_became_peered":"2024-06-14T23:15:09.028078+0000","last_unstale":"2024-06-14T23:16:11.074507+0000","last_undegraded":"2024-06-14T23:16:11.074507+0000","last_fullsized":"2024-06-14T23:16:11.074507+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:36:57.434975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":224,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.081768+0000","last_change":"2024-06-14T23:15:09.037746+0000","last_active":"2024-06-14T23:16:11.081768+0000","last_peered":"2024-06-14T23:16:11.081768+0000","last_clean":"2024-06-14T23:16:11.081768+0000","last_became_active":"2024-06-14T23:15:09.037427+0000","last_became_peered":"2024-06-14T23:15:09.037427+0000","last_unstale":"2024-06-14T23:16:11.081768+0000","last_undegraded":"2024-06-14T23:16:11.081768+0000","last_fullsized":"2024-06-14T23:16:11.081768+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:40:34.511973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":754,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073362+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:16:11.073362+0000","last_peered":"2024-06-14T23:16:11.073362+0000","last_clean":"2024-06-14T23:16:11.073362+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:16:11.073362+0000","last_undegraded":"2024-06-14T23:16:11.073362+0000","last_fullsized":"2024-06-14T23:16:11.073362+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.5","version":"372'8166","reported_seq":8204,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:23.379827+0000","last_change":"2024-06-14T23:15:09.032574+0000","last_active":"2024-06-14T23:16:23.379827+0000","last_peered":"2024-06-14T23:16:23.379827+0000","last_clean":"2024-06-14T23:16:23.379827+0000","last_became_active":"2024-06-14T23:15:09.027289+0000","last_became_peered":"2024-06-14T23:15:09.027289+0000","last_unstale":"2024-06-14T23:16:23.379827+0000","last_undegraded":"2024-06-14T23:16:23.379827+0000","last_fullsized":"2024-06-14T23:16:23.379827+0000","mapping_epoch":359,"log_start":"360'3400","ondisk_log_start":"360'3400","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:31:01.504384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465960960,"num_objects":7110,"num_object_clones":0,"num_object_copies":21330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8166,"num_write_kb":488832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":266,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.082760+0000","last_change":"2024-06-14T23:15:09.038060+0000","last_active":"2024-06-14T23:16:11.082760+0000","last_peered":"2024-06-14T23:16:11.082760+0000","last_clean":"2024-06-14T23:16:11.082760+0000","last_became_active":"2024-06-14T23:15:09.037814+0000","last_became_peered":"2024-06-14T23:15:09.037814+0000","last_unstale":"2024-06-14T23:16:11.082760+0000","last_undegraded":"2024-06-14T23:16:11.082760+0000","last_fullsized":"2024-06-14T23:16:11.082760+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:48.977466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":120,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.069339+0000","last_change":"2024-06-14T23:15:09.024949+0000","last_active":"2024-06-14T23:16:11.069339+0000","last_peered":"2024-06-14T23:16:11.069339+0000","last_clean":"2024-06-14T23:16:11.069339+0000","last_became_active":"2024-06-14T23:15:09.023224+0000","last_became_peered":"2024-06-14T23:15:09.023224+0000","last_unstale":"2024-06-14T23:16:11.069339+0000","last_undegraded":"2024-06-14T23:16:11.069339+0000","last_fullsized":"2024-06-14T23:16:11.069339+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:12.547802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":656,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076053+0000","last_change":"2024-06-14T23:14:14.052730+0000","last_active":"2024-06-14T23:16:11.076053+0000","last_peered":"2024-06-14T23:16:11.076053+0000","last_clean":"2024-06-14T23:16:11.076053+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:16:11.076053+0000","last_undegraded":"2024-06-14T23:16:11.076053+0000","last_fullsized":"2024-06-14T23:16:11.076053+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:14.052631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:14.052631+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:51:25.441426+0000","scrub_duration":0.0017584180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"372'7988","reported_seq":8017,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:23.404390+0000","last_change":"2024-06-14T23:14:31.003884+0000","last_active":"2024-06-14T23:16:23.404390+0000","last_peered":"2024-06-14T23:16:23.404390+0000","last_clean":"2024-06-14T23:16:23.404390+0000","last_became_active":"2024-06-14T23:14:31.003542+0000","last_became_peered":"2024-06-14T23:14:31.003542+0000","last_unstale":"2024-06-14T23:16:23.404390+0000","last_undegraded":"2024-06-14T23:16:23.404390+0000","last_fullsized":"2024-06-14T23:16:23.404390+0000","mapping_epoch":351,"log_start":"360'3200","ondisk_log_start":"360'3200","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4788,"log_dups_size":0,"ondisk_log_size":4788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:27:38.082967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448397312,"num_objects":6842,"num_object_clones":0,"num_object_copies":20526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7988,"num_write_kb":474560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":179,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074436+0000","last_change":"2024-06-14T23:15:09.022932+0000","last_active":"2024-06-14T23:16:11.074436+0000","last_peered":"2024-06-14T23:16:11.074436+0000","last_clean":"2024-06-14T23:16:11.074436+0000","last_became_active":"2024-06-14T23:15:09.022603+0000","last_became_peered":"2024-06-14T23:15:09.022603+0000","last_unstale":"2024-06-14T23:16:11.074436+0000","last_undegraded":"2024-06-14T23:16:11.074436+0000","last_fullsized":"2024-06-14T23:16:11.074436+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:03:53.567467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":350,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.078814+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:16:11.078814+0000","last_peered":"2024-06-14T23:16:11.078814+0000","last_clean":"2024-06-14T23:16:11.078814+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:16:11.078814+0000","last_undegraded":"2024-06-14T23:16:11.078814+0000","last_fullsized":"2024-06-14T23:16:11.078814+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":493,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.072722+0000","last_change":"2024-06-14T23:15:12.943908+0000","last_active":"2024-06-14T23:16:11.072722+0000","last_peered":"2024-06-14T23:16:11.072722+0000","last_clean":"2024-06-14T23:16:11.072722+0000","last_became_active":"2024-06-14T23:15:09.031629+0000","last_became_peered":"2024-06-14T23:15:09.031629+0000","last_unstale":"2024-06-14T23:16:11.072722+0000","last_undegraded":"2024-06-14T23:16:11.072722+0000","last_fullsized":"2024-06-14T23:16:11.072722+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:12.943827+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:15:12.943827+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:48:02.155745+0000","scrub_duration":0.0029576450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3","version":"372'8360","reported_seq":8397,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:25.684454+0000","last_change":"2024-06-14T23:15:09.025883+0000","last_active":"2024-06-14T23:16:25.684454+0000","last_peered":"2024-06-14T23:16:25.684454+0000","last_clean":"2024-06-14T23:16:25.684454+0000","last_became_active":"2024-06-14T23:15:09.022582+0000","last_became_peered":"2024-06-14T23:15:09.022582+0000","last_unstale":"2024-06-14T23:16:25.684454+0000","last_undegraded":"2024-06-14T23:16:25.684454+0000","last_fullsized":"2024-06-14T23:16:25.684454+0000","mapping_epoch":359,"log_start":"360'3500","ondisk_log_start":"360'3500","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:20:48.026157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444334080,"num_objects":6780,"num_object_clones":0,"num_object_copies":20340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8360,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":323,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.081562+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:16:11.081562+0000","last_peered":"2024-06-14T23:16:11.081562+0000","last_clean":"2024-06-14T23:16:11.081562+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:16:11.081562+0000","last_undegraded":"2024-06-14T23:16:11.081562+0000","last_fullsized":"2024-06-14T23:16:11.081562+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":267,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074311+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:16:11.074311+0000","last_peered":"2024-06-14T23:16:11.074311+0000","last_clean":"2024-06-14T23:16:11.074311+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:16:11.074311+0000","last_undegraded":"2024-06-14T23:16:11.074311+0000","last_fullsized":"2024-06-14T23:16:11.074311+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":651,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.085826+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:16:11.085826+0000","last_peered":"2024-06-14T23:16:11.085826+0000","last_clean":"2024-06-14T23:16:11.085826+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:16:11.085826+0000","last_undegraded":"2024-06-14T23:16:11.085826+0000","last_fullsized":"2024-06-14T23:16:11.085826+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"372'8132","reported_seq":8162,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:22.363149+0000","last_change":"2024-06-14T23:14:30.738940+0000","last_active":"2024-06-14T23:16:22.363149+0000","last_peered":"2024-06-14T23:16:22.363149+0000","last_clean":"2024-06-14T23:16:22.363149+0000","last_became_active":"2024-06-14T23:14:30.738700+0000","last_became_peered":"2024-06-14T23:14:30.738700+0000","last_unstale":"2024-06-14T23:16:22.363149+0000","last_undegraded":"2024-06-14T23:16:22.363149+0000","last_fullsized":"2024-06-14T23:16:22.363149+0000","mapping_epoch":351,"log_start":"358'2300","ondisk_log_start":"358'2300","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:29:01.585691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463994880,"num_objects":7080,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8132,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":312,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075454+0000","last_change":"2024-06-14T23:15:09.030332+0000","last_active":"2024-06-14T23:16:11.075454+0000","last_peered":"2024-06-14T23:16:11.075454+0000","last_clean":"2024-06-14T23:16:11.075454+0000","last_became_active":"2024-06-14T23:15:09.030035+0000","last_became_peered":"2024-06-14T23:15:09.030035+0000","last_unstale":"2024-06-14T23:16:11.075454+0000","last_undegraded":"2024-06-14T23:16:11.075454+0000","last_fullsized":"2024-06-14T23:16:11.075454+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:14:00.226413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":337,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073073+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:16:11.073073+0000","last_peered":"2024-06-14T23:16:11.073073+0000","last_clean":"2024-06-14T23:16:11.073073+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:16:11.073073+0000","last_undegraded":"2024-06-14T23:16:11.073073+0000","last_fullsized":"2024-06-14T23:16:11.073073+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":781,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076107+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:16:11.076107+0000","last_peered":"2024-06-14T23:16:11.076107+0000","last_clean":"2024-06-14T23:16:11.076107+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:16:11.076107+0000","last_undegraded":"2024-06-14T23:16:11.076107+0000","last_fullsized":"2024-06-14T23:16:11.076107+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"372'7994","reported_seq":8023,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-06-14T23:16:21.851959+0000","last_change":"2024-06-14T23:14:30.735686+0000","last_active":"2024-06-14T23:16:21.851959+0000","last_peered":"2024-06-14T23:16:21.851959+0000","last_clean":"2024-06-14T23:16:21.851959+0000","last_became_active":"2024-06-14T23:14:30.735396+0000","last_became_peered":"2024-06-14T23:14:30.735396+0000","last_unstale":"2024-06-14T23:16:21.851959+0000","last_undegraded":"2024-06-14T23:16:21.851959+0000","last_fullsized":"2024-06-14T23:16:21.851959+0000","mapping_epoch":351,"log_start":"358'2900","ondisk_log_start":"358'2900","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:25:23.806204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459014144,"num_objects":7004,"num_object_clones":0,"num_object_copies":21012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7994,"num_write_kb":479936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":332,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073594+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:16:11.073594+0000","last_peered":"2024-06-14T23:16:11.073594+0000","last_clean":"2024-06-14T23:16:11.073594+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:16:11.073594+0000","last_undegraded":"2024-06-14T23:16:11.073594+0000","last_fullsized":"2024-06-14T23:16:11.073594+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":350,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074856+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:16:11.074856+0000","last_peered":"2024-06-14T23:16:11.074856+0000","last_clean":"2024-06-14T23:16:11.074856+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:16:11.074856+0000","last_undegraded":"2024-06-14T23:16:11.074856+0000","last_fullsized":"2024-06-14T23:16:11.074856+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":734,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074432+0000","last_change":"2024-06-14T23:15:09.028234+0000","last_active":"2024-06-14T23:16:11.074432+0000","last_peered":"2024-06-14T23:16:11.074432+0000","last_clean":"2024-06-14T23:16:11.074432+0000","last_became_active":"2024-06-14T23:15:09.027710+0000","last_became_peered":"2024-06-14T23:15:09.027710+0000","last_unstale":"2024-06-14T23:16:11.074432+0000","last_undegraded":"2024-06-14T23:16:11.074432+0000","last_fullsized":"2024-06-14T23:16:11.074432+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:46:47.401663+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":358,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.090406+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:16:11.090406+0000","last_peered":"2024-06-14T23:16:11.090406+0000","last_clean":"2024-06-14T23:16:11.090406+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:16:11.090406+0000","last_undegraded":"2024-06-14T23:16:11.090406+0000","last_fullsized":"2024-06-14T23:16:11.090406+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":356,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076719+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:16:11.076719+0000","last_peered":"2024-06-14T23:16:11.076719+0000","last_clean":"2024-06-14T23:16:11.076719+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:16:11.076719+0000","last_undegraded":"2024-06-14T23:16:11.076719+0000","last_fullsized":"2024-06-14T23:16:11.076719+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":727,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073765+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:16:11.073765+0000","last_peered":"2024-06-14T23:16:11.073765+0000","last_clean":"2024-06-14T23:16:11.073765+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:16:11.073765+0000","last_undegraded":"2024-06-14T23:16:11.073765+0000","last_fullsized":"2024-06-14T23:16:11.073765+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":173,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.081994+0000","last_change":"2024-06-14T23:15:09.039681+0000","last_active":"2024-06-14T23:16:11.081994+0000","last_peered":"2024-06-14T23:16:11.081994+0000","last_clean":"2024-06-14T23:16:11.081994+0000","last_became_active":"2024-06-14T23:15:09.039180+0000","last_became_peered":"2024-06-14T23:15:09.039180+0000","last_unstale":"2024-06-14T23:16:11.081994+0000","last_undegraded":"2024-06-14T23:16:11.081994+0000","last_fullsized":"2024-06-14T23:16:11.081994+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:54.312788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":237,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.077341+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:16:11.077341+0000","last_peered":"2024-06-14T23:16:11.077341+0000","last_clean":"2024-06-14T23:16:11.077341+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:16:11.077341+0000","last_undegraded":"2024-06-14T23:16:11.077341+0000","last_fullsized":"2024-06-14T23:16:11.077341+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":694,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075949+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:16:11.075949+0000","last_peered":"2024-06-14T23:16:11.075949+0000","last_clean":"2024-06-14T23:16:11.075949+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:16:11.075949+0000","last_undegraded":"2024-06-14T23:16:11.075949+0000","last_fullsized":"2024-06-14T23:16:11.075949+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":75,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.075699+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:16:11.075699+0000","last_peered":"2024-06-14T23:16:11.075699+0000","last_clean":"2024-06-14T23:16:11.075699+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:16:11.075699+0000","last_undegraded":"2024-06-14T23:16:11.075699+0000","last_fullsized":"2024-06-14T23:16:11.075699+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":332,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.074384+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:16:11.074384+0000","last_peered":"2024-06-14T23:16:11.074384+0000","last_clean":"2024-06-14T23:16:11.074384+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:16:11.074384+0000","last_undegraded":"2024-06-14T23:16:11.074384+0000","last_fullsized":"2024-06-14T23:16:11.074384+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":691,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.073272+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:16:11.073272+0000","last_peered":"2024-06-14T23:16:11.073272+0000","last_clean":"2024-06-14T23:16:11.073272+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:16:11.073272+0000","last_undegraded":"2024-06-14T23:16:11.073272+0000","last_fullsized":"2024-06-14T23:16:11.073272+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":384,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.069517+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:16:11.069517+0000","last_peered":"2024-06-14T23:16:11.069517+0000","last_clean":"2024-06-14T23:16:11.069517+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:16:11.069517+0000","last_undegraded":"2024-06-14T23:16:11.069517+0000","last_fullsized":"2024-06-14T23:16:11.069517+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":75,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.076191+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:16:11.076191+0000","last_peered":"2024-06-14T23:16:11.076191+0000","last_clean":"2024-06-14T23:16:11.076191+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:16:11.076191+0000","last_undegraded":"2024-06-14T23:16:11.076191+0000","last_fullsized":"2024-06-14T23:16:11.076191+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":804,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-06-14T23:16:11.071934+0000","last_change":"2024-06-14T23:11:23.429703+0000","last_active":"2024-06-14T23:16:11.071934+0000","last_peered":"2024-06-14T23:16:11.071934+0000","last_clean":"2024-06-14T23:16:11.071934+0000","last_became_active":"2024-06-14T23:11:23.428535+0000","last_became_peered":"2024-06-14T23:11:23.428535+0000","last_unstale":"2024-06-14T23:16:11.071934+0000","last_undegraded":"2024-06-14T23:16:11.071934+0000","last_fullsized":"2024-06-14T23:16:11.071934+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:08:06.719704+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":7317159960,"num_objects":111652,"num_object_clones":0,"num_object_copies":334956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111652,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130020,"num_write_kb":7733441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21896048640,"data_stored":21896036424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81620,"ondisk_log_size":81620,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":682,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234508,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4637160,"kb_used_data":3130272,"kb_used_omap":1,"kb_used_meta":1506878,"kb_avail":89107992,"statfs":{"total":95995035648,"available":91246583808,"internally_reserved":0,"allocated":3205398528,"data_stored":3203466204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1543043416},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.1399999999999999,"15min":0.93300000000000005},"min":{"1min":0.25900000000000001,"5min":0.24199999999999999,"15min":0.19800000000000001},"max":{"1min":2.3809999999999998,"5min":5.8259999999999996,"15min":5.8259999999999996},"last":1.0900000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.171,"15min":1.1120000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.224},"max":{"1min":3.4089999999999998,"5min":6.5149999999999997,"15min":6.9550000000000001},"last":0.61199999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:16:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1599999999999999,"15min":1.23},"min":{"1min":0.29699999999999999,"5min":0.25600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1719999999999997,"5min":6.0229999999999997,"15min":12.010999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":2.3740000000000001,"5min":1.512,"15min":1.331},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.22900000000000001},"max":{"1min":12.295,"5min":12.295,"15min":13.531000000000001},"last":1.0600000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.242,"15min":1.3640000000000001},"min":{"1min":0.309,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.9220000000000002,"5min":10.539,"15min":10.539},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.2230000000000001,"15min":1.3100000000000001},"min":{"1min":0.39900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.9849999999999999,"5min":10.56,"15min":10.56},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.3979999999999999,"15min":1.3859999999999999},"min":{"1min":0.34999999999999998,"5min":0.30599999999999999,"15min":0.28299999999999997},"max":{"1min":2.956,"5min":17.991,"15min":20.645},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.498,"15min":1.466},"min":{"1min":0.318,"5min":0.318,"15min":0.28299999999999997},"max":{"1min":3.3780000000000001,"5min":17.824999999999999,"15min":22.042999999999999},"last":0.65900000000000003}]},{"osd":4,"last update":"Fri Jun 14 23:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.2689999999999999,"15min":1.079},"min":{"1min":0.311,"5min":0.26300000000000001,"15min":0.23100000000000001},"max":{"1min":2.6200000000000001,"5min":6.0490000000000004,"15min":11.342000000000001},"last":2.7919999999999998},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.5229999999999999,"15min":1.1479999999999999},"min":{"1min":0.33400000000000002,"5min":0.27000000000000002,"15min":0.22700000000000001},"max":{"1min":3.0070000000000001,"5min":10.499000000000001,"15min":12.651},"last":2.9340000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:15:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.3149999999999999,"15min":1.048},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.23100000000000001},"max":{"1min":4.5330000000000004,"5min":5.734,"15min":6.4320000000000004},"last":2.2530000000000001},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.3640000000000001,"15min":1.1160000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.24399999999999999},"max":{"1min":5.3410000000000002,"5min":6.0899999999999999,"15min":6.2809999999999997},"last":0.63800000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:16:06 2024","interfaces":[{"interface":"back","average":{"1min":3.4940000000000002,"5min":2.2930000000000001,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":12.317,"5min":12.317,"15min":12.317},"last":2.335},{"interface":"front","average":{"1min":2.2919999999999998,"5min":1.8720000000000001,"15min":1.5369999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.32600000000000001},"max":{"1min":4.843,"5min":10.087999999999999,"15min":10.087999999999999},"last":2.3929999999999998}]}]},{"osd":6,"up_from":272,"seq":1168231104614,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4159124,"kb_used_data":3190028,"kb_used_omap":7,"kb_used_meta":969080,"kb_avail":89586028,"statfs":{"total":95995035648,"available":91736092672,"internally_reserved":0,"allocated":3266588672,"data_stored":3264676361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":992338362},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.95099999999999996,"15min":1.4099999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.5709999999999997,"5min":4.5709999999999997,"15min":6.6440000000000001},"last":6.109},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.028,"15min":1.4710000000000001},"min":{"1min":0.32200000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.6420000000000003,"5min":4.6420000000000003,"15min":6.6790000000000003},"last":6.1280000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.113,"15min":1.8149999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.25800000000000001},"max":{"1min":5.6509999999999998,"5min":5.6509999999999998,"15min":8.173},"last":1.8839999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.97299999999999998,"15min":1.704},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":10.380000000000001},"last":6.0279999999999996}]},{"osd":2,"last update":"Fri Jun 14 23:15:32 2024","interfaces":[{"interface":"back","average":{"1min":2.5310000000000001,"5min":1.3979999999999999,"15min":1.21},"min":{"1min":0.32100000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":29.940999999999999,"5min":29.940999999999999,"15min":29.940999999999999},"last":2.1680000000000001},{"interface":"front","average":{"1min":2.5939999999999999,"5min":1.4990000000000001,"15min":1.2889999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":30.135999999999999,"5min":30.135999999999999,"15min":30.135999999999999},"last":2.2029999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.0349999999999999,"15min":1.548},"min":{"1min":0.36299999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.5330000000000004,"5min":4.5330000000000004,"15min":6.7089999999999996},"last":6.0839999999999996},{"interface":"front","average":{"1min":1.635,"5min":1.3899999999999999,"15min":1.8149999999999999},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.1449999999999996,"5min":7.4969999999999999,"15min":8.4740000000000002},"last":2.2309999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":2.2349999999999999,"5min":1.484,"15min":1.45},"min":{"1min":0.51900000000000002,"5min":0.24399999999999999,"15min":0.22900000000000001},"max":{"1min":5.758,"5min":7.3159999999999998,"15min":11.010999999999999},"last":1.855},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.325,"15min":1.27},"min":{"1min":0.28799999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":6.4269999999999996,"5min":6.4269999999999996,"15min":6.4269999999999996},"last":6.1719999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":1.4490000000000001,"15min":1.379},"min":{"1min":0.33000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.0670000000000002,"5min":7.0670000000000002,"15min":7.7469999999999999},"last":6.1500000000000004},{"interface":"front","average":{"1min":2.7429999999999999,"5min":1.597,"15min":1.593},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":6.7889999999999997,"5min":7.4359999999999999,"15min":7.8680000000000003},"last":2.2599999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":2.4489999999999998,"5min":1.7549999999999999,"15min":1.738},"min":{"1min":0.40000000000000002,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":6.827,"5min":10.016999999999999,"15min":10.016999999999999},"last":5.9189999999999996},{"interface":"front","average":{"1min":2.9350000000000001,"5min":1.839,"15min":1.625},"min":{"1min":0.36599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.906000000000001,"5min":10.906000000000001,"15min":10.906000000000001},"last":8.25}]}]},{"osd":5,"up_from":38,"seq":163208757783,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5967888,"kb_used_data":4488200,"kb_used_omap":1,"kb_used_meta":1479678,"kb_avail":87777264,"statfs":{"total":95995035648,"available":89883918336,"internally_reserved":0,"allocated":4595916800,"data_stored":4594004395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1515190616},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":1.1910000000000001,"15min":1.3979999999999999},"min":{"1min":0.26500000000000001,"5min":0.23999999999999999,"15min":0.20100000000000001},"max":{"1min":8.3770000000000007,"5min":8.3770000000000007,"15min":14.94},"last":4.601},{"interface":"front","average":{"1min":2.7250000000000001,"5min":1.0940000000000001,"15min":1.383},"min":{"1min":0.245,"5min":0.245,"15min":0.219},"max":{"1min":13.398999999999999,"5min":13.398999999999999,"15min":13.398999999999999},"last":0.30099999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:16:18 2024","interfaces":[{"interface":"back","average":{"1min":2.1970000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":7.6219999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.9980000000000002,"5min":1.224,"15min":1.234},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":8.4480000000000004,"5min":8.4480000000000004,"15min":9.5039999999999996},"last":4.6849999999999996}]},{"osd":2,"last update":"Fri Jun 14 23:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":0.96099999999999997,"15min":0.753},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.7709999999999999,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":4.6589999999999998},{"interface":"front","average":{"1min":1.798,"5min":0.999,"15min":0.75700000000000001},"min":{"1min":0.29699999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.718,"5min":6.0659999999999998,"15min":6.0659999999999998},"last":4.7409999999999997}]},{"osd":3,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":3.044,"5min":1.462,"15min":1.444},"min":{"1min":0.34000000000000002,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":13.352,"5min":21.815000000000001,"15min":21.815000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":2.9729999999999999,"5min":1.5509999999999999,"15min":1.4750000000000001},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":8.4269999999999996,"5min":21.797000000000001,"15min":21.797000000000001},"last":4.7869999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:15:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":0.91700000000000004,"15min":0.92600000000000005},"min":{"1min":0.29299999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":5.0839999999999996,"5min":5.0839999999999996,"15min":14.352},"last":0.81200000000000006},{"interface":"front","average":{"1min":2.5009999999999999,"5min":1.151,"15min":1.1539999999999999},"min":{"1min":0.36499999999999999,"5min":0.27000000000000002,"15min":0.253},"max":{"1min":8.0350000000000001,"5min":8.0350000000000001,"15min":9.5980000000000008},"last":4.7009999999999996}]},{"osd":6,"last update":"Fri Jun 14 23:16:18 2024","interfaces":[{"interface":"back","average":{"1min":3.258,"5min":1.671,"15min":1.4870000000000001},"min":{"1min":0.42899999999999999,"5min":0.29599999999999999,"15min":0.26100000000000001},"max":{"1min":10.893000000000001,"5min":30.934000000000001,"15min":30.934000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":3.827,"5min":1.9219999999999999,"15min":1.804},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.26100000000000001},"max":{"1min":13.465999999999999,"5min":31.059999999999999,"15min":31.059999999999999},"last":0.36799999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:15:29 2024","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":1.738,"15min":1.347},"min":{"1min":0.38800000000000001,"5min":0.25,"15min":0.25},"max":{"1min":8.9659999999999993,"5min":26.981999999999999,"15min":26.981999999999999},"last":4.8159999999999998},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.7030000000000001,"15min":1.2849999999999999},"min":{"1min":0.40500000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":8.407,"5min":23.326000000000001,"15min":23.326000000000001},"last":0.79100000000000004}]}]},{"osd":4,"up_from":30,"seq":128849019419,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5514012,"kb_used_data":4021524,"kb_used_omap":1,"kb_used_meta":1492222,"kb_avail":88231140,"statfs":{"total":95995035648,"available":90348687360,"internally_reserved":0,"allocated":4118040576,"data_stored":4116119661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1528035672},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":0.86599999999999999,"15min":1.1100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.17100000000000001},"max":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":10.567},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.93600000000000005,"15min":1.0880000000000001},"min":{"1min":0.32900000000000001,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3920000000000003,"5min":6.7779999999999996,"15min":7.4900000000000002},"last":4.8540000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:16:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.88600000000000001,"15min":1.3660000000000001},"min":{"1min":0.33600000000000002,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":3.944,"5min":4.476,"15min":56.917000000000002},"last":1.667},{"interface":"front","average":{"1min":1.052,"5min":0.92800000000000005,"15min":1.431},"min":{"1min":0.34799999999999998,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":3.7890000000000001,"5min":4.6600000000000001,"15min":56.789999999999999},"last":0.505}]},{"osd":2,"last update":"Fri Jun 14 23:15:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.85299999999999998,"15min":0.82599999999999996},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.5899999999999999,"5min":3.468,"15min":3.468},"last":1.714},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.90500000000000003,"15min":0.88900000000000001},"min":{"1min":0.32700000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.5129999999999999,"5min":3.4980000000000002,"15min":3.4980000000000002},"last":1.74}]},{"osd":3,"last update":"Fri Jun 14 23:16:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.91500000000000004,"15min":1.274},"min":{"1min":0.38300000000000001,"5min":0.30599999999999999,"15min":0.25},"max":{"1min":5.3540000000000001,"5min":5.3540000000000001,"15min":44.389000000000003},"last":4.8159999999999998},{"interface":"front","average":{"1min":1.222,"5min":1.016,"15min":1.4530000000000001},"min":{"1min":0.33500000000000002,"5min":0.29599999999999999,"15min":0.27600000000000002},"max":{"1min":4.6120000000000001,"5min":4.6120000000000001,"15min":46.101999999999997},"last":0.45700000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:15:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.0529999999999999,"15min":1.0109999999999999},"min":{"1min":0.61699999999999999,"5min":0.24399999999999999,"15min":0.20300000000000001},"max":{"1min":5.4160000000000004,"5min":5.4160000000000004,"15min":12.315},"last":4.8789999999999996},{"interface":"front","average":{"1min":1.972,"5min":1.1399999999999999,"15min":1.038},"min":{"1min":0.51500000000000001,"5min":0.28799999999999998,"15min":0.23300000000000001},"max":{"1min":6.5289999999999999,"5min":6.9290000000000003,"15min":10.959},"last":3.4670000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":2.548,"15min":1.5680000000000001},"min":{"1min":0.28399999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.5780000000000003,"5min":122.88500000000001,"15min":122.88500000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":2.1269999999999998,"5min":2.6640000000000001,"15min":1.863},"min":{"1min":0.65100000000000002,"5min":0.30499999999999999,"15min":0.25700000000000001},"max":{"1min":7.2439999999999998,"5min":122.21299999999999,"15min":122.21299999999999},"last":4.9020000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:15:27 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.2949999999999999,"15min":1.0629999999999999},"min":{"1min":0.40799999999999997,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":5.4699999999999998,"5min":6.5910000000000002,"15min":6.5910000000000002},"last":2.968},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.3779999999999999,"15min":1.0860000000000001},"min":{"1min":0.46899999999999997,"5min":0.33300000000000002,"15min":0.27300000000000002},"max":{"1min":2.028,"5min":8.5340000000000007,"15min":8.5340000000000007},"last":4.9269999999999996}]}]},{"osd":3,"up_from":138,"seq":592705487118,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3605580,"kb_used_data":2648900,"kb_used_omap":4,"kb_used_meta":956539,"kb_avail":90139572,"statfs":{"total":95995035648,"available":92302921728,"internally_reserved":0,"allocated":2712473600,"data_stored":2710564522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":979496414},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35799999999999998,"15min":1.1339999999999999},"min":{"1min":0.17199999999999999,"5min":0.16900000000000001,"15min":0.16800000000000001},"max":{"1min":0.84399999999999997,"5min":1.175,"15min":25.190000000000001},"last":0.217},{"interface":"front","average":{"1min":0.38,"5min":0.53400000000000003,"15min":1.23},"min":{"1min":0.246,"5min":0.20699999999999999,"15min":0.17799999999999999},"max":{"1min":0.93400000000000005,"5min":4.6369999999999996,"15min":25.652000000000001},"last":0.313}]},{"osd":1,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.56100000000000005,"15min":1.2090000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20300000000000001},"max":{"1min":0.998,"5min":4.6840000000000002,"15min":25.754000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.55400000000000005,"15min":1.3560000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.0189999999999999,"5min":2.6899999999999999,"15min":27.992999999999999},"last":0.35699999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:15:29 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.72899999999999998,"15min":0.84899999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.1659999999999999,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.27000000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.2709999999999999,"5min":3.5550000000000002,"15min":3.5550000000000002},"last":0.34000000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.1519999999999999,"15min":1.762},"min":{"1min":0.29699999999999999,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":2.419,"5min":17.018999999999998,"15min":73.263000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.2609999999999999,"15min":1.651},"min":{"1min":0.30199999999999999,"5min":0.29499999999999998,"15min":0.27300000000000002},"max":{"1min":3.6699999999999999,"5min":17.074000000000002,"15min":73.5},"last":0.626}]},{"osd":5,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":0.92000000000000004,"15min":1.643},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.28699999999999998},"max":{"1min":3.339,"5min":3.4740000000000002,"15min":81.438999999999993},"last":1.232},{"interface":"front","average":{"1min":1.2869999999999999,"5min":0.93400000000000005,"15min":1.6930000000000001},"min":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.29299999999999998},"max":{"1min":2.8530000000000002,"5min":4.5990000000000002,"15min":81.626999999999995},"last":0.41599999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:16:12 2024","interfaces":[{"interface":"back","average":{"1min":1.867,"5min":1.5089999999999999,"15min":1.6579999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.34599999999999997},"max":{"1min":15.458,"5min":17.141999999999999,"15min":25.869},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.599,"15min":1.5489999999999999},"min":{"1min":0.38600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":15.699,"5min":20.056999999999999,"15min":25.143000000000001},"last":0.40200000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.4850000000000001,"15min":1.4810000000000001},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":3.4060000000000001,"5min":44.887999999999998,"15min":44.887999999999998},"last":0.374},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.635,"15min":1.427},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":3.5259999999999998,"5min":45.835000000000001,"15min":45.835000000000001},"last":0.44800000000000001}]}]},{"osd":2,"up_from":322,"seq":1382979469373,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4330124,"kb_used_data":4057604,"kb_used_omap":10,"kb_used_meta":272501,"kb_avail":89415028,"statfs":{"total":95995035648,"available":91560988672,"internally_reserved":0,"allocated":4154986496,"data_stored":4153078152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":279041574},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.44600000000000001,"15min":0.36299999999999999},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":0.76600000000000001,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.498,"15min":0.45800000000000002},"min":{"1min":0.27000000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.90700000000000003,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.371}]},{"osd":1,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.45300000000000001,"15min":0.41999999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.64800000000000002,"5min":3.4340000000000002,"15min":3.4340000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.54500000000000004,"15min":0.47299999999999998},"min":{"1min":0.26300000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.1799999999999999,"5min":5.0389999999999997,"15min":5.0389999999999997},"last":0.27500000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.65000000000000002,"15min":0.60999999999999999},"min":{"1min":0.24299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.86799999999999999,"5min":3.7519999999999998,"15min":3.7519999999999998},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.63600000000000001,"15min":0.55500000000000005},"min":{"1min":0.245,"5min":0.223,"15min":0.223},"max":{"1min":1.1439999999999999,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.499}]},{"osd":4,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.0860000000000001,"15min":0.96399999999999997},"min":{"1min":0.47299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.7859999999999996,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.89100000000000001,"15min":0.73999999999999999},"min":{"1min":0.35599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.8220000000000001,"5min":4.3680000000000003,"15min":4.3680000000000003},"last":0.88800000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":0.86199999999999999,"15min":0.80600000000000005},"min":{"1min":0.33200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":5.9100000000000001,"5min":5.9100000000000001,"15min":5.9100000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.375,"5min":0.93300000000000005,"15min":0.89100000000000001},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.7810000000000001,"5min":3.7810000000000001,"15min":3.7810000000000001},"last":4.1429999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.0629999999999999,"15min":0.86899999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.6989999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0049999999999999,"15min":0.98299999999999998},"min":{"1min":0.48699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.9209999999999998,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":0.31900000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:15:36 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.0509999999999999,"15min":0.97699999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":4.4480000000000004,"5min":4.4480000000000004,"15min":4.4480000000000004},"last":1.234},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.2529999999999999,"15min":1.151},"min":{"1min":0.45700000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.879,"5min":5.2130000000000001,"15min":5.2130000000000001},"last":1.3049999999999999}]}]},{"osd":1,"up_from":0,"seq":498216206662,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024317,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4135518208,"data_stored":4135518208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2685538304,"data_stored":2685534232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096069632,"data_stored":4096065560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4561502208,"data_stored":4561502208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3242065920,"data_stored":3242065920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3175354368,"data_stored":3175350296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:16:27.045 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-14T23:16:27.045 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:16:27.046 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.7062498438213645 2024-06-14T23:16:27.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 2 0.7062498438213645 2024-06-14T23:16:27.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:27.116 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:27.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:27.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:27.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:27.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:26 smithi137 ceph-mon[90634]: from='client.47000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:16:27.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:27.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:27.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:27.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:27.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:27.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:27.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:27.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:27.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:27.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:27.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:27.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:27.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:27.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:27.934 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:27.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:28.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:28.078 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:28.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:28.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:28.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:28.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:28.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:28.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:16:28.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:28.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:28.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:28.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:28.480 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:28.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:28.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:28.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:28.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:28.712 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:28.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:28.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:28 smithi086 ceph-mon[97913]: pgmap v1401: 119 pgs: 119 active+clean; 6.7 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 95 B/s rd, 4.7 MiB/s wr, 1.08k op/s 2024-06-14T23:16:28.713 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:28.713 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:28.713 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:28 smithi086 ceph-mon[102754]: pgmap v1401: 119 pgs: 119 active+clean; 6.7 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 95 B/s rd, 4.7 MiB/s wr, 1.08k op/s 2024-06-14T23:16:28.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:28 smithi137 ceph-mon[90634]: pgmap v1401: 119 pgs: 119 active+clean; 6.7 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 95 B/s rd, 4.7 MiB/s wr, 1.08k op/s 2024-06-14T23:16:28.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:28.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:28.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:28.899 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:28.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:28.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:29.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:29.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:29.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:29.167 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:29.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:29.211 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:29.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:29.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:29.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:29.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:29.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:29.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:29.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:29.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:29.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:29.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:29.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:29.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:29.828 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:29.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:29.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:29.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:29.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:30.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:30.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:30.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:30.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:30.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:30.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:30.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:30.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:30.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:30.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:30.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:30.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:30.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:30.543 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:30.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:30.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:30.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:30.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:30.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:30.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:30.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:30.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:30.991 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:31.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:31.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:31.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:30 smithi086 ceph-mon[97913]: pgmap v1402: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 85 B/s rd, 2.4 MiB/s wr, 1.15k op/s 2024-06-14T23:16:31.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:30 smithi086 ceph-mon[102754]: pgmap v1402: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 85 B/s rd, 2.4 MiB/s wr, 1.15k op/s 2024-06-14T23:16:31.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:31.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:31.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:31.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:31.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:30 smithi137 ceph-mon[90634]: pgmap v1402: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 85 B/s rd, 2.4 MiB/s wr, 1.15k op/s 2024-06-14T23:16:31.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:31.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:31.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:31.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:31.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:31.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:31.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:31.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:31.582 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:31.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:31.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:31.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:31.721 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:31.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:31.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:31.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:31.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:31.850 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:16:31.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:31.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:31.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:32.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:32.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1628469449' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:32.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:31 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:32.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1628469449' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:32.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:31 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:32.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:32.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:32.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:32.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1628469449' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:32.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:31 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:32.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:32.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:32.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:32.392 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:32.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:32.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:32.517 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:32.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:32.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:32.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:32.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:32.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:32.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:32.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:32.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:32.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:32.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:32.935 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:32 smithi086 ceph-mon[102754]: pgmap v1403: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 937 op/s 2024-06-14T23:16:32.935 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:32 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:32.935 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:32 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:16:32.936 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:32 smithi086 ceph-mon[102754]: osdmap e373: 8 total, 8 up, 6 in 2024-06-14T23:16:32.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:32 smithi086 ceph-mon[97913]: pgmap v1403: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 937 op/s 2024-06-14T23:16:32.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:32 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:32.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:32 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:16:32.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:32 smithi086 ceph-mon[97913]: osdmap e373: 8 total, 8 up, 6 in 2024-06-14T23:16:33.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:33.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:33.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:33.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:33.165 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:33.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:33.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:32 smithi137 ceph-mon[90634]: pgmap v1403: 119 pgs: 119 active+clean; 6.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 937 op/s 2024-06-14T23:16:33.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:32 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:33.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:32 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:16:33.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:32 smithi137 ceph-mon[90634]: osdmap e373: 8 total, 8 up, 6 in 2024-06-14T23:16:33.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:33.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:33.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:33.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:33.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:33.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:33.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:33.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:33.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:33.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:33.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:33.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:33.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:33.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:33.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:33.984 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:33.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:34.017 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:16:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:16:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:141299 13.8 13:171c1d5d:::benchmark_data_smithi086_299538_object20461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-06-14T23:16:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:33.179543+0000", 2024-06-14T23:16:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0080351439999999993, 2024-06-14T23:16:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0080567220000000005, 2024-06-14T23:16:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141299 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.179543+0000", 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.179543+0000", 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.179542+0000", 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.179548+0000", 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0059999999999998e-06 2024-06-14T23:16:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.179563+0000", 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4657999999999999e-05 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.179569+0000", 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7739999999999999e-06 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.179620+0000", 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0319999999999999e-05 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.179753+0000", 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013378700000000001 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.179870+0000", 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116983 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.182950+0000", 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003079624 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184684+0000", 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017342099999999999 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:141295 13.1 e373/351 13:8451602a:::benchmark_data_smithi086_299538_object20457:head v 373'8798, mlcod=373'8795)", 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:33.184647+0000", 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0029312769999999999, 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0030035800000000001, 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184647+0000", 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184647+0000", 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184647+0000", 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184651+0000", 2024-06-14T23:16:34.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3259999999999997e-06 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184658+0000", 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1180000000000003e-06 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184663+0000", 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7759999999999997e-06 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:34.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184705+0000", 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2014999999999997e-05 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184727+0000", 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1895e-05 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:141303 13.0 13:050d908f:::benchmark_data_smithi086_299538_object20465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-06-14T23:16:34.026 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:33.184701+0000", 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0028770779999999999, 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029826929999999998, 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141303 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184701+0000", 2024-06-14T23:16:34.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184701+0000", 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184700+0000", 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184705+0000", 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4920000000000004e-06 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184715+0000", 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0419e-05 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184722+0000", 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8469999999999998e-06 2024-06-14T23:16:34.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184757+0000", 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4912e-05 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184884+0000", 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012718699999999999 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:16:34.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.185007+0000", 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123282 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:141304 13.8 13:198c4f40:::benchmark_data_smithi086_299538_object20466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:33.184833+0000", 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0027452209999999999, 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028873649999999998, 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:34.031 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141304 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184833+0000", 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184833+0000", 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184832+0000", 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184838+0000", 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8799999999999996e-06 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:34.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184861+0000", 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2577000000000001e-05 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184868+0000", 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6159999999999996e-06 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184906+0000", 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7305999999999998e-05 2024-06-14T23:16:34.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.185012+0000", 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106338 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.185121+0000", 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108966 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:141305 13.a 13:5e3df262:::benchmark_data_smithi086_299538_object20467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:33.184888+0000", 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0026906059999999999, 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028935720000000001, 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 141305 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184888+0000", 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184888+0000", 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184887+0000", 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184891+0000", 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6119999999999999e-06 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184898+0000", 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3919999999999997e-06 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184903+0000", 2024-06-14T23:16:34.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1930000000000001e-06 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.184953+0000", 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9901000000000002e-05 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.185052+0000", 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8677000000000004e-05 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.185123+0000", 2024-06-14T23:16:34.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0939999999999995e-05 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.187451+0000", 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023283589999999999 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:141302 13.1 e373/351 13:88d8e763:::benchmark_data_smithi086_299538_object20464:head v 373'8799, mlcod=373'8795)", 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:33.186604+0000", 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.00097474300000000003, 2024-06-14T23:16:34.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012342480000000001, 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.186604+0000", 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.186604+0000", 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.186603+0000", 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.186608+0000", 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4160000000000003e-06 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.186620+0000", 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1336e-05 2024-06-14T23:16:34.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.186627+0000", 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7989999999999996e-06 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.186759+0000", 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000132426 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:33.186781+0000", 2024-06-14T23:16:34.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1590000000000002e-05 2024-06-14T23:16:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-06-14T23:16:34.044 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 6 2024-06-14T23:16:34.044 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:16:34.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:34.114 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:34.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:34.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:34.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:34.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:34.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:34.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:34.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:34.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:34.500 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:34.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:34.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:34.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:34.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:34.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:34.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:34.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:34.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:34.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:34.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:16:35.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:35.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:35.068 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:34 smithi086 ceph-mon[102754]: pgmap v1405: 119 pgs: 119 active+clean; 6.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-14T23:16:35.068 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:16:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:16:34] "GET /metrics HTTP/1.1" 200 36677 "" "Prometheus/2.43.0" 2024-06-14T23:16:35.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:34 smithi086 ceph-mon[97913]: pgmap v1405: 119 pgs: 119 active+clean; 6.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-14T23:16:35.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:35.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:35.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:35.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:34 smithi137 ceph-mon[90634]: pgmap v1405: 119 pgs: 119 active+clean; 6.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-14T23:16:35.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:35.303 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:35.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:35.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:35.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:35.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:35.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:35.586 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:35.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:35.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:35.714 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:35.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:35.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:35.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:35.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:35.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:35.869 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.2 to 0.70625 (b4cc) 2024-06-14T23:16:35.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:35.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:35.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:35.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:36.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:36.113 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:36.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:36.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:36.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:36.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3563881119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.7062498438213645}]: dispatch 2024-06-14T23:16:36.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:35 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.7062498438213645}]: dispatch 2024-06-14T23:16:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3563881119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.7062498438213645}]: dispatch 2024-06-14T23:16:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:35 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.7062498438213645}]: dispatch 2024-06-14T23:16:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3563881119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.7062498438213645}]: dispatch 2024-06-14T23:16:36.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:35 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 0.7062498438213645}]: dispatch 2024-06-14T23:16:36.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:36.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:36.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:36.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:36.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:36.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:36.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:36.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:36.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:36.732 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:36.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:36.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:36.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:36.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:36.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-14T23:16:36.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:36.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:36.995 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:36.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:37.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:37.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:37.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:37.179 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:36 smithi086 ceph-mon[97913]: pgmap v1406: 119 pgs: 119 active+clean; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-14T23:16:37.179 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:36 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.7062498438213645}]': finished 2024-06-14T23:16:37.179 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:36 smithi086 ceph-mon[97913]: osdmap e374: 8 total, 8 up, 6 in 2024-06-14T23:16:37.179 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:36 smithi086 ceph-mon[102754]: pgmap v1406: 119 pgs: 119 active+clean; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-14T23:16:37.179 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:36 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.7062498438213645}]': finished 2024-06-14T23:16:37.180 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:36 smithi086 ceph-mon[102754]: osdmap e374: 8 total, 8 up, 6 in 2024-06-14T23:16:37.180 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:16:37.180 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:16:37.197 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:16:37.197 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:16:37.213 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:16:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:16:37.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:37.229 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:16:37.229 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:16:37.245 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:16:37.245 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:16:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:36 smithi137 ceph-mon[90634]: pgmap v1406: 119 pgs: 119 active+clean; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.12k op/s 2024-06-14T23:16:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:36 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.7062498438213645}]': finished 2024-06-14T23:16:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:36 smithi137 ceph-mon[90634]: osdmap e374: 8 total, 8 up, 6 in 2024-06-14T23:16:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:37.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:37.262 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:16:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:16:37.278 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:16:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:16:37.295 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:16:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:16:37.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:37.370 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:37.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:37.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:37.498 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:37.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:37.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:37.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:37.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:37.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:37.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:37.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:37.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:37.864 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:37.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:37.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:37.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:38.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:38.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:38.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:38.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:38.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:38.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:38.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:37 smithi137 ceph-mon[90634]: osdmap e375: 8 total, 8 up, 6 in 2024-06-14T23:16:38.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:37 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/744866053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:38.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:37 smithi137 ceph-mon[90634]: osdmap e376: 8 total, 8 up, 6 in 2024-06-14T23:16:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:38.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:38.322 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:37 smithi086 ceph-mon[97913]: osdmap e375: 8 total, 8 up, 6 in 2024-06-14T23:16:38.322 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:37 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/744866053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:38.323 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:37 smithi086 ceph-mon[97913]: osdmap e376: 8 total, 8 up, 6 in 2024-06-14T23:16:38.323 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:37 smithi086 ceph-mon[102754]: osdmap e375: 8 total, 8 up, 6 in 2024-06-14T23:16:38.323 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:37 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/744866053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:38.323 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:37 smithi086 ceph-mon[102754]: osdmap e376: 8 total, 8 up, 6 in 2024-06-14T23:16:38.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:38.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:38.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:38.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:38.548 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:38.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:38.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:38.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:38.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:38.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:38.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:38.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:38.880 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:38 smithi086 ceph-mon[97913]: pgmap v1409: 119 pgs: 2 remapped+peering, 117 active+clean; 6.0 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 3.3 KiB/s rd, 9.2 KiB/s wr, 1.18k op/s; 6.5 KiB/s, 0 objects/s recovering 2024-06-14T23:16:38.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:38.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:38.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:39.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:39.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:39.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:39.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:39.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:39.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:39.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:38 smithi137 ceph-mon[90634]: pgmap v1409: 119 pgs: 2 remapped+peering, 117 active+clean; 6.0 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 3.3 KiB/s rd, 9.2 KiB/s wr, 1.18k op/s; 6.5 KiB/s, 0 objects/s recovering 2024-06-14T23:16:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:38 smithi137 ceph-mon[90634]: 13.a starting backfill to osd.3 from (0'0,0'0] MAX to 373'8890 2024-06-14T23:16:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:38 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:16:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:38 smithi137 ceph-mon[90634]: osdmap e377: 8 total, 8 up, 6 in 2024-06-14T23:16:39.285 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:38 smithi086 ceph-mon[97913]: 13.a starting backfill to osd.3 from (0'0,0'0] MAX to 373'8890 2024-06-14T23:16:39.286 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:38 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:16:39.286 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:38 smithi086 ceph-mon[97913]: osdmap e377: 8 total, 8 up, 6 in 2024-06-14T23:16:39.286 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:38 smithi086 ceph-mon[102754]: pgmap v1409: 119 pgs: 2 remapped+peering, 117 active+clean; 6.0 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 3.3 KiB/s rd, 9.2 KiB/s wr, 1.18k op/s; 6.5 KiB/s, 0 objects/s recovering 2024-06-14T23:16:39.286 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:38 smithi086 ceph-mon[102754]: 13.a starting backfill to osd.3 from (0'0,0'0] MAX to 373'8890 2024-06-14T23:16:39.286 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:38 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:16:39.286 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:38 smithi086 ceph-mon[102754]: osdmap e377: 8 total, 8 up, 6 in 2024-06-14T23:16:39.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:39.353 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:39.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:39.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:39.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:39.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:39.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:39.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:39.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:39.713 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:39.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:39.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:39.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:39.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:39.896 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:39.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:39.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:40.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:40.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:40.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:40.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:40.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:40.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:40.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:40.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:16:40.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:40.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:40.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:40.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:40.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:40 smithi086 ceph-mon[97913]: osdmap e378: 8 total, 8 up, 6 in 2024-06-14T23:16:40.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:40 smithi086 ceph-mon[97913]: pgmap v1413: 119 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 remapped+peering, 115 active+clean; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 5.1 KiB/s rd, 14 KiB/s wr, 952 op/s; 5860/290088 objects misplaced (2.020%); 1.2 MiB/s, 19 objects/s recovering 2024-06-14T23:16:40.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:40 smithi086 ceph-mon[102754]: osdmap e378: 8 total, 8 up, 6 in 2024-06-14T23:16:40.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:40 smithi086 ceph-mon[102754]: pgmap v1413: 119 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 remapped+peering, 115 active+clean; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 5.1 KiB/s rd, 14 KiB/s wr, 952 op/s; 5860/290088 objects misplaced (2.020%); 1.2 MiB/s, 19 objects/s recovering 2024-06-14T23:16:40.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:40.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:40.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:40.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:40.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:40.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:40 smithi137 ceph-mon[90634]: osdmap e378: 8 total, 8 up, 6 in 2024-06-14T23:16:40.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:40 smithi137 ceph-mon[90634]: pgmap v1413: 119 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 remapped+peering, 115 active+clean; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 5.1 KiB/s rd, 14 KiB/s wr, 952 op/s; 5860/290088 objects misplaced (2.020%); 1.2 MiB/s, 19 objects/s recovering 2024-06-14T23:16:40.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:40.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:40.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:40.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:40.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:40.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:41.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:41.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:41.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:41.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:41.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:41.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:41.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:41.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:41.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:41.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:41.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:41.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:41.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:41.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:41.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:41.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:41.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:41.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:41.830 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:16:41.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:41.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:41 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3270739551' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:16:41.919 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:41 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3270739551' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:16:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:41.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:41.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:42.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:42.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:42.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:42.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:42.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:42.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:42.244 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:16:42.245 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:16:42.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:41 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3270739551' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:16:42.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:42.343 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:42.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:42.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:42.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:42.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:42.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:42.596 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 2] out_osds: [0, 1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:16:42.597 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:16:42.597 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_8 2024-06-14T23:16:42.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:16:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:42.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:42.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:42.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:42.739 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:42.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:42.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:42.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:42.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:42.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:43.006 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:43.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[102754]: pgmap v1414: 119 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 remapped+peering, 115 active+clean; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 4.1 KiB/s rd, 11 KiB/s wr, 800 op/s; 5860/289470 objects misplaced (2.024%); 996 KiB/s, 15 objects/s recovering 2024-06-14T23:16:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3270739551' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:16:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[102754]: osdmap e379: 8 total, 8 up, 6 in 2024-06-14T23:16:43.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:43.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:43.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[97913]: pgmap v1414: 119 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 remapped+peering, 115 active+clean; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 4.1 KiB/s rd, 11 KiB/s wr, 800 op/s; 5860/289470 objects misplaced (2.024%); 996 KiB/s, 15 objects/s recovering 2024-06-14T23:16:43.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:43.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3270739551' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:16:43.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[97913]: osdmap e379: 8 total, 8 up, 6 in 2024-06-14T23:16:43.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:43.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:43.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:43.126 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:42 smithi137 ceph-mon[90634]: pgmap v1414: 119 pgs: 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 remapped+peering, 115 active+clean; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 4.1 KiB/s rd, 11 KiB/s wr, 800 op/s; 5860/289470 objects misplaced (2.024%); 996 KiB/s, 15 objects/s recovering 2024-06-14T23:16:43.126 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:42 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:16:43.126 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:42 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3270739551' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:16:43.127 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:42 smithi137 ceph-mon[90634]: osdmap e379: 8 total, 8 up, 6 in 2024-06-14T23:16:43.127 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:43.127 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:43.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:43.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:43.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:43.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:43.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:43.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:43.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:43.405 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:43.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-14T23:16:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:43.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:43.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:43.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:43.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:43.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:43.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:43.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:43.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:43.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:43.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:43.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:43.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:43.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:44.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:44.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:44.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:43 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:16:44.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:16:44.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:43 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:16:44.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:16:44.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:44.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:44.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:44.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:43 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:16:44.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:16:44.255 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:44.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:44.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:44.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:44.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:44.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:44.508 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:44.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:44.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:44.633 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:44.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:44.657 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:44.662 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:44.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:44.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:44.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:44.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:44.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:44.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:44.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:44 smithi137 ceph-mon[90634]: pgmap v1416: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.6 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.14k op/s; 4900/276801 objects misplaced (1.770%); 7.4 MiB/s, 118 objects/s recovering 2024-06-14T23:16:44.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:44 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:16:44.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:44 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:16:45.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:45.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:45.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:16:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:16:44] "GET /metrics HTTP/1.1" 200 36714 "" "Prometheus/2.43.0" 2024-06-14T23:16:45.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:44 smithi086 ceph-mon[102754]: pgmap v1416: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.6 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.14k op/s; 4900/276801 objects misplaced (1.770%); 7.4 MiB/s, 118 objects/s recovering 2024-06-14T23:16:45.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:44 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:16:45.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:44 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:16:45.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:44 smithi086 ceph-mon[97913]: pgmap v1416: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.6 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 1.14k op/s; 4900/276801 objects misplaced (1.770%); 7.4 MiB/s, 118 objects/s recovering 2024-06-14T23:16:45.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:44 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:16:45.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:44 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:16:45.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:45.188 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:45.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:45.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:45.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:45.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:45.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:45.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:45.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:45.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:45.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:45.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:45.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:45.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:45.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:45.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:45.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:16:45.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:45.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:45.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:45.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:45 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:16:46.014 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:46.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:46.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:45 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:16:46.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:45 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:16:46.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:46.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:46.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:46.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:46.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:46.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:46.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:46.491 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:46.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:46.570 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:16:46.570 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:16:46.570 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:16:46.570 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:16:46.571 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:16:46.571 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:16:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:16:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:16:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:16:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:16:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:16:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:16:46.572 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:16:46.572 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:16:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:16:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:16:46.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:46.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:46.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:46.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:46.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:46.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:46.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:46.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:46.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:46.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:46.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:47.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:47.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:47.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:46 smithi086 ceph-mon[102754]: pgmap v1417: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 834 op/s; 4900/273846 objects misplaced (1.789%); 6.0 MiB/s, 95 objects/s recovering 2024-06-14T23:16:47.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3259647941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:47.107 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:46 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:16:47.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:46 smithi086 ceph-mon[97913]: pgmap v1417: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 834 op/s; 4900/273846 objects misplaced (1.789%); 6.0 MiB/s, 95 objects/s recovering 2024-06-14T23:16:47.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3259647941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:47.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:46 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:16:47.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:47.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:47.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:47.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:47.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:46 smithi137 ceph-mon[90634]: pgmap v1417: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 834 op/s; 4900/273846 objects misplaced (1.789%); 6.0 MiB/s, 95 objects/s recovering 2024-06-14T23:16:47.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3259647941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:16:47.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:46 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:16:47.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:47.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:47.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:47.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:47.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:47.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:47.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:47.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:47.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:47.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:47.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:47.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:47.854 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:47.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:47.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:47.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:47.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:48.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:48.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:48.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:48.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:48.236 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:48.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:48.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:48.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:48.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:48.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:48.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:48.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:48.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:48.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:48.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:48.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:48.783 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:48.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:48.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:48.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:48.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:49.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:49.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:49.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:49.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:49.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:49.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:49.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:48 smithi137 ceph-mon[90634]: pgmap v1418: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 832 op/s; 4900/269235 objects misplaced (1.820%); 4.7 MiB/s, 74 objects/s recovering 2024-06-14T23:16:49.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:48 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:49.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:48 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:16:49.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:49.321 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:49.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:48 smithi086 ceph-mon[97913]: pgmap v1418: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 832 op/s; 4900/269235 objects misplaced (1.820%); 4.7 MiB/s, 74 objects/s recovering 2024-06-14T23:16:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:48 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:48 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:16:49.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:48 smithi086 ceph-mon[102754]: pgmap v1418: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 832 op/s; 4900/269235 objects misplaced (1.820%); 4.7 MiB/s, 74 objects/s recovering 2024-06-14T23:16:49.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:48 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:49.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:48 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:16:49.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:49.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:49.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:49.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:49.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:49.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:49.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:16:49.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:49.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:49.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:49.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:49.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:49.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:50.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:50.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:50.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:50.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:50.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:50.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:49 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:16:50.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:50.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:50.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:50.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:49 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:16:50.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:49 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:16:50.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:50.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:50.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:50.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:50.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:50.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:50.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:50.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:50.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:50.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:50.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:50.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:50.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:50.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:50.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:51.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:51.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:51.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:51.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:51.213 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:50 smithi086 ceph-mon[97913]: pgmap v1419: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 4023/259839 objects misplaced (1.548%); 7.9 MiB/s, 126 objects/s recovering 2024-06-14T23:16:51.213 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:50 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:51.213 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:50 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:51.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:50 smithi086 ceph-mon[102754]: pgmap v1419: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 4023/259839 objects misplaced (1.548%); 7.9 MiB/s, 126 objects/s recovering 2024-06-14T23:16:51.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:50 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:51.214 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:50 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:51.234 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:51.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:50 smithi137 ceph-mon[90634]: pgmap v1419: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 4023/259839 objects misplaced (1.548%); 7.9 MiB/s, 126 objects/s recovering 2024-06-14T23:16:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:50 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:50 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:51.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:51.363 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:51.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:51.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:51.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:51.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:51.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:51.635 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:51.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:51.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:51.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:51.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:51.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:51.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:51.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:52.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:52.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:52.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:52.091 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:16:52.091 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:16:52.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:16:52.092 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:16:52.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.092 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232556 12.d e344/323 12:bf0779b7:::benchmark_data_smithi086_261492_object116051:head v 344'14541, mlcod=314'4802)", 2024-06-14T23:16:52.092 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.887331+0000", 2024-06-14T23:16:52.092 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 177.197284862, 2024-06-14T23:16:52.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059639240000000003, 2024-06-14T23:16:52.093 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.093 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.093 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887331+0000", 2024-06-14T23:16:52.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887331+0000", 2024-06-14T23:16:52.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887331+0000", 2024-06-14T23:16:52.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887345+0000", 2024-06-14T23:16:52.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.454e-05 2024-06-14T23:16:52.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887356+0000", 2024-06-14T23:16:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1266e-05 2024-06-14T23:16:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887363+0000", 2024-06-14T23:16:52.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2820000000000002e-06 2024-06-14T23:16:52.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887424+0000", 2024-06-14T23:16:52.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1023999999999997e-05 2024-06-14T23:16:52.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.887448+0000", 2024-06-14T23:16:52.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4295e-05 2024-06-14T23:16:52.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.946945+0000", 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059496926999999998 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.946971+0000", 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5774000000000001e-05 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46469.0:232566 12.d e344/323 12:b387c827:::benchmark_data_smithi086_261492_object116061:head v 344'14543, mlcod=314'4802)", 2024-06-14T23:16:52.100 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:13:54.898063+0000", 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 177.18655284799999, 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.062928186999999997, 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.898063+0000", 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.898063+0000", 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.898063+0000", 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.898069+0000", 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8710000000000002e-06 2024-06-14T23:16:52.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936761+0000", 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.038692336000000001 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.936771+0000", 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0461e-05 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.941716+0000", 2024-06-14T23:16:52.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049446239999999999 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.941739+0000", 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3218e-05 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.960919+0000", 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019180049000000001 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:13:54.960992+0000", 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2385999999999994e-05 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:35122 13.8 13:1f37731d:::benchmark_data_smithi086_299538_object35121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:14:58.368656+0000", 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 113.71596012000001, 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.070183514000000002, 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.105 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.106 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.106 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.106 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 35122 2024-06-14T23:16:52.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.106 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.368656+0000", 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.368656+0000", 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.368655+0000", 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.368679+0000", 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4071999999999999e-05 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.368689+0000", 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8910000000000007e-06 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.108 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.368695+0000", 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.037e-06 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.368741+0000", 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5510000000000003e-05 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.368883+0000", 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014252399999999999 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.369086+0000", 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000203222 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.371822+0000", 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002735061 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.372611+0000", 2024-06-14T23:16:52.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00078912199999999996 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.438746+0000", 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.066135042000000005 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.438796+0000", 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0643999999999998e-05 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.438840+0000", 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3368999999999998e-05 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:35123 13.8 13:1da91a8f:::benchmark_data_smithi086_299538_object35122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:14:58.369018+0000", 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 113.715598547, 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.075035150999999994, 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 35123 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.369018+0000", 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.369018+0000", 2024-06-14T23:16:52.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.369017+0000", 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.369039+0000", 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2357999999999998e-05 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.369050+0000", 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0936e-05 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.369061+0000", 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0556e-05 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.370331+0000", 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012700540000000001 2024-06-14T23:16:52.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.370474+0000", 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000143361 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.370631+0000", 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015710600000000001 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.372316+0000", 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016841740000000001 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.372673+0000", 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00035718799999999998 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.443928+0000", 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.071255308000000003 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.443987+0000", 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9141999999999997e-05 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.444053+0000", 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5658000000000001e-05 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:35129 13.0 13:01071272:::benchmark_data_smithi086_299538_object35128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:14:58.371780+0000", 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 113.71283580799999, 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.073417719000000006, 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 35129 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.371780+0000", 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.371780+0000", 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.371779+0000", 2024-06-14T23:16:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.371828+0000", 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8990999999999999e-05 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.371840+0000", 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1956999999999999e-05 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.371847+0000", 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8210000000000004e-06 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.371889+0000", 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1941999999999999e-05 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.372011+0000", 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000121897 2024-06-14T23:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.372144+0000", 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013279500000000001 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373414+0000", 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001270333 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.374964+0000", 2024-06-14T23:16:52.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015498829999999999 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.445112+0000", 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.070148021000000005 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.445160+0000", 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8154e-05 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.445198+0000", 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8105000000000002e-05 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:35131 13.0 13:0034b10f:::benchmark_data_smithi086_299538_object35130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:14:58.373378+0000", 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 113.711238236, 2024-06-14T23:16:52.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.071965035999999996, 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 35131 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373378+0000", 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373378+0000", 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373377+0000", 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373399+0000", 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1348999999999998e-05 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373411+0000", 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2218999999999999e-05 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373418+0000", 2024-06-14T23:16:52.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6190000000000002e-06 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373451+0000", 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2431000000000002e-05 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373572+0000", 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000121221 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.373712+0000", 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014013400000000001 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.374857+0000", 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011451320000000001 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.375029+0000", 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017185 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.445265+0000", 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.070236220000000002 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.445313+0000", 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7553999999999999e-05 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.445343+0000", 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0187000000000001e-05 2024-06-14T23:16:52.132 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:35134 13.0 13:0b2bb66e:::benchmark_data_smithi086_299538_object35133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:14:58.375802+0000", 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 113.708814706, 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.069729996000000002, 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.133 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 35134 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.375802+0000", 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.375802+0000", 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.375801+0000", 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.375820+0000", 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.91e-05 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.375830+0000", 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0172e-05 2024-06-14T23:16:52.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.375836+0000", 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9580000000000004e-06 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.375873+0000", 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7540000000000003e-05 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.376053+0000", 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017918599999999999 2024-06-14T23:16:52.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.376201+0000", 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014834599999999999 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.377100+0000", 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00089874300000000002 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.377322+0000", 2024-06-14T23:16:52.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00022262900000000001 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.445395+0000", 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.068072199999999999 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.445434+0000", 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9311999999999999e-05 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:58.445532+0000", 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7671999999999996e-05 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:36783 13.f e358/351 13:f45ff6ef:::benchmark_data_smithi086_299538_object36782:head v 358'2288, mlcod=358'2285)", 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:14:59.326367+0000", 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 112.75824970399999, 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059290313999999997, 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.139 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.326367+0000", 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.326367+0000", 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.326366+0000", 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.326413+0000", 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6650999999999998e-05 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.326421+0000", 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9699999999999999e-06 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.326425+0000", 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0289999999999999e-06 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.326454+0000", 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9825999999999999e-05 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.326467+0000", 2024-06-14T23:16:52.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2502e-05 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.385634+0000", 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059166599 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.385657+0000", 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3385999999999999e-05 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:36792 13.a 13:52d5535d:::benchmark_data_smithi086_299538_object36791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:14:59.328326+0000", 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 112.756290304, 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.057060765999999999, 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 36792 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.328326+0000", 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.328326+0000", 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.328325+0000", 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.328344+0000", 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9219999999999999e-05 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.381512+0000", 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.053167552999999999 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.381524+0000", 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2249000000000001e-05 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.381568+0000", 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3608e-05 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.381709+0000", 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014100000000000001 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.381854+0000", 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014585299999999999 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.383257+0000", 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014022240000000001 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.384171+0000", 2024-06-14T23:16:52.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00091482200000000003 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.385326+0000", 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001154487 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.385354+0000", 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7710999999999999e-05 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:14:59.385387+0000", 2024-06-14T23:16:52.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.311e-05 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:95709 13.0 13:0520ffed:::benchmark_data_smithi086_299538_object95708:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:15:48.953164+0000", 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.131451873000003, 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.064817476999999998, 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.150 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 95709 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.953164+0000", 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.953164+0000", 2024-06-14T23:16:52.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.953163+0000", 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.953185+0000", 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1863000000000001e-05 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.953196+0000", 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1046999999999999e-05 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.953203+0000", 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9190000000000002e-06 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.953260+0000", 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6860000000000001e-05 2024-06-14T23:16:52.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.953414+0000", 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000154289 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.953535+0000", 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120769 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956167+0000", 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026313690000000002 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.960860+0000", 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046935149999999997 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:49.017890+0000", 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.057029994000000001 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:49.017942+0000", 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1943000000000001e-05 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:49.017982+0000", 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9925000000000001e-05 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.156 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:95710 13.0 13:030cbdf5:::benchmark_data_smithi086_299538_object95709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:15:48.956142+0000", 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 63.128474375000003, 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.061934213000000002, 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 95710 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956142+0000", 2024-06-14T23:16:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956142+0000", 2024-06-14T23:16:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956141+0000", 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956165+0000", 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3944999999999999e-05 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956174+0000", 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.397e-06 2024-06-14T23:16:52.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956181+0000", 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8499999999999996e-06 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956211+0000", 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0329999999999999e-05 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956301+0000", 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0334000000000006e-05 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.956402+0000", 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100599 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:48.959453+0000", 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0030505710000000002 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:49.011305+0000", 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051852239000000001 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:49.018017+0000", 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067123790000000001 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:49.018050+0000", 2024-06-14T23:16:52.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2246999999999998e-05 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:15:49.018076+0000", 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6614e-05 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:112324 13.8 13:11c3d3fe:::benchmark_data_smithi086_299538_object112323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:05.592413+0000", 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 46.492203175, 2024-06-14T23:16:52.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.067416063999999998, 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112324 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592413+0000", 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592413+0000", 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592412+0000", 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592435+0000", 2024-06-14T23:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2719999999999999e-05 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592445+0000", 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0711e-05 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592452+0000", 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7769999999999997e-06 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592495+0000", 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2676999999999998e-05 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592626+0000", 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013097200000000001 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592773+0000", 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000146914 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.598029+0000", 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052557619999999998 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.598077+0000", 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8173000000000001e-05 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.659766+0000", 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.061689581 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.659801+0000", 2024-06-14T23:16:52.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5039000000000001e-05 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.659829+0000", 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7827999999999999e-05 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:112325 13.0 13:091b8c08:::benchmark_data_smithi086_299538_object112324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:05.592466+0000", 2024-06-14T23:16:52.170 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 46.492150387000002, 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.067475263999999993, 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112325 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592466+0000", 2024-06-14T23:16:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592466+0000", 2024-06-14T23:16:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592465+0000", 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592483+0000", 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7554000000000002e-05 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592490+0000", 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3839999999999999e-06 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592496+0000", 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.874e-06 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592526+0000", 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0001e-05 2024-06-14T23:16:52.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592615+0000", 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9017000000000005e-05 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.592726+0000", 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110333 2024-06-14T23:16:52.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.598061+0000", 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053359009999999997 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.598067+0000", 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5890000000000002e-06 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.659857+0000", 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.061789572000000001 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.659902+0000", 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5086999999999997e-05 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.659941+0000", 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9452000000000001e-05 2024-06-14T23:16:52.177 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:112326 13.8 13:1c9830f6:::benchmark_data_smithi086_299538_object112325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-06-14T23:16:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:05.593847+0000", 2024-06-14T23:16:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 46.490769536000002, 2024-06-14T23:16:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.066194840000000005, 2024-06-14T23:16:52.178 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 112326 2024-06-14T23:16:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.593847+0000", 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.593847+0000", 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.593846+0000", 2024-06-14T23:16:52.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.593865+0000", 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9307e-05 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.593876+0000", 2024-06-14T23:16:52.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1277000000000001e-05 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.593883+0000", 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8040000000000004e-06 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.595541+0000", 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016576360000000001 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.595693+0000", 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015257899999999999 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.595845+0000", 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015187499999999999 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.598151+0000", 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023056650000000001 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.599541+0000", 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001389883 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.659954+0000", 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060412644000000001 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.660006+0000", 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1999000000000001e-05 2024-06-14T23:16:52.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:05.660042+0000", 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6044999999999998e-05 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:131557 13.8 13:167c081a:::benchmark_data_smithi086_299538_object10719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:24.569996+0000", 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.514619830000001, 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.062111397999999998, 2024-06-14T23:16:52.185 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 131557 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.569996+0000", 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.569996+0000", 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.569996+0000", 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.570001+0000", 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0980000000000001e-06 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.570011+0000", 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9480000000000003e-06 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.570015+0000", 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5340000000000001e-06 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.570053+0000", 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7697999999999997e-05 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.570173+0000", 2024-06-14T23:16:52.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119725 2024-06-14T23:16:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:16:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.570294+0000", 2024-06-14T23:16:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012139299999999999 2024-06-14T23:16:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.573288+0000", 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029934919999999999 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.631964+0000", 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.058675988999999998 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632021+0000", 2024-06-14T23:16:52.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7309000000000003e-05 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632064+0000", 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3303000000000002e-05 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632108+0000", 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3677999999999998e-05 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:131560 13.a 13:51ad8719:::benchmark_data_smithi086_299538_object10722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:24.572001+0000", 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.512615156999999, 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060269862, 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 131560 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.572001+0000", 2024-06-14T23:16:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.572001+0000", 2024-06-14T23:16:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.572000+0000", 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.572005+0000", 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1159999999999998e-06 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.572016+0000", 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1113e-05 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.572023+0000", 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3509999999999998e-06 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.572058+0000", 2024-06-14T23:16:52.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5608000000000002e-05 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.572135+0000", 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6748000000000004e-05 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.572201+0000", 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6314000000000002e-05 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.631991+0000", 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059789488000000002 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632000+0000", 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9150000000000002e-06 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632210+0000", 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000209707 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632242+0000", 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2249000000000002e-05 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632271+0000", 2024-06-14T23:16:52.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9162e-05 2024-06-14T23:16:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.46844.0:131557 13.8 e372/351 ondisk, result = 0)", 2024-06-14T23:16:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:24.574131+0000", 2024-06-14T23:16:52.200 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.510485412000001, 2024-06-14T23:16:52.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.057837064000000001, 2024-06-14T23:16:52.201 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.201 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:16:52.201 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.574131+0000", 2024-06-14T23:16:52.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.574131+0000", 2024-06-14T23:16:52.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:52.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.574130+0000", 2024-06-14T23:16:52.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:52.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.574134+0000", 2024-06-14T23:16:52.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2310000000000001e-06 2024-06-14T23:16:52.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.574141+0000", 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2130000000000002e-06 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.574145+0000", 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8410000000000002e-06 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.631946+0000", 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.057800820000000003 2024-06-14T23:16:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.631956+0000", 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0916e-05 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.631968+0000", 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1484000000000001e-05 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.46844.0:131557 13.8 e372/351 ondisk, result = 0)", 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:24.575184+0000", 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.509432091000001, 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.056928985000000001, 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.575184+0000", 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.575184+0000", 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.575184+0000", 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.575187+0000", 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1599999999999998e-06 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.631904+0000", 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.056717298999999999 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.631925+0000", 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0992000000000002e-05 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632010+0000", 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5138999999999999e-05 2024-06-14T23:16:52.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632016+0000", 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0379999999999996e-06 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:24.632113+0000", 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6785000000000003e-05 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.210 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.211 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.211 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:139083 13.8 13:1597139c:::benchmark_data_smithi086_299538_object18245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-06-14T23:16:52.211 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:31.170037+0000", 2024-06-14T23:16:52.211 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 20.914579184000001, 2024-06-14T23:16:52.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.081799470999999999, 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 139083 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.170037+0000", 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.170037+0000", 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.170036+0000", 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.170042+0000", 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1349999999999997e-06 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.170054+0000", 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2072000000000001e-05 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.170061+0000", 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.72e-06 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.170108+0000", 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7209000000000002e-05 2024-06-14T23:16:52.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.170225+0000", 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011752 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 5,6", 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.170357+0000", 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000132075 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.172942+0000", 2024-06-14T23:16:52.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025850830000000002 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.173890+0000", 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00094804600000000002 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.251751+0000", 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.077860156 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.251796+0000", 2024-06-14T23:16:52.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5050999999999999e-05 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:31.251837+0000", 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.091e-05 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:154407 13.0 13:05690450:::benchmark_data_smithi086_299538_object33569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:16:47.242723+0000", 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 4.8418935300000001, 2024-06-14T23:16:52.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.058550552999999998, 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 154407 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.242723+0000", 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.242723+0000", 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.242722+0000", 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.242739+0000", 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6884e-05 2024-06-14T23:16:52.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.242746+0000", 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2250000000000003e-06 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.242752+0000", 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3530000000000002e-06 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.245464+0000", 2024-06-14T23:16:52.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027126950000000002 2024-06-14T23:16:52.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:16:52.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.245533+0000", 2024-06-14T23:16:52.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8325999999999996e-05 2024-06-14T23:16:52.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 6,7", 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.245608+0000", 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5630999999999995e-05 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.248094+0000", 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024859859999999999 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.259698+0000", 2024-06-14T23:16:52.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011603561 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.301194+0000", 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.041496400000000003 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.301232+0000", 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7737999999999997e-05 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:16:52.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:16:52.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:16:47.301273+0000", 2024-06-14T23:16:52.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1298999999999999e-05 2024-06-14T23:16:52.224 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.224 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.224 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.224 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:16:52.224 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:16:52.224 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:16:52.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:52.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:52.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:51 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:16:52.308 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:51 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:16:52.308 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:51 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:16:52.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:52.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:52.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:52.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:52.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:52.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:52.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:52.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:52.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:52.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:52.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:52.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:52.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:52.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:52.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:53.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:53.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:53.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:53.161 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:53.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:53.228 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:52 smithi086 ceph-mon[102754]: pgmap v1420: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 4023/258840 objects misplaced (1.554%); 7.9 MiB/s, 126 objects/s recovering 2024-06-14T23:16:53.228 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:52 smithi086 ceph-mon[97913]: pgmap v1420: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 4023/258840 objects misplaced (1.554%); 7.9 MiB/s, 126 objects/s recovering 2024-06-14T23:16:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:52 smithi137 ceph-mon[90634]: pgmap v1420: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 1.02k op/s; 4023/258840 objects misplaced (1.554%); 7.9 MiB/s, 126 objects/s recovering 2024-06-14T23:16:53.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:53.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:53.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:53.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:53.420 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:16:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:16:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:53.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:53.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:53.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:53.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:53.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:53.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:53.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:53.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:53.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:53.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:53.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:53.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:53.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:53.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:53.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:54.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:54.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:54.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:54.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:54.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:53 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:16:54.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:53 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:16:54.247 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:53 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:16:54.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:54.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:54.278 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1420,"stamp":"2024-06-14T23:16:51.787338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5653417031,"num_objects":86280,"num_object_clones":0,"num_object_copies":258840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4023,"num_objects_unfound":0,"num_objects_dirty":86280,"num_whiteouts":0,"num_read":12,"num_read_kb":21,"num_write":155482,"num_write_kb":7733525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1355,"num_bytes_recovered":87663136,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84391,"ondisk_log_size":84391,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":358,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":23499944,"kb_used_data":16754680,"kb_used_omap":27,"kb_used_meta":6744996,"kb_avail":538970968,"statfs":{"total":575970213888,"available":551906271232,"internally_reserved":0,"allocated":17156792320,"data_stored":17145085767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28218,"internal_metadata":6906876358},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,3,4,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-669122560,"num_objects":-10210,"num_object_clones":0,"num_object_copies":-30630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1837,"num_objects_unfound":0,"num_objects_dirty":-10210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10210,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":82640896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002741"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":759,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.442315+0000","last_change":"2024-06-14T23:11:23.417363+0000","last_active":"2024-06-14T23:16:39.442315+0000","last_peered":"2024-06-14T23:16:39.442315+0000","last_clean":"2024-06-14T23:16:39.442315+0000","last_became_active":"2024-06-14T23:11:23.417050+0000","last_became_peered":"2024-06-14T23:11:23.417050+0000","last_unstale":"2024-06-14T23:16:39.442315+0000","last_undegraded":"2024-06-14T23:16:39.442315+0000","last_fullsized":"2024-06-14T23:16:39.442315+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:57.710155+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":212,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.445509+0000","last_change":"2024-06-14T23:16:36.883392+0000","last_active":"2024-06-14T23:16:39.445509+0000","last_peered":"2024-06-14T23:16:39.445509+0000","last_clean":"2024-06-14T23:16:39.445509+0000","last_became_active":"2024-06-14T23:16:36.879421+0000","last_became_peered":"2024-06-14T23:16:36.879421+0000","last_unstale":"2024-06-14T23:16:39.445509+0000","last_undegraded":"2024-06-14T23:16:39.445509+0000","last_fullsized":"2024-06-14T23:16:39.445509+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:15.243934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":366,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.439289+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:16:39.439289+0000","last_peered":"2024-06-14T23:16:39.439289+0000","last_clean":"2024-06-14T23:16:39.439289+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:16:39.439289+0000","last_undegraded":"2024-06-14T23:16:39.439289+0000","last_fullsized":"2024-06-14T23:16:39.439289+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":273,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434984+0000","last_change":"2024-06-14T23:11:23.400983+0000","last_active":"2024-06-14T23:16:39.434984+0000","last_peered":"2024-06-14T23:16:39.434984+0000","last_clean":"2024-06-14T23:16:39.434984+0000","last_became_active":"2024-06-14T23:11:23.397987+0000","last_became_peered":"2024-06-14T23:11:23.397987+0000","last_unstale":"2024-06-14T23:16:39.434984+0000","last_undegraded":"2024-06-14T23:16:39.434984+0000","last_fullsized":"2024-06-14T23:16:39.434984+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:47:04.395690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":163,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434792+0000","last_change":"2024-06-14T23:11:23.428660+0000","last_active":"2024-06-14T23:16:39.434792+0000","last_peered":"2024-06-14T23:16:39.434792+0000","last_clean":"2024-06-14T23:16:39.434792+0000","last_became_active":"2024-06-14T23:11:23.426381+0000","last_became_peered":"2024-06-14T23:11:23.426381+0000","last_unstale":"2024-06-14T23:16:39.434792+0000","last_undegraded":"2024-06-14T23:16:39.434792+0000","last_fullsized":"2024-06-14T23:16:39.434792+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:28:11.185682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":823,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447978+0000","last_change":"2024-06-14T23:14:16.297931+0000","last_active":"2024-06-14T23:16:39.447978+0000","last_peered":"2024-06-14T23:16:39.447978+0000","last_clean":"2024-06-14T23:16:39.447978+0000","last_became_active":"2024-06-14T23:10:29.075070+0000","last_became_peered":"2024-06-14T23:10:29.075070+0000","last_unstale":"2024-06-14T23:16:39.447978+0000","last_undegraded":"2024-06-14T23:16:39.447978+0000","last_fullsized":"2024-06-14T23:16:39.447978+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:14:16.297860+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:14:16.297860+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:50:46.938506+0000","scrub_duration":0.0016077540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":342,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.445512+0000","last_change":"2024-06-14T23:07:58.013543+0000","last_active":"2024-06-14T23:16:39.445512+0000","last_peered":"2024-06-14T23:16:39.445512+0000","last_clean":"2024-06-14T23:16:39.445512+0000","last_became_active":"2024-06-14T23:07:58.011942+0000","last_became_peered":"2024-06-14T23:07:58.011942+0000","last_unstale":"2024-06-14T23:16:39.445512+0000","last_undegraded":"2024-06-14T23:16:39.445512+0000","last_fullsized":"2024-06-14T23:16:39.445512+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:16:27.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":349,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.435457+0000","last_change":"2024-06-14T23:15:09.033723+0000","last_active":"2024-06-14T23:16:39.435457+0000","last_peered":"2024-06-14T23:16:39.435457+0000","last_clean":"2024-06-14T23:16:39.435457+0000","last_became_active":"2024-06-14T23:15:09.033433+0000","last_became_peered":"2024-06-14T23:15:09.033433+0000","last_unstale":"2024-06-14T23:16:39.435457+0000","last_undegraded":"2024-06-14T23:16:39.435457+0000","last_fullsized":"2024-06-14T23:16:39.435457+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:39.865328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":531,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.442157+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:16:39.442157+0000","last_peered":"2024-06-14T23:16:39.442157+0000","last_clean":"2024-06-14T23:16:39.442157+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:16:39.442157+0000","last_undegraded":"2024-06-14T23:16:39.442157+0000","last_fullsized":"2024-06-14T23:16:39.442157+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":356,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.448213+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:16:39.448213+0000","last_peered":"2024-06-14T23:16:39.448213+0000","last_clean":"2024-06-14T23:16:39.448213+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:16:39.448213+0000","last_undegraded":"2024-06-14T23:16:39.448213+0000","last_fullsized":"2024-06-14T23:16:39.448213+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":358,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441685+0000","last_change":"2024-06-14T23:16:36.874086+0000","last_active":"2024-06-14T23:16:39.441685+0000","last_peered":"2024-06-14T23:16:39.441685+0000","last_clean":"2024-06-14T23:16:39.441685+0000","last_became_active":"2024-06-14T23:16:36.873767+0000","last_became_peered":"2024-06-14T23:16:36.873767+0000","last_unstale":"2024-06-14T23:16:39.441685+0000","last_undegraded":"2024-06-14T23:16:39.441685+0000","last_fullsized":"2024-06-14T23:16:39.441685+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:19:06.918622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":125,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441607+0000","last_change":"2024-06-14T23:15:09.030207+0000","last_active":"2024-06-14T23:16:39.441607+0000","last_peered":"2024-06-14T23:16:39.441607+0000","last_clean":"2024-06-14T23:16:39.441607+0000","last_became_active":"2024-06-14T23:15:09.029856+0000","last_became_peered":"2024-06-14T23:15:09.029856+0000","last_unstale":"2024-06-14T23:16:39.441607+0000","last_undegraded":"2024-06-14T23:16:39.441607+0000","last_fullsized":"2024-06-14T23:16:39.441607+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:50:21.877412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":131,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:40.706620+0000","last_change":"2024-06-14T23:16:36.876835+0000","last_active":"2024-06-14T23:16:40.706620+0000","last_peered":"2024-06-14T23:16:40.706620+0000","last_clean":"2024-06-14T23:16:40.706620+0000","last_became_active":"2024-06-14T23:16:36.876556+0000","last_became_peered":"2024-06-14T23:16:36.876556+0000","last_unstale":"2024-06-14T23:16:40.706620+0000","last_undegraded":"2024-06-14T23:16:40.706620+0000","last_fullsized":"2024-06-14T23:16:40.706620+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:11:22.609810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":682,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.453460+0000","last_change":"2024-06-14T23:09:57.294540+0000","last_active":"2024-06-14T23:16:39.453460+0000","last_peered":"2024-06-14T23:16:39.453460+0000","last_clean":"2024-06-14T23:16:39.453460+0000","last_became_active":"2024-06-14T23:09:44.363850+0000","last_became_peered":"2024-06-14T23:09:44.363850+0000","last_unstale":"2024-06-14T23:16:39.453460+0000","last_undegraded":"2024-06-14T23:16:39.453460+0000","last_fullsized":"2024-06-14T23:16:39.453460+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:32:47.314487+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":251,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.435340+0000","last_change":"2024-06-14T23:15:09.033745+0000","last_active":"2024-06-14T23:16:39.435340+0000","last_peered":"2024-06-14T23:16:39.435340+0000","last_clean":"2024-06-14T23:16:39.435340+0000","last_became_active":"2024-06-14T23:15:09.033449+0000","last_became_peered":"2024-06-14T23:15:09.033449+0000","last_unstale":"2024-06-14T23:16:39.435340+0000","last_undegraded":"2024-06-14T23:16:39.435340+0000","last_fullsized":"2024-06-14T23:16:39.435340+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:33.546853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":170,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.442184+0000","last_change":"2024-06-14T23:15:09.028225+0000","last_active":"2024-06-14T23:16:39.442184+0000","last_peered":"2024-06-14T23:16:39.442184+0000","last_clean":"2024-06-14T23:16:39.442184+0000","last_became_active":"2024-06-14T23:15:09.027806+0000","last_became_peered":"2024-06-14T23:15:09.027806+0000","last_unstale":"2024-06-14T23:16:39.442184+0000","last_undegraded":"2024-06-14T23:16:39.442184+0000","last_fullsized":"2024-06-14T23:16:39.442184+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:43.248153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":785,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.452261+0000","last_change":"2024-06-14T23:11:23.393747+0000","last_active":"2024-06-14T23:16:39.452261+0000","last_peered":"2024-06-14T23:16:39.452261+0000","last_clean":"2024-06-14T23:16:39.452261+0000","last_became_active":"2024-06-14T23:11:23.393434+0000","last_became_peered":"2024-06-14T23:11:23.393434+0000","last_unstale":"2024-06-14T23:16:39.452261+0000","last_undegraded":"2024-06-14T23:16:39.452261+0000","last_fullsized":"2024-06-14T23:16:39.452261+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:36:03.304837+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":177,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441262+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:16:39.441262+0000","last_peered":"2024-06-14T23:16:39.441262+0000","last_clean":"2024-06-14T23:16:39.441262+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:16:39.441262+0000","last_undegraded":"2024-06-14T23:16:39.441262+0000","last_fullsized":"2024-06-14T23:16:39.441262+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":240,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.442110+0000","last_change":"2024-06-14T23:10:26.053206+0000","last_active":"2024-06-14T23:16:39.442110+0000","last_peered":"2024-06-14T23:16:39.442110+0000","last_clean":"2024-06-14T23:16:39.442110+0000","last_became_active":"2024-06-14T23:10:26.049410+0000","last_became_peered":"2024-06-14T23:10:26.049410+0000","last_unstale":"2024-06-14T23:16:39.442110+0000","last_undegraded":"2024-06-14T23:16:39.442110+0000","last_fullsized":"2024-06-14T23:16:39.442110+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:49.210233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":733,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447796+0000","last_change":"2024-06-14T23:11:23.429241+0000","last_active":"2024-06-14T23:16:39.447796+0000","last_peered":"2024-06-14T23:16:39.447796+0000","last_clean":"2024-06-14T23:16:39.447796+0000","last_became_active":"2024-06-14T23:11:23.428734+0000","last_became_peered":"2024-06-14T23:11:23.428734+0000","last_unstale":"2024-06-14T23:16:39.447796+0000","last_undegraded":"2024-06-14T23:16:39.447796+0000","last_fullsized":"2024-06-14T23:16:39.447796+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:59.825217+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":101,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434369+0000","last_change":"2024-06-14T23:11:23.408505+0000","last_active":"2024-06-14T23:16:39.434369+0000","last_peered":"2024-06-14T23:16:39.434369+0000","last_clean":"2024-06-14T23:16:39.434369+0000","last_became_active":"2024-06-14T23:11:23.403962+0000","last_became_peered":"2024-06-14T23:11:23.403962+0000","last_unstale":"2024-06-14T23:16:39.434369+0000","last_undegraded":"2024-06-14T23:16:39.434369+0000","last_fullsized":"2024-06-14T23:16:39.434369+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:42:32.761215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":330,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.450107+0000","last_change":"2024-06-14T23:10:26.047177+0000","last_active":"2024-06-14T23:16:39.450107+0000","last_peered":"2024-06-14T23:16:39.450107+0000","last_clean":"2024-06-14T23:16:39.450107+0000","last_became_active":"2024-06-14T23:10:26.046706+0000","last_became_peered":"2024-06-14T23:10:26.046706+0000","last_unstale":"2024-06-14T23:16:39.450107+0000","last_undegraded":"2024-06-14T23:16:39.450107+0000","last_fullsized":"2024-06-14T23:16:39.450107+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:01:03.373542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":632,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447938+0000","last_change":"2024-06-14T23:15:09.027462+0000","last_active":"2024-06-14T23:16:39.447938+0000","last_peered":"2024-06-14T23:16:39.447938+0000","last_clean":"2024-06-14T23:16:39.447938+0000","last_became_active":"2024-06-14T23:15:09.027093+0000","last_became_peered":"2024-06-14T23:15:09.027093+0000","last_unstale":"2024-06-14T23:16:39.447938+0000","last_undegraded":"2024-06-14T23:16:39.447938+0000","last_fullsized":"2024-06-14T23:16:39.447938+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:41:49.451565+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":363,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.440996+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:16:39.440996+0000","last_peered":"2024-06-14T23:16:39.440996+0000","last_clean":"2024-06-14T23:16:39.440996+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:16:39.440996+0000","last_undegraded":"2024-06-14T23:16:39.440996+0000","last_fullsized":"2024-06-14T23:16:39.440996+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":226,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.446485+0000","last_change":"2024-06-14T23:09:41.993634+0000","last_active":"2024-06-14T23:16:39.446485+0000","last_peered":"2024-06-14T23:16:39.446485+0000","last_clean":"2024-06-14T23:16:39.446485+0000","last_became_active":"2024-06-14T23:09:41.992810+0000","last_became_peered":"2024-06-14T23:09:41.992810+0000","last_unstale":"2024-06-14T23:16:39.446485+0000","last_undegraded":"2024-06-14T23:16:39.446485+0000","last_fullsized":"2024-06-14T23:16:39.446485+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:55:28.663164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":685,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.445102+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:16:39.445102+0000","last_peered":"2024-06-14T23:16:39.445102+0000","last_clean":"2024-06-14T23:16:39.445102+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:16:39.445102+0000","last_undegraded":"2024-06-14T23:16:39.445102+0000","last_fullsized":"2024-06-14T23:16:39.445102+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":246,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441154+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:16:39.441154+0000","last_peered":"2024-06-14T23:16:39.441154+0000","last_clean":"2024-06-14T23:16:39.441154+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:16:39.441154+0000","last_undegraded":"2024-06-14T23:16:39.441154+0000","last_fullsized":"2024-06-14T23:16:39.441154+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434891+0000","last_change":"2024-06-14T23:14:44.938003+0000","last_active":"2024-06-14T23:16:39.434891+0000","last_peered":"2024-06-14T23:16:39.434891+0000","last_clean":"2024-06-14T23:16:39.434891+0000","last_became_active":"2024-06-14T23:14:44.937736+0000","last_became_peered":"2024-06-14T23:14:44.937736+0000","last_unstale":"2024-06-14T23:16:39.434891+0000","last_undegraded":"2024-06-14T23:16:39.434891+0000","last_fullsized":"2024-06-14T23:16:39.434891+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:13:31.172524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":723,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434948+0000","last_change":"2024-06-14T23:14:17.053346+0000","last_active":"2024-06-14T23:16:39.434948+0000","last_peered":"2024-06-14T23:16:39.434948+0000","last_clean":"2024-06-14T23:16:39.434948+0000","last_became_active":"2024-06-14T23:11:23.399840+0000","last_became_peered":"2024-06-14T23:11:23.399840+0000","last_unstale":"2024-06-14T23:16:39.434948+0000","last_undegraded":"2024-06-14T23:16:39.434948+0000","last_fullsized":"2024-06-14T23:16:39.434948+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:17.053258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:17.053258+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:03.297247+0000","scrub_duration":0.0014699820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":264,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.442337+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:16:39.442337+0000","last_peered":"2024-06-14T23:16:39.442337+0000","last_clean":"2024-06-14T23:16:39.442337+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:16:39.442337+0000","last_undegraded":"2024-06-14T23:16:39.442337+0000","last_fullsized":"2024-06-14T23:16:39.442337+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":392,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.449847+0000","last_change":"2024-06-14T23:11:23.382757+0000","last_active":"2024-06-14T23:16:39.449847+0000","last_peered":"2024-06-14T23:16:39.449847+0000","last_clean":"2024-06-14T23:16:39.449847+0000","last_became_active":"2024-06-14T23:11:23.382237+0000","last_became_peered":"2024-06-14T23:11:23.382237+0000","last_unstale":"2024-06-14T23:16:39.449847+0000","last_undegraded":"2024-06-14T23:16:39.449847+0000","last_fullsized":"2024-06-14T23:16:39.449847+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:08:40.459305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":555,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.446528+0000","last_change":"2024-06-14T23:10:03.262706+0000","last_active":"2024-06-14T23:16:39.446528+0000","last_peered":"2024-06-14T23:16:39.446528+0000","last_clean":"2024-06-14T23:16:39.446528+0000","last_became_active":"2024-06-14T23:09:44.363698+0000","last_became_peered":"2024-06-14T23:09:44.363698+0000","last_unstale":"2024-06-14T23:16:39.446528+0000","last_undegraded":"2024-06-14T23:16:39.446528+0000","last_fullsized":"2024-06-14T23:16:39.446528+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:03:35.413193+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":78,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.443828+0000","last_change":"2024-06-14T23:15:09.030189+0000","last_active":"2024-06-14T23:16:39.443828+0000","last_peered":"2024-06-14T23:16:39.443828+0000","last_clean":"2024-06-14T23:16:39.443828+0000","last_became_active":"2024-06-14T23:15:09.029757+0000","last_became_peered":"2024-06-14T23:15:09.029757+0000","last_unstale":"2024-06-14T23:16:39.443828+0000","last_undegraded":"2024-06-14T23:16:39.443828+0000","last_fullsized":"2024-06-14T23:16:39.443828+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:58:48.305434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":320,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.450223+0000","last_change":"2024-06-14T23:15:09.044391+0000","last_active":"2024-06-14T23:16:39.450223+0000","last_peered":"2024-06-14T23:16:39.450223+0000","last_clean":"2024-06-14T23:16:39.450223+0000","last_became_active":"2024-06-14T23:15:09.043763+0000","last_became_peered":"2024-06-14T23:15:09.043763+0000","last_unstale":"2024-06-14T23:16:39.450223+0000","last_undegraded":"2024-06-14T23:16:39.450223+0000","last_fullsized":"2024-06-14T23:16:39.450223+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:46:25.597564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":545,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.448024+0000","last_change":"2024-06-14T23:15:09.438359+0000","last_active":"2024-06-14T23:16:39.448024+0000","last_peered":"2024-06-14T23:16:39.448024+0000","last_clean":"2024-06-14T23:16:39.448024+0000","last_became_active":"2024-06-14T23:15:09.026760+0000","last_became_peered":"2024-06-14T23:15:09.026760+0000","last_unstale":"2024-06-14T23:16:39.448024+0000","last_undegraded":"2024-06-14T23:16:39.448024+0000","last_fullsized":"2024-06-14T23:16:39.448024+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:09.438276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:15:09.438276+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:07.051915+0000","scrub_duration":0.011767011000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.8","version":"379'9662","reported_seq":9700,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:47.391547+0000","last_change":"2024-06-14T23:14:30.741601+0000","last_active":"2024-06-14T23:16:47.391547+0000","last_peered":"2024-06-14T23:16:47.391547+0000","last_clean":"2024-06-14T23:16:47.391547+0000","last_became_active":"2024-06-14T23:14:30.741403+0000","last_became_peered":"2024-06-14T23:14:30.741403+0000","last_unstale":"2024-06-14T23:16:47.391547+0000","last_undegraded":"2024-06-14T23:16:47.391547+0000","last_fullsized":"2024-06-14T23:16:47.391547+0000","mapping_epoch":351,"log_start":"358'3200","ondisk_log_start":"358'3200","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:19:32.635229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351272960,"num_objects":5360,"num_object_clones":0,"num_object_copies":16080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9662,"num_write_kb":480704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":256,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447531+0000","last_change":"2024-06-14T23:15:09.039701+0000","last_active":"2024-06-14T23:16:39.447531+0000","last_peered":"2024-06-14T23:16:39.447531+0000","last_clean":"2024-06-14T23:16:39.447531+0000","last_became_active":"2024-06-14T23:15:09.039273+0000","last_became_peered":"2024-06-14T23:15:09.039273+0000","last_unstale":"2024-06-14T23:16:39.447531+0000","last_undegraded":"2024-06-14T23:16:39.447531+0000","last_fullsized":"2024-06-14T23:16:39.447531+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:51:34.296727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":351,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.449889+0000","last_change":"2024-06-14T23:11:23.440378+0000","last_active":"2024-06-14T23:16:39.449889+0000","last_peered":"2024-06-14T23:16:39.449889+0000","last_clean":"2024-06-14T23:16:39.449889+0000","last_became_active":"2024-06-14T23:11:23.439637+0000","last_became_peered":"2024-06-14T23:11:23.439637+0000","last_unstale":"2024-06-14T23:16:39.449889+0000","last_undegraded":"2024-06-14T23:16:39.449889+0000","last_fullsized":"2024-06-14T23:16:39.449889+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:32:03.026532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":63,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.444102+0000","last_change":"2024-06-14T23:15:09.032054+0000","last_active":"2024-06-14T23:16:39.444102+0000","last_peered":"2024-06-14T23:16:39.444102+0000","last_clean":"2024-06-14T23:16:39.444102+0000","last_became_active":"2024-06-14T23:15:09.031613+0000","last_became_peered":"2024-06-14T23:15:09.031613+0000","last_unstale":"2024-06-14T23:16:39.444102+0000","last_undegraded":"2024-06-14T23:16:39.444102+0000","last_fullsized":"2024-06-14T23:16:39.444102+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:18:14.012846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":705,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441703+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:16:39.441703+0000","last_peered":"2024-06-14T23:16:39.441703+0000","last_clean":"2024-06-14T23:16:39.441703+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:16:39.441703+0000","last_undegraded":"2024-06-14T23:16:39.441703+0000","last_fullsized":"2024-06-14T23:16:39.441703+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.9","version":"379'9830","reported_seq":9865,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:48.411860+0000","last_change":"2024-06-14T23:14:30.745071+0000","last_active":"2024-06-14T23:16:48.411860+0000","last_peered":"2024-06-14T23:16:48.411860+0000","last_clean":"2024-06-14T23:16:48.411860+0000","last_became_active":"2024-06-14T23:14:30.744589+0000","last_became_peered":"2024-06-14T23:14:30.744589+0000","last_unstale":"2024-06-14T23:16:48.411860+0000","last_undegraded":"2024-06-14T23:16:48.411860+0000","last_fullsized":"2024-06-14T23:16:48.411860+0000","mapping_epoch":351,"log_start":"363'5187","ondisk_log_start":"363'5187","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:20:16.097928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353501184,"num_objects":5394,"num_object_clones":0,"num_object_copies":16182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":487168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":294,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441000+0000","last_change":"2024-06-14T23:10:26.052898+0000","last_active":"2024-06-14T23:16:39.441000+0000","last_peered":"2024-06-14T23:16:39.441000+0000","last_clean":"2024-06-14T23:16:39.441000+0000","last_became_active":"2024-06-14T23:10:26.052440+0000","last_became_peered":"2024-06-14T23:10:26.052440+0000","last_unstale":"2024-06-14T23:16:39.441000+0000","last_undegraded":"2024-06-14T23:16:39.441000+0000","last_fullsized":"2024-06-14T23:16:39.441000+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:32:27.288680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":295,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441909+0000","last_change":"2024-06-14T23:15:09.029369+0000","last_active":"2024-06-14T23:16:39.441909+0000","last_peered":"2024-06-14T23:16:39.441909+0000","last_clean":"2024-06-14T23:16:39.441909+0000","last_became_active":"2024-06-14T23:15:09.029112+0000","last_became_peered":"2024-06-14T23:15:09.029112+0000","last_unstale":"2024-06-14T23:16:39.441909+0000","last_undegraded":"2024-06-14T23:16:39.441909+0000","last_fullsized":"2024-06-14T23:16:39.441909+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:39:40.009379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":93,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.445201+0000","last_change":"2024-06-14T23:09:42.002569+0000","last_active":"2024-06-14T23:16:39.445201+0000","last_peered":"2024-06-14T23:16:39.445201+0000","last_clean":"2024-06-14T23:16:39.445201+0000","last_became_active":"2024-06-14T23:09:41.993878+0000","last_became_peered":"2024-06-14T23:09:41.993878+0000","last_unstale":"2024-06-14T23:16:39.445201+0000","last_undegraded":"2024-06-14T23:16:39.445201+0000","last_fullsized":"2024-06-14T23:16:39.445201+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:18.967782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":438,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:46.404216+0000","last_change":"2024-06-14T23:16:46.404216+0000","last_active":"2024-06-14T23:16:46.404216+0000","last_peered":"2024-06-14T23:16:46.404216+0000","last_clean":"2024-06-14T23:16:46.404216+0000","last_became_active":"2024-06-14T23:16:39.459236+0000","last_became_peered":"2024-06-14T23:16:39.459236+0000","last_unstale":"2024-06-14T23:16:46.404216+0000","last_undegraded":"2024-06-14T23:16:46.404216+0000","last_fullsized":"2024-06-14T23:16:46.404216+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:10.023159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:15:10.023159+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":5,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.a","version":"379'9611","reported_seq":12334,"reported_epoch":379,"state":"active+remapped+backfilling","last_fresh":"2024-06-14T23:16:48.597508+0000","last_change":"2024-06-14T23:16:38.005423+0000","last_active":"2024-06-14T23:16:48.597508+0000","last_peered":"2024-06-14T23:16:48.597508+0000","last_clean":"2024-06-14T23:16:35.789491+0000","last_became_active":"2024-06-14T23:16:37.947914+0000","last_became_peered":"2024-06-14T23:16:37.947914+0000","last_unstale":"2024-06-14T23:16:48.597508+0000","last_undegraded":"2024-06-14T23:16:48.597508+0000","last_fullsized":"2024-06-14T23:16:48.597508+0000","mapping_epoch":375,"log_start":"362'4903","ondisk_log_start":"362'4903","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4708,"log_dups_size":0,"ondisk_log_size":4708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:07:26.892686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345702400,"num_objects":5275,"num_object_clones":0,"num_object_copies":15825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4023,"num_objects_unfound":0,"num_objects_dirty":5275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9611,"num_write_kb":476352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1337,"num_bytes_recovered":87621632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,2],"avail_no_missing":["4","2","7"],"object_location_counts":[{"shards":"2,4,7","objects":5275}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":200,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.448366+0000","last_change":"2024-06-14T23:16:36.878268+0000","last_active":"2024-06-14T23:16:39.448366+0000","last_peered":"2024-06-14T23:16:39.448366+0000","last_clean":"2024-06-14T23:16:39.448366+0000","last_became_active":"2024-06-14T23:16:36.877796+0000","last_became_peered":"2024-06-14T23:16:36.877796+0000","last_unstale":"2024-06-14T23:16:39.448366+0000","last_undegraded":"2024-06-14T23:16:39.448366+0000","last_fullsized":"2024-06-14T23:16:39.448366+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:22:05.069447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":364,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.445175+0000","last_change":"2024-06-14T23:10:26.044452+0000","last_active":"2024-06-14T23:16:39.445175+0000","last_peered":"2024-06-14T23:16:39.445175+0000","last_clean":"2024-06-14T23:16:39.445175+0000","last_became_active":"2024-06-14T23:10:26.044150+0000","last_became_peered":"2024-06-14T23:10:26.044150+0000","last_unstale":"2024-06-14T23:16:39.445175+0000","last_undegraded":"2024-06-14T23:16:39.445175+0000","last_fullsized":"2024-06-14T23:16:39.445175+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:20.245909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":268,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.448069+0000","last_change":"2024-06-14T23:16:36.879206+0000","last_active":"2024-06-14T23:16:39.448069+0000","last_peered":"2024-06-14T23:16:39.448069+0000","last_clean":"2024-06-14T23:16:39.448069+0000","last_became_active":"2024-06-14T23:16:36.877376+0000","last_became_peered":"2024-06-14T23:16:36.877376+0000","last_unstale":"2024-06-14T23:16:39.448069+0000","last_undegraded":"2024-06-14T23:16:39.448069+0000","last_fullsized":"2024-06-14T23:16:39.448069+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:53:15.055456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":678,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.450480+0000","last_change":"2024-06-14T23:09:55.324347+0000","last_active":"2024-06-14T23:16:39.450480+0000","last_peered":"2024-06-14T23:16:39.450480+0000","last_clean":"2024-06-14T23:16:39.450480+0000","last_became_active":"2024-06-14T23:09:44.366217+0000","last_became_peered":"2024-06-14T23:09:44.366217+0000","last_unstale":"2024-06-14T23:16:39.450480+0000","last_undegraded":"2024-06-14T23:16:39.450480+0000","last_fullsized":"2024-06-14T23:16:39.450480+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:13:30.574851+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.b","version":"379'9866","reported_seq":9902,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:48.408752+0000","last_change":"2024-06-14T23:14:31.004195+0000","last_active":"2024-06-14T23:16:48.408752+0000","last_peered":"2024-06-14T23:16:48.408752+0000","last_clean":"2024-06-14T23:16:48.408752+0000","last_became_active":"2024-06-14T23:14:31.003779+0000","last_became_peered":"2024-06-14T23:14:31.003779+0000","last_unstale":"2024-06-14T23:16:48.408752+0000","last_undegraded":"2024-06-14T23:16:48.408752+0000","last_fullsized":"2024-06-14T23:16:48.408752+0000","mapping_epoch":351,"log_start":"364'5316","ondisk_log_start":"364'5316","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:42:29.429100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356646912,"num_objects":5442,"num_object_clones":0,"num_object_copies":16326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9866,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":165,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447848+0000","last_change":"2024-06-14T23:16:36.879133+0000","last_active":"2024-06-14T23:16:39.447848+0000","last_peered":"2024-06-14T23:16:39.447848+0000","last_clean":"2024-06-14T23:16:39.447848+0000","last_became_active":"2024-06-14T23:16:36.878088+0000","last_became_peered":"2024-06-14T23:16:36.878088+0000","last_unstale":"2024-06-14T23:16:39.447848+0000","last_undegraded":"2024-06-14T23:16:39.447848+0000","last_fullsized":"2024-06-14T23:16:39.447848+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:12.952411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":101,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434860+0000","last_change":"2024-06-14T23:11:23.412890+0000","last_active":"2024-06-14T23:16:39.434860+0000","last_peered":"2024-06-14T23:16:39.434860+0000","last_clean":"2024-06-14T23:16:39.434860+0000","last_became_active":"2024-06-14T23:11:23.400770+0000","last_became_peered":"2024-06-14T23:11:23.400770+0000","last_unstale":"2024-06-14T23:16:39.434860+0000","last_undegraded":"2024-06-14T23:16:39.434860+0000","last_fullsized":"2024-06-14T23:16:39.434860+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:42:01.828949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":394,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.446556+0000","last_change":"2024-06-14T23:11:23.401754+0000","last_active":"2024-06-14T23:16:39.446556+0000","last_peered":"2024-06-14T23:16:39.446556+0000","last_clean":"2024-06-14T23:16:39.446556+0000","last_became_active":"2024-06-14T23:11:23.401486+0000","last_became_peered":"2024-06-14T23:11:23.401486+0000","last_unstale":"2024-06-14T23:16:39.446556+0000","last_undegraded":"2024-06-14T23:16:39.446556+0000","last_fullsized":"2024-06-14T23:16:39.446556+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:27:13.954111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":658,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.449967+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:16:39.449967+0000","last_peered":"2024-06-14T23:16:39.449967+0000","last_clean":"2024-06-14T23:16:39.449967+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:16:39.449967+0000","last_undegraded":"2024-06-14T23:16:39.449967+0000","last_fullsized":"2024-06-14T23:16:39.449967+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.d","version":"379'9785","reported_seq":9830,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:48.964468+0000","last_change":"2024-06-14T23:15:09.042072+0000","last_active":"2024-06-14T23:16:48.964468+0000","last_peered":"2024-06-14T23:16:48.964468+0000","last_clean":"2024-06-14T23:16:48.964468+0000","last_became_active":"2024-06-14T23:15:09.034036+0000","last_became_peered":"2024-06-14T23:15:09.034036+0000","last_unstale":"2024-06-14T23:16:48.964468+0000","last_undegraded":"2024-06-14T23:16:48.964468+0000","last_fullsized":"2024-06-14T23:16:48.964468+0000","mapping_epoch":359,"log_start":"362'4850","ondisk_log_start":"362'4850","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4935,"log_dups_size":0,"ondisk_log_size":4935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:49:17.304835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356778008,"num_objects":5445,"num_object_clones":0,"num_object_copies":16335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9785,"num_write_kb":487297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":332,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434786+0000","last_change":"2024-06-14T23:11:23.414308+0000","last_active":"2024-06-14T23:16:39.434786+0000","last_peered":"2024-06-14T23:16:39.434786+0000","last_clean":"2024-06-14T23:16:39.434786+0000","last_became_active":"2024-06-14T23:11:23.402373+0000","last_became_peered":"2024-06-14T23:11:23.402373+0000","last_unstale":"2024-06-14T23:16:39.434786+0000","last_undegraded":"2024-06-14T23:16:39.434786+0000","last_fullsized":"2024-06-14T23:16:39.434786+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:47:44.213774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":331,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.450086+0000","last_change":"2024-06-14T23:15:09.044549+0000","last_active":"2024-06-14T23:16:39.450086+0000","last_peered":"2024-06-14T23:16:39.450086+0000","last_clean":"2024-06-14T23:16:39.450086+0000","last_became_active":"2024-06-14T23:15:09.044122+0000","last_became_peered":"2024-06-14T23:15:09.044122+0000","last_unstale":"2024-06-14T23:16:39.450086+0000","last_undegraded":"2024-06-14T23:16:39.450086+0000","last_fullsized":"2024-06-14T23:16:39.450086+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:11:38.624888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":371,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.448151+0000","last_change":"2024-06-14T23:10:26.051396+0000","last_active":"2024-06-14T23:16:39.448151+0000","last_peered":"2024-06-14T23:16:39.448151+0000","last_clean":"2024-06-14T23:16:39.448151+0000","last_became_active":"2024-06-14T23:10:26.051111+0000","last_became_peered":"2024-06-14T23:10:26.051111+0000","last_unstale":"2024-06-14T23:16:39.448151+0000","last_undegraded":"2024-06-14T23:16:39.448151+0000","last_fullsized":"2024-06-14T23:16:39.448151+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:40:41.180971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":772,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447431+0000","last_change":"2024-06-14T23:11:23.429488+0000","last_active":"2024-06-14T23:16:39.447431+0000","last_peered":"2024-06-14T23:16:39.447431+0000","last_clean":"2024-06-14T23:16:39.447431+0000","last_became_active":"2024-06-14T23:11:23.428966+0000","last_became_peered":"2024-06-14T23:11:23.428966+0000","last_unstale":"2024-06-14T23:16:39.447431+0000","last_undegraded":"2024-06-14T23:16:39.447431+0000","last_fullsized":"2024-06-14T23:16:39.447431+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:11.050681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:11.050681+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:27:43.377798+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"379'9814","reported_seq":9849,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:48.414534+0000","last_change":"2024-06-14T23:14:30.743278+0000","last_active":"2024-06-14T23:16:48.414534+0000","last_peered":"2024-06-14T23:16:48.414534+0000","last_clean":"2024-06-14T23:16:48.414534+0000","last_became_active":"2024-06-14T23:14:30.743000+0000","last_became_peered":"2024-06-14T23:14:30.743000+0000","last_unstale":"2024-06-14T23:16:48.414534+0000","last_undegraded":"2024-06-14T23:16:48.414534+0000","last_fullsized":"2024-06-14T23:16:48.414534+0000","mapping_epoch":351,"log_start":"362'5259","ondisk_log_start":"362'5259","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4555,"log_dups_size":0,"ondisk_log_size":4555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:22:35.618792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350879744,"num_objects":5354,"num_object_clones":0,"num_object_copies":16062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9814,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":339,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.445324+0000","last_change":"2024-06-14T23:10:26.051058+0000","last_active":"2024-06-14T23:16:39.445324+0000","last_peered":"2024-06-14T23:16:39.445324+0000","last_clean":"2024-06-14T23:16:39.445324+0000","last_became_active":"2024-06-14T23:10:26.050230+0000","last_became_peered":"2024-06-14T23:10:26.050230+0000","last_unstale":"2024-06-14T23:16:39.445324+0000","last_undegraded":"2024-06-14T23:16:39.445324+0000","last_fullsized":"2024-06-14T23:16:39.445324+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:06:35.905404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":377,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434221+0000","last_change":"2024-06-14T23:11:23.408258+0000","last_active":"2024-06-14T23:16:39.434221+0000","last_peered":"2024-06-14T23:16:39.434221+0000","last_clean":"2024-06-14T23:16:39.434221+0000","last_became_active":"2024-06-14T23:11:23.401586+0000","last_became_peered":"2024-06-14T23:11:23.401586+0000","last_unstale":"2024-06-14T23:16:39.434221+0000","last_undegraded":"2024-06-14T23:16:39.434221+0000","last_fullsized":"2024-06-14T23:16:39.434221+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:07.297376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":182,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434720+0000","last_change":"2024-06-14T23:11:23.428105+0000","last_active":"2024-06-14T23:16:39.434720+0000","last_peered":"2024-06-14T23:16:39.434720+0000","last_clean":"2024-06-14T23:16:39.434720+0000","last_became_active":"2024-06-14T23:11:23.426376+0000","last_became_peered":"2024-06-14T23:11:23.426376+0000","last_unstale":"2024-06-14T23:16:39.434720+0000","last_undegraded":"2024-06-14T23:16:39.434720+0000","last_fullsized":"2024-06-14T23:16:39.434720+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:17:00.917875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":595,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.458855+0000","last_change":"2024-06-14T23:16:39.458474+0000","last_active":"2024-06-14T23:16:39.458855+0000","last_peered":"2024-06-14T23:16:39.458855+0000","last_clean":"2024-06-14T23:16:39.458855+0000","last_became_active":"2024-06-14T23:16:39.457700+0000","last_became_peered":"2024-06-14T23:16:39.457700+0000","last_unstale":"2024-06-14T23:16:39.458855+0000","last_undegraded":"2024-06-14T23:16:39.458855+0000","last_fullsized":"2024-06-14T23:16:39.458855+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.876902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.876902+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:05:44.616044+0000","scrub_duration":0.0055302119999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.f","version":"379'9582","reported_seq":9618,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:46.882382+0000","last_change":"2024-06-14T23:14:30.740225+0000","last_active":"2024-06-14T23:16:46.882382+0000","last_peered":"2024-06-14T23:16:46.882382+0000","last_clean":"2024-06-14T23:16:46.882382+0000","last_became_active":"2024-06-14T23:14:30.739814+0000","last_became_peered":"2024-06-14T23:14:30.739814+0000","last_unstale":"2024-06-14T23:16:46.882382+0000","last_undegraded":"2024-06-14T23:16:46.882382+0000","last_fullsized":"2024-06-14T23:16:46.882382+0000","mapping_epoch":351,"log_start":"362'4512","ondisk_log_start":"362'4512","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:27:30.267316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358219776,"num_objects":5466,"num_object_clones":0,"num_object_copies":16398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9582,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":314,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.448273+0000","last_change":"2024-06-14T23:16:36.880634+0000","last_active":"2024-06-14T23:16:39.448273+0000","last_peered":"2024-06-14T23:16:39.448273+0000","last_clean":"2024-06-14T23:16:39.448273+0000","last_became_active":"2024-06-14T23:16:36.880005+0000","last_became_peered":"2024-06-14T23:16:36.880005+0000","last_unstale":"2024-06-14T23:16:39.448273+0000","last_undegraded":"2024-06-14T23:16:39.448273+0000","last_fullsized":"2024-06-14T23:16:39.448273+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:44:44.891888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":170,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441188+0000","last_change":"2024-06-14T23:15:09.025369+0000","last_active":"2024-06-14T23:16:39.441188+0000","last_peered":"2024-06-14T23:16:39.441188+0000","last_clean":"2024-06-14T23:16:39.441188+0000","last_became_active":"2024-06-14T23:15:09.025108+0000","last_became_peered":"2024-06-14T23:15:09.025108+0000","last_unstale":"2024-06-14T23:16:39.441188+0000","last_undegraded":"2024-06-14T23:16:39.441188+0000","last_fullsized":"2024-06-14T23:16:39.441188+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:04:22.310031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":346,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.440912+0000","last_change":"2024-06-14T23:10:26.050884+0000","last_active":"2024-06-14T23:16:39.440912+0000","last_peered":"2024-06-14T23:16:39.440912+0000","last_clean":"2024-06-14T23:16:39.440912+0000","last_became_active":"2024-06-14T23:10:26.050441+0000","last_became_peered":"2024-06-14T23:10:26.050441+0000","last_unstale":"2024-06-14T23:16:39.440912+0000","last_undegraded":"2024-06-14T23:16:39.440912+0000","last_fullsized":"2024-06-14T23:16:39.440912+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:26:36.746843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":533,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447606+0000","last_change":"2024-06-14T23:15:09.035281+0000","last_active":"2024-06-14T23:16:39.447606+0000","last_peered":"2024-06-14T23:16:39.447606+0000","last_clean":"2024-06-14T23:16:39.447606+0000","last_became_active":"2024-06-14T23:15:09.034141+0000","last_became_peered":"2024-06-14T23:15:09.034141+0000","last_unstale":"2024-06-14T23:16:39.447606+0000","last_undegraded":"2024-06-14T23:16:39.447606+0000","last_fullsized":"2024-06-14T23:16:39.447606+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.102085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.102085+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:12:36.329142+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.c","version":"379'9585","reported_seq":9621,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:48.420441+0000","last_change":"2024-06-14T23:14:30.745351+0000","last_active":"2024-06-14T23:16:48.420441+0000","last_peered":"2024-06-14T23:16:48.420441+0000","last_clean":"2024-06-14T23:16:48.420441+0000","last_became_active":"2024-06-14T23:14:30.745017+0000","last_became_peered":"2024-06-14T23:14:30.745017+0000","last_unstale":"2024-06-14T23:16:48.420441+0000","last_undegraded":"2024-06-14T23:16:48.420441+0000","last_fullsized":"2024-06-14T23:16:48.420441+0000","mapping_epoch":351,"log_start":"362'4972","ondisk_log_start":"362'4972","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:32.069491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350158848,"num_objects":5343,"num_object_clones":0,"num_object_copies":16029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9585,"num_write_kb":477696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":294,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.451178+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:16:39.451178+0000","last_peered":"2024-06-14T23:16:39.451178+0000","last_clean":"2024-06-14T23:16:39.451178+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:16:39.451178+0000","last_undegraded":"2024-06-14T23:16:39.451178+0000","last_fullsized":"2024-06-14T23:16:39.451178+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":300,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.448077+0000","last_change":"2024-06-14T23:07:58.017235+0000","last_active":"2024-06-14T23:16:39.448077+0000","last_peered":"2024-06-14T23:16:39.448077+0000","last_clean":"2024-06-14T23:16:39.448077+0000","last_became_active":"2024-06-14T23:07:58.014518+0000","last_became_peered":"2024-06-14T23:07:58.014518+0000","last_unstale":"2024-06-14T23:16:39.448077+0000","last_undegraded":"2024-06-14T23:16:39.448077+0000","last_fullsized":"2024-06-14T23:16:39.448077+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:08.231103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":381,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.449876+0000","last_change":"2024-06-14T23:11:23.429064+0000","last_active":"2024-06-14T23:16:39.449876+0000","last_peered":"2024-06-14T23:16:39.449876+0000","last_clean":"2024-06-14T23:16:39.449876+0000","last_became_active":"2024-06-14T23:11:23.428470+0000","last_became_peered":"2024-06-14T23:11:23.428470+0000","last_unstale":"2024-06-14T23:16:39.449876+0000","last_undegraded":"2024-06-14T23:16:39.449876+0000","last_fullsized":"2024-06-14T23:16:39.449876+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:03:38.544095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":672,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434440+0000","last_change":"2024-06-14T23:14:18.033116+0000","last_active":"2024-06-14T23:16:39.434440+0000","last_peered":"2024-06-14T23:16:39.434440+0000","last_clean":"2024-06-14T23:16:39.434440+0000","last_became_active":"2024-06-14T23:11:23.401110+0000","last_became_peered":"2024-06-14T23:11:23.401110+0000","last_unstale":"2024-06-14T23:16:39.434440+0000","last_undegraded":"2024-06-14T23:16:39.434440+0000","last_fullsized":"2024-06-14T23:16:39.434440+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:18.032997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:18.032997+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:57:13.313132+0000","scrub_duration":0.0019777449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.6","version":"379'9667","reported_seq":9711,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:48.397570+0000","last_change":"2024-06-14T23:15:09.037260+0000","last_active":"2024-06-14T23:16:48.397570+0000","last_peered":"2024-06-14T23:16:48.397570+0000","last_clean":"2024-06-14T23:16:48.397570+0000","last_became_active":"2024-06-14T23:15:09.033102+0000","last_became_peered":"2024-06-14T23:15:09.033102+0000","last_unstale":"2024-06-14T23:16:48.397570+0000","last_undegraded":"2024-06-14T23:16:48.397570+0000","last_fullsized":"2024-06-14T23:16:48.397570+0000","mapping_epoch":359,"log_start":"363'5041","ondisk_log_start":"363'5041","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4626,"log_dups_size":0,"ondisk_log_size":4626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:35:04.940982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356319232,"num_objects":5437,"num_object_clones":0,"num_object_copies":16311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9667,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":178,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.442235+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:16:39.442235+0000","last_peered":"2024-06-14T23:16:39.442235+0000","last_clean":"2024-06-14T23:16:39.442235+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:16:39.442235+0000","last_undegraded":"2024-06-14T23:16:39.442235+0000","last_fullsized":"2024-06-14T23:16:39.442235+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":350,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441146+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:16:39.441146+0000","last_peered":"2024-06-14T23:16:39.441146+0000","last_clean":"2024-06-14T23:16:39.441146+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:16:39.441146+0000","last_undegraded":"2024-06-14T23:16:39.441146+0000","last_fullsized":"2024-06-14T23:16:39.441146+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":770,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447689+0000","last_change":"2024-06-14T23:11:23.385899+0000","last_active":"2024-06-14T23:16:39.447689+0000","last_peered":"2024-06-14T23:16:39.447689+0000","last_clean":"2024-06-14T23:16:39.447689+0000","last_became_active":"2024-06-14T23:11:23.385576+0000","last_became_peered":"2024-06-14T23:11:23.385576+0000","last_unstale":"2024-06-14T23:16:39.447689+0000","last_undegraded":"2024-06-14T23:16:39.447689+0000","last_fullsized":"2024-06-14T23:16:39.447689+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.260300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.260300+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:09.659202+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.7","version":"379'9836","reported_seq":9872,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:48.937472+0000","last_change":"2024-06-14T23:14:30.742252+0000","last_active":"2024-06-14T23:16:48.937472+0000","last_peered":"2024-06-14T23:16:48.937472+0000","last_clean":"2024-06-14T23:16:48.937472+0000","last_became_active":"2024-06-14T23:14:30.741955+0000","last_became_peered":"2024-06-14T23:14:30.741955+0000","last_unstale":"2024-06-14T23:16:48.937472+0000","last_undegraded":"2024-06-14T23:16:48.937472+0000","last_fullsized":"2024-06-14T23:16:48.937472+0000","mapping_epoch":351,"log_start":"362'4989","ondisk_log_start":"362'4989","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:37.550055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354549760,"num_objects":5410,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":487872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":366,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.443957+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:16:39.443957+0000","last_peered":"2024-06-14T23:16:39.443957+0000","last_clean":"2024-06-14T23:16:39.443957+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:16:39.443957+0000","last_undegraded":"2024-06-14T23:16:39.443957+0000","last_fullsized":"2024-06-14T23:16:39.443957+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":343,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434645+0000","last_change":"2024-06-14T23:15:09.029095+0000","last_active":"2024-06-14T23:16:39.434645+0000","last_peered":"2024-06-14T23:16:39.434645+0000","last_clean":"2024-06-14T23:16:39.434645+0000","last_became_active":"2024-06-14T23:15:09.028817+0000","last_became_peered":"2024-06-14T23:15:09.028817+0000","last_unstale":"2024-06-14T23:16:39.434645+0000","last_undegraded":"2024-06-14T23:16:39.434645+0000","last_fullsized":"2024-06-14T23:16:39.434645+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:43:20.264367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":738,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447939+0000","last_change":"2024-06-14T23:11:23.428760+0000","last_active":"2024-06-14T23:16:39.447939+0000","last_peered":"2024-06-14T23:16:39.447939+0000","last_clean":"2024-06-14T23:16:39.447939+0000","last_became_active":"2024-06-14T23:11:23.427691+0000","last_became_peered":"2024-06-14T23:11:23.427691+0000","last_unstale":"2024-06-14T23:16:39.447939+0000","last_undegraded":"2024-06-14T23:16:39.447939+0000","last_fullsized":"2024-06-14T23:16:39.447939+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:07:44.833166+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"379'9607","reported_seq":9643,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:46.923207+0000","last_change":"2024-06-14T23:14:30.742997+0000","last_active":"2024-06-14T23:16:46.923207+0000","last_peered":"2024-06-14T23:16:46.923207+0000","last_clean":"2024-06-14T23:16:46.923207+0000","last_became_active":"2024-06-14T23:14:30.742733+0000","last_became_peered":"2024-06-14T23:14:30.742733+0000","last_unstale":"2024-06-14T23:16:46.923207+0000","last_undegraded":"2024-06-14T23:16:46.923207+0000","last_fullsized":"2024-06-14T23:16:46.923207+0000","mapping_epoch":351,"log_start":"362'4600","ondisk_log_start":"362'4600","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:48.248776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356843520,"num_objects":5445,"num_object_clones":0,"num_object_copies":16335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":481664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":362,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434765+0000","last_change":"2024-06-14T23:15:09.028336+0000","last_active":"2024-06-14T23:16:39.434765+0000","last_peered":"2024-06-14T23:16:39.434765+0000","last_clean":"2024-06-14T23:16:39.434765+0000","last_became_active":"2024-06-14T23:15:09.028078+0000","last_became_peered":"2024-06-14T23:15:09.028078+0000","last_unstale":"2024-06-14T23:16:39.434765+0000","last_undegraded":"2024-06-14T23:16:39.434765+0000","last_fullsized":"2024-06-14T23:16:39.434765+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:36:57.434975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":231,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.449929+0000","last_change":"2024-06-14T23:15:09.037746+0000","last_active":"2024-06-14T23:16:39.449929+0000","last_peered":"2024-06-14T23:16:39.449929+0000","last_clean":"2024-06-14T23:16:39.449929+0000","last_became_active":"2024-06-14T23:15:09.037427+0000","last_became_peered":"2024-06-14T23:15:09.037427+0000","last_unstale":"2024-06-14T23:16:39.449929+0000","last_undegraded":"2024-06-14T23:16:39.449929+0000","last_fullsized":"2024-06-14T23:16:39.449929+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:40:34.511973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":761,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.452181+0000","last_change":"2024-06-14T23:11:23.393675+0000","last_active":"2024-06-14T23:16:39.452181+0000","last_peered":"2024-06-14T23:16:39.452181+0000","last_clean":"2024-06-14T23:16:39.452181+0000","last_became_active":"2024-06-14T23:11:23.393376+0000","last_became_peered":"2024-06-14T23:11:23.393376+0000","last_unstale":"2024-06-14T23:16:39.452181+0000","last_undegraded":"2024-06-14T23:16:39.452181+0000","last_fullsized":"2024-06-14T23:16:39.452181+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.280724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.280724+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:31:22.875615+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.5","version":"379'9797","reported_seq":9842,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:48.431479+0000","last_change":"2024-06-14T23:15:09.032574+0000","last_active":"2024-06-14T23:16:48.431479+0000","last_peered":"2024-06-14T23:16:48.431479+0000","last_clean":"2024-06-14T23:16:48.431479+0000","last_became_active":"2024-06-14T23:15:09.027289+0000","last_became_peered":"2024-06-14T23:15:09.027289+0000","last_unstale":"2024-06-14T23:16:48.431479+0000","last_undegraded":"2024-06-14T23:16:48.431479+0000","last_fullsized":"2024-06-14T23:16:48.431479+0000","mapping_epoch":359,"log_start":"362'5160","ondisk_log_start":"362'5160","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:31:01.504384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359071744,"num_objects":5479,"num_object_clones":0,"num_object_copies":16437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9797,"num_write_kb":488832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":273,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.451775+0000","last_change":"2024-06-14T23:15:09.038060+0000","last_active":"2024-06-14T23:16:39.451775+0000","last_peered":"2024-06-14T23:16:39.451775+0000","last_clean":"2024-06-14T23:16:39.451775+0000","last_became_active":"2024-06-14T23:15:09.037814+0000","last_became_peered":"2024-06-14T23:15:09.037814+0000","last_unstale":"2024-06-14T23:16:39.451775+0000","last_undegraded":"2024-06-14T23:16:39.451775+0000","last_fullsized":"2024-06-14T23:16:39.451775+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:48.977466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":127,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.443398+0000","last_change":"2024-06-14T23:15:09.024949+0000","last_active":"2024-06-14T23:16:39.443398+0000","last_peered":"2024-06-14T23:16:39.443398+0000","last_clean":"2024-06-14T23:16:39.443398+0000","last_became_active":"2024-06-14T23:15:09.023224+0000","last_became_peered":"2024-06-14T23:15:09.023224+0000","last_unstale":"2024-06-14T23:16:39.443398+0000","last_undegraded":"2024-06-14T23:16:39.443398+0000","last_fullsized":"2024-06-14T23:16:39.443398+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:12.547802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":663,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434560+0000","last_change":"2024-06-14T23:14:14.052730+0000","last_active":"2024-06-14T23:16:39.434560+0000","last_peered":"2024-06-14T23:16:39.434560+0000","last_clean":"2024-06-14T23:16:39.434560+0000","last_became_active":"2024-06-14T23:11:23.427349+0000","last_became_peered":"2024-06-14T23:11:23.427349+0000","last_unstale":"2024-06-14T23:16:39.434560+0000","last_undegraded":"2024-06-14T23:16:39.434560+0000","last_fullsized":"2024-06-14T23:16:39.434560+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:14.052631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:14.052631+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:51:25.441426+0000","scrub_duration":0.0017584180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"379'9597","reported_seq":9633,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:48.397397+0000","last_change":"2024-06-14T23:14:31.003884+0000","last_active":"2024-06-14T23:16:48.397397+0000","last_peered":"2024-06-14T23:16:48.397397+0000","last_clean":"2024-06-14T23:16:48.397397+0000","last_became_active":"2024-06-14T23:14:31.003542+0000","last_became_peered":"2024-06-14T23:14:31.003542+0000","last_unstale":"2024-06-14T23:16:48.397397+0000","last_undegraded":"2024-06-14T23:16:48.397397+0000","last_fullsized":"2024-06-14T23:16:48.397397+0000","mapping_epoch":351,"log_start":"362'5034","ondisk_log_start":"362'5034","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4563,"log_dups_size":0,"ondisk_log_size":4563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:27:38.082967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342949888,"num_objects":5233,"num_object_clones":0,"num_object_copies":15699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9597,"num_write_kb":474560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":186,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.435200+0000","last_change":"2024-06-14T23:15:09.022932+0000","last_active":"2024-06-14T23:16:39.435200+0000","last_peered":"2024-06-14T23:16:39.435200+0000","last_clean":"2024-06-14T23:16:39.435200+0000","last_became_active":"2024-06-14T23:15:09.022603+0000","last_became_peered":"2024-06-14T23:15:09.022603+0000","last_unstale":"2024-06-14T23:16:39.435200+0000","last_undegraded":"2024-06-14T23:16:39.435200+0000","last_fullsized":"2024-06-14T23:16:39.435200+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:03:53.567467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":357,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.448292+0000","last_change":"2024-06-14T23:07:58.013148+0000","last_active":"2024-06-14T23:16:39.448292+0000","last_peered":"2024-06-14T23:16:39.448292+0000","last_clean":"2024-06-14T23:16:39.448292+0000","last_became_active":"2024-06-14T23:07:58.012576+0000","last_became_peered":"2024-06-14T23:07:58.012576+0000","last_unstale":"2024-06-14T23:16:39.448292+0000","last_undegraded":"2024-06-14T23:16:39.448292+0000","last_fullsized":"2024-06-14T23:16:39.448292+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:38:44.712123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":500,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.444020+0000","last_change":"2024-06-14T23:15:12.943908+0000","last_active":"2024-06-14T23:16:39.444020+0000","last_peered":"2024-06-14T23:16:39.444020+0000","last_clean":"2024-06-14T23:16:39.444020+0000","last_became_active":"2024-06-14T23:15:09.031629+0000","last_became_peered":"2024-06-14T23:15:09.031629+0000","last_unstale":"2024-06-14T23:16:39.444020+0000","last_undegraded":"2024-06-14T23:16:39.444020+0000","last_fullsized":"2024-06-14T23:16:39.444020+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:12.943827+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:15:12.943827+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:48:02.155745+0000","scrub_duration":0.0029576450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3","version":"379'9949","reported_seq":9993,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:50.683521+0000","last_change":"2024-06-14T23:15:09.025883+0000","last_active":"2024-06-14T23:16:50.683521+0000","last_peered":"2024-06-14T23:16:50.683521+0000","last_clean":"2024-06-14T23:16:50.683521+0000","last_became_active":"2024-06-14T23:15:09.022582+0000","last_became_peered":"2024-06-14T23:15:09.022582+0000","last_unstale":"2024-06-14T23:16:50.683521+0000","last_undegraded":"2024-06-14T23:16:50.683521+0000","last_fullsized":"2024-06-14T23:16:50.683521+0000","mapping_epoch":359,"log_start":"363'5245","ondisk_log_start":"363'5245","created":351,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:20:48.026157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340197376,"num_objects":5191,"num_object_clones":0,"num_object_copies":15573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":330,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.449715+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:16:39.449715+0000","last_peered":"2024-06-14T23:16:39.449715+0000","last_clean":"2024-06-14T23:16:39.449715+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:16:39.449715+0000","last_undegraded":"2024-06-14T23:16:39.449715+0000","last_fullsized":"2024-06-14T23:16:39.449715+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":274,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.445205+0000","last_change":"2024-06-14T23:07:58.012386+0000","last_active":"2024-06-14T23:16:39.445205+0000","last_peered":"2024-06-14T23:16:39.445205+0000","last_clean":"2024-06-14T23:16:39.445205+0000","last_became_active":"2024-06-14T23:07:58.011446+0000","last_became_peered":"2024-06-14T23:07:58.011446+0000","last_unstale":"2024-06-14T23:16:39.445205+0000","last_undegraded":"2024-06-14T23:16:39.445205+0000","last_fullsized":"2024-06-14T23:16:39.445205+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:11:46.532842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":658,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.450040+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:16:39.450040+0000","last_peered":"2024-06-14T23:16:39.450040+0000","last_clean":"2024-06-14T23:16:39.450040+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:16:39.450040+0000","last_undegraded":"2024-06-14T23:16:39.450040+0000","last_fullsized":"2024-06-14T23:16:39.450040+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.0","version":"379'9665","reported_seq":9703,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:47.427012+0000","last_change":"2024-06-14T23:14:30.738940+0000","last_active":"2024-06-14T23:16:47.427012+0000","last_peered":"2024-06-14T23:16:47.427012+0000","last_clean":"2024-06-14T23:16:47.427012+0000","last_became_active":"2024-06-14T23:14:30.738700+0000","last_became_peered":"2024-06-14T23:14:30.738700+0000","last_unstale":"2024-06-14T23:16:47.427012+0000","last_undegraded":"2024-06-14T23:16:47.427012+0000","last_fullsized":"2024-06-14T23:16:47.427012+0000","mapping_epoch":351,"log_start":"360'3300","ondisk_log_start":"360'3300","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":6365,"log_dups_size":0,"ondisk_log_size":6365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:29:01.585691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363528192,"num_objects":5547,"num_object_clones":0,"num_object_copies":16641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9665,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":319,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441227+0000","last_change":"2024-06-14T23:15:09.030332+0000","last_active":"2024-06-14T23:16:39.441227+0000","last_peered":"2024-06-14T23:16:39.441227+0000","last_clean":"2024-06-14T23:16:39.441227+0000","last_became_active":"2024-06-14T23:15:09.030035+0000","last_became_peered":"2024-06-14T23:15:09.030035+0000","last_unstale":"2024-06-14T23:16:39.441227+0000","last_undegraded":"2024-06-14T23:16:39.441227+0000","last_fullsized":"2024-06-14T23:16:39.441227+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:14:00.226413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":344,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.444375+0000","last_change":"2024-06-14T23:10:26.060530+0000","last_active":"2024-06-14T23:16:39.444375+0000","last_peered":"2024-06-14T23:16:39.444375+0000","last_clean":"2024-06-14T23:16:39.444375+0000","last_became_active":"2024-06-14T23:10:26.051425+0000","last_became_peered":"2024-06-14T23:10:26.051425+0000","last_unstale":"2024-06-14T23:16:39.444375+0000","last_undegraded":"2024-06-14T23:16:39.444375+0000","last_fullsized":"2024-06-14T23:16:39.444375+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:49:00.824452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":788,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441804+0000","last_change":"2024-06-14T23:10:06.551626+0000","last_active":"2024-06-14T23:16:39.441804+0000","last_peered":"2024-06-14T23:16:39.441804+0000","last_clean":"2024-06-14T23:16:39.441804+0000","last_became_active":"2024-06-14T23:09:44.365474+0000","last_became_peered":"2024-06-14T23:09:44.365474+0000","last_unstale":"2024-06-14T23:16:39.441804+0000","last_undegraded":"2024-06-14T23:16:39.441804+0000","last_fullsized":"2024-06-14T23:16:39.441804+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:03.640733+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1","version":"379'9563","reported_seq":9598,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-06-14T23:16:46.877646+0000","last_change":"2024-06-14T23:14:30.735686+0000","last_active":"2024-06-14T23:16:46.877646+0000","last_peered":"2024-06-14T23:16:46.877646+0000","last_clean":"2024-06-14T23:16:46.877646+0000","last_became_active":"2024-06-14T23:14:30.735396+0000","last_became_peered":"2024-06-14T23:14:30.735396+0000","last_unstale":"2024-06-14T23:16:46.877646+0000","last_undegraded":"2024-06-14T23:16:46.877646+0000","last_fullsized":"2024-06-14T23:16:46.877646+0000","mapping_epoch":351,"log_start":"362'4505","ondisk_log_start":"362'4505","created":351,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:14:29.726619+0000","last_clean_scrub_stamp":"2024-06-14T23:14:29.726619+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:25:23.806204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356188160,"num_objects":5435,"num_object_clones":0,"num_object_copies":16305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":479936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":339,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.452329+0000","last_change":"2024-06-14T23:07:58.010931+0000","last_active":"2024-06-14T23:16:39.452329+0000","last_peered":"2024-06-14T23:16:39.452329+0000","last_clean":"2024-06-14T23:16:39.452329+0000","last_became_active":"2024-06-14T23:07:58.009957+0000","last_became_peered":"2024-06-14T23:07:58.009957+0000","last_unstale":"2024-06-14T23:16:39.452329+0000","last_undegraded":"2024-06-14T23:16:39.452329+0000","last_fullsized":"2024-06-14T23:16:39.452329+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:51:37.698849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":357,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.446834+0000","last_change":"2024-06-14T23:10:26.054371+0000","last_active":"2024-06-14T23:16:39.446834+0000","last_peered":"2024-06-14T23:16:39.446834+0000","last_clean":"2024-06-14T23:16:39.446834+0000","last_became_active":"2024-06-14T23:10:26.045587+0000","last_became_peered":"2024-06-14T23:10:26.045587+0000","last_unstale":"2024-06-14T23:16:39.446834+0000","last_undegraded":"2024-06-14T23:16:39.446834+0000","last_fullsized":"2024-06-14T23:16:39.446834+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:50:02.479130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":741,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.434681+0000","last_change":"2024-06-14T23:15:09.028234+0000","last_active":"2024-06-14T23:16:39.434681+0000","last_peered":"2024-06-14T23:16:39.434681+0000","last_clean":"2024-06-14T23:16:39.434681+0000","last_became_active":"2024-06-14T23:15:09.027710+0000","last_became_peered":"2024-06-14T23:15:09.027710+0000","last_unstale":"2024-06-14T23:16:39.434681+0000","last_undegraded":"2024-06-14T23:16:39.434681+0000","last_fullsized":"2024-06-14T23:16:39.434681+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:46:47.401663+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":365,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.454041+0000","last_change":"2024-06-14T23:09:42.001143+0000","last_active":"2024-06-14T23:16:39.454041+0000","last_peered":"2024-06-14T23:16:39.454041+0000","last_clean":"2024-06-14T23:16:39.454041+0000","last_became_active":"2024-06-14T23:09:41.994136+0000","last_became_peered":"2024-06-14T23:09:41.994136+0000","last_unstale":"2024-06-14T23:16:39.454041+0000","last_undegraded":"2024-06-14T23:16:39.454041+0000","last_fullsized":"2024-06-14T23:16:39.454041+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:57:18.298778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":363,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.442159+0000","last_change":"2024-06-14T23:11:23.437069+0000","last_active":"2024-06-14T23:16:39.442159+0000","last_peered":"2024-06-14T23:16:39.442159+0000","last_clean":"2024-06-14T23:16:39.442159+0000","last_became_active":"2024-06-14T23:11:23.436836+0000","last_became_peered":"2024-06-14T23:11:23.436836+0000","last_unstale":"2024-06-14T23:16:39.442159+0000","last_undegraded":"2024-06-14T23:16:39.442159+0000","last_fullsized":"2024-06-14T23:16:39.442159+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:49:54.213514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":734,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.447773+0000","last_change":"2024-06-14T23:10:29.052895+0000","last_active":"2024-06-14T23:16:39.447773+0000","last_peered":"2024-06-14T23:16:39.447773+0000","last_clean":"2024-06-14T23:16:39.447773+0000","last_became_active":"2024-06-14T23:10:29.052403+0000","last_became_peered":"2024-06-14T23:10:29.052403+0000","last_unstale":"2024-06-14T23:16:39.447773+0000","last_undegraded":"2024-06-14T23:16:39.447773+0000","last_fullsized":"2024-06-14T23:16:39.447773+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:49:13.454357+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":180,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.453573+0000","last_change":"2024-06-14T23:15:09.039681+0000","last_active":"2024-06-14T23:16:39.453573+0000","last_peered":"2024-06-14T23:16:39.453573+0000","last_clean":"2024-06-14T23:16:39.453573+0000","last_became_active":"2024-06-14T23:15:09.039180+0000","last_became_peered":"2024-06-14T23:15:09.039180+0000","last_unstale":"2024-06-14T23:16:39.453573+0000","last_undegraded":"2024-06-14T23:16:39.453573+0000","last_fullsized":"2024-06-14T23:16:39.453573+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:54.312788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":244,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.442223+0000","last_change":"2024-06-14T23:11:23.388147+0000","last_active":"2024-06-14T23:16:39.442223+0000","last_peered":"2024-06-14T23:16:39.442223+0000","last_clean":"2024-06-14T23:16:39.442223+0000","last_became_active":"2024-06-14T23:11:23.387722+0000","last_became_peered":"2024-06-14T23:11:23.387722+0000","last_unstale":"2024-06-14T23:16:39.442223+0000","last_undegraded":"2024-06-14T23:16:39.442223+0000","last_fullsized":"2024-06-14T23:16:39.442223+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:50.614582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":702,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.441668+0000","last_change":"2024-06-14T23:10:29.065669+0000","last_active":"2024-06-14T23:16:39.441668+0000","last_peered":"2024-06-14T23:16:39.441668+0000","last_clean":"2024-06-14T23:16:39.441668+0000","last_became_active":"2024-06-14T23:10:29.065296+0000","last_became_peered":"2024-06-14T23:10:29.065296+0000","last_unstale":"2024-06-14T23:16:39.441668+0000","last_undegraded":"2024-06-14T23:16:39.441668+0000","last_fullsized":"2024-06-14T23:16:39.441668+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:28.891573+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":82,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.435417+0000","last_change":"2024-06-14T23:11:23.414065+0000","last_active":"2024-06-14T23:16:39.435417+0000","last_peered":"2024-06-14T23:16:39.435417+0000","last_clean":"2024-06-14T23:16:39.435417+0000","last_became_active":"2024-06-14T23:11:23.412140+0000","last_became_peered":"2024-06-14T23:11:23.412140+0000","last_unstale":"2024-06-14T23:16:39.435417+0000","last_undegraded":"2024-06-14T23:16:39.435417+0000","last_fullsized":"2024-06-14T23:16:39.435417+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:04:26.450978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":339,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.445420+0000","last_change":"2024-06-14T22:54:06.274802+0000","last_active":"2024-06-14T23:16:39.445420+0000","last_peered":"2024-06-14T23:16:39.445420+0000","last_clean":"2024-06-14T23:16:39.445420+0000","last_became_active":"2024-06-14T22:54:06.274400+0000","last_became_peered":"2024-06-14T22:54:06.274400+0000","last_unstale":"2024-06-14T23:16:39.445420+0000","last_undegraded":"2024-06-14T23:16:39.445420+0000","last_fullsized":"2024-06-14T23:16:39.445420+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:45:22.307005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":698,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.445535+0000","last_change":"2024-06-14T23:07:59.004905+0000","last_active":"2024-06-14T23:16:39.445535+0000","last_peered":"2024-06-14T23:16:39.445535+0000","last_clean":"2024-06-14T23:16:39.445535+0000","last_became_active":"2024-06-14T23:07:59.002821+0000","last_became_peered":"2024-06-14T23:07:59.002821+0000","last_unstale":"2024-06-14T23:16:39.445535+0000","last_undegraded":"2024-06-14T23:16:39.445535+0000","last_fullsized":"2024-06-14T23:16:39.445535+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:23:44.168417+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":391,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.443481+0000","last_change":"2024-06-14T23:11:23.430102+0000","last_active":"2024-06-14T23:16:39.443481+0000","last_peered":"2024-06-14T23:16:39.443481+0000","last_clean":"2024-06-14T23:16:39.443481+0000","last_became_active":"2024-06-14T23:11:23.428174+0000","last_became_peered":"2024-06-14T23:11:23.428174+0000","last_unstale":"2024-06-14T23:16:39.443481+0000","last_undegraded":"2024-06-14T23:16:39.443481+0000","last_fullsized":"2024-06-14T23:16:39.443481+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:31.070260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":82,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.435016+0000","last_change":"2024-06-14T23:11:23.415403+0000","last_active":"2024-06-14T23:16:39.435016+0000","last_peered":"2024-06-14T23:16:39.435016+0000","last_clean":"2024-06-14T23:16:39.435016+0000","last_became_active":"2024-06-14T23:11:23.404045+0000","last_became_peered":"2024-06-14T23:11:23.404045+0000","last_unstale":"2024-06-14T23:16:39.435016+0000","last_undegraded":"2024-06-14T23:16:39.435016+0000","last_fullsized":"2024-06-14T23:16:39.435016+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:47.221517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":831,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-06-14T23:16:39.470690+0000","last_change":"2024-06-14T23:16:39.469324+0000","last_active":"2024-06-14T23:16:39.470690+0000","last_peered":"2024-06-14T23:16:39.470690+0000","last_clean":"2024-06-14T23:16:39.470690+0000","last_became_active":"2024-06-14T23:16:39.468817+0000","last_became_peered":"2024-06-14T23:16:39.468817+0000","last_unstale":"2024-06-14T23:16:39.470690+0000","last_undegraded":"2024-06-14T23:16:39.470690+0000","last_fullsized":"2024-06-14T23:16:39.470690+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:14:29.368195+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":5652807704,"num_objects":86256,"num_object_clones":0,"num_object_copies":258768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4023,"num_objects_unfound":0,"num_objects_dirty":86256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155416,"num_write_kb":7733441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1337,"num_bytes_recovered":87621632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17018925056,"data_stored":17018912840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79343,"ondisk_log_size":79343,"up":48,"acting":48,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1252,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603785,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":5,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234513,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3939844,"kb_used_data":2428796,"kb_used_omap":1,"kb_used_meta":1510974,"kb_avail":89805308,"statfs":{"total":95995035648,"available":91960635392,"internally_reserved":0,"allocated":2487087104,"data_stored":2485119584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1547237720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:40 2024","interfaces":[{"interface":"back","average":{"1min":2.3260000000000001,"5min":1.3109999999999999,"15min":1.0249999999999999},"min":{"1min":0.25600000000000001,"5min":0.24199999999999999,"15min":0.19800000000000001},"max":{"1min":12.263999999999999,"5min":12.263999999999999,"15min":12.263999999999999},"last":1.605},{"interface":"front","average":{"1min":1.165,"5min":1.0840000000000001,"15min":1.155},"min":{"1min":0.26400000000000001,"5min":0.26300000000000001,"15min":0.224},"max":{"1min":4.1349999999999998,"5min":6.5149999999999997,"15min":6.9550000000000001},"last":5.5129999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:16:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.1599999999999999,"15min":1.23},"min":{"1min":0.29699999999999999,"5min":0.25600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1719999999999997,"5min":6.0229999999999997,"15min":12.010999999999999},"last":5.5419999999999998},{"interface":"front","average":{"1min":2.3740000000000001,"5min":1.512,"15min":1.331},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.22900000000000001},"max":{"1min":12.295,"5min":12.295,"15min":13.531000000000001},"last":1.5840000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:16:32 2024","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.2769999999999999,"15min":1.3759999999999999},"min":{"1min":0.29999999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.0030000000000001,"5min":10.539,"15min":10.539},"last":1.212},{"interface":"front","average":{"1min":1.504,"5min":1.2529999999999999,"15min":1.3200000000000001},"min":{"1min":0.373,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.0300000000000002,"5min":10.56,"15min":10.56},"last":1.2949999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:16:40 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":1.5349999999999999,"15min":1.46},"min":{"1min":0.32000000000000001,"5min":0.30599999999999999,"15min":0.28299999999999997},"max":{"1min":6.1130000000000004,"5min":17.991,"15min":20.645},"last":1.274},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.4119999999999999,"15min":1.51},"min":{"1min":0.315,"5min":0.315,"15min":0.28299999999999997},"max":{"1min":4.2000000000000002,"5min":17.824999999999999,"15min":22.042999999999999},"last":5.4500000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:16:32 2024","interfaces":[{"interface":"back","average":{"1min":2.3010000000000002,"5min":1.3740000000000001,"15min":1.1990000000000001},"min":{"1min":0.28399999999999997,"5min":0.26300000000000001,"15min":0.23100000000000001},"max":{"1min":6.1539999999999999,"5min":6.1539999999999999,"15min":11.342000000000001},"last":5.9089999999999998},{"interface":"front","average":{"1min":2.117,"5min":1.486,"15min":1.2529999999999999},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.22700000000000001},"max":{"1min":4.5339999999999998,"5min":10.499000000000001,"15min":12.651},"last":1.248}]},{"osd":5,"last update":"Fri Jun 14 23:16:32 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":1.5840000000000001,"15min":1.1919999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.23699999999999999},"max":{"1min":12.236000000000001,"5min":12.236000000000001,"15min":12.236000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":2.6600000000000001,"5min":1.504,"15min":1.26},"min":{"1min":0.35099999999999998,"5min":0.29299999999999998,"15min":0.24399999999999999},"max":{"1min":12.842000000000001,"5min":12.842000000000001,"15min":12.842000000000001},"last":5.4859999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:16:06 2024","interfaces":[{"interface":"back","average":{"1min":3.4940000000000002,"5min":2.2930000000000001,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":12.317,"5min":12.317,"15min":12.317},"last":1.5609999999999999},{"interface":"front","average":{"1min":2.2919999999999998,"5min":1.8720000000000001,"15min":1.5369999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.32600000000000001},"max":{"1min":4.843,"5min":10.087999999999999,"15min":10.087999999999999},"last":10.050000000000001}]}]},{"osd":6,"up_from":272,"seq":1168231104619,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3436400,"kb_used_data":2475496,"kb_used_omap":7,"kb_used_meta":960888,"kb_avail":90308752,"statfs":{"total":95995035648,"available":92476162048,"internally_reserved":0,"allocated":2534907904,"data_stored":2532960397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":983949754},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.95099999999999996,"15min":1.4099999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.5709999999999997,"5min":4.5709999999999997,"15min":6.6440000000000001},"last":4.2510000000000003},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.028,"15min":1.4710000000000001},"min":{"1min":0.32200000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.6420000000000003,"5min":4.6420000000000003,"15min":6.6790000000000003},"last":4.3410000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.113,"15min":1.8149999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.25800000000000001},"max":{"1min":5.6509999999999998,"5min":5.6509999999999998,"15min":8.173},"last":1.381},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.97299999999999998,"15min":1.704},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":10.380000000000001},"last":4.3159999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:16:32 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.407,"15min":1.2130000000000001},"min":{"1min":0.32400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.8399999999999999,"5min":29.940999999999999,"15min":29.940999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.101,"5min":1.482,"15min":1.284},"min":{"1min":0.315,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.9910000000000001,"5min":30.135999999999999,"15min":30.135999999999999},"last":0.5}]},{"osd":3,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.0349999999999999,"15min":1.548},"min":{"1min":0.36299999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.5330000000000004,"5min":4.5330000000000004,"15min":6.7089999999999996},"last":4.2800000000000002},{"interface":"front","average":{"1min":1.635,"5min":1.3899999999999999,"15min":1.8149999999999999},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.1449999999999996,"5min":7.4969999999999999,"15min":8.4740000000000002},"last":0.44600000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":2.2349999999999999,"5min":1.484,"15min":1.45},"min":{"1min":0.51900000000000002,"5min":0.24399999999999999,"15min":0.22900000000000001},"max":{"1min":5.758,"5min":7.3159999999999998,"15min":11.010999999999999},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.8700000000000001,"5min":1.325,"15min":1.27},"min":{"1min":0.28799999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":6.4269999999999996,"5min":6.4269999999999996,"15min":6.4269999999999996},"last":4.3639999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":1.4490000000000001,"15min":1.379},"min":{"1min":0.33000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.0670000000000002,"5min":7.0670000000000002,"15min":7.7469999999999999},"last":4.3860000000000001},{"interface":"front","average":{"1min":2.7429999999999999,"5min":1.597,"15min":1.593},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.26900000000000002},"max":{"1min":6.7889999999999997,"5min":7.4359999999999999,"15min":7.8680000000000003},"last":0.72799999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":2.4489999999999998,"5min":1.7549999999999999,"15min":1.738},"min":{"1min":0.40000000000000002,"5min":0.33100000000000002,"15min":0.29399999999999998},"max":{"1min":6.827,"5min":10.016999999999999,"15min":10.016999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":2.9350000000000001,"5min":1.839,"15min":1.625},"min":{"1min":0.36599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.906000000000001,"5min":10.906000000000001,"15min":10.906000000000001},"last":1.3440000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757788,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4950892,"kb_used_data":3467044,"kb_used_omap":1,"kb_used_meta":1483774,"kb_avail":88794260,"statfs":{"total":95995035648,"available":90925322240,"internally_reserved":0,"allocated":3550253056,"data_stored":3548305455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1519384920},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":1.1910000000000001,"15min":1.3979999999999999},"min":{"1min":0.26500000000000001,"5min":0.23999999999999999,"15min":0.20100000000000001},"max":{"1min":8.3770000000000007,"5min":8.3770000000000007,"15min":14.94},"last":4.8680000000000003},{"interface":"front","average":{"1min":2.7250000000000001,"5min":1.0940000000000001,"15min":1.383},"min":{"1min":0.245,"5min":0.245,"15min":0.219},"max":{"1min":13.398999999999999,"5min":13.398999999999999,"15min":13.398999999999999},"last":1.8460000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:16:18 2024","interfaces":[{"interface":"back","average":{"1min":2.1970000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":7.6219999999999999,"5min":7.6219999999999999,"15min":7.6219999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.9980000000000002,"5min":1.224,"15min":1.234},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":8.4480000000000004,"5min":8.4480000000000004,"15min":9.5039999999999996},"last":4.8979999999999997}]},{"osd":2,"last update":"Fri Jun 14 23:16:33 2024","interfaces":[{"interface":"back","average":{"1min":2.7429999999999999,"5min":1.3799999999999999,"15min":0.89200000000000002},"min":{"1min":0.38900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":8.3989999999999991,"5min":8.3989999999999991,"15min":8.3989999999999991},"last":4.8369999999999997},{"interface":"front","average":{"1min":2.746,"5min":1.421,"15min":0.89700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":8.4760000000000009,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":4.9199999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":3.044,"5min":1.462,"15min":1.444},"min":{"1min":0.34000000000000002,"5min":0.26900000000000002,"15min":0.249},"max":{"1min":13.352,"5min":21.815000000000001,"15min":21.815000000000001},"last":1.8200000000000001},{"interface":"front","average":{"1min":2.9729999999999999,"5min":1.5509999999999999,"15min":1.4750000000000001},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":8.4269999999999996,"5min":21.797000000000001,"15min":21.797000000000001},"last":4.9400000000000004}]},{"osd":4,"last update":"Fri Jun 14 23:16:33 2024","interfaces":[{"interface":"back","average":{"1min":3.0430000000000001,"5min":1.3680000000000001,"15min":1.0820000000000001},"min":{"1min":0.35899999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":7.468,"5min":7.468,"15min":14.352},"last":0.57099999999999995},{"interface":"front","average":{"1min":4.1470000000000002,"5min":1.849,"15min":1.371},"min":{"1min":0.28699999999999998,"5min":0.27000000000000002,"15min":0.253},"max":{"1min":13.443,"5min":13.443,"15min":13.443},"last":2.1469999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:16:18 2024","interfaces":[{"interface":"back","average":{"1min":3.258,"5min":1.671,"15min":1.4870000000000001},"min":{"1min":0.42899999999999999,"5min":0.29599999999999999,"15min":0.26100000000000001},"max":{"1min":10.893000000000001,"5min":30.934000000000001,"15min":30.934000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":3.827,"5min":1.9219999999999999,"15min":1.804},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.26100000000000001},"max":{"1min":13.465999999999999,"5min":31.059999999999999,"15min":31.059999999999999},"last":1.869}]},{"osd":7,"last update":"Fri Jun 14 23:16:33 2024","interfaces":[{"interface":"back","average":{"1min":3.8969999999999998,"5min":2.282,"15min":1.5600000000000001},"min":{"1min":0.36299999999999999,"5min":0.25,"15min":0.25},"max":{"1min":8.4309999999999992,"5min":26.981999999999999,"15min":26.981999999999999},"last":9.7040000000000006},{"interface":"front","average":{"1min":3.282,"5min":2.0030000000000001,"15min":1.464},"min":{"1min":0.36699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":10.925000000000001,"5min":23.326000000000001,"15min":23.326000000000001},"last":0.54200000000000004}]}]},{"osd":4,"up_from":30,"seq":128849019424,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4601340,"kb_used_data":3109108,"kb_used_omap":1,"kb_used_meta":1492222,"kb_avail":89143812,"statfs":{"total":95995035648,"available":91283263488,"internally_reserved":0,"allocated":3183726592,"data_stored":3181766955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1528035672},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":0.86599999999999999,"15min":1.1100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.17100000000000001},"max":{"1min":3.9369999999999998,"5min":3.9369999999999998,"15min":10.567},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":0.93600000000000005,"15min":1.0880000000000001},"min":{"1min":0.32900000000000001,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3920000000000003,"5min":6.7779999999999996,"15min":7.4900000000000002},"last":1.7789999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:16:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":0.88600000000000001,"15min":1.3660000000000001},"min":{"1min":0.33600000000000002,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":3.944,"5min":4.476,"15min":56.917000000000002},"last":2.25},{"interface":"front","average":{"1min":1.052,"5min":0.92800000000000005,"15min":1.431},"min":{"1min":0.34799999999999998,"5min":0.29099999999999998,"15min":0.249},"max":{"1min":3.7890000000000001,"5min":4.6600000000000001,"15min":56.789999999999999},"last":0.56599999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.877,"15min":0.83399999999999996},"min":{"1min":0.27800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.9670000000000001,"5min":3.9670000000000001,"15min":3.9670000000000001},"last":2.2130000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.92700000000000005,"15min":0.89600000000000002},"min":{"1min":0.33900000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.9900000000000002,"5min":3.9900000000000002,"15min":3.9900000000000002},"last":2.2839999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:16:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":0.91500000000000004,"15min":1.274},"min":{"1min":0.38300000000000001,"5min":0.30599999999999999,"15min":0.25},"max":{"1min":5.3540000000000001,"5min":5.3540000000000001,"15min":44.389000000000003},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.016,"15min":1.4530000000000001},"min":{"1min":0.33500000000000002,"5min":0.29599999999999999,"15min":0.27600000000000002},"max":{"1min":4.6120000000000001,"5min":4.6120000000000001,"15min":46.101999999999997},"last":0.53900000000000003}]},{"osd":5,"last update":"Fri Jun 14 23:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.6680000000000001,"5min":1.3520000000000001,"15min":1.153},"min":{"1min":0.29899999999999999,"5min":0.24399999999999999,"15min":0.20300000000000001},"max":{"1min":8.173,"5min":8.173,"15min":12.315},"last":5.8540000000000001},{"interface":"front","average":{"1min":2.1819999999999999,"5min":1.3280000000000001,"15min":1.1379999999999999},"min":{"1min":0.32800000000000001,"5min":0.28799999999999998,"15min":0.23300000000000001},"max":{"1min":7.7789999999999999,"5min":7.7789999999999999,"15min":10.959},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:16:13 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":2.548,"15min":1.5680000000000001},"min":{"1min":0.28399999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.5780000000000003,"5min":122.88500000000001,"15min":122.88500000000001},"last":2.6869999999999998},{"interface":"front","average":{"1min":2.1269999999999998,"5min":2.6640000000000001,"15min":1.863},"min":{"1min":0.65100000000000002,"5min":0.30499999999999999,"15min":0.25700000000000001},"max":{"1min":7.2439999999999998,"5min":122.21299999999999,"15min":122.21299999999999},"last":5.9000000000000004}]},{"osd":7,"last update":"Fri Jun 14 23:16:32 2024","interfaces":[{"interface":"back","average":{"1min":3.2480000000000002,"5min":1.534,"15min":1.2410000000000001},"min":{"1min":0.432,"5min":0.27000000000000002,"15min":0.26400000000000001},"max":{"1min":11.430999999999999,"5min":11.430999999999999,"15min":11.430999999999999},"last":2.3159999999999998},{"interface":"front","average":{"1min":2.9140000000000001,"5min":1.571,"15min":1.2390000000000001},"min":{"1min":0.36299999999999999,"5min":0.33300000000000002,"15min":0.27300000000000002},"max":{"1min":7.2930000000000001,"5min":7.2930000000000001,"15min":8.5340000000000007},"last":1.8420000000000001}]}]},{"osd":3,"up_from":138,"seq":592705487123,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3109864,"kb_used_data":2129504,"kb_used_omap":4,"kb_used_meta":980347,"kb_avail":90635288,"statfs":{"total":95995035648,"available":92810534912,"internally_reserved":0,"allocated":2180612096,"data_stored":2178667822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":1003875806},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35799999999999998,"15min":1.1339999999999999},"min":{"1min":0.17199999999999999,"5min":0.16900000000000001,"15min":0.16800000000000001},"max":{"1min":0.84399999999999997,"5min":1.175,"15min":25.190000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.53400000000000003,"15min":1.23},"min":{"1min":0.246,"5min":0.20699999999999999,"15min":0.17799999999999999},"max":{"1min":0.93400000000000005,"5min":4.6369999999999996,"15min":25.652000000000001},"last":0.252}]},{"osd":1,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.56100000000000005,"15min":1.2090000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20300000000000001},"max":{"1min":0.998,"5min":4.6840000000000002,"15min":25.754000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.55400000000000005,"15min":1.3560000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.0189999999999999,"5min":2.6899999999999999,"15min":27.992999999999999},"last":0.40100000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.628,"15min":0.81599999999999995},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.0269999999999999,"5min":5.7549999999999999,"15min":5.7549999999999999},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.65500000000000003,"15min":0.68600000000000005},"min":{"1min":0.30599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.97199999999999998,"5min":3.5550000000000002,"15min":3.5550000000000002},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.1519999999999999,"15min":1.762},"min":{"1min":0.29699999999999999,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":2.419,"5min":17.018999999999998,"15min":73.263000000000005},"last":2.7450000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.2609999999999999,"15min":1.651},"min":{"1min":0.30199999999999999,"5min":0.29499999999999998,"15min":0.27300000000000002},"max":{"1min":3.6699999999999999,"5min":17.074000000000002,"15min":73.5},"last":0.49199999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":0.92000000000000004,"15min":1.643},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.28699999999999998},"max":{"1min":3.339,"5min":3.4740000000000002,"15min":81.438999999999993},"last":1.514},{"interface":"front","average":{"1min":1.2869999999999999,"5min":0.93400000000000005,"15min":1.6930000000000001},"min":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.29299999999999998},"max":{"1min":2.8530000000000002,"5min":4.5990000000000002,"15min":81.626999999999995},"last":0.98299999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:16:12 2024","interfaces":[{"interface":"back","average":{"1min":1.867,"5min":1.5089999999999999,"15min":1.6579999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.34599999999999997},"max":{"1min":15.458,"5min":17.141999999999999,"15min":25.869},"last":1.7949999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.599,"15min":1.5489999999999999},"min":{"1min":0.38600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":15.699,"5min":20.056999999999999,"15min":25.143000000000001},"last":2.8159999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.4850000000000001,"15min":1.4810000000000001},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":3.4060000000000001,"5min":44.887999999999998,"15min":44.887999999999998},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.635,"15min":1.427},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33800000000000002},"max":{"1min":3.5259999999999998,"5min":45.835000000000001,"15min":45.835000000000001},"last":0.51300000000000001}]}]},{"osd":2,"up_from":322,"seq":1382979469378,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3461604,"kb_used_data":3144732,"kb_used_omap":10,"kb_used_meta":316789,"kb_avail":90283548,"statfs":{"total":95995035648,"available":92450353152,"internally_reserved":0,"allocated":3220205568,"data_stored":3218265554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":324392486},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:16:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.45300000000000001,"15min":0.36499999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.91000000000000003,"5min":6.1319999999999997,"15min":6.1319999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.495,"15min":0.45700000000000002},"min":{"1min":0.313,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65200000000000002,"5min":3.0699999999999998,"15min":3.0699999999999998},"last":0.28699999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:16:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.45100000000000001,"15min":0.41899999999999998},"min":{"1min":0.23899999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.69199999999999995,"5min":3.4340000000000002,"15min":3.4340000000000002},"last":0.217},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.53800000000000003,"15min":0.47099999999999997},"min":{"1min":0.23300000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.81499999999999995,"5min":5.0389999999999997,"15min":5.0389999999999997},"last":0.252}]},{"osd":3,"last update":"Fri Jun 14 23:16:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.625,"15min":0.60199999999999998},"min":{"1min":0.224,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.1259999999999999,"5min":3.7519999999999998,"15min":3.7519999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.625,"15min":0.55100000000000005},"min":{"1min":0.27000000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.77400000000000002,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Jun 14 23:16:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.1120000000000001,"15min":0.97199999999999998},"min":{"1min":0.34499999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.8079999999999998,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":0.99099999999999999,"15min":0.77300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.653,"5min":4.3680000000000003,"15min":4.3680000000000003},"last":0.36899999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:16:36 2024","interfaces":[{"interface":"back","average":{"1min":4.4649999999999999,"5min":1.599,"15min":1.052},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":54.969999999999999,"5min":54.969999999999999,"15min":54.969999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":4.782,"5min":1.7150000000000001,"15min":1.1519999999999999},"min":{"1min":0.38800000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":57.284999999999997,"5min":57.284999999999997,"15min":57.284999999999997},"last":2.52}]},{"osd":6,"last update":"Fri Jun 14 23:16:36 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.151,"15min":0.89800000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.7999999999999998,"5min":6.6989999999999998,"15min":6.6989999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.0069999999999999,"15min":0.98399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.9740000000000002,"5min":4.6950000000000003,"15min":4.6950000000000003},"last":2.903}]},{"osd":7,"last update":"Fri Jun 14 23:16:36 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.0309999999999999,"15min":0.97099999999999997},"min":{"1min":0.40200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.869,"5min":4.4480000000000004,"15min":4.4480000000000004},"last":0.501},{"interface":"front","average":{"1min":4.2759999999999998,"5min":1.8879999999999999,"15min":1.363},"min":{"1min":0.46800000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":49.552999999999997,"5min":49.552999999999997,"15min":49.552999999999997},"last":0.65200000000000002}]}]},{"osd":1,"up_from":0,"seq":498216206667,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":390842024322,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3202613248,"data_stored":3202613248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2163281920,"data_stored":2163277848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3165392896,"data_stored":3165388824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3516071936,"data_stored":3516071936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2513567744,"data_stored":2513567744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2457997312,"data_stored":2457993240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:16:54.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set unique_pool_8 pgp_num 16 2024-06-14T23:16:54.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:54.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:54.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:54.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:54.526 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:54.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:54.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:54.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:54.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:54.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:54.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:54.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:54.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:54.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:54.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:55.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:55.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:55.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:55.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:55.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:55.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:55.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:54 smithi137 ceph-mon[90634]: from='client.37104 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:16:55.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:54 smithi137 ceph-mon[90634]: pgmap v1421: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 3100/245805 objects misplaced (1.261%); 10 MiB/s, 163 objects/s recovering 2024-06-14T23:16:55.320 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:54 smithi086 ceph-mon[97913]: from='client.37104 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:16:55.320 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:54 smithi086 ceph-mon[97913]: pgmap v1421: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 3100/245805 objects misplaced (1.261%); 10 MiB/s, 163 objects/s recovering 2024-06-14T23:16:55.321 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:16:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:16:54] "GET /metrics HTTP/1.1" 200 36714 "" "Prometheus/2.43.0" 2024-06-14T23:16:55.321 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:54 smithi086 ceph-mon[102754]: from='client.37104 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:16:55.321 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:54 smithi086 ceph-mon[102754]: pgmap v1421: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 5.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 3100/245805 objects misplaced (1.261%); 10 MiB/s, 163 objects/s recovering 2024-06-14T23:16:55.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:55.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:55.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:55.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:55.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:55.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:55.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:55.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:55.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:55.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:55.755 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:55.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:55.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-14T23:16:55.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:55.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:55.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:55.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:56.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:56.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:56.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:56.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:56.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:56.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:56.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:56.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:56.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:56.442 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:56.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:56.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:56.459 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:16:56.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:56.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:56.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:56.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:56.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:56.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:56.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:56.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:56.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:56.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:56.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:56.987 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:16:57.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:57.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:57.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:57.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:57.228 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:57.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:57.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:56 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:16:57.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:56 smithi137 ceph-mon[90634]: pgmap v1422: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 944 op/s; 3100/242787 objects misplaced (1.277%); 7.0 MiB/s, 111 objects/s recovering 2024-06-14T23:16:57.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3641390118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:57.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:57.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:56 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:16:57.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:56 smithi086 ceph-mon[97913]: pgmap v1422: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 944 op/s; 3100/242787 objects misplaced (1.277%); 7.0 MiB/s, 111 objects/s recovering 2024-06-14T23:16:57.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3641390118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:57.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:56 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:16:57.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:56 smithi086 ceph-mon[102754]: pgmap v1422: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 944 op/s; 3100/242787 objects misplaced (1.277%); 7.0 MiB/s, 111 objects/s recovering 2024-06-14T23:16:57.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3641390118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:16:57.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:57.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:57.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:57.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:57.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:57.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:57.628 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:57.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:57.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:57.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:57.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:57.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:57.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:57.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:57.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:57.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:58.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:58.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:58.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:58.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:58.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:16:58.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:16:58.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:58.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:16:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3641390118' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[97913]: osdmap e380: 8 total, 8 up, 6 in 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3641390118' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[102754]: osdmap e380: 8 total, 8 up, 6 in 2024-06-14T23:16:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:58.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:58.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:58.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:58.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:57 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:16:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:57 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:16:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3641390118' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:16:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:57 smithi137 ceph-mon[90634]: osdmap e380: 8 total, 8 up, 6 in 2024-06-14T23:16:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:16:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:16:58.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:58.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:58.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:58.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:58.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:58.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:58.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:16:58.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:58.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:16:58.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:58.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:58.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:59.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:16:59.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:59.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:16:59.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:16:59.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:59.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:16:59.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:16:59.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:16:58 smithi086 ceph-mon[97913]: pgmap v1424: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 3100/237654 objects misplaced (1.304%); 8.3 MiB/s, 133 objects/s recovering 2024-06-14T23:16:59.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:16:58 smithi086 ceph-mon[102754]: pgmap v1424: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 3100/237654 objects misplaced (1.304%); 8.3 MiB/s, 133 objects/s recovering 2024-06-14T23:16:59.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:59.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:16:59.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:16:59.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:16:58 smithi137 ceph-mon[90634]: pgmap v1424: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 3100/237654 objects misplaced (1.304%); 8.3 MiB/s, 133 objects/s recovering 2024-06-14T23:16:59.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:59.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:16:59.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:59.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:59.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:59.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:16:59.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:16:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:59.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:16:59.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:16:59.918 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:16:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:16:59.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:00.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:00.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:00.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:00.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:00.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:00.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:00.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:00.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:00.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:00.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:00.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:00.447 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:00.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:00.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:00.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:00.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:00.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:00.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:00.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:00.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:00.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:00.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:00.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:00.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:00.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:01.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:01.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:01.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:01.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:01.255 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:01.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:01.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:01 smithi086 ceph-mon[97913]: pgmap v1425: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 2202/227688 objects misplaced (0.967%); 9.0 MiB/s, 143 objects/s recovering 2024-06-14T23:17:01.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:01 smithi086 ceph-mon[102754]: pgmap v1425: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 2202/227688 objects misplaced (0.967%); 9.0 MiB/s, 143 objects/s recovering 2024-06-14T23:17:01.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:01.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:01.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:01.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:01.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:01 smithi137 ceph-mon[90634]: pgmap v1425: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 2202/227688 objects misplaced (0.967%); 9.0 MiB/s, 143 objects/s recovering 2024-06-14T23:17:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:01.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:01.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:01.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:17:01.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:01.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:01.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:01.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:01.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:01.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:01.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:01.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:01.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:02.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:02.065 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:02.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:02.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:02.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:02.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:02.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:02.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:02.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:02.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:02.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:02.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:02.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:02.608 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:02.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:02.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:02.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:02.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:02.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:02.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:02.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:02.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:03.011 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 13 pgp_num to 16 2024-06-14T23:17:03.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:03.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:03.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:03.158 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:03.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:03.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:03.283 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:03.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:03.327 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:03 smithi086 ceph-mon[102754]: pgmap v1426: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 2202/226722 objects misplaced (0.971%); 9.0 MiB/s, 143 objects/s recovering 2024-06-14T23:17:03.327 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:03 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/309031970' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:17:03.327 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:03 smithi086 ceph-mon[97913]: pgmap v1426: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 2202/226722 objects misplaced (0.971%); 9.0 MiB/s, 143 objects/s recovering 2024-06-14T23:17:03.327 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:03 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/309031970' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:17:03.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:03.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:03.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:03.452 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:17:03.452 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:17:03.475 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:17:03.476 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:17:03.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:03 smithi137 ceph-mon[90634]: pgmap v1426: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 2202/226722 objects misplaced (0.971%); 9.0 MiB/s, 143 objects/s recovering 2024-06-14T23:17:03.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:03 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/309031970' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-14T23:17:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:17:03.496 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:17:03.517 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:17:03.517 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:17:03.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:03.547 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:03.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:17:03.555 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:17:03.579 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:17:03.579 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:17:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:17:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:17:03.639 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:17:03.639 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:17:03.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:03.684 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:03.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:03.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:03.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:03.827 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:03.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:03.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:03.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:03.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:04.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:04.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:04.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:04.187 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:04 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:17:04.187 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:04 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/309031970' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:17:04.187 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:04 smithi086 ceph-mon[102754]: osdmap e381: 8 total, 8 up, 6 in 2024-06-14T23:17:04.188 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:04 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3526347836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:04.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:04 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:17:04.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:04 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/309031970' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:17:04.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:04 smithi086 ceph-mon[97913]: osdmap e381: 8 total, 8 up, 6 in 2024-06-14T23:17:04.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:04 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3526347836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:04.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:04.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:04.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:04.327 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:04.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:04.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:04.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:04.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:04 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:17:04.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:04 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/309031970' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"}]': finished 2024-06-14T23:17:04.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:04 smithi137 ceph-mon[90634]: osdmap e381: 8 total, 8 up, 6 in 2024-06-14T23:17:04.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:04 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3526347836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:04.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:04.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:04.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:04.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:04.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:04.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:04.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:04.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:04.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:04.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:04.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:05.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:05.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:05.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:05.027 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:17:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:17:04] "GET /metrics HTTP/1.1" 200 36721 "" "Prometheus/2.43.0" 2024-06-14T23:17:05.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:05.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:05.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:05.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:05.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:05.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:05.314 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:05 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:17:05.314 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:05 smithi086 ceph-mon[97913]: pgmap v1428: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 1366/213561 objects misplaced (0.640%); 11 MiB/s, 178 objects/s recovering 2024-06-14T23:17:05.314 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:05 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:17:05.314 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:05 smithi086 ceph-mon[102754]: pgmap v1428: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 1366/213561 objects misplaced (0.640%); 11 MiB/s, 178 objects/s recovering 2024-06-14T23:17:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:17:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:17:05.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:05.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:05.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:05.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:05 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:17:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:05 smithi137 ceph-mon[90634]: pgmap v1428: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s; 1366/213561 objects misplaced (0.640%); 11 MiB/s, 178 objects/s recovering 2024-06-14T23:17:05.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:05.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:05.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:05.700 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:05.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:05.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:05.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:05.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:05.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:05.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:06.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-14T23:17:06.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:06.074 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:06.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:06.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:06.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:06.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:06.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:06.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:06.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:06 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:17:06.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:06 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:17:06.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:06.392 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:06.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:06.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:06.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:06 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:17:06.520 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:06.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:06.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:06.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:06.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:06.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:06.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:06.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:06.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:06.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:06.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:06.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:07.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:07.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:07.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:07.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:07.241 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:07 smithi086 ceph-mon[97913]: pgmap v1429: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 1366/210483 objects misplaced (0.649%); 10 MiB/s, 162 objects/s recovering 2024-06-14T23:17:07.242 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:07 smithi086 ceph-mon[102754]: pgmap v1429: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 1366/210483 objects misplaced (0.649%); 10 MiB/s, 162 objects/s recovering 2024-06-14T23:17:07.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:07.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:07.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:07.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:07.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:07.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:07.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:07 smithi137 ceph-mon[90634]: pgmap v1429: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.3 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 1366/210483 objects misplaced (0.649%); 10 MiB/s, 162 objects/s recovering 2024-06-14T23:17:07.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:07.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:07.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:07.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:07.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:07.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:07.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:07.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:07.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:07.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:07.998 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:07.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:08.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:08.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:08.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:08.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:08.261 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:08.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:08.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:08 smithi086 ceph-mon[97913]: 1.5 deep-scrub starts 2024-06-14T23:17:08.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:08 smithi086 ceph-mon[102754]: 1.5 deep-scrub starts 2024-06-14T23:17:08.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:08.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:08.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:08.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:08.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:08.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:08 smithi137 ceph-mon[90634]: 1.5 deep-scrub starts 2024-06-14T23:17:08.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:08.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:08.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:17:08.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:08.670 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:08.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:08.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:08.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:08.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:08.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:08.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:08.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:09.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:09.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:09.080 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:09.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:09.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:09.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:09.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:09.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:09.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:09.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:09 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:17:09.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:09 smithi086 ceph-mon[97913]: pgmap v1430: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 1366/205371 objects misplaced (0.665%); 8.9 MiB/s, 143 objects/s recovering 2024-06-14T23:17:09.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:09 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:17:09.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:09 smithi086 ceph-mon[102754]: pgmap v1430: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 1366/205371 objects misplaced (0.665%); 8.9 MiB/s, 143 objects/s recovering 2024-06-14T23:17:09.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:09.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:09.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:09.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:09.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:09.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:09 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:17:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:09 smithi137 ceph-mon[90634]: pgmap v1430: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 1366/205371 objects misplaced (0.665%); 8.9 MiB/s, 143 objects/s recovering 2024-06-14T23:17:09.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:09.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:09.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:09.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:09.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:09.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:09.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:09.873 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:09.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:09.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:10.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:10.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:10.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:10.129 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 2] out_osds: [0, 1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:17:10.129 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:17:10.129 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-06-14T23:17:10.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd in 0 2024-06-14T23:17:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:10.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:10.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:10.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:10.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:10.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:10.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:10.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:10.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:10.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:10.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:10.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:10.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:10.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:10.673 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:10.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:10.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:10.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:10.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:10.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:10.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:10.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:10.982 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:10.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:10 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:17:10.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:10 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:10.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:10 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:10.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:10 smithi086 ceph-mon[97913]: pgmap v1431: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.0 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 603/195093 objects misplaced (0.309%); 8.6 MiB/s, 137 objects/s recovering 2024-06-14T23:17:10.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:10 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:17:10.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:10 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:10.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:10 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:10.994 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:10 smithi086 ceph-mon[102754]: pgmap v1431: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.0 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 603/195093 objects misplaced (0.309%); 8.6 MiB/s, 137 objects/s recovering 2024-06-14T23:17:10.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:10 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:17:10.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:10 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:10.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:10 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:10.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:10 smithi137 ceph-mon[90634]: pgmap v1431: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 4.0 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 603/195093 objects misplaced (0.309%); 8.6 MiB/s, 137 objects/s recovering 2024-06-14T23:17:11.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:11.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:11.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:11.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:11.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:11.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:11.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:11.376 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:11.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:11.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:11.507 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:11.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:11.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:11.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:11.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:11.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:11.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:11.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:11.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:17:11.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:17:11.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:11.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1051304342' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:17:11.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:17:11.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:17:11.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:17:11.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:11.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1051304342' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:17:11.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:11 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:17:11.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:11.904 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:17:11.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:11.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:11.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:17:11.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:17:11.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:11.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1051304342' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:17:11.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:11 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:17:12.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:12.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:12.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:12.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:12.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:12.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:12.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:12.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:12.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:12.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:12.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:12.478 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:12.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:12.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:12.625 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:12.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:12.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:12.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:12.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:12.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:12.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:12.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:12.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[97913]: pgmap v1432: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 3.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 603/194055 objects misplaced (0.311%); 8.6 MiB/s, 137 objects/s recovering 2024-06-14T23:17:12.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:17:12.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:17:12.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[97913]: osdmap e382: 8 total, 8 up, 6 in 2024-06-14T23:17:12.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:12.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:12.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[102754]: pgmap v1432: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 3.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 603/194055 objects misplaced (0.311%); 8.6 MiB/s, 137 objects/s recovering 2024-06-14T23:17:12.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:17:12.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:17:12.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[102754]: osdmap e382: 8 total, 8 up, 6 in 2024-06-14T23:17:12.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:12.946 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:12.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:12 smithi137 ceph-mon[90634]: pgmap v1432: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 3.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 1.09k op/s; 603/194055 objects misplaced (0.311%); 8.6 MiB/s, 137 objects/s recovering 2024-06-14T23:17:12.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:12 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:17:12.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:12 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:17:12.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:12 smithi137 ceph-mon[90634]: osdmap e382: 8 total, 8 up, 6 in 2024-06-14T23:17:12.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:12.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:13.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:13.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:13.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:13.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:13.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:13.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:13.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:13.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:13.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:13.451 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:13.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:13.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:13.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:13.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:13.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:13.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:13.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:13.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:13.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:13.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:13.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:13.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:13.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:13.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:14.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:14.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:14.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:14.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:14.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:14.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:14.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:14.210 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:17:14.210 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:17:14.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:14.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:14.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:14.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:14.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:14.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:14.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:14.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:14 smithi086 ceph-mon[97913]: osdmap e383: 8 total, 8 up, 6 in 2024-06-14T23:17:14.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:14 smithi086 ceph-mon[97913]: pgmap v1435: 119 pgs: 1 peering, 118 active+clean; 3.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s; 9.8 MiB/s, 157 objects/s recovering 2024-06-14T23:17:14.546 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:14 smithi086 ceph-mon[102754]: osdmap e383: 8 total, 8 up, 6 in 2024-06-14T23:17:14.546 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:14 smithi086 ceph-mon[102754]: pgmap v1435: 119 pgs: 1 peering, 118 active+clean; 3.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s; 9.8 MiB/s, 157 objects/s recovering 2024-06-14T23:17:14.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:14.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:14.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:14.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:14.696 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:14.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:14.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:14 smithi137 ceph-mon[90634]: osdmap e383: 8 total, 8 up, 6 in 2024-06-14T23:17:14.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:14 smithi137 ceph-mon[90634]: pgmap v1435: 119 pgs: 1 peering, 118 active+clean; 3.7 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s; 9.8 MiB/s, 157 objects/s recovering 2024-06-14T23:17:14.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:14.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:14.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:14.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:14.979 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:14.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:15.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:15.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:15.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:15.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:15.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:15.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:15.327 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:17:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:17:14] "GET /metrics HTTP/1.1" 200 36719 "" "Prometheus/2.43.0" 2024-06-14T23:17:15.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:15.370 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:15.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:15.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:15.497 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:15.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:15.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:15 smithi086 ceph-mon[97913]: osdmap e384: 8 total, 8 up, 6 in 2024-06-14T23:17:15.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:15 smithi086 ceph-mon[102754]: osdmap e384: 8 total, 8 up, 6 in 2024-06-14T23:17:15.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:15.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:15.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:15.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:15.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:15 smithi137 ceph-mon[90634]: osdmap e384: 8 total, 8 up, 6 in 2024-06-14T23:17:15.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:15.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:15.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:15.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:15.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:15.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:17:16.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:16.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:16.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:16.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:16.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:16.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:16.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:16.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:16.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:16.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:16.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:16.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:16.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:16.608 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:16.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:16.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:16.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:16.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:16 smithi137 ceph-mon[90634]: pgmap v1437: 119 pgs: 1 peering, 118 active+clean; 3.6 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 927 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-06-14T23:17:16.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:16 smithi086 ceph-mon[97913]: pgmap v1437: 119 pgs: 1 peering, 118 active+clean; 3.6 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 927 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-06-14T23:17:16.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:16 smithi086 ceph-mon[102754]: pgmap v1437: 119 pgs: 1 peering, 118 active+clean; 3.6 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 927 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-06-14T23:17:16.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:16.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:16.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:16.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:16.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:16.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:17.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:17.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:17.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:17.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:17.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:17.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:17.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:17.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:17.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:17.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:17.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:17.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:17.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:17.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:17.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:17.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:17.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:17.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:17.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:17.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-14T23:17:17.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:17.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:17.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:18.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:18.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:18.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:18.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:18.194 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:18.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:18.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:18.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:18.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:18.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:18.460 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:18.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:18.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:18.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:18.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:18.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:18.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:18.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:18.789 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:17:18.789 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:17:18.819 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:17:18.820 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:17:18.842 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:17:18.843 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:17:18.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:18.866 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:17:18.867 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:17:18.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:18.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:18.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:17:18.887 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:17:18.918 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:17:18.919 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:17:18.919 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:18 smithi086 ceph-mon[102754]: pgmap v1438: 119 pgs: 1 peering, 118 active+clean; 3.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 6.1 MiB/s, 97 objects/s recovering 2024-06-14T23:17:18.919 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2376157159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:18.919 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:18 smithi086 ceph-mon[97913]: pgmap v1438: 119 pgs: 1 peering, 118 active+clean; 3.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 6.1 MiB/s, 97 objects/s recovering 2024-06-14T23:17:18.919 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2376157159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:18.940 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:17:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:17:18.966 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:17:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:17:18.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:18.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:19.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:19.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:19.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:19.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:19.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:18 smithi137 ceph-mon[90634]: pgmap v1438: 119 pgs: 1 peering, 118 active+clean; 3.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 6.1 MiB/s, 97 objects/s recovering 2024-06-14T23:17:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2376157159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:19.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:19.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:19.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:19.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:19.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:19.500 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:19.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:19.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:19.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:19.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:19.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:19.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:19.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:19.835 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:19.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:19.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:19.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:20.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:20.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:20.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:20.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:20.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:20.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:20.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:20.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:20.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:20.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:20.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:20.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:20.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:20.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:20.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:20.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:20.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:20.748 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:20.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:20.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:20.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:20.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:20.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:21.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:21.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:21.124 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:20 smithi086 ceph-mon[102754]: pgmap v1439: 119 pgs: 119 active+clean; 3.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-06-14T23:17:21.124 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:20 smithi086 ceph-mon[97913]: pgmap v1439: 119 pgs: 119 active+clean; 3.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-06-14T23:17:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:21.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:21.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:21.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:20 smithi137 ceph-mon[90634]: pgmap v1439: 119 pgs: 119 active+clean; 3.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.23k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-06-14T23:17:21.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:21.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:21.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:21.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:21.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:21.441 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:21.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:21.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:21.582 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:21.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:21.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:21.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:21.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:21.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:21.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:21.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:21.876 INFO:teuthology.orchestra.run.smithi086.stderr:marked in osd.0. 2024-06-14T23:17:21.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:21.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:22.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:22.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2548665770' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-06-14T23:17:22.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:17:21.722+0000 7fe0161db640 -1 mon.a@0(leader).osd e384 definitely_dead 0 2024-06-14T23:17:22.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2548665770' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-06-14T23:17:22.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:22.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:22.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:22.227 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:22.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:22.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2548665770' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-06-14T23:17:22.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:22.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:22.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:22.409 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:22.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:17:22.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:22.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:22.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:22.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:22.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:22.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:22.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:22.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:22.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:22.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:22.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:22.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:22.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:22.860 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-06-14T23:17:22.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:22.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:22.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:23.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:23.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:22 smithi086 ceph-mon[97913]: pgmap v1440: 119 pgs: 119 active+clean; 3.3 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 4.3 MiB/s, 69 objects/s recovering 2024-06-14T23:17:23.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2548665770' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-06-14T23:17:23.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:22 smithi086 ceph-mon[97913]: osdmap e385: 8 total, 8 up, 7 in 2024-06-14T23:17:23.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:22 smithi086 ceph-mon[102754]: pgmap v1440: 119 pgs: 119 active+clean; 3.3 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 4.3 MiB/s, 69 objects/s recovering 2024-06-14T23:17:23.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2548665770' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-06-14T23:17:23.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:22 smithi086 ceph-mon[102754]: osdmap e385: 8 total, 8 up, 7 in 2024-06-14T23:17:23.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:23.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:23.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:23.226 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:23.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:23.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:22 smithi137 ceph-mon[90634]: pgmap v1440: 119 pgs: 119 active+clean; 3.3 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 4.3 MiB/s, 69 objects/s recovering 2024-06-14T23:17:23.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2548665770' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-06-14T23:17:23.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:22 smithi137 ceph-mon[90634]: osdmap e385: 8 total, 8 up, 7 in 2024-06-14T23:17:23.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:23.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:23.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:23.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:23.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:23.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:23.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:23.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:23.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:23.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:23.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:23.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:23.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:23.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:23.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:23.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:23.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:23.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:23 smithi086 ceph-mon[102754]: osdmap e386: 8 total, 8 up, 7 in 2024-06-14T23:17:23.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:23 smithi086 ceph-mon[102754]: osdmap e387: 8 total, 8 up, 7 in 2024-06-14T23:17:23.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:23 smithi086 ceph-mon[97913]: osdmap e386: 8 total, 8 up, 7 in 2024-06-14T23:17:23.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:23 smithi086 ceph-mon[97913]: osdmap e387: 8 total, 8 up, 7 in 2024-06-14T23:17:24.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:24.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:24.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:24.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:23 smithi137 ceph-mon[90634]: osdmap e386: 8 total, 8 up, 7 in 2024-06-14T23:17:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:23 smithi137 ceph-mon[90634]: osdmap e387: 8 total, 8 up, 7 in 2024-06-14T23:17:24.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:24.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:24.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:24.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:24.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:24.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:24.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:24.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:24.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:24.649 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:24.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:24.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:24.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:24.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:24.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:24.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:24.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:25.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:25.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:25.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:25.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:25.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[102754]: pgmap v1443: 119 pgs: 7 remapped+peering, 13 peering, 99 active+clean; 3.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 71 B/s, 0 objects/s recovering 2024-06-14T23:17:25.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[102754]: 13.3 starting backfill to osd.0 from (0'0,0'0] MAX to 384'12004 2024-06-14T23:17:25.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[102754]: 13.5 starting backfill to osd.0 from (0'0,0'0] MAX to 384'12057 2024-06-14T23:17:25.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[102754]: 13.9 starting backfill to osd.0 from (0'0,0'0] MAX to 384'12047 2024-06-14T23:17:25.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[102754]: osdmap e388: 8 total, 8 up, 7 in 2024-06-14T23:17:25.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[97913]: pgmap v1443: 119 pgs: 7 remapped+peering, 13 peering, 99 active+clean; 3.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 71 B/s, 0 objects/s recovering 2024-06-14T23:17:25.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[97913]: 13.3 starting backfill to osd.0 from (0'0,0'0] MAX to 384'12004 2024-06-14T23:17:25.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[97913]: 13.5 starting backfill to osd.0 from (0'0,0'0] MAX to 384'12057 2024-06-14T23:17:25.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[97913]: 13.9 starting backfill to osd.0 from (0'0,0'0] MAX to 384'12047 2024-06-14T23:17:25.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-mon[97913]: osdmap e388: 8 total, 8 up, 7 in 2024-06-14T23:17:25.175 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:17:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:17:24] "GET /metrics HTTP/1.1" 200 36719 "" "Prometheus/2.43.0" 2024-06-14T23:17:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:25.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:25.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:24 smithi137 ceph-mon[90634]: pgmap v1443: 119 pgs: 7 remapped+peering, 13 peering, 99 active+clean; 3.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 1.05k op/s; 71 B/s, 0 objects/s recovering 2024-06-14T23:17:25.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:24 smithi137 ceph-mon[90634]: 13.3 starting backfill to osd.0 from (0'0,0'0] MAX to 384'12004 2024-06-14T23:17:25.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:24 smithi137 ceph-mon[90634]: 13.5 starting backfill to osd.0 from (0'0,0'0] MAX to 384'12057 2024-06-14T23:17:25.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:24 smithi137 ceph-mon[90634]: 13.9 starting backfill to osd.0 from (0'0,0'0] MAX to 384'12047 2024-06-14T23:17:25.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:24 smithi137 ceph-mon[90634]: osdmap e388: 8 total, 8 up, 7 in 2024-06-14T23:17:25.282 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:25.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:25.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:25.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:25.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:25.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:25.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:25.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:25.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:25.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:25.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:25.736 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:25.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:25.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:25.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:25.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:25.892 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:17:25.912 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:17:25.913 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:17:25.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:17:25.913 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:17:25.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.913 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191096 13.9 e387/386 13:984f9f0e:::benchmark_data_smithi086_299538_object70258:head v 387'12053, mlcod=0'0)", 2024-06-14T23:17:25.913 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:23.992519+0000", 2024-06-14T23:17:25.913 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.9148111289999998, 2024-06-14T23:17:25.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054241960000000001, 2024-06-14T23:17:25.913 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.992519+0000", 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.992519+0000", 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.992518+0000", 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.992523+0000", 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0050000000000004e-06 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.992535+0000", 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2265e-05 2024-06-14T23:17:25.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.992541+0000", 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1099999999999999e-06 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.992589+0000", 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7305999999999997e-05 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.992608+0000", 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9023000000000001e-05 2024-06-14T23:17:25.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.997879+0000", 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052715410000000002 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:23.997943+0000", 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3559999999999995e-05 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.917 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191071 13.5 e387/386 13:af722e94:::benchmark_data_smithi086_299538_object70233:head v 387'12058, mlcod=0'0)", 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.000430+0000", 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.9068996679999999, 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066446830000000002, 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.000430+0000", 2024-06-14T23:17:25.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.000430+0000", 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.000429+0000", 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.000437+0000", 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9670000000000001e-06 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.000454+0000", 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6444999999999999e-05 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.000461+0000", 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6939999999999995e-06 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.000599+0000", 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013728199999999999 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.000619+0000", 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0058999999999999e-05 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.007006+0000", 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063877509999999997 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.007075+0000", 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8429999999999994e-05 2024-06-14T23:17:25.922 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191108 13.5 e387/386 13:a6fd28e8:::benchmark_data_smithi086_299538_object70270:head v 387'12059, mlcod=0'0)", 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.002348+0000", 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.90498174, 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047574949999999996, 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.002348+0000", 2024-06-14T23:17:25.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.002348+0000", 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.002347+0000", 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.002356+0000", 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2290000000000003e-06 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.002371+0000", 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5529e-05 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.002378+0000", 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8179999999999998e-06 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.002766+0000", 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00038846399999999999 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.002784+0000", 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8002000000000001e-05 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.007079+0000", 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004294511 2024-06-14T23:17:25.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.007106+0000", 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6787e-05 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191360 13.9 e387/386 13:9159e606:::benchmark_data_smithi086_299538_object70522:head v 387'12069, mlcod=387'12067)", 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.181399+0000", 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.7259303620000002, 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052829549999999998, 2024-06-14T23:17:25.927 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.181399+0000", 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.181399+0000", 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.181399+0000", 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.181405+0000", 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8449999999999999e-06 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.181418+0000", 2024-06-14T23:17:25.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3842999999999999e-05 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.181427+0000", 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8179999999999993e-06 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.181491+0000", 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3627000000000006e-05 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.181511+0000", 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0449e-05 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.186641+0000", 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0051292619999999999 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.186682+0000", 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1810000000000001e-05 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.931 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191374 13.5 e387/386 13:aa7d9b22:::benchmark_data_smithi086_299538_object70536:head v 387'12075, mlcod=387'12073)", 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.198570+0000", 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.7087598069999999, 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0065620740000000002, 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198570+0000", 2024-06-14T23:17:25.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198570+0000", 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198569+0000", 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198577+0000", 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6340000000000001e-06 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198592+0000", 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5597999999999999e-05 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198599+0000", 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5819999999999998e-06 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198682+0000", 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3138000000000003e-05 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198705+0000", 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2759e-05 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.205061+0000", 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063557350000000004 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.205132+0000", 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1410999999999998e-05 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191377 13.5 e387/386 13:a8699744:::benchmark_data_smithi086_299538_object70539:head v 387'12076, mlcod=387'12073)", 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.198613+0000", 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.708717211, 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070345060000000003, 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198613+0000", 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198613+0000", 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198612+0000", 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198619+0000", 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.708e-06 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198625+0000", 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2709999999999998e-06 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.198630+0000", 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5020000000000003e-06 2024-06-14T23:17:25.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.200893+0000", 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00226275 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.200911+0000", 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8723e-05 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.205587+0000", 2024-06-14T23:17:25.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046757760000000004 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.205647+0000", 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0115000000000003e-05 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191390 13.3 e387/386 13:c7fec1af:::benchmark_data_smithi086_299538_object70552:head v 387'12024, mlcod=387'12022)", 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.208751+0000", 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.6985785259999999, 2024-06-14T23:17:25.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0067057610000000002, 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.208751+0000", 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.208751+0000", 2024-06-14T23:17:25.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.208751+0000", 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.208756+0000", 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5570000000000003e-06 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.208773+0000", 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6313000000000001e-05 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.208778+0000", 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8610000000000003e-06 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.210562+0000", 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001783139 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.210582+0000", 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9970999999999999e-05 2024-06-14T23:17:25.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.215203+0000", 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046214619999999998 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.215457+0000", 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00025411500000000002 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191404 13.3 e387/386 13:ccc8a738:::benchmark_data_smithi086_299538_object70566:head v 387'12026, mlcod=387'12022)", 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.218299+0000", 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.6890309480000001, 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047705860000000003, 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.218299+0000", 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.218299+0000", 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.218298+0000", 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.218304+0000", 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2110000000000003e-06 2024-06-14T23:17:25.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.218317+0000", 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2515e-05 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.218323+0000", 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.93e-06 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.220479+0000", 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021563429999999998 2024-06-14T23:17:25.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.220499+0000", 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0092999999999999e-05 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.223000+0000", 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025008539999999998 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.223069+0000", 2024-06-14T23:17:25.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9309999999999999e-05 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191686 13.9 e387/386 13:966adbda:::benchmark_data_smithi086_299538_object70848:head v 387'12088, mlcod=387'12085)", 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.513010+0000", 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.394319581, 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017458325, 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.513010+0000", 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.513010+0000", 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.513009+0000", 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524375+0000", 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011366090000000001 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524387+0000", 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1914e-05 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524393+0000", 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2550000000000003e-06 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524438+0000", 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4567999999999997e-05 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524454+0000", 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6348999999999999e-05 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.530445+0000", 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059907449999999996 2024-06-14T23:17:25.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.530469+0000", 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3669e-05 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191694 13.9 e387/386 13:9070ce8b:::benchmark_data_smithi086_299538_object70856:head v 387'12089, mlcod=387'12086)", 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.524416+0000", 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.382913936, 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006075698, 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.954 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524416+0000", 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524416+0000", 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524415+0000", 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524419+0000", 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7969999999999998e-06 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524426+0000", 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6449999999999999e-06 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524430+0000", 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2300000000000002e-06 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.526463+0000", 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002033071 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.526476+0000", 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2574e-05 2024-06-14T23:17:25.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.530472+0000", 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039963400000000001 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.530492+0000", 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9538000000000002e-05 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191696 13.9 e387/386 13:95e46e21:::benchmark_data_smithi086_299538_object70858:head v 387'12090, mlcod=387'12086)", 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.524458+0000", 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.3828720370000001, 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060572660000000004, 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524458+0000", 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524458+0000", 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524458+0000", 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967296 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524461+0000", 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0240000000000002e-06 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524467+0000", 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5010000000000001e-06 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.524471+0000", 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9369999999999997e-06 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.528101+0000", 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036301390000000001 2024-06-14T23:17:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.528122+0000", 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1033e-05 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.530493+0000", 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023708420000000002 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.530515+0000", 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1956000000000001e-05 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:191831 13.5 e387/386 13:a8e784c5:::benchmark_data_smithi086_299538_object70993:head v 387'12102, mlcod=387'12099)", 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.649681+0000", 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.257648689, 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048892290000000001, 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.649681+0000", 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.649681+0000", 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.649680+0000", 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.649687+0000", 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.131e-06 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.649703+0000", 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5109999999999999e-05 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.649709+0000", 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5139999999999998e-06 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.649751+0000", 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2308000000000002e-05 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.649771+0000", 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9196999999999999e-05 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.654510+0000", 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004738913 2024-06-14T23:17:25.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.654570+0000", 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0897000000000003e-05 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:192087 13.3 e387/386 13:c771b7db:::benchmark_data_smithi086_299538_object71249:head v 387'12063, mlcod=387'12059)", 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.882943+0000", 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.0243873480000001, 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0096864050000000004, 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.967 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.882943+0000", 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.882943+0000", 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.882942+0000", 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.882948+0000", 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5529999999999998e-06 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.882964+0000", 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5393e-05 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.882971+0000", 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.294e-06 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.883032+0000", 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310999999999996e-05 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.883055+0000", 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2988000000000001e-05 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.892597+0000", 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0095417839999999993 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.892629+0000", 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2021999999999999e-05 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.971 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:192090 13.3 e387/386 13:cfa6a98d:::benchmark_data_smithi086_299538_object71252:head v 387'12064, mlcod=387'12059)", 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:24.887936+0000", 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 1.0193942540000001, 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0079129170000000002, 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.887936+0000", 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.887936+0000", 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.887935+0000", 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.887941+0000", 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3380000000000004e-06 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.887949+0000", 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0099999999999995e-06 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.887955+0000", 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5649999999999998e-06 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.888099+0000", 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000143585 2024-06-14T23:17:25.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.888119+0000", 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0160999999999999e-05 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.895817+0000", 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0076980679999999998 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:24.895849+0000", 2024-06-14T23:17:25.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1606999999999997e-05 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:192258 13.5 e388/386 13:a057e82e:::benchmark_data_smithi086_299538_object71420:head v 388'12135, mlcod=388'12131)", 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:25.043568+0000", 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.86376185500000002, 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049421350000000003, 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.043568+0000", 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.043568+0000", 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.043567+0000", 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.043573+0000", 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1129999999999997e-06 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.043584+0000", 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0766e-05 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.043589+0000", 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5940000000000004e-06 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.043626+0000", 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7020000000000001e-05 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.043647+0000", 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1665999999999999e-05 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.048435+0000", 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047877370000000002 2024-06-14T23:17:25.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.048510+0000", 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4917000000000001e-05 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:192617 13.9 e388/386 13:9241b9b3:::benchmark_data_smithi086_299538_object71779:head v 388'12142, mlcod=388'12139)", 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:25.361458+0000", 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.54587216299999997, 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005551945, 2024-06-14T23:17:25.980 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.361458+0000", 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.361458+0000", 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.361457+0000", 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.361462+0000", 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0579999999999997e-06 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.361470+0000", 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2029999999999992e-06 2024-06-14T23:17:25.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.361477+0000", 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.669e-06 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.361526+0000", 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8857000000000003e-05 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.361548+0000", 2024-06-14T23:17:25.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2257000000000001e-05 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.366979+0000", 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054307849999999996 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.367010+0000", 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0725999999999997e-05 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.985 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:192864 13.3 e388/386 13:cc6a01ac:::benchmark_data_smithi086_299538_object72026:head v 388'12106, mlcod=388'12103)", 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:25.573800+0000", 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.33352994800000002, 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046923920000000001, 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.573800+0000", 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.573800+0000", 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.573799+0000", 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.573806+0000", 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5459999999999997e-06 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.573817+0000", 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1822e-05 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.573823+0000", 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3190000000000002e-06 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.573873+0000", 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0386000000000002e-05 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.573892+0000", 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8652999999999999e-05 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.578452+0000", 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045600900000000002 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.578492+0000", 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0524e-05 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:193083 13.9 e388/386 13:97f30f8d:::benchmark_data_smithi086_299538_object72245:head v 388'12174, mlcod=388'12171)", 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:25.774152+0000", 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.13317833100000001, 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047038399999999999, 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.990 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.774152+0000", 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.774152+0000", 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.774151+0000", 2024-06-14T23:17:25.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.774163+0000", 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2266000000000001e-05 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.774190+0000", 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7192999999999999e-05 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.774198+0000", 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6079999999999998e-06 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.774241+0000", 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3049000000000001e-05 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.774258+0000", 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7294000000000001e-05 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.778820+0000", 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045616090000000003 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.778855+0000", 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5728000000000001e-05 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:193184 13.5 e388/386 13:a66799f6:::benchmark_data_smithi086_299538_object72346:head v 388'12188, mlcod=388'12186)", 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:25.867178+0000", 2024-06-14T23:17:25.994 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.040151569999999998, 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055117930000000001, 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.867178+0000", 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.867178+0000", 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:25.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.867177+0000", 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.867185+0000", 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4070000000000004e-06 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.867197+0000", 2024-06-14T23:17:25.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2251e-05 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.867204+0000", 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.584e-06 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.867700+0000", 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00049619799999999999 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:25.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.867725+0000", 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4859999999999999e-05 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.872650+0000", 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049250939999999997 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.872690+0000", 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0309999999999999e-05 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.998 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.46844.0:193212 13.9 e388/386 13:9d9dd977:::benchmark_data_smithi086_299538_object72374:head v 388'12184, mlcod=388'12182)", 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:25.892483+0000", 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.014847074, 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.007743331, 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.892483+0000", 2024-06-14T23:17:25.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.892483+0000", 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.892482+0000", 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.892489+0000", 2024-06-14T23:17:26.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4409999999999998e-06 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.892497+0000", 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4470000000000006e-06 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.892504+0000", 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4889999999999998e-06 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:26.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.892572+0000", 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7341000000000005e-05 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.892592+0000", 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0599e-05 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.900182+0000", 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0075896330000000001 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:26.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:26.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:25.900226+0000", 2024-06-14T23:17:26.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4056e-05 2024-06-14T23:17:26.003 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:26.003 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:26.003 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:26.003 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:26.003 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:26.003 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:17:26.042 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/12683667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:26.043 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:26.043 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:17:26.043 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:17:26.043 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[102754]: osdmap e389: 8 total, 8 up, 7 in 2024-06-14T23:17:26.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/12683667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:26.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:26.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:17:26.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:17:26.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:25 smithi086 ceph-mon[97913]: osdmap e389: 8 total, 8 up, 7 in 2024-06-14T23:17:26.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:26.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:26.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:26.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:26.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/12683667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:25 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:25 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:17:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:25 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:17:26.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:25 smithi137 ceph-mon[90634]: osdmap e389: 8 total, 8 up, 7 in 2024-06-14T23:17:26.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:26.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:26.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:26.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:26.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:26.446 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:26.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:26.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:26.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:26.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:26.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:26.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:26.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:26.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:26.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:26.984 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:26.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:27.019 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:26 smithi086 ceph-mon[102754]: pgmap v1446: 119 pgs: 1 active+remapped+backfilling, 2 active+undersized+remapped, 7 remapped+peering, 15 peering, 94 active+clean; 3.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 10 KiB/s rd, 27 KiB/s wr, 1.05k op/s; 2412/151347 objects misplaced (1.594%); 9.7 MiB/s, 155 objects/s recovering 2024-06-14T23:17:27.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:26 smithi086 ceph-mon[97913]: pgmap v1446: 119 pgs: 1 active+remapped+backfilling, 2 active+undersized+remapped, 7 remapped+peering, 15 peering, 94 active+clean; 3.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 10 KiB/s rd, 27 KiB/s wr, 1.05k op/s; 2412/151347 objects misplaced (1.594%); 9.7 MiB/s, 155 objects/s recovering 2024-06-14T23:17:27.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:27.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:27.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:27.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:26 smithi137 ceph-mon[90634]: pgmap v1446: 119 pgs: 1 active+remapped+backfilling, 2 active+undersized+remapped, 7 remapped+peering, 15 peering, 94 active+clean; 3.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 10 KiB/s rd, 27 KiB/s wr, 1.05k op/s; 2412/151347 objects misplaced (1.594%); 9.7 MiB/s, 155 objects/s recovering 2024-06-14T23:17:27.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:27.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:27.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:27.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:27.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:27.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:27.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:27.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:27.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:27.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-14T23:17:27.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:27.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:27.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:27.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:27.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:27.811 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:27.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:27.860 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 2, 0] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:17:27.860 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:17:27.860 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T23:17:27.860 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.724724 2024-06-14T23:17:27.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 5 0.7247241023810629 2024-06-14T23:17:27.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:27.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:27.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:28.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:28.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:28.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:28.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:28.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:28.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:28.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:28.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:28.293 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:28.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:28.377 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:28.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:28.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:28.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:28.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:28.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:28.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:28.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:28.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:28.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:28.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:28.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:28.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:28.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:29.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:29.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:29.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:29.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:29.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:29.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:29.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:28 smithi137 ceph-mon[90634]: pgmap v1448: 119 pgs: 1 active+remapped+backfilling, 2 active+undersized+remapped, 7 remapped+peering, 2 peering, 107 active+clean; 3.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 8.5 KiB/s rd, 22 KiB/s wr, 716 op/s; 2412/148170 objects misplaced (1.628%); 7.9 MiB/s, 127 objects/s recovering 2024-06-14T23:17:29.272 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:28 smithi086 ceph-mon[97913]: pgmap v1448: 119 pgs: 1 active+remapped+backfilling, 2 active+undersized+remapped, 7 remapped+peering, 2 peering, 107 active+clean; 3.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 8.5 KiB/s rd, 22 KiB/s wr, 716 op/s; 2412/148170 objects misplaced (1.628%); 7.9 MiB/s, 127 objects/s recovering 2024-06-14T23:17:29.272 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:28 smithi086 ceph-mon[102754]: pgmap v1448: 119 pgs: 1 active+remapped+backfilling, 2 active+undersized+remapped, 7 remapped+peering, 2 peering, 107 active+clean; 3.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 8.5 KiB/s rd, 22 KiB/s wr, 716 op/s; 2412/148170 objects misplaced (1.628%); 7.9 MiB/s, 127 objects/s recovering 2024-06-14T23:17:29.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:29.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:29.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:29.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:29.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:29.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:29.575 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:29.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:29.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:29.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:29.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:29.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:29.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:29.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:29.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:29.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:30.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:30.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:30.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:30.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:30.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:30.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:30.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:30.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:30.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:17:30.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:30.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:30.551 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:30.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:30.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:30.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:30.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:30.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:30.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:30.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:30.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:30.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:30.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:30.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:30.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:30.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:31.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:31.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:31.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:31.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:30 smithi086 ceph-mon[97913]: pgmap v1449: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 peering, 114 active+clean; 2.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 801 op/s; 8184/138843 objects misplaced (5.894%); 6.5 MiB/s, 104 objects/s recovering 2024-06-14T23:17:31.188 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:30 smithi086 ceph-mon[102754]: pgmap v1449: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 peering, 114 active+clean; 2.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 801 op/s; 8184/138843 objects misplaced (5.894%); 6.5 MiB/s, 104 objects/s recovering 2024-06-14T23:17:31.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:31.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:30 smithi137 ceph-mon[90634]: pgmap v1449: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 peering, 114 active+clean; 2.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 801 op/s; 8184/138843 objects misplaced (5.894%); 6.5 MiB/s, 104 objects/s recovering 2024-06-14T23:17:31.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:31.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:31.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:31.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:31.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:31.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:31.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:31.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:31.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:31.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:31.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:31.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:31.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:31.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:31.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:31.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:31.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:31.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:32.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:32.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:32.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:32.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:32.230 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:32.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:32.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:32.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:32.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:32.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:32.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:32.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:32.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:32.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:32.625 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:32.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:32.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:32.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:32.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:32.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:32.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:32.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:33.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:33.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:33.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:33.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:33.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:33.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:33.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:33.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:33.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:32 smithi137 ceph-mon[90634]: pgmap v1450: 119 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 116 active+clean; 2.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 7.9 KiB/s rd, 21 KiB/s wr, 654 op/s; 6317/137784 objects misplaced (4.585%); 18 MiB/s, 290 objects/s recovering 2024-06-14T23:17:33.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:33.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:33.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:32 smithi086 ceph-mon[97913]: pgmap v1450: 119 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 116 active+clean; 2.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 7.9 KiB/s rd, 21 KiB/s wr, 654 op/s; 6317/137784 objects misplaced (4.585%); 18 MiB/s, 290 objects/s recovering 2024-06-14T23:17:33.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:32 smithi086 ceph-mon[102754]: pgmap v1450: 119 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 116 active+clean; 2.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 7.9 KiB/s rd, 21 KiB/s wr, 654 op/s; 6317/137784 objects misplaced (4.585%); 18 MiB/s, 290 objects/s recovering 2024-06-14T23:17:33.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:33.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:33.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:33.519 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:17:33.519 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:17:33.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:33.538 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:17:33.538 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:17:33.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:33.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:33.557 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:17:33.557 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:17:33.576 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:17:33.576 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:17:33.594 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:17:33.594 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:17:33.612 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:17:33.612 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:17:33.629 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:17:33.629 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:17:33.646 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:17:33.646 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:17:33.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:33.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:33.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:33.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:33.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:33.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:33.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:33.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:33.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:34.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:34.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:34.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:34.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:34.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:34.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:34.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:33 smithi137 ceph-mon[90634]: osdmap e390: 8 total, 8 up, 7 in 2024-06-14T23:17:34.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2070770357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:34.261 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:33 smithi086 ceph-mon[97913]: osdmap e390: 8 total, 8 up, 7 in 2024-06-14T23:17:34.261 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2070770357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:34.261 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:33 smithi086 ceph-mon[102754]: osdmap e390: 8 total, 8 up, 7 in 2024-06-14T23:17:34.261 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2070770357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:34.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:34.320 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:34.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:34.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:34.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:34.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:34.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:34.603 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:34.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:34.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:34.733 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:34.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:34.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:34.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:34.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:34.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:34.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:34.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:34.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:35.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:34 smithi086 ceph-mon[97913]: pgmap v1452: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 2.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 2.6 KiB/s rd, 6.9 KiB/s wr, 1.08k op/s; 5297/125541 objects misplaced (4.219%); 16 MiB/s, 260 objects/s recovering 2024-06-14T23:17:35.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:34 smithi086 ceph-mon[97913]: osdmap e391: 8 total, 8 up, 7 in 2024-06-14T23:17:35.088 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:17:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:17:34] "GET /metrics HTTP/1.1" 200 36732 "" "Prometheus/2.43.0" 2024-06-14T23:17:35.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:34 smithi086 ceph-mon[102754]: pgmap v1452: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 2.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 2.6 KiB/s rd, 6.9 KiB/s wr, 1.08k op/s; 5297/125541 objects misplaced (4.219%); 16 MiB/s, 260 objects/s recovering 2024-06-14T23:17:35.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:34 smithi086 ceph-mon[102754]: osdmap e391: 8 total, 8 up, 7 in 2024-06-14T23:17:35.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:35.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:35.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:35.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:35.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:34 smithi137 ceph-mon[90634]: pgmap v1452: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 2.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 2.6 KiB/s rd, 6.9 KiB/s wr, 1.08k op/s; 5297/125541 objects misplaced (4.219%); 16 MiB/s, 260 objects/s recovering 2024-06-14T23:17:35.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:34 smithi137 ceph-mon[90634]: osdmap e391: 8 total, 8 up, 7 in 2024-06-14T23:17:35.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:35.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:35.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:35.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:35.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:35.506 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:17:35.506 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:17:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:35.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:35.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:35.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:35.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:35.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:35.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:35.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:35.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:35.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:35.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:35.959 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.5 primary-affinity to 0.724724 (8474952) 2024-06-14T23:17:36.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:36.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:36.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:36.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/829419558' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7247241023810629}]: dispatch 2024-06-14T23:17:36.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/829419558' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7247241023810629}]: dispatch 2024-06-14T23:17:36.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:36.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:36.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:36.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/829419558' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7247241023810629}]: dispatch 2024-06-14T23:17:36.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-14T23:17:36.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:36.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:36.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:36.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:36.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:36.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:36.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:36.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:36.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:36.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:36.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:36.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:36.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:36.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:36.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:36.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:36.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:36.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:37.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:37.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:37.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:37.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:37.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:37.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:36 smithi137 ceph-mon[90634]: pgmap v1454: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 2.6 KiB/s rd, 6.9 KiB/s wr, 1.04k op/s; 4752/123210 objects misplaced (3.857%); 20 MiB/s, 326 objects/s recovering 2024-06-14T23:17:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/829419558' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7247241023810629}]': finished 2024-06-14T23:17:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:36 smithi137 ceph-mon[90634]: osdmap e392: 8 total, 8 up, 7 in 2024-06-14T23:17:37.327 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:37.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:36 smithi086 ceph-mon[97913]: pgmap v1454: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 2.6 KiB/s rd, 6.9 KiB/s wr, 1.04k op/s; 4752/123210 objects misplaced (3.857%); 20 MiB/s, 326 objects/s recovering 2024-06-14T23:17:37.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/829419558' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7247241023810629}]': finished 2024-06-14T23:17:37.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:36 smithi086 ceph-mon[97913]: osdmap e392: 8 total, 8 up, 7 in 2024-06-14T23:17:37.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:36 smithi086 ceph-mon[102754]: pgmap v1454: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 2.6 KiB/s rd, 6.9 KiB/s wr, 1.04k op/s; 4752/123210 objects misplaced (3.857%); 20 MiB/s, 326 objects/s recovering 2024-06-14T23:17:37.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/829419558' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.7247241023810629}]': finished 2024-06-14T23:17:37.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:36 smithi086 ceph-mon[102754]: osdmap e392: 8 total, 8 up, 7 in 2024-06-14T23:17:37.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:37.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:37.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:37.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:37.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:37.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:37.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:37.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:37.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:37.704 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:37.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:37.760 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:37.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:37.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:37.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:37.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:37.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:38.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:38.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:38.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:38.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:38.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:38.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:37 smithi086 ceph-mon[102754]: osdmap e393: 8 total, 8 up, 7 in 2024-06-14T23:17:38.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:37 smithi086 ceph-mon[97913]: osdmap e393: 8 total, 8 up, 7 in 2024-06-14T23:17:38.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:37 smithi137 ceph-mon[90634]: osdmap e393: 8 total, 8 up, 7 in 2024-06-14T23:17:38.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:38.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:38.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:38.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:38.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:38.457 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:38.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:38.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:38.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:38.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:38.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:38.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:38.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:38.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:38.857 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:38.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:38.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:38.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:38.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:39.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:17:39.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:39.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:39.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:39.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:38 smithi137 ceph-mon[90634]: pgmap v1457: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 5.5 KiB/s rd, 15 KiB/s wr, 897 op/s; 4752/119688 objects misplaced (3.970%); 12 MiB/s, 190 objects/s recovering 2024-06-14T23:17:39.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:39.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:39.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:39.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:38 smithi086 ceph-mon[97913]: pgmap v1457: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 5.5 KiB/s rd, 15 KiB/s wr, 897 op/s; 4752/119688 objects misplaced (3.970%); 12 MiB/s, 190 objects/s recovering 2024-06-14T23:17:39.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:38 smithi086 ceph-mon[102754]: pgmap v1457: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 5.5 KiB/s rd, 15 KiB/s wr, 897 op/s; 4752/119688 objects misplaced (3.970%); 12 MiB/s, 190 objects/s recovering 2024-06-14T23:17:39.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:39.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:39.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:39.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:39.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:39.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:39.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:39.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:39.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:39.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:39.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:39.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:39.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:39.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:39.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:39.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:40.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:40.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:40.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:40.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:40.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:40.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:40.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:40.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:40.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:40.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:40.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:40.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:40.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:40.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:40.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:40.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:40.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:40.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:40.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:40.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:40.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:40.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:40.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:41.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:41.033 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:41.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:41.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:41.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:41.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:41.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:40 smithi137 ceph-mon[90634]: pgmap v1458: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4.5 KiB/s rd, 12 KiB/s wr, 652 op/s; 2967/113859 objects misplaced (2.606%); 20 MiB/s, 323 objects/s recovering 2024-06-14T23:17:41.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:41.308 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:41.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:41.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:40 smithi086 ceph-mon[97913]: pgmap v1458: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4.5 KiB/s rd, 12 KiB/s wr, 652 op/s; 2967/113859 objects misplaced (2.606%); 20 MiB/s, 323 objects/s recovering 2024-06-14T23:17:41.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:40 smithi086 ceph-mon[102754]: pgmap v1458: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4.5 KiB/s rd, 12 KiB/s wr, 652 op/s; 2967/113859 objects misplaced (2.606%); 20 MiB/s, 323 objects/s recovering 2024-06-14T23:17:41.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:41.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:41.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:41.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:41.573 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:41.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:41.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:41.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:41.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:41.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:41.845 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:41.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:41.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:41.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:41.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:42.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:42.102 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:42.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:42.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:42.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:42.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:42.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:42.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:42.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:42.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:42.510 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:42.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:42.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:42.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:42.626 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 2, 0] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:17:42.627 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:17:42.627 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-06-14T23:17:42.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-06-14T23:17:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:42.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:42.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:42.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:42.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:42.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:42.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:42.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:42.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:42.988 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:17:43.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:43.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:43.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:43.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:43.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:43.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:42 smithi137 ceph-mon[90634]: pgmap v1459: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 3.4 KiB/s rd, 9.3 KiB/s wr, 575 op/s; 2967/112050 objects misplaced (2.648%); 15 MiB/s, 246 objects/s recovering 2024-06-14T23:17:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:42 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/79342935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:17:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:42 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:17:43.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:43.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:43.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:42 smithi086 ceph-mon[97913]: pgmap v1459: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 3.4 KiB/s rd, 9.3 KiB/s wr, 575 op/s; 2967/112050 objects misplaced (2.648%); 15 MiB/s, 246 objects/s recovering 2024-06-14T23:17:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:42 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/79342935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:17:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:42 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:17:43.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:42 smithi086 ceph-mon[102754]: pgmap v1459: 119 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 117 active+clean; 2.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 3.4 KiB/s rd, 9.3 KiB/s wr, 575 op/s; 2967/112050 objects misplaced (2.648%); 15 MiB/s, 246 objects/s recovering 2024-06-14T23:17:43.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:43.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:42 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/79342935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:17:43.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:42 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:17:43.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:43.432 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:43.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:43.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:43.555 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:43.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:43.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:43.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:43.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:43.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:43.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:43.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:43.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:43.926 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:43.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:44.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:44.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:44.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:44.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:44.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:44.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:43 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:17:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:43 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:17:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:43 smithi137 ceph-mon[90634]: osdmap e394: 8 total, 8 up, 7 in 2024-06-14T23:17:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:43 smithi137 ceph-mon[90634]: 1.a scrub starts 2024-06-14T23:17:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:43 smithi137 ceph-mon[90634]: 1.a scrub ok 2024-06-14T23:17:44.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:44.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:44.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:17:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:17:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[97913]: osdmap e394: 8 total, 8 up, 7 in 2024-06-14T23:17:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[97913]: 1.a scrub starts 2024-06-14T23:17:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[97913]: 1.a scrub ok 2024-06-14T23:17:44.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:17:44.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:17:44.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[102754]: osdmap e394: 8 total, 8 up, 7 in 2024-06-14T23:17:44.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[102754]: 1.a scrub starts 2024-06-14T23:17:44.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:43 smithi086 ceph-mon[102754]: 1.a scrub ok 2024-06-14T23:17:44.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:44.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:44.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:44.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:44.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:44.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:44.733 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:44.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:44.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:44.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:44.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:44.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:45.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:45.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:45.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:45.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:45.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:45.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:45.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:45.140 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:17:44] "GET /metrics HTTP/1.1" 200 36731 "" "Prometheus/2.43.0" 2024-06-14T23:17:45.140 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[97913]: pgmap v1461: 119 pgs: 1 active+remapped+backfilling, 1 peering, 117 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 3.4 KiB/s rd, 9.3 KiB/s wr, 949 op/s; 1710/100920 objects misplaced (1.694%); 19 MiB/s, 298 objects/s recovering 2024-06-14T23:17:45.140 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[97913]: osdmap e395: 8 total, 8 up, 7 in 2024-06-14T23:17:45.141 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[97913]: 1.2 scrub starts 2024-06-14T23:17:45.141 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:17:45.141 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[97913]: 1.8 scrub ok 2024-06-14T23:17:45.141 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[102754]: pgmap v1461: 119 pgs: 1 active+remapped+backfilling, 1 peering, 117 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 3.4 KiB/s rd, 9.3 KiB/s wr, 949 op/s; 1710/100920 objects misplaced (1.694%); 19 MiB/s, 298 objects/s recovering 2024-06-14T23:17:45.141 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[102754]: osdmap e395: 8 total, 8 up, 7 in 2024-06-14T23:17:45.141 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[102754]: 1.2 scrub starts 2024-06-14T23:17:45.141 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:17:45.141 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:44 smithi086 ceph-mon[102754]: 1.8 scrub ok 2024-06-14T23:17:45.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:44 smithi137 ceph-mon[90634]: pgmap v1461: 119 pgs: 1 active+remapped+backfilling, 1 peering, 117 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 3.4 KiB/s rd, 9.3 KiB/s wr, 949 op/s; 1710/100920 objects misplaced (1.694%); 19 MiB/s, 298 objects/s recovering 2024-06-14T23:17:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:44 smithi137 ceph-mon[90634]: osdmap e395: 8 total, 8 up, 7 in 2024-06-14T23:17:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:44 smithi137 ceph-mon[90634]: 1.2 scrub starts 2024-06-14T23:17:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:44 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:17:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:44 smithi137 ceph-mon[90634]: 1.8 scrub ok 2024-06-14T23:17:45.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:45.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:45.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:45.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:45.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:45.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:45.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:45.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:45.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:45.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:45.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:45.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:45.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:45.822 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:45.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:45.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:45.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:46.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:46.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:46.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:46.104 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:17:46.104 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:17:46.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:45 smithi086 ceph-mon[97913]: 1.1 scrub starts 2024-06-14T23:17:46.116 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:45 smithi086 ceph-mon[102754]: 1.1 scrub starts 2024-06-14T23:17:46.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:46.213 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:46.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:46.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:46.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:46.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:46.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:46.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:17:46.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:46.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:46.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:46 smithi137 ceph-mon[90634]: 1.1 scrub starts 2024-06-14T23:17:46.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:46.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:46.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:46.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:46.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:46.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:46.876 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:17:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:17:46.892 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:46.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:46.903 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:17:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:17:46.925 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:17:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:17:46.946 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:17:46.947 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:17:46.973 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:17:46.973 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:17:46.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:47.009 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:17:47.009 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:17:47.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:47.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:47.041 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:17:47.041 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:17:47.066 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:17:47.066 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:17:47.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:47.157 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:47.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:47.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:47.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:47.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:47 smithi086 ceph-mon[97913]: pgmap v1463: 119 pgs: 1 active+remapped+backfilling, 1 peering, 117 active+clean; 2.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 915 op/s; 1710/97722 objects misplaced (1.750%); 18 MiB/s, 292 objects/s recovering 2024-06-14T23:17:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:47 smithi086 ceph-mon[97913]: 1.0 scrub starts 2024-06-14T23:17:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:47 smithi086 ceph-mon[97913]: 1.1 scrub starts 2024-06-14T23:17:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/123602705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:47.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:47 smithi086 ceph-mon[102754]: pgmap v1463: 119 pgs: 1 active+remapped+backfilling, 1 peering, 117 active+clean; 2.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 915 op/s; 1710/97722 objects misplaced (1.750%); 18 MiB/s, 292 objects/s recovering 2024-06-14T23:17:47.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:47 smithi086 ceph-mon[102754]: 1.0 scrub starts 2024-06-14T23:17:47.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:47 smithi086 ceph-mon[102754]: 1.1 scrub starts 2024-06-14T23:17:47.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/123602705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:47.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:47.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:47.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:47 smithi137 ceph-mon[90634]: pgmap v1463: 119 pgs: 1 active+remapped+backfilling, 1 peering, 117 active+clean; 2.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 915 op/s; 1710/97722 objects misplaced (1.750%); 18 MiB/s, 292 objects/s recovering 2024-06-14T23:17:47.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:47 smithi137 ceph-mon[90634]: 1.0 scrub starts 2024-06-14T23:17:47.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:47 smithi137 ceph-mon[90634]: 1.1 scrub starts 2024-06-14T23:17:47.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/123602705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:17:47.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-14T23:17:47.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:47.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:47.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:47.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:47.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:47.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:47.738 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:47.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:47.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:47.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:47.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:47.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:47.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:47.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:48.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:48.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:48.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:48.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:48.254 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:48.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:48.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:48 smithi086 ceph-mon[97913]: 1.2 scrub starts 2024-06-14T23:17:48.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:48 smithi086 ceph-mon[97913]: 1.1 scrub starts 2024-06-14T23:17:48.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:48 smithi086 ceph-mon[102754]: 1.2 scrub starts 2024-06-14T23:17:48.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:48 smithi086 ceph-mon[102754]: 1.1 scrub starts 2024-06-14T23:17:48.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:48.398 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:48.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:48.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:48 smithi137 ceph-mon[90634]: 1.2 scrub starts 2024-06-14T23:17:48.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:48 smithi137 ceph-mon[90634]: 1.1 scrub starts 2024-06-14T23:17:48.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:48.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:48.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:48.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:48.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:48.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:48.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:48.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:48.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:48.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:48.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:48.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:48.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:49.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:49.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:49.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:49.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:49.178 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:49 smithi086 ceph-mon[97913]: pgmap v1464: 119 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 peering, 116 active+clean; 1.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 865 op/s; 1710/93084 objects misplaced (1.837%); 7.3 MiB/s, 116 objects/s recovering 2024-06-14T23:17:49.178 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:49 smithi086 ceph-mon[102754]: pgmap v1464: 119 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 peering, 116 active+clean; 1.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 865 op/s; 1710/93084 objects misplaced (1.837%); 7.3 MiB/s, 116 objects/s recovering 2024-06-14T23:17:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:49.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:49.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:49.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:49.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:49.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:49.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:49.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:49.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:49 smithi137 ceph-mon[90634]: pgmap v1464: 119 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 peering, 116 active+clean; 1.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 865 op/s; 1710/93084 objects misplaced (1.837%); 7.3 MiB/s, 116 objects/s recovering 2024-06-14T23:17:49.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:49.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:49.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:49.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:49.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:49.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:49.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:49.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:49.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:49.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:49.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:49.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:50.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:50.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:50.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:50.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:50.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:50.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:50.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:50 smithi086 ceph-mon[97913]: 1.1 scrub starts 2024-06-14T23:17:50.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:50 smithi086 ceph-mon[102754]: 1.1 scrub starts 2024-06-14T23:17:50.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:50.390 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:50.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:50.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:50.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:50 smithi137 ceph-mon[90634]: 1.1 scrub starts 2024-06-14T23:17:50.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:50.520 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:50.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:50.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:50.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:50.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:50.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:50.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:50.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:50.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:50.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:50.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:51.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:51.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:51.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:51.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:51.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:17:51.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:51.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:51.283 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:51 smithi086 ceph-mon[97913]: pgmap v1465: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 117 active+clean; 1.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 552/86313 objects misplaced (0.640%); 15 MiB/s, 245 objects/s recovering 2024-06-14T23:17:51.283 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:51 smithi086 ceph-mon[97913]: 1.1 scrub starts 2024-06-14T23:17:51.284 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:51 smithi086 ceph-mon[102754]: pgmap v1465: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 117 active+clean; 1.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 552/86313 objects misplaced (0.640%); 15 MiB/s, 245 objects/s recovering 2024-06-14T23:17:51.284 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:51 smithi086 ceph-mon[102754]: 1.1 scrub starts 2024-06-14T23:17:51.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:51.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:51.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:51.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:51.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:51.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:51 smithi137 ceph-mon[90634]: pgmap v1465: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 117 active+clean; 1.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 552/86313 objects misplaced (0.640%); 15 MiB/s, 245 objects/s recovering 2024-06-14T23:17:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:51 smithi137 ceph-mon[90634]: 1.1 scrub starts 2024-06-14T23:17:51.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:51.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:51.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:51.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:51.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:51.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:51.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:51.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:51.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:51.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:52.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:52.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:52.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:52.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:52.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:52.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:52.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:52.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:52.264 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:52.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:52.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:52 smithi086 ceph-mon[97913]: 1.0 scrub starts 2024-06-14T23:17:52.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:52 smithi086 ceph-mon[97913]: 1.1 scrub starts 2024-06-14T23:17:52.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:52 smithi086 ceph-mon[102754]: 1.0 scrub starts 2024-06-14T23:17:52.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:52 smithi086 ceph-mon[102754]: 1.1 scrub starts 2024-06-14T23:17:52.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:52.396 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:52.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:52.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:52 smithi137 ceph-mon[90634]: 1.0 scrub starts 2024-06-14T23:17:52.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:52 smithi137 ceph-mon[90634]: 1.1 scrub starts 2024-06-14T23:17:52.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:52.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:52.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:52.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:52.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:52.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:52.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:52.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:52.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:52.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:52.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:52.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:17:53.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:53.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:53.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:53.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:53.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:53.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:53.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[97913]: 1.5 scrub starts 2024-06-14T23:17:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[97913]: 1.5 scrub ok 2024-06-14T23:17:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[97913]: pgmap v1466: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 117 active+clean; 1.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 552/84387 objects misplaced (0.654%); 14 MiB/s, 222 objects/s recovering 2024-06-14T23:17:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[97913]: 1.2 scrub starts 2024-06-14T23:17:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[97913]: 1.2 scrub ok 2024-06-14T23:17:53.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/26582740' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:53.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[102754]: 1.5 scrub starts 2024-06-14T23:17:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[102754]: 1.5 scrub ok 2024-06-14T23:17:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[102754]: pgmap v1466: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 117 active+clean; 1.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 552/84387 objects misplaced (0.654%); 14 MiB/s, 222 objects/s recovering 2024-06-14T23:17:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[102754]: 1.2 scrub starts 2024-06-14T23:17:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[102754]: 1.2 scrub ok 2024-06-14T23:17:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/26582740' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:53 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:53.343 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:53.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:53.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:53.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:53.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:53.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:53 smithi137 ceph-mon[90634]: 1.5 scrub starts 2024-06-14T23:17:53.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:53 smithi137 ceph-mon[90634]: 1.5 scrub ok 2024-06-14T23:17:53.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:53 smithi137 ceph-mon[90634]: pgmap v1466: 119 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 117 active+clean; 1.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 552/84387 objects misplaced (0.654%); 14 MiB/s, 222 objects/s recovering 2024-06-14T23:17:53.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:53 smithi137 ceph-mon[90634]: 1.2 scrub starts 2024-06-14T23:17:53.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:53 smithi137 ceph-mon[90634]: 1.2 scrub ok 2024-06-14T23:17:53.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/26582740' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:53.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:53 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:17:53.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:53.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:53.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:53.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:53.714 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:53.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:53.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:53.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:53.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:53.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:53.961 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:53.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:54.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:54.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:54.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:54.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:54.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:54.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:54.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:17:54.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:17:54.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:17:54.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[97913]: osdmap e396: 8 total, 8 up, 7 in 2024-06-14T23:17:54.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[97913]: 1.0 scrub starts 2024-06-14T23:17:54.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[97913]: 1.0 scrub ok 2024-06-14T23:17:54.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[97913]: 1.1 scrub starts 2024-06-14T23:17:54.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[97913]: 1.1 scrub ok 2024-06-14T23:17:54.302 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:17:54.302 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:17:54.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:17:54.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[102754]: osdmap e396: 8 total, 8 up, 7 in 2024-06-14T23:17:54.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[102754]: 1.0 scrub starts 2024-06-14T23:17:54.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[102754]: 1.0 scrub ok 2024-06-14T23:17:54.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[102754]: 1.1 scrub starts 2024-06-14T23:17:54.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-mon[102754]: 1.1 scrub ok 2024-06-14T23:17:54.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:54.346 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:54.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:54.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:54.491 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:54.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:54.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:54 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:17:54.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:54 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:17:54.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:54 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:17:54.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:54 smithi137 ceph-mon[90634]: osdmap e396: 8 total, 8 up, 7 in 2024-06-14T23:17:54.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:54 smithi137 ceph-mon[90634]: 1.0 scrub starts 2024-06-14T23:17:54.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:54 smithi137 ceph-mon[90634]: 1.0 scrub ok 2024-06-14T23:17:54.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:54 smithi137 ceph-mon[90634]: 1.1 scrub starts 2024-06-14T23:17:54.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:54 smithi137 ceph-mon[90634]: 1.1 scrub ok 2024-06-14T23:17:54.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:54.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:54.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:54.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:54.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:54.742 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:17:54.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:54.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:54.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:54.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:54.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:54.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:55.017 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:17:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:17:54] "GET /metrics HTTP/1.1" 200 36731 "" "Prometheus/2.43.0" 2024-06-14T23:17:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:55.021 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:55.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:55.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:55.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:55.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:55.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:55.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:55.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:55.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:55.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[97913]: 1.16 scrub starts 2024-06-14T23:17:55.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[97913]: 1.16 scrub ok 2024-06-14T23:17:55.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[97913]: 1.3 deep-scrub starts 2024-06-14T23:17:55.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[97913]: 1.3 deep-scrub ok 2024-06-14T23:17:55.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[97913]: pgmap v1468: 119 pgs: 1 peering, 1 active+clean+scrubbing, 117 active+clean; 1.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 930 op/s; 9.9 MiB/s, 158 objects/s recovering 2024-06-14T23:17:55.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[97913]: osdmap e397: 8 total, 8 up, 7 in 2024-06-14T23:17:55.336 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[102754]: 1.16 scrub starts 2024-06-14T23:17:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[102754]: 1.16 scrub ok 2024-06-14T23:17:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[102754]: 1.3 deep-scrub starts 2024-06-14T23:17:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[102754]: 1.3 deep-scrub ok 2024-06-14T23:17:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[102754]: pgmap v1468: 119 pgs: 1 peering, 1 active+clean+scrubbing, 117 active+clean; 1.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 930 op/s; 9.9 MiB/s, 158 objects/s recovering 2024-06-14T23:17:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:55 smithi086 ceph-mon[102754]: osdmap e397: 8 total, 8 up, 7 in 2024-06-14T23:17:55.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:55.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:55.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:55 smithi137 ceph-mon[90634]: 1.16 scrub starts 2024-06-14T23:17:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:55 smithi137 ceph-mon[90634]: 1.16 scrub ok 2024-06-14T23:17:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:55 smithi137 ceph-mon[90634]: 1.3 deep-scrub starts 2024-06-14T23:17:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:55 smithi137 ceph-mon[90634]: 1.3 deep-scrub ok 2024-06-14T23:17:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:55 smithi137 ceph-mon[90634]: pgmap v1468: 119 pgs: 1 peering, 1 active+clean+scrubbing, 117 active+clean; 1.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 930 op/s; 9.9 MiB/s, 158 objects/s recovering 2024-06-14T23:17:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:55 smithi137 ceph-mon[90634]: osdmap e397: 8 total, 8 up, 7 in 2024-06-14T23:17:55.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:55.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:55.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:55.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:55.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:55.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:55.692 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:17:55.692 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:17:55.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:17:55.692 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:17:55.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.692 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:218471 11.10 11:0e8f4d20:::benchmark_data_smithi086_226760_object91150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:08:35.027427+0000", 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 560.66008054300005, 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.106725873, 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 218471 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:55.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.027427+0000", 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.027427+0000", 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.027426+0000", 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.027431+0000", 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7400000000000001e-06 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.027443+0000", 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1482e-05 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.027449+0000", 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1589999999999998e-06 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.027481+0000", 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2246999999999998e-05 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.027551+0000", 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9420999999999998e-05 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.027769+0000", 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000218474 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.029504+0000", 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017352089999999999 2024-06-14T23:17:55.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030608+0000", 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011037849999999999 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.134070+0000", 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10346232900000001 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.134117+0000", 2024-06-14T23:17:55.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6999999999999997e-05 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.134153+0000", 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5154999999999998e-05 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.36237.0:218475 11.c 11:3e865de5:::benchmark_data_smithi086_226760_object91154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:08:35.030663+0000", 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 560.65684425100005, 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.114635056, 2024-06-14T23:17:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.36237", 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/2536578420", 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 218475 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030663+0000", 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030663+0000", 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030662+0000", 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030667+0000", 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1089999999999997e-06 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030683+0000", 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5693e-05 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030689+0000", 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6880000000000002e-06 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030735+0000", 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5694e-05 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030854+0000", 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119094 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.030982+0000", 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012765500000000001 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.133502+0000", 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.102520025 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.137779+0000", 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042775110000000003 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.145227+0000", 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0074477320000000003 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.145258+0000", 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0925e-05 2024-06-14T23:17:55.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.145298+0000", 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0145000000000003e-05 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.36237.0:218462 11.3 e280/279 11:c4079cb8:::benchmark_data_smithi086_226760_object91141:head v 280'13376, mlcod=262'8411)", 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:08:35.031184+0000", 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 560.6563228, 2024-06-14T23:17:55.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.099057728999999997, 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.031184+0000", 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.031184+0000", 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.031184+0000", 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.031188+0000", 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.369e-06 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.031195+0000", 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3790000000000003e-06 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.031198+0000", 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5930000000000002e-06 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.031233+0000", 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4487999999999999e-05 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.031248+0000", 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5242000000000001e-05 2024-06-14T23:17:55.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.032941+0000", 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016926949999999999 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.130242+0000", 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.097301427999999995 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.36237.0:218475 11.c e280/237 ondisk, result = 0)", 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:08:35.034973+0000", 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 560.65253423700005, 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.098531361999999997, 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.034973+0000", 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.034973+0000", 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.034972+0000", 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.034982+0000", 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0122999999999999e-05 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.131266+0000", 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.096284083000000006 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.131277+0000", 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0883000000000001e-05 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.133487+0000", 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002210315 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.133495+0000", 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0409999999999998e-06 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.133504+0000", 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3070000000000008e-06 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.714 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.36237.0:218478 11.17 e280/278 11:eb74b28a:::benchmark_data_smithi086_226760_object91157:head v 280'13546, mlcod=280'13544)", 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:08:35.044193+0000", 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 560.64331469800004, 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.089078009999999999, 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.044193+0000", 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.044193+0000", 2024-06-14T23:17:55.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.044191+0000", 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.044204+0000", 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2663000000000001e-05 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.128786+0000", 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.084582184000000005 2024-06-14T23:17:55.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.128803+0000", 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.683e-05 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.129035+0000", 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00023273799999999999 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.129066+0000", 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0695999999999999e-05 2024-06-14T23:17:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.133228+0000", 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041617329999999999 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:08:35.133271+0000", 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2809999999999998e-05 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:17:55.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:17:55.719 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:17:55.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:55.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:55.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:55.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:55.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:55.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:56.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:56.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:56.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:56.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:56.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:56.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:56.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:56.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:56.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:56.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:56.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:56.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:56.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:56.632 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:56.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:56.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:56.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:56.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:56.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:17:56.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:56.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:56.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:56.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:57.021 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:57.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:57.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:57 smithi086 ceph-mon[97913]: pgmap v1470: 119 pgs: 1 peering, 1 active+clean+scrubbing, 117 active+clean; 1.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 947 op/s; 12 MiB/s, 194 objects/s recovering 2024-06-14T23:17:57.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:57 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]: dispatch 2024-06-14T23:17:57.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:57 smithi086 ceph-mon[102754]: pgmap v1470: 119 pgs: 1 peering, 1 active+clean+scrubbing, 117 active+clean; 1.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 947 op/s; 12 MiB/s, 194 objects/s recovering 2024-06-14T23:17:57.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:57 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]: dispatch 2024-06-14T23:17:57.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:57.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:57.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:57.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:57.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:57.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:57.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:57.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:57.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:57.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:57 smithi137 ceph-mon[90634]: pgmap v1470: 119 pgs: 1 peering, 1 active+clean+scrubbing, 117 active+clean; 1.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 947 op/s; 12 MiB/s, 194 objects/s recovering 2024-06-14T23:17:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:57 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]: dispatch 2024-06-14T23:17:57.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:57.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:57.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout:moved 7 / 357 (1.96078%) 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout:avg 51 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout:stddev 7.0102 -> 4.78091 (expected baseline 6.61168) 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout:min osd.2 with 36 -> 43 pgs (0.705882 -> 0.843137 * mean) 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout:max osd.3 with 57 -> 56 pgs (1.11765 -> 1.09804 * mean) 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout:oload 1000 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout:max_change 1 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout:max_change_osds 4 2024-06-14T23:17:57.625 INFO:teuthology.orchestra.run.smithi086.stdout:average_utilization 0.0192 2024-06-14T23:17:57.626 INFO:teuthology.orchestra.run.smithi086.stdout:overload_utilization 0.1923 2024-06-14T23:17:57.626 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2 weight 0.7062 -> 1.0000 2024-06-14T23:17:57.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:57.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:57.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:17:57.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:57.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:57.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:57.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:57.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:57.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:58.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:58.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:58.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:58.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:58.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:58.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:58.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:58.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:58.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[97913]: from='client.47111 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:17:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]': finished 2024-06-14T23:17:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[97913]: osdmap e398: 8 total, 8 up, 7 in 2024-06-14T23:17:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:58.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[102754]: from='client.47111 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:17:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]': finished 2024-06-14T23:17:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[102754]: osdmap e398: 8 total, 8 up, 7 in 2024-06-14T23:17:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:58.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:17:58.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:58.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:58.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:17:58.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:58 smithi137 ceph-mon[90634]: from='client.47111 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:17:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'65536'}"}]': finished 2024-06-14T23:17:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:58 smithi137 ceph-mon[90634]: osdmap e398: 8 total, 8 up, 7 in 2024-06-14T23:17:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:17:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:17:58.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:58.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:58.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:58.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:58.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:58.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:58.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:58.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:58.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-14T23:17:58.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:17:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:58.889 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:58.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:17:58.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:59.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:59.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:59.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:59.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:59.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:59.189 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:59 smithi086 ceph-mon[97913]: pgmap v1472: 119 pgs: 2 unknown, 5 peering, 112 active+clean; 1.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail 2024-06-14T23:17:59.190 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:17:59 smithi086 ceph-mon[97913]: osdmap e399: 8 total, 8 up, 7 in 2024-06-14T23:17:59.190 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:59 smithi086 ceph-mon[102754]: pgmap v1472: 119 pgs: 2 unknown, 5 peering, 112 active+clean; 1.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail 2024-06-14T23:17:59.190 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:17:59 smithi086 ceph-mon[102754]: osdmap e399: 8 total, 8 up, 7 in 2024-06-14T23:17:59.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:17:59.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:59.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:17:59.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:17:59.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:59.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:17:59.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:59 smithi137 ceph-mon[90634]: pgmap v1472: 119 pgs: 2 unknown, 5 peering, 112 active+clean; 1.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail 2024-06-14T23:17:59.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:17:59 smithi137 ceph-mon[90634]: osdmap e399: 8 total, 8 up, 7 in 2024-06-14T23:17:59.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:17:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:59.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:59.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:17:59.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:17:59.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:59.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:17:59.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:17:59.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:17:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:59.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:17:59.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:17:59.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:17:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:17:59.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:00.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:00.104 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:00.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:00.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:00.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:00.241 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:00.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:00.245 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-06-14T23:18:00.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-06-14T23:18:00.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:00.373 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:00 smithi086 ceph-mon[97913]: osdmap e400: 8 total, 8 up, 7 in 2024-06-14T23:18:00.373 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:00 smithi086 ceph-mon[97913]: 13.a starting backfill to osd.2 from (0'0,0'0] MAX to 397'13664 2024-06-14T23:18:00.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:00 smithi086 ceph-mon[102754]: osdmap e400: 8 total, 8 up, 7 in 2024-06-14T23:18:00.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:00 smithi086 ceph-mon[102754]: 13.a starting backfill to osd.2 from (0'0,0'0] MAX to 397'13664 2024-06-14T23:18:00.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:00.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:00.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:00.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:00.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:00 smithi137 ceph-mon[90634]: osdmap e400: 8 total, 8 up, 7 in 2024-06-14T23:18:00.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:00 smithi137 ceph-mon[90634]: 13.a starting backfill to osd.2 from (0'0,0'0] MAX to 397'13664 2024-06-14T23:18:00.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:00.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:00.540 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:18:00.541 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:18:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:18:00.574 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:18:00.597 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:18:00.612 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:18:00.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:00.649 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:18:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:18:00.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:00.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:18:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:18:00.725 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:18:00.725 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:18:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:18:00.758 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:18:00.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:00.796 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:18:00.797 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:18:00.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:00.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:00.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:00.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:00.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:01.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:01.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:01.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:01.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:01.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:01.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:01.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:01 smithi086 ceph-mon[97913]: pgmap v1475: 119 pgs: 1 remapped+peering, 1 unknown, 4 peering, 113 active+clean; 1.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail 2024-06-14T23:18:01.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:01 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:18:01.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:01 smithi086 ceph-mon[97913]: osdmap e401: 8 total, 8 up, 7 in 2024-06-14T23:18:01.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3023725304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:01.252 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:01 smithi086 ceph-mon[102754]: pgmap v1475: 119 pgs: 1 remapped+peering, 1 unknown, 4 peering, 113 active+clean; 1.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail 2024-06-14T23:18:01.252 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:01 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:18:01.253 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:01 smithi086 ceph-mon[102754]: osdmap e401: 8 total, 8 up, 7 in 2024-06-14T23:18:01.253 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3023725304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:01.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:01.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:01.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:01.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:01.476 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:01.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:01.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:01 smithi137 ceph-mon[90634]: pgmap v1475: 119 pgs: 1 remapped+peering, 1 unknown, 4 peering, 113 active+clean; 1.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail 2024-06-14T23:18:01.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:01 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-14T23:18:01.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:01 smithi137 ceph-mon[90634]: osdmap e401: 8 total, 8 up, 7 in 2024-06-14T23:18:01.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3023725304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:01.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:01.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:01.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:01.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:01.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:01.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:01.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:01.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:01.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:01.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:02.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:02.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:02.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:02.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:02.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:02.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:02.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:02.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:02.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:02 smithi086 ceph-mon[97913]: osdmap e402: 8 total, 8 up, 7 in 2024-06-14T23:18:02.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:02 smithi086 ceph-mon[102754]: osdmap e402: 8 total, 8 up, 7 in 2024-06-14T23:18:02.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:02.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:02.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:02.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:02 smithi137 ceph-mon[90634]: osdmap e402: 8 total, 8 up, 7 in 2024-06-14T23:18:02.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:02.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:02.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:02.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:02.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:02.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:02.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:02.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:02.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:02.924 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:02.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:02.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:02.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:03.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:03.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:03.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:03.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:03.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:03.216 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:03 smithi086 ceph-mon[97913]: pgmap v1478: 119 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 4 peering, 113 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 116 MiB/s wr, 3.89k op/s; 690/59004 objects misplaced (1.169%); 81 MiB/s, 1.29k objects/s recovering 2024-06-14T23:18:03.217 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:03 smithi086 ceph-mon[102754]: pgmap v1478: 119 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 4 peering, 113 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 116 MiB/s wr, 3.89k op/s; 690/59004 objects misplaced (1.169%); 81 MiB/s, 1.29k objects/s recovering 2024-06-14T23:18:03.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:03.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:03.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:03.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:03.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:03.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:03.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:03.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:03.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:03 smithi137 ceph-mon[90634]: pgmap v1478: 119 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 4 peering, 113 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 116 MiB/s wr, 3.89k op/s; 690/59004 objects misplaced (1.169%); 81 MiB/s, 1.29k objects/s recovering 2024-06-14T23:18:03.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:03.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:03.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:03.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:03.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:03.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:03.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:03.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:03.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:03.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:03.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:03.989 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:03.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:04.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:04.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:04.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:04.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:04.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:04.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:04.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:04.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:04.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:04.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:04.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:04 smithi137 ceph-mon[90634]: osdmap e403: 8 total, 8 up, 7 in 2024-06-14T23:18:04.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:04.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:04.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:04.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:04 smithi086 ceph-mon[97913]: osdmap e403: 8 total, 8 up, 7 in 2024-06-14T23:18:04.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:04 smithi086 ceph-mon[102754]: osdmap e403: 8 total, 8 up, 7 in 2024-06-14T23:18:04.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:04.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:04.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:04.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:04.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:04.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:04.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:04.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:04.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:18:04.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:05.034 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:05.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:05.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:18:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:18:04] "GET /metrics HTTP/1.1" 200 36692 "" "Prometheus/2.43.0" 2024-06-14T23:18:05.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:05.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:05.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:05.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:05.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:05.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:05.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:05.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:05.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:05 smithi137 ceph-mon[90634]: pgmap v1480: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 98 MiB/s wr, 3.55k op/s; 690/51105 objects misplaced (1.350%); 68 MiB/s, 1.09k objects/s recovering 2024-06-14T23:18:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:05 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:18:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:05 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:18:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:05 smithi137 ceph-mon[90634]: osdmap e404: 8 total, 8 up, 7 in 2024-06-14T23:18:05.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:05.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:05 smithi086 ceph-mon[97913]: pgmap v1480: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 98 MiB/s wr, 3.55k op/s; 690/51105 objects misplaced (1.350%); 68 MiB/s, 1.09k objects/s recovering 2024-06-14T23:18:05.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:05 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:18:05.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:05 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:18:05.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:05 smithi086 ceph-mon[97913]: osdmap e404: 8 total, 8 up, 7 in 2024-06-14T23:18:05.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:05 smithi086 ceph-mon[102754]: pgmap v1480: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 98 MiB/s wr, 3.55k op/s; 690/51105 objects misplaced (1.350%); 68 MiB/s, 1.09k objects/s recovering 2024-06-14T23:18:05.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:05 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-14T23:18:05.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:05 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:18:05.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:05 smithi086 ceph-mon[102754]: osdmap e404: 8 total, 8 up, 7 in 2024-06-14T23:18:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:05.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:05.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:05.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:05.750 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:05.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:05.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:05.880 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:05.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:05.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:05.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:06.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:06.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:06.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:06.144 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:06.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:06.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:06.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:06.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:06.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:06.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:06.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:06.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:06.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:06.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:06.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:06.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:06.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:06.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:06.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:06.895 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:18:06.895 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:18:06.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:06.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:06.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:07.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:07.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:07.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:18:07.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:07.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:07.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:07.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:07.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:07.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:07.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:07.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:07 smithi137 ceph-mon[90634]: pgmap v1482: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 997 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 81 MiB/s wr, 3.12k op/s; 690/47919 objects misplaced (1.440%); 56 MiB/s, 903 objects/s recovering 2024-06-14T23:18:07.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:07 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2121661397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:07.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:07 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:07.505 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:07.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:07.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:07 smithi086 ceph-mon[102754]: pgmap v1482: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 997 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 81 MiB/s wr, 3.12k op/s; 690/47919 objects misplaced (1.440%); 56 MiB/s, 903 objects/s recovering 2024-06-14T23:18:07.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:07 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2121661397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:07.515 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:07 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:07.515 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:07 smithi086 ceph-mon[97913]: pgmap v1482: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 997 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 81 MiB/s wr, 3.12k op/s; 690/47919 objects misplaced (1.440%); 56 MiB/s, 903 objects/s recovering 2024-06-14T23:18:07.515 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:07 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2121661397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:07.516 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:07 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:07.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:07.633 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:07.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:07.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:07.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:07.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:07.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:07.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:07.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:07.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:07.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:07.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:08.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:08.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:08.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:08.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:08.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:08.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:08.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:08.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:08.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:08.424 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:08.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:08.425 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:08 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:08.425 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:08 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:18:08.425 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:08 smithi086 ceph-mon[97913]: osdmap e405: 8 total, 8 up, 7 in 2024-06-14T23:18:08.426 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:08 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:08.426 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:08 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:18:08.426 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:08 smithi086 ceph-mon[102754]: osdmap e405: 8 total, 8 up, 7 in 2024-06-14T23:18:08.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:08 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:08.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:08 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:18:08.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:08 smithi137 ceph-mon[90634]: osdmap e405: 8 total, 8 up, 7 in 2024-06-14T23:18:08.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:08.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:08.559 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:08.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:08.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:08.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:08.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:08.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:08.844 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:08.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:08.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:08.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:08.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:09.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:09.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:09.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:09.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:09.226 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:09.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:09.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:09 smithi086 ceph-mon[97913]: pgmap v1484: 119 pgs: 119 active+clean; 875 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 6.8 MiB/s, 109 objects/s recovering 2024-06-14T23:18:09.302 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:09 smithi086 ceph-mon[102754]: pgmap v1484: 119 pgs: 119 active+clean; 875 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 6.8 MiB/s, 109 objects/s recovering 2024-06-14T23:18:09.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:09.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:09.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:09.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:09.488 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:09.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:09 smithi137 ceph-mon[90634]: pgmap v1484: 119 pgs: 119 active+clean; 875 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 6.8 MiB/s, 109 objects/s recovering 2024-06-14T23:18:09.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:09.618 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:09.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:09.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:09.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:09.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:09.854 INFO:teuthology.orchestra.run.smithi086.stdout:moved 0 / 357 (0%) 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout:avg 51 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout:stddev 4.78091 -> 4.78091 (expected baseline 6.61168) 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout:min osd.2 with 43 -> 43 pgs (0.843137 -> 0.843137 * mean) 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout:max osd.3 with 56 -> 56 pgs (1.09804 -> 1.09804 * mean) 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout:oload 1000 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout:max_change 1 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout:max_change_osds 4 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout:average_utilization 0.0159 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stdout:overload_utilization 0.1588 2024-06-14T23:18:09.855 INFO:teuthology.orchestra.run.smithi086.stderr:no change 2024-06-14T23:18:09.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:09.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:09.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:09.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:10.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:10.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:10.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:10.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:10.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:10.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:10.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:10.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:10.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:10.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-14T23:18:10.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:10.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:10.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:10.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:10.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:10.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:10.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:10.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:10.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:10.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:10.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:10.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:11.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:11.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:11.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:18:11.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:11.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:11.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:11.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:11.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:11.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:11.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:11 smithi086 ceph-mon[97913]: pgmap v1485: 119 pgs: 119 active+clean; 749 MiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-06-14T23:18:11.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:11 smithi086 ceph-mon[97913]: from='client.37200 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:11.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:11 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:18:11.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:11 smithi086 ceph-mon[102754]: pgmap v1485: 119 pgs: 119 active+clean; 749 MiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-06-14T23:18:11.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:11 smithi086 ceph-mon[102754]: from='client.37200 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:11.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:11 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:18:11.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:11.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:11.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:11.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:11.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:11 smithi137 ceph-mon[90634]: pgmap v1485: 119 pgs: 119 active+clean; 749 MiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 956 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-06-14T23:18:11.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:11 smithi137 ceph-mon[90634]: from='client.37200 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:11.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:11 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:18:11.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:11.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:11.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:11.574 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-06-14T23:18:11.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-06-14T23:18:11.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:11.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:11.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:11.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:11.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:11.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:11.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:12.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:12.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:12.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:12.160 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:12.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:12.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:12 smithi137 ceph-mon[90634]: pgmap v1486: 119 pgs: 119 active+clean; 708 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 5.1 MiB/s, 81 objects/s recovering 2024-06-14T23:18:12.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:12.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:12.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:12.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:12.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:12.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:12.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:12.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:12.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:12.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:12.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:12.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:12 smithi086 ceph-mon[97913]: pgmap v1486: 119 pgs: 119 active+clean; 708 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 5.1 MiB/s, 81 objects/s recovering 2024-06-14T23:18:12.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:12 smithi086 ceph-mon[102754]: pgmap v1486: 119 pgs: 119 active+clean; 708 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 5.1 MiB/s, 81 objects/s recovering 2024-06-14T23:18:12.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:12.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:12.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:12.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:12.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:12.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:12.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:12.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:12.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:13.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:13.116 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:13.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:13.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:13.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:13.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:13.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:13.404 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:13.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:13.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:13.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:13.545 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:13.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:13.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:13.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:13.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:13.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:13.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:13.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:13.692 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:13.692 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:13.692 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:13.729 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:13.729 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:13.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:13.798 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:13.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:13.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:13.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:13.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:14.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:14.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:14.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:14.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:14.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:14.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:14.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:14.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:14.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:14.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:14.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:14.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:14.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:14.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:14.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:14.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:14 smithi086 ceph-mon[97913]: pgmap v1487: 119 pgs: 119 active+clean; 475 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 4.2 MiB/s, 67 objects/s recovering 2024-06-14T23:18:14.691 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:14 smithi086 ceph-mon[102754]: pgmap v1487: 119 pgs: 119 active+clean; 475 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 4.2 MiB/s, 67 objects/s recovering 2024-06-14T23:18:14.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:14.696 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:18:14.697 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:18:14.720 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:14.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:14.726 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:18:14.726 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:18:14.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:14 smithi137 ceph-mon[90634]: pgmap v1487: 119 pgs: 119 active+clean; 475 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 4.2 MiB/s, 67 objects/s recovering 2024-06-14T23:18:14.750 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:18:14.750 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:18:14.771 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:18:14.771 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:18:14.798 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:18:14.798 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:18:14.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:14.843 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:18:14.843 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:18:14.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:14.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:14.871 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:18:14.871 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:18:14.904 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:18:14.905 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:18:14.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:14.976 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:18:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:18:14] "GET /metrics HTTP/1.1" 200 36688 "" "Prometheus/2.43.0" 2024-06-14T23:18:14.977 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:14.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:15.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:15.106 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:15.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:15.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:15.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:15.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:15.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:15.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:15.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:15.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:15.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:15.508 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:15.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:15.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:15 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2804617136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:15.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:15 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2804617136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:15.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:15.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:15.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:15.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:15 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2804617136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:15.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:15.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:15.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:15.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:15.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:15.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:15.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:16.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:16.098 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:16.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:16.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:16.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:16.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:16.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:16.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:16.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:16.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:16.546 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:16.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:16.590 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:16 smithi137 ceph-mon[90634]: pgmap v1488: 119 pgs: 119 active+clean; 400 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 955 op/s; 4.1 MiB/s, 65 objects/s recovering 2024-06-14T23:18:16.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:16.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:16.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:16.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:16 smithi086 ceph-mon[97913]: pgmap v1488: 119 pgs: 119 active+clean; 400 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 955 op/s; 4.1 MiB/s, 65 objects/s recovering 2024-06-14T23:18:16.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:16 smithi086 ceph-mon[102754]: pgmap v1488: 119 pgs: 119 active+clean; 400 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 955 op/s; 4.1 MiB/s, 65 objects/s recovering 2024-06-14T23:18:16.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:16.816 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:16.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:16.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:16.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:16.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:16.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:17.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:17.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:17.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:17.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:17.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:17.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:17.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:17.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:17.462 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:17.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:17.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:17.587 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:17.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:17.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:17.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:17.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:17.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:17.874 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:17.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:17.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:18.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:18.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:18.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:18.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:18.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:18.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:18.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:18.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:18.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:18.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:18.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:18 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:18 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:18 smithi137 ceph-mon[90634]: pgmap v1489: 119 pgs: 119 active+clean; 271 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-06-14T23:18:18.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:18.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:18.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:18 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:18.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:18 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:18.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:18 smithi086 ceph-mon[97913]: pgmap v1489: 119 pgs: 119 active+clean; 271 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-06-14T23:18:18.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:18 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:18.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:18 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:18.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:18 smithi086 ceph-mon[102754]: pgmap v1489: 119 pgs: 119 active+clean; 271 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-06-14T23:18:18.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:18.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:18.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:18.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:18:18.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:18.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:18.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:18.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:18.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:18.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:19.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:19.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:19.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:19.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:19.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:19.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:19.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:19.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:19.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:19.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:19.462 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:19.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:19.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:19.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:19.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:19.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:19.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:19.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:19.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:19.874 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:19.882 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:19.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:19.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:20.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:20.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:20.071 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 120835 objects 2024-06-14T23:18:20.071 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :125.718 2024-06-14T23:18:20.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:20.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:20.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:20.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:20.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:20.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:20.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:20.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:20.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:20.410 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:18:20.410 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:18:20.410 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:18:20.410 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:18:20.410 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:18:20.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:20.525 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:20.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:20.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:20.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:20.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:20.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:20.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:20.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:20.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:20.877 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:18:20.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:20.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:20.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:21.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:21.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:21.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:21.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:21.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:21.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:21.167 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:20 smithi086 ceph-mon[97913]: pgmap v1490: 119 pgs: 119 active+clean; 145 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-06-14T23:18:21.167 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2833269823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:21.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:20 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:21.168 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:20 smithi086 ceph-mon[102754]: pgmap v1490: 119 pgs: 119 active+clean; 145 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-06-14T23:18:21.168 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2833269823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:21.168 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:20 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:21.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:20 smithi137 ceph-mon[90634]: pgmap v1490: 119 pgs: 119 active+clean; 145 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-06-14T23:18:21.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2833269823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:21.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:20 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:21.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:21.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:21.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:21.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:21.405 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:21.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:21.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:21.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:21.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:21.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:21.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:21.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:21.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:21.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:21.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:21.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:21.905 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:21.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:22.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:22.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:22.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:22.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:22.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:22.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:21 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:22.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:21 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:18:22.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:21 smithi137 ceph-mon[90634]: osdmap e406: 8 total, 8 up, 7 in 2024-06-14T23:18:22.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:21 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:22.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:21 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:18:22.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:21 smithi086 ceph-mon[97913]: osdmap e406: 8 total, 8 up, 7 in 2024-06-14T23:18:22.256 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:21 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:22.256 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:21 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:18:22.256 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:21 smithi086 ceph-mon[102754]: osdmap e406: 8 total, 8 up, 7 in 2024-06-14T23:18:22.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:22.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:22.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:22.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-14T23:18:22.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:22.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:22.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:22.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:22.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:22.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:22.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:22.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:22.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:22.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:22.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:22.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:22.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:22.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:22.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:22.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:22.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:23.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:23.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:22 smithi086 ceph-mon[97913]: pgmap v1492: 119 pgs: 119 active+clean; 114 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 951 op/s 2024-06-14T23:18:23.059 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:22 smithi086 ceph-mon[102754]: pgmap v1492: 119 pgs: 119 active+clean; 114 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 951 op/s 2024-06-14T23:18:23.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:23.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:23.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:23.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:23.188 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:23.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:23.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:22 smithi137 ceph-mon[90634]: pgmap v1492: 119 pgs: 119 active+clean; 114 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 951 op/s 2024-06-14T23:18:23.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:23.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:23.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:23.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:23.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:23.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:23.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:18:23.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:23.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:23.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:23.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:23.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:23.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:23.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:23.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:23.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:23.940 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_8 2024-06-14T23:18:23.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_8 unique_pool_8 --yes-i-really-really-mean-it 2024-06-14T23:18:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:23.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:23.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:24.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:24.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:24.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:24.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:24.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:24.246 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:18:23 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a[77437]: logger=cleanup t=2024-06-14T23:18:23.767707354Z level=info msg="Completed cleanup jobs" duration=13.360044ms 2024-06-14T23:18:24.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:24.312 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:24.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:24.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:24.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:24.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:24.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:24.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:24.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:24.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:24.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:24.697 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:24.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:24.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:24.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:24.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:24.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:24.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:24.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:24.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:25.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:24 smithi086 ceph-mon[97913]: pgmap v1493: 119 pgs: 119 active+clean; 9.4 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 744 op/s 2024-06-14T23:18:25.008 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:18:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:18:24] "GET /metrics HTTP/1.1" 200 36688 "" "Prometheus/2.43.0" 2024-06-14T23:18:25.008 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:24 smithi086 ceph-mon[102754]: pgmap v1493: 119 pgs: 119 active+clean; 9.4 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 744 op/s 2024-06-14T23:18:25.014 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:25.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:25.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:25.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:25.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:25.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:25.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:25.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:24 smithi137 ceph-mon[90634]: pgmap v1493: 119 pgs: 119 active+clean; 9.4 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 744 op/s 2024-06-14T23:18:25.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:25.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:25.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:25.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:25.446 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:25.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:25.532 INFO:teuthology.orchestra.run.smithi086.stdout:moved 0 / 357 (0%) 2024-06-14T23:18:25.532 INFO:teuthology.orchestra.run.smithi086.stdout:avg 51 2024-06-14T23:18:25.532 INFO:teuthology.orchestra.run.smithi086.stdout:stddev 4.78091 -> 4.78091 (expected baseline 6.61168) 2024-06-14T23:18:25.532 INFO:teuthology.orchestra.run.smithi086.stdout:min osd.2 with 43 -> 43 pgs (0.843137 -> 0.843137 * mean) 2024-06-14T23:18:25.533 INFO:teuthology.orchestra.run.smithi086.stdout:max osd.3 with 56 -> 56 pgs (1.09804 -> 1.09804 * mean) 2024-06-14T23:18:25.533 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:18:25.533 INFO:teuthology.orchestra.run.smithi086.stdout:oload 1000 2024-06-14T23:18:25.533 INFO:teuthology.orchestra.run.smithi086.stdout:max_change 1 2024-06-14T23:18:25.533 INFO:teuthology.orchestra.run.smithi086.stdout:max_change_osds 4 2024-06-14T23:18:25.533 INFO:teuthology.orchestra.run.smithi086.stdout:average_utilization 0.0123 2024-06-14T23:18:25.534 INFO:teuthology.orchestra.run.smithi086.stdout:overload_utilization 0.1228 2024-06-14T23:18:25.534 INFO:teuthology.orchestra.run.smithi086.stderr:no change 2024-06-14T23:18:25.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:25.570 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:25.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:25.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:25.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:25.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:25.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:25.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:25.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:25.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:25.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:25.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:26.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:26.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:26.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:26.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:26.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:26.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:26.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:26.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:26.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:26.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:26.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:26.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:26.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:26.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:26.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:26.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:26.718 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:26.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:26.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:26.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:26.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:26.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:26.970 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:26.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:27.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:27.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:27.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:27.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:27.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:27.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:26 smithi137 ceph-mon[90634]: from='client.37218 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:27.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:26 smithi137 ceph-mon[90634]: pgmap v1494: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 638 op/s 2024-06-14T23:18:27.273 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:26 smithi086 ceph-mon[97913]: from='client.37218 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:27.273 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:26 smithi086 ceph-mon[97913]: pgmap v1494: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 638 op/s 2024-06-14T23:18:27.273 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:26 smithi086 ceph-mon[102754]: from='client.37218 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:27.274 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:26 smithi086 ceph-mon[102754]: pgmap v1494: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 638 op/s 2024-06-14T23:18:27.327 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:27.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:27.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:27.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:27.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:27.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:27.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:27.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:27.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:27.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:27.721 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:27.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:27.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:27.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:27.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:27.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:27.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:27.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:28.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:28.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:28.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:28.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:28.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:28.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:28.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:28.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:28.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:28.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:28.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:28.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:28.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:28.460 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:28.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:28.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:28.564 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-06-14T23:18:28.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-06-14T23:18:28.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:28.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:28.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:28.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:28.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:28.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:28.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:28.875 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:18:28.875 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:18:28.894 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:18:28.895 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:18:28.912 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:18:28.913 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:18:28.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:18:28.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:18:28.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:28.946 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:18:28.946 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:18:28.962 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:18:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:18:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:28.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:28.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:28.980 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:18:28.980 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:18:28.997 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:18:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:18:29.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:29.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:28 smithi086 ceph-mon[97913]: pgmap v1495: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 432 op/s 2024-06-14T23:18:29.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:28 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/294440849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:29.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:28 smithi086 ceph-mon[102754]: pgmap v1495: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 432 op/s 2024-06-14T23:18:29.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:28 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/294440849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:29.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:29.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:29.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:29.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:29.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:29.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:28 smithi137 ceph-mon[90634]: pgmap v1495: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 432 op/s 2024-06-14T23:18:29.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:28 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/294440849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:29.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:29.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:29.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:29.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:29.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:29.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:29.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:29.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:29.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:29.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:29.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:29.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:29.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:29.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:29.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:29.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:29.970 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:29.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:30.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:30.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:30.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:30.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:30.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:30.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:30.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:30.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:30.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:30.471 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:30.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:30.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:30.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:30.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:30.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:30.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:30.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:30.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:30.844 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:18:30.844 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:18:30.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:30.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:30.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:30.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:30.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:31.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:31.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:31.181 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:30 smithi086 ceph-mon[97913]: pgmap v1496: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 231 op/s 2024-06-14T23:18:31.181 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:30 smithi086 ceph-mon[102754]: pgmap v1496: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 231 op/s 2024-06-14T23:18:31.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:31.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:31.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:31.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:30 smithi137 ceph-mon[90634]: pgmap v1496: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 231 op/s 2024-06-14T23:18:31.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:31.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:31.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:31.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:31.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:31.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:31.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:31.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-14T23:18:31.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:31.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:31.616 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:31.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:31.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:31.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:31.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:31.839 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:31.840 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:31.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:31.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:31.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:32.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:32.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:32.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:32.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:32.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:32.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:32.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:32.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:32.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:32.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:32.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:32.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:32.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:32.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:32.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:32.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:32.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:32.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:32.829 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:32.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:32.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:32.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:32.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:33.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:33.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:32 smithi086 ceph-mon[97913]: pgmap v1497: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 165 op/s 2024-06-14T23:18:33.067 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:32 smithi086 ceph-mon[102754]: pgmap v1497: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 165 op/s 2024-06-14T23:18:33.077 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:33.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:33.155 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:32 smithi137 ceph-mon[90634]: pgmap v1497: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 165 op/s 2024-06-14T23:18:33.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:33.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:33.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:33.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:33.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:33.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:33.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:33.447 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:33.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:33.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:33.570 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:33.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:33.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:33.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:33.714 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:18:33.715 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:18:33.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:33.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:33.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:33.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:33.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:33.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:34.043 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:34.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:34.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:18:34.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:34.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:34.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:34.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:34.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:34.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:34.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:34.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:34.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:34.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:34.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:34.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:34.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-14T23:18:34.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:34.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:34.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:34.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:34.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:34.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:34.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:34.721 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:34.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:34.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:34.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:34.902 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:18:34.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:34.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:35.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:35.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:35.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:35.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:35.166 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:34 smithi137 ceph-mon[90634]: pgmap v1498: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 150 op/s 2024-06-14T23:18:35.166 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2994517216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:18:35.166 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:34 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:18:35.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:35.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:35.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:35.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:35.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:35.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:18:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:18:34] "GET /metrics HTTP/1.1" 200 36665 "" "Prometheus/2.43.0" 2024-06-14T23:18:35.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:34 smithi086 ceph-mon[97913]: pgmap v1498: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 150 op/s 2024-06-14T23:18:35.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2994517216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:18:35.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:34 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:18:35.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:34 smithi086 ceph-mon[102754]: pgmap v1498: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 150 op/s 2024-06-14T23:18:35.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2994517216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:18:35.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:34 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:18:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:35.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:35.426 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:35.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:35.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:35.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:35.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:35.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:35.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:35.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:35.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:35.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:35.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:35.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:35.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:35.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:36.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:36.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:36.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:36.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:36.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:36.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:36.184 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:35 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:36.184 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:35 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:18:36.184 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:35 smithi137 ceph-mon[90634]: osdmap e407: 8 total, 8 up, 7 in 2024-06-14T23:18:36.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:36.274 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:35 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:36.274 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:35 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:18:36.274 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:35 smithi086 ceph-mon[97913]: osdmap e407: 8 total, 8 up, 7 in 2024-06-14T23:18:36.275 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:35 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:36.275 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:35 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:18:36.275 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:35 smithi086 ceph-mon[102754]: osdmap e407: 8 total, 8 up, 7 in 2024-06-14T23:18:36.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:36.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:36.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:36.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:36.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:36.440 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:18:36.440 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:18:36.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:36.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:36.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:36.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:36.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:36.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:36.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:36.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:36.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:36.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:36.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:36.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:36.918 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_8' removed 2024-06-14T23:18:37.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:37.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:37.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:37.088 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:36 smithi137 ceph-mon[90634]: pgmap v1500: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:37.089 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3936122771' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:18:37.089 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:36 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:18:37.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-14T23:18:37.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:37.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:37.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:37.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:37.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:37.303 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:37.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:37.306 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:36 smithi086 ceph-mon[97913]: pgmap v1500: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:37.306 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3936122771' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:18:37.306 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:36 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:18:37.306 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:36 smithi086 ceph-mon[102754]: pgmap v1500: 119 pgs: 119 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:37.306 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3936122771' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:18:37.307 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:36 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:18:37.367 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:37.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:37.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:37.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:37.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:37.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:37.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:37.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:37.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:37.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:37.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:37.811 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:37.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:37.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:37.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:37.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:37.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:38.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:38.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:38.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:38.100 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-14T23:18:38.101 INFO:tasks.radosbench:Beginning radosbench... 2024-06-14T23:18:38.101 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_9 2024-06-14T23:18:38.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool create unique_pool_9 16 2024-06-14T23:18:38.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:18:38.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:38.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:38.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:38.204 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:37 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:18:38.204 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:37 smithi086 ceph-mon[97913]: osdmap e408: 8 total, 8 up, 7 in 2024-06-14T23:18:38.204 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:37 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:18:38.204 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:37 smithi086 ceph-mon[102754]: osdmap e408: 8 total, 8 up, 7 in 2024-06-14T23:18:38.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:37 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:18:38.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:37 smithi137 ceph-mon[90634]: osdmap e408: 8 total, 8 up, 7 in 2024-06-14T23:18:38.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:38.305 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:38.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:38.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:38.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:38.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:38.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:38.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:38.552 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:38.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:38.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:38.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:38.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:38.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:38.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:38.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:38.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:38.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:38.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:39.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:39.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:39.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:39.056 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:38 smithi137 ceph-mon[90634]: pgmap v1502: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:39.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:38 smithi086 ceph-mon[97913]: pgmap v1502: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:39.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:38 smithi086 ceph-mon[102754]: pgmap v1502: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:39.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:39.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:39.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:39.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:39.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:39.367 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:18:39.367 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:18:39.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:39.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:39.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:39.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:39.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:39.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:39.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:39.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:39.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:39.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:39.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:39.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:39.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:39.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:39.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:40.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:40.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:40.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:40.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:40.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T23:18:40.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:40.158 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:40.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:40.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:40.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:40.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:40.338 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:40.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:40.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:40.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:40.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:40.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:40.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:40.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:40.652 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:40.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:40.726 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:18:40.726 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:18:40.742 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:18:40.742 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:18:40.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:40.758 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:18:40.758 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:18:40.774 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:18:40.775 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:18:40.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:40.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:40.790 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:18:40.791 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:18:40.807 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:18:40.807 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:18:40.824 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:18:40.824 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:18:40.840 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:18:40.840 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:18:40.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:40.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:40.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:40.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:41.021 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:41.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:41.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:41.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:41.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:41.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:41.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:41.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:40 smithi086 ceph-mon[97913]: pgmap v1503: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:41.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3255035619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:41.246 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:40 smithi086 ceph-mon[102754]: pgmap v1503: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:41.247 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3255035619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:41.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:40 smithi137 ceph-mon[90634]: pgmap v1503: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:41.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3255035619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:41.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:41.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:41.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:41.390 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:41.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:41.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:41.511 INFO:teuthology.orchestra.run.smithi086.stdout:moved 0 / 309 (0%) 2024-06-14T23:18:41.511 INFO:teuthology.orchestra.run.smithi086.stdout:avg 44.1429 2024-06-14T23:18:41.511 INFO:teuthology.orchestra.run.smithi086.stdout:stddev 5.74279 -> 5.74279 (expected baseline 6.15116) 2024-06-14T23:18:41.512 INFO:teuthology.orchestra.run.smithi086.stdout:min osd.2 with 34 -> 34 pgs (0.770227 -> 0.770227 * mean) 2024-06-14T23:18:41.512 INFO:teuthology.orchestra.run.smithi086.stdout:max osd.3 with 51 -> 51 pgs (1.15534 -> 1.15534 * mean) 2024-06-14T23:18:41.512 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:18:41.512 INFO:teuthology.orchestra.run.smithi086.stdout:oload 1000 2024-06-14T23:18:41.512 INFO:teuthology.orchestra.run.smithi086.stdout:max_change 1 2024-06-14T23:18:41.512 INFO:teuthology.orchestra.run.smithi086.stdout:max_change_osds 4 2024-06-14T23:18:41.512 INFO:teuthology.orchestra.run.smithi086.stdout:average_utilization 0.0121 2024-06-14T23:18:41.512 INFO:teuthology.orchestra.run.smithi086.stdout:overload_utilization 0.1210 2024-06-14T23:18:41.512 INFO:teuthology.orchestra.run.smithi086.stderr:no change 2024-06-14T23:18:41.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:41.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:41.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:41.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:41.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:41.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:41.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:41.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:41.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:41.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:41.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:41.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:41.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:42.008 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:42.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:42.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:42.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:42.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:42.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:42.147 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:18:42.147 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:18:42.148 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-06-14T23:18:42.148 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:18:42.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:42.254 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:42.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:42.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:42.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:42.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:42.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:42.479 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:42.504 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:42.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:42.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:42.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:42.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:42.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:42.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:42.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:42.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-14T23:18:42.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:42.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:42.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:42.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:42.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:42.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:43.068 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:43.087 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-06-14T23:18:43.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-06-14T23:18:43.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:43.153 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:43.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:43.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:43.191 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:42 smithi137 ceph-mon[90634]: from='client.47171 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:43.191 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:42 smithi137 ceph-mon[90634]: pgmap v1504: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:43.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:43.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:43.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:43.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:18:43.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:18:43.192 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:43.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[97913]: from='client.47171 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:43.258 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[97913]: pgmap v1504: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[102754]: from='client.47171 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[102754]: pgmap v1504: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:43.259 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:43.260 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:18:43.260 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:18:43.260 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:43.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:43.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:43.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:43.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:43.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:43.405 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:43.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:43.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:43.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:43.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:43.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:43.650 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:43.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:43.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:43.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:43.773 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:43.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:43.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:43.899 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:43.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:44.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:44.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:44.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:44.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:44.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:18:44.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:44.144 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:44.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:44.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:44.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:44.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:44.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:44.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:44.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:44.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:44.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:44.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:44.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:44.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:44.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:44.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:44.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:44.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:44.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:44.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:44.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:44.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:44.982 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:44 smithi137 ceph-mon[90634]: pgmap v1505: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:44.982 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:44 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]: dispatch 2024-06-14T23:18:44.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:45.002 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:18:45.003 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:18:45.003 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:18:45.003 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:18:45.003 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:18:45.013 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:45.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:45.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:45.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:44 smithi086 ceph-mon[97913]: pgmap v1505: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:45.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:44 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]: dispatch 2024-06-14T23:18:45.088 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:18:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:18:44] "GET /metrics HTTP/1.1" 200 34270 "" "Prometheus/2.43.0" 2024-06-14T23:18:45.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:44 smithi086 ceph-mon[102754]: pgmap v1505: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:45.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:44 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]: dispatch 2024-06-14T23:18:45.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:45.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:45.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:45.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:45.262 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:45.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:45.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:45.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:45.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:45.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:45.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:45.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:45.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:45.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:45.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:45.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:45.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-14T23:18:45.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:45.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:45.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:45.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:45.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:45.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:45.951 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]': finished 2024-06-14T23:18:45.951 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:45 smithi137 ceph-mon[90634]: osdmap e409: 8 total, 8 up, 7 in 2024-06-14T23:18:45.964 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:45.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:46.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:46.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:46.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:46.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]': finished 2024-06-14T23:18:46.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:45 smithi086 ceph-mon[97913]: osdmap e409: 8 total, 8 up, 7 in 2024-06-14T23:18:46.119 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 5]}]': finished 2024-06-14T23:18:46.119 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:45 smithi086 ceph-mon[102754]: osdmap e409: 8 total, 8 up, 7 in 2024-06-14T23:18:46.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:46.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:46.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:46.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:46.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:46.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:46.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:46.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:46.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:46.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:46.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:46.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:46.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:46.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:46.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:46.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:46.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:46.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:46.880 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:46.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:46.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:46.954 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_9' created 2024-06-14T23:18:46.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:47.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:47.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:47.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:46 smithi086 ceph-mon[97913]: pgmap v1507: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:47.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:46 smithi086 ceph-mon[97913]: osdmap e410: 8 total, 8 up, 7 in 2024-06-14T23:18:47.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1659335265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]: dispatch 2024-06-14T23:18:47.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:46 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]: dispatch 2024-06-14T23:18:47.047 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:46 smithi086 ceph-mon[102754]: pgmap v1507: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:47.047 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:46 smithi086 ceph-mon[102754]: osdmap e410: 8 total, 8 up, 7 in 2024-06-14T23:18:47.047 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1659335265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]: dispatch 2024-06-14T23:18:47.047 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:46 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]: dispatch 2024-06-14T23:18:47.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:47.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:47.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:47.167 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:46 smithi137 ceph-mon[90634]: pgmap v1507: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:47.167 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:46 smithi137 ceph-mon[90634]: osdmap e410: 8 total, 8 up, 7 in 2024-06-14T23:18:47.167 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1659335265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]: dispatch 2024-06-14T23:18:47.167 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:46 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]: dispatch 2024-06-14T23:18:47.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:47.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:47.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:47.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:47.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:47.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:47.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:47.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:47.496 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:47.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:47.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:47.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:47.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:47.688 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:18:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:18:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:18:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:18:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.688 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:131509 12.c 12:363e613a:::benchmark_data_smithi086_261492_object15004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:49.551046+0000", 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 418.12099973699998, 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0172304130000001, 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 131509 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551046+0000", 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551046+0000", 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551044+0000", 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999976 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551055+0000", 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0774000000000001e-05 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551078+0000", 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3337999999999999e-05 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551087+0000", 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8300000000000002e-06 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551216+0000", 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012913700000000001 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.551228+0000", 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2318e-05 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.564927+0000", 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.013698679 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.565051+0000", 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012433 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.565184+0000", 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013285700000000001 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568141+0000", 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029568350000000001 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568191+0000", 2024-06-14T23:18:47.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9478000000000003e-05 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568201+0000", 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0181e-05 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568244+0000", 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3458000000000003e-05 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568277+0000", 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2592999999999998e-05 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:131550 12.c 12:34f200d1:::benchmark_data_smithi086_261492_object15045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:49.636597+0000", 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 418.03544888800002, 2024-06-14T23:18:47.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94182777699999998, 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 131550 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636597+0000", 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636597+0000", 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636595+0000", 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636604+0000", 2024-06-14T23:18:47.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9789999999999999e-06 2024-06-14T23:18:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636624+0000", 2024-06-14T23:18:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9910000000000001e-05 2024-06-14T23:18:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636633+0000", 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5720000000000003e-06 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636699+0000", 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6550000000000005e-05 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:49.636712+0000", 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2938e-05 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568307+0000", 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93159522699999997 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568391+0000", 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3325999999999995e-05 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.568460+0000", 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9091999999999995e-05 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.573215+0000", 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047549130000000004 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.573267+0000", 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2782999999999999e-05 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578327+0000", 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050593189999999996 2024-06-14T23:18:47.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578388+0000", 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1035000000000001e-05 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:50.578425+0000", 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7168000000000001e-05 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191053 13.5 13:a1e52043:::benchmark_data_smithi086_299538_object70215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.870982+0000", 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.801063862000007, 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1272406730000002, 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191053 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.870982+0000", 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.870982+0000", 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.870981+0000", 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.870988+0000", 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2710000000000003e-06 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.870999+0000", 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0951000000000001e-05 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.871004+0000", 2024-06-14T23:18:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4190000000000001e-06 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.871048+0000", 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3850000000000002e-05 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.871147+0000", 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8795999999999994e-05 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.871244+0000", 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7459000000000003e-05 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.904511+0000", 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033266531000000002 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.904519+0000", 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0709999999999995e-06 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.893093+0000", 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98857427099999995 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.893104+0000", 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0740999999999999e-05 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.998156+0000", 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1050514920000001 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.998223+0000", 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7342999999999995e-05 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191059 13.9 13:9e01cc54:::benchmark_data_smithi086_299538_object70221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.876487+0000", 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.795558851999999, 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1277087579999998, 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.713 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191059 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.876487+0000", 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.876487+0000", 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.876486+0000", 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.876491+0000", 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6869999999999999e-06 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.876501+0000", 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8900000000000002e-06 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.876507+0000", 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4840000000000003e-06 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.893755+0000", 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017248593 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.893765+0000", 2024-06-14T23:18:47.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4520000000000003e-06 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883416+0000", 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98965153299999997 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883428+0000", 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.203e-05 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.980142+0000", 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0967138160000001 2024-06-14T23:18:47.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.980281+0000", 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013922500000000001 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,6", 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.980435+0000", 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015324499999999999 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.992102+0000", 2024-06-14T23:18:47.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011667653 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003938+0000", 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011835513000000001 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003976+0000", 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.871e-05 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004005+0000", 2024-06-14T23:18:47.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8255e-05 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004099+0000", 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4136999999999997e-05 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004196+0000", 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7229e-05 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191071 13.5 13:af722e94:::benchmark_data_smithi086_299538_object70233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.905287+0000", 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.766759016999998, 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.108800741, 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191071 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.905287+0000", 2024-06-14T23:18:47.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.905287+0000", 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.905286+0000", 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.905292+0000", 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6860000000000003e-06 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.905304+0000", 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1948e-05 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.905319+0000", 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4857e-05 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.905518+0000", 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019929600000000001 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.905526+0000", 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3049999999999993e-06 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.893121+0000", 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98759503400000004 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.893127+0000", 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6280000000000002e-06 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.998459+0000", 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1053322590000001 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.998566+0000", 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010684900000000001 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,4", 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.998674+0000", 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107809 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006666+0000", 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079922929999999993 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.010007+0000", 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033406949999999999 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.010013+0000", 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5100000000000004e-06 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.013944+0000", 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039305709999999999 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.013988+0000", 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3615000000000002e-05 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.014088+0000", 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100319 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191080 13.9 13:93b2590c:::benchmark_data_smithi086_299538_object70242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.919370+0000", 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.752675941999996, 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0851213679999998, 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.728 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191080 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.919370+0000", 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.919370+0000", 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.919369+0000", 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.919374+0000", 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7110000000000003e-06 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.919385+0000", 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0801999999999999e-05 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.919389+0000", 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7350000000000004e-06 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.919445+0000", 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5948e-05 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.919453+0000", 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9850000000000006e-06 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883457+0000", 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96400332700000002 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883464+0000", 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5290000000000003e-06 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.982992+0000", 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0995278500000001 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.983139+0000", 2024-06-14T23:18:47.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000146761 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,6", 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.983269+0000", 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013015100000000001 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.992110+0000", 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088409179999999997 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004274+0000", 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012163953 2024-06-14T23:18:47.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004304+0000", 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0006999999999999e-05 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004332+0000", 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7664000000000001e-05 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004399+0000", 2024-06-14T23:18:47.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7421000000000004e-05 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004492+0000", 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2528000000000006e-05 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191082 13.9 13:9a2c247f:::benchmark_data_smithi086_299538_object70244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.920567+0000", 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.751479412999998, 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0842202630000002, 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.735 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191082 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.920567+0000", 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.920567+0000", 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.920566+0000", 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.920568+0000", 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9999999999999999e-06 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.920572+0000", 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5329999999999999e-06 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.920574+0000", 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4260000000000002e-06 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.920625+0000", 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0207999999999998e-05 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.920632+0000", 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2509999999999997e-06 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883488+0000", 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96285586599999995 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883495+0000", 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.883e-06 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.985645+0000", 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1021503509999999 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.985761+0000", 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116173 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,6", 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.985895+0000", 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000134203 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003636+0000", 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017740881 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004535+0000", 2024-06-14T23:18:47.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00089864899999999995 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004562+0000", 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7614e-05 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004616+0000", 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3236000000000003e-05 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004689+0000", 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2909000000000003e-05 2024-06-14T23:18:47.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004787+0000", 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8408999999999999e-05 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191083 13.9 13:93a1ba69:::benchmark_data_smithi086_299538_object70245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.921079+0000", 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.750967007, 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.084029498, 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.742 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191083 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.921079+0000", 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.921079+0000", 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.921078+0000", 2024-06-14T23:18:47.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:18:47.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:47.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.921083+0000", 2024-06-14T23:18:47.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9080000000000003e-06 2024-06-14T23:18:47.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.921095+0000", 2024-06-14T23:18:47.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2085e-05 2024-06-14T23:18:47.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.921101+0000", 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8769999999999998e-06 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.921143+0000", 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1610000000000003e-05 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.921150+0000", 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.199e-06 2024-06-14T23:18:47.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883520+0000", 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96237004500000001 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883528+0000", 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9799999999999998e-06 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.987710+0000", 2024-06-14T23:18:47.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1041823879999999 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.987829+0000", 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011896300000000001 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,6", 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.987955+0000", 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012556000000000001 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003642+0000", 2024-06-14T23:18:47.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015686927 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004589+0000", 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00094738099999999996 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004821+0000", 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023144300000000001 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004950+0000", 2024-06-14T23:18:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012915100000000001 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.005016+0000", 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6143999999999996e-05 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.005109+0000", 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2639000000000005e-05 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191090 13.9 13:9cfe4390:::benchmark_data_smithi086_299538_object70252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.926461+0000", 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.745584750000006, 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.078891579, 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191090 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.926461+0000", 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.926461+0000", 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.926460+0000", 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.926466+0000", 2024-06-14T23:18:47.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5050000000000004e-06 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.926479+0000", 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2478e-05 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.926484+0000", 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1880000000000001e-06 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.926547+0000", 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3360000000000003e-05 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.926556+0000", 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0089999999999996e-06 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883554+0000", 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95699770299999998 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883562+0000", 2024-06-14T23:18:47.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5639999999999997e-06 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.990148+0000", 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.106585739 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.990257+0000", 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108389 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,6", 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.990329+0000", 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2861999999999995e-05 2024-06-14T23:18:47.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003695+0000", 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013365836000000001 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.004921+0000", 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012254760000000001 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.005142+0000", 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022153799999999999 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.005197+0000", 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4898000000000002e-05 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.005261+0000", 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3484000000000004e-05 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.005353+0000", 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2264999999999998e-05 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.758 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191096 13.9 13:984f9f0e:::benchmark_data_smithi086_299538_object70258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.928728+0000", 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.743317619999999, 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0778869809999998, 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191096 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.928728+0000", 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.928728+0000", 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.928727+0000", 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.928733+0000", 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5500000000000002e-06 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.928746+0000", 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3223e-05 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.928752+0000", 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4859999999999997e-06 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.928798+0000", 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6665000000000001e-05 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.928807+0000", 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.303e-06 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883588+0000", 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95478174999999998 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.883596+0000", 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4850000000000003e-06 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.992152+0000", 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1085563 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.992257+0000", 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010437099999999999 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,6", 2024-06-14T23:18:47.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.992382+0000", 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012589900000000001 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003701+0000", 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011318379999999999 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.005170+0000", 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001469092 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006104+0000", 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00093389500000000002 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006164+0000", 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9809999999999998e-05 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006517+0000", 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000353326 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006615+0000", 2024-06-14T23:18:47.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8498999999999993e-05 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191102 13.9 13:960a7158:::benchmark_data_smithi086_299538_object70264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.934918+0000", 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.737127823999998, 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0719056280000001, 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.766 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191102 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.934918+0000", 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.934918+0000", 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.934917+0000", 2024-06-14T23:18:47.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.934922+0000", 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1699999999999996e-06 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.934932+0000", 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6770000000000007e-06 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.934937+0000", 2024-06-14T23:18:47.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0100000000000003e-06 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.935000+0000", 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2774000000000006e-05 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.935009+0000", 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3309999999999993e-06 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.890402+0000", 2024-06-14T23:18:47.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95539278599999999 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.890419+0000", 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7232999999999999e-05 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.995426+0000", 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.105006524 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.995576+0000", 2024-06-14T23:18:47.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000149906 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,6", 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.995726+0000", 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015066 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003706+0000", 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079795879999999993 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006135+0000", 2024-06-14T23:18:47.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024292490000000001 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006653+0000", 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00051728500000000003 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006681+0000", 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8795000000000001e-05 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006720+0000", 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9042000000000003e-05 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006824+0000", 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103556 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191108 13.5 13:a6fd28e8:::benchmark_data_smithi086_299538_object70270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.939884+0000", 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.732161755999996, 2024-06-14T23:18:47.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0743909999999999, 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191108 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.939884+0000", 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.939884+0000", 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.939884+0000", 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.939887+0000", 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8990000000000001e-06 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.939892+0000", 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4820000000000002e-06 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.939895+0000", 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0469999999999998e-06 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.939946+0000", 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0345e-05 2024-06-14T23:18:47.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.939953+0000", 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.644e-06 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.893143+0000", 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.953190027 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.893149+0000", 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3569999999999997e-06 2024-06-14T23:18:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.001193+0000", 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1080444389999999 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.001311+0000", 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118194 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,4", 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.001450+0000", 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000138262 2024-06-14T23:18:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.010019+0000", 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0085689649999999996 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.010057+0000", 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8813000000000003e-05 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.010088+0000", 2024-06-14T23:18:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1102000000000001e-05 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.014146+0000", 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040572899999999999 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.014185+0000", 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9580999999999999e-05 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.014275+0000", 2024-06-14T23:18:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0045999999999998e-05 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191141 13.5 13:aaf1c10d:::benchmark_data_smithi086_299538_object70303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:21.987140+0000", 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.684905622000002, 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0282231959999999, 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.781 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191141 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.987140+0000", 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.987140+0000", 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.987139+0000", 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.987146+0000", 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3769999999999998e-06 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.987162+0000", 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6059999999999999e-05 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.987169+0000", 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7290000000000003e-06 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.987221+0000", 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1900000000000001e-05 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:21.987229+0000", 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6059999999999997e-06 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.893165+0000", 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90593639500000001 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.893170+0000", 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4890000000000003e-06 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003773+0000", 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1106026520000001 2024-06-14T23:18:47.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003871+0000", 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8369999999999995e-05 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,4", 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003988+0000", 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116204 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.010024+0000", 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060365210000000004 2024-06-14T23:18:47.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.010119+0000", 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5076999999999998e-05 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.010338+0000", 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021894300000000001 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.014304+0000", 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039655690000000004 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.014339+0000", 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5089999999999998e-05 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.015364+0000", 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010248639999999999 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.788 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:191153 13.9 13:9b19ee15:::benchmark_data_smithi086_299538_object70315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:22.020742+0000", 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 85.651303706999997, 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9863386840000001, 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 191153 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.020742+0000", 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.020742+0000", 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.020741+0000", 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.020746+0000", 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0499999999999999e-06 2024-06-14T23:18:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.020757+0000", 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0708e-05 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.020761+0000", 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3150000000000002e-06 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.020801+0000", 2024-06-14T23:18:47.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9453000000000003e-05 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.020808+0000", 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6270000000000002e-06 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.890460+0000", 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86965283199999999 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:22.890468+0000", 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3950000000000003e-06 2024-06-14T23:18:47.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.998961+0000", 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1084927630000001 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.999064+0000", 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103329 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,6", 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:23.999168+0000", 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010433399999999999 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.003944+0000", 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004775746 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006860+0000", 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029159059999999998 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006888+0000", 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8580000000000001e-05 2024-06-14T23:18:47.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006942+0000", 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3987999999999999e-05 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.006988+0000", 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5943999999999999e-05 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:24.007081+0000", 2024-06-14T23:18:47.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2705000000000001e-05 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:218177 13.9 13:9fb1370d:::benchmark_data_smithi086_299538_object97339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:53.047548+0000", 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 54.624498127000003, 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0557280609999999, 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.797 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218177 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.047548+0000", 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.047548+0000", 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.047547+0000", 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.047551+0000", 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4789999999999998e-06 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.047558+0000", 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4429999999999997e-06 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.047562+0000", 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9779999999999999e-06 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.048539+0000", 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00097630700000000004 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.048548+0000", 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8890000000000008e-06 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.076120+0000", 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.027572186 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.076254+0000", 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013402499999999999 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.076387+0000", 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013281600000000001 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.084701+0000", 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0083145530000000006 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103140+0000", 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018439212 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103185+0000", 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4079000000000003e-05 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103236+0000", 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1065e-05 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103276+0000", 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0431e-05 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:218184 13.9 13:9b1a0d75:::benchmark_data_smithi086_299538_object97346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:53.055454+0000", 2024-06-14T23:18:47.803 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 54.616592066999999, 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.04798586, 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218184 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.055454+0000", 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.055454+0000", 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.055454+0000", 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.055456+0000", 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.543e-06 2024-06-14T23:18:47.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.055462+0000", 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6720000000000001e-06 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.055465+0000", 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.354e-06 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.055950+0000", 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048455299999999998 2024-06-14T23:18:47.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.055957+0000", 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2540000000000003e-06 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.079002+0000", 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0230450639999999 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.079081+0000", 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8620000000000003e-05 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.079157+0000", 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5809999999999994e-05 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.087816+0000", 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086596880000000005 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103322+0000", 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015505639 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103383+0000", 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1153000000000002e-05 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103412+0000", 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8581e-05 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103440+0000", 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8317e-05 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.809 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:218208 13.9 13:9cef9f72:::benchmark_data_smithi086_299538_object97370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:53.084525+0000", 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 54.587520986999998, 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0191108600000001, 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218208 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.084525+0000", 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.084525+0000", 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.084523+0000", 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.084530+0000", 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3559999999999998e-06 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.084544+0000", 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3968000000000001e-05 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.084550+0000", 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5529999999999998e-06 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.084597+0000", 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6860000000000002e-05 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.084605+0000", 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.14e-06 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.081222+0000", 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.996616218 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.081332+0000", 2024-06-14T23:18:47.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110874 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.081456+0000", 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012318100000000001 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.087822+0000", 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063661919999999997 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103357+0000", 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015534839999999999 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103539+0000", 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018218100000000001 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103602+0000", 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2794999999999998e-05 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103636+0000", 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4356000000000001e-05 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:218234 13.9 13:9222d050:::benchmark_data_smithi086_299538_object97396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:53.106779+0000", 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 54.565267573, 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99717761699999996, 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.816 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218234 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.106779+0000", 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.106779+0000", 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.106777+0000", 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.106782+0000", 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3809999999999998e-06 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.106792+0000", 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3540000000000006e-06 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.106797+0000", 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8559999999999998e-06 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.107551+0000", 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00075414700000000004 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.107558+0000", 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7560000000000002e-06 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.083090+0000", 2024-06-14T23:18:47.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.975531963 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.083185+0000", 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5291999999999995e-05 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.083269+0000", 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4178999999999995e-05 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.090858+0000", 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075892609999999999 2024-06-14T23:18:47.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103751+0000", 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012892898 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103849+0000", 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8010999999999994e-05 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103910+0000", 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0509e-05 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103956+0000", 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6480000000000002e-05 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:218256 13.9 13:97a54ac5:::benchmark_data_smithi086_299538_object97418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:17:53.128797+0000", 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 54.543249463999999, 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97538096299999999, 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:18:47.822 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 218256 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.128797+0000", 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.128797+0000", 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.128796+0000", 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.128799+0000", 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.892e-06 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.128805+0000", 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9610000000000002e-06 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.128809+0000", 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1949999999999996e-06 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.128854+0000", 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4523e-05 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:53.128860+0000", 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4579999999999998e-06 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.084744+0000", 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95588356699999999 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.084851+0000", 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107657 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.084978+0000", 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126728 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.090865+0000", 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058868159999999996 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.103804+0000", 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012938923 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.104109+0000", 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000305383 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:18:47.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.104136+0000", 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6262999999999999e-05 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:17:54.104178+0000", 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2027000000000003e-05 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:18:47.828 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:18:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:47.849 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:47.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:47.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool application enable unique_pool_9 rados --yes-i-really-mean-it || true 2024-06-14T23:18:47.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:47.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:47.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:48.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:48.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:48.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:48.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:48.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:48.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:47 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-06-14T23:18:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:47 smithi137 ceph-mon[90634]: osdmap e411: 8 total, 8 up, 7 in 2024-06-14T23:18:48.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:48.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:47 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-06-14T23:18:48.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:47 smithi086 ceph-mon[102754]: osdmap e411: 8 total, 8 up, 7 in 2024-06-14T23:18:48.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:47 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-06-14T23:18:48.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:47 smithi086 ceph-mon[97913]: osdmap e411: 8 total, 8 up, 7 in 2024-06-14T23:18:48.346 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:48.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:48.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:48.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:48.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:48.569 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:48.591 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:48.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:48.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:48.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:48.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:48.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:48.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-14T23:18:48.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:48.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:48.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:48.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:48.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:48.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:48.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:48.969 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:18:48.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:48 smithi137 ceph-mon[90634]: pgmap v1510: 119 pgs: 9 creating+peering, 7 unknown, 103 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:48.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:48 smithi137 ceph-mon[90634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:18:48.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:48 smithi137 ceph-mon[90634]: osdmap e412: 8 total, 8 up, 7 in 2024-06-14T23:18:48.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2851787628' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:48.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:48 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:49.027 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:49.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:49.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:49.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:49.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:49.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:49.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:49.210 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[97913]: pgmap v1510: 119 pgs: 9 creating+peering, 7 unknown, 103 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:49.210 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[97913]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:18:49.210 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[97913]: osdmap e412: 8 total, 8 up, 7 in 2024-06-14T23:18:49.210 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2851787628' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:49.210 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:49.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[102754]: pgmap v1510: 119 pgs: 9 creating+peering, 7 unknown, 103 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:49.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[102754]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-14T23:18:49.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[102754]: osdmap e412: 8 total, 8 up, 7 in 2024-06-14T23:18:49.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2851787628' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:49.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:48 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:18:49.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:49.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:49.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:49.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:49.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:49.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:49.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:49.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:49.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:49.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:49.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:49.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:49.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:49.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:49.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:49.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:49.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:49.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:50.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:50.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:50.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:50.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:50.196 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:50.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:50.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:49 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:18:50.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:49 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:18:50.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:49 smithi137 ceph-mon[90634]: osdmap e413: 8 total, 8 up, 7 in 2024-06-14T23:18:50.274 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:49 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:18:50.274 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:49 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:18:50.274 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:49 smithi086 ceph-mon[102754]: osdmap e413: 8 total, 8 up, 7 in 2024-06-14T23:18:50.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:49 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:18:50.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:49 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:18:50.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:49 smithi086 ceph-mon[97913]: osdmap e413: 8 total, 8 up, 7 in 2024-06-14T23:18:50.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:50.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:50.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:50.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:50.443 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:50.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:50.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:50.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:50.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:50.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:50.603 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:50.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:50.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:50.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:50.732 INFO:teuthology.orchestra.run.smithi086.stdout:moved 0 / 357 (0%) 2024-06-14T23:18:50.732 INFO:teuthology.orchestra.run.smithi086.stdout:avg 51 2024-06-14T23:18:50.732 INFO:teuthology.orchestra.run.smithi086.stdout:stddev 5.68205 -> 5.68205 (expected baseline 6.61168) 2024-06-14T23:18:50.732 INFO:teuthology.orchestra.run.smithi086.stdout:min osd.2 with 41 -> 41 pgs (0.803922 -> 0.803922 * mean) 2024-06-14T23:18:50.732 INFO:teuthology.orchestra.run.smithi086.stdout:max osd.3 with 59 -> 59 pgs (1.15686 -> 1.15686 * mean) 2024-06-14T23:18:50.732 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:18:50.732 INFO:teuthology.orchestra.run.smithi086.stdout:oload 1000 2024-06-14T23:18:50.733 INFO:teuthology.orchestra.run.smithi086.stdout:max_change 0.05 2024-06-14T23:18:50.733 INFO:teuthology.orchestra.run.smithi086.stdout:max_change_osds 4 2024-06-14T23:18:50.733 INFO:teuthology.orchestra.run.smithi086.stdout:average_utilization 0.0120 2024-06-14T23:18:50.733 INFO:teuthology.orchestra.run.smithi086.stdout:overload_utilization 0.1205 2024-06-14T23:18:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:no change 2024-06-14T23:18:50.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:50.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:50.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:50.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:50.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:50.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:50.964 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:18:50.964 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:18:50.964 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-06-14T23:18:50.964 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:18:51.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:51.062 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:51.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:51.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:51.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:51.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:50 smithi137 ceph-mon[90634]: pgmap v1513: 119 pgs: 9 creating+peering, 3 unknown, 107 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:51.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:51.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:51.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:50 smithi086 ceph-mon[97913]: pgmap v1513: 119 pgs: 9 creating+peering, 3 unknown, 107 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:51.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:50 smithi086 ceph-mon[102754]: pgmap v1513: 119 pgs: 9 creating+peering, 3 unknown, 107 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:51.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:51.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:51.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:51.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:51.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:51.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:51.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-14T23:18:51.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:51.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:51.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:51.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:51.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:51.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:51.852 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:51.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:51.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:51.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:52.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:52.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:52.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:52.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:52.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:52.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:51 smithi137 ceph-mon[90634]: from='client.37266 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:52.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:52.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:18:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:52.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:52.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:52.323 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:51 smithi086 ceph-mon[97913]: from='client.37266 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:52.323 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:51 smithi086 ceph-mon[102754]: from='client.37266 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-14T23:18:52.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:52.424 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:52.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:52.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:52.546 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:52.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:52.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:52.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:52.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:52.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:52.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:52.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:52.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:52.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:52.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:53.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:53.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:53.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:52 smithi086 ceph-mon[97913]: pgmap v1514: 119 pgs: 9 creating+peering, 110 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:53.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:52 smithi086 ceph-mon[102754]: pgmap v1514: 119 pgs: 9 creating+peering, 110 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:53.135 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:52 smithi137 ceph-mon[90634]: pgmap v1514: 119 pgs: 9 creating+peering, 110 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:53.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:53.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:53.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:53.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:53.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:53.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:53.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:53.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:53.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:53.503 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:18:53.503 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:18:53.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:53.520 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:18:53.521 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:18:53.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:53.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:53.536 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:18:53.537 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:18:53.553 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:18:53.553 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:18:53.570 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:18:53.570 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:18:53.586 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:18:53.586 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:18:53.602 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:18:53.603 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:18:53.619 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:18:53.620 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:18:53.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:53.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:53.694 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:18:53.694 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:18:53.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:53.780 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:53.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:53.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:53.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:53.986 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_9' 2024-06-14T23:18:54.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:54.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:54.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:54.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:54.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:54.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:54.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2534104565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:54.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2547162308' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:18:54.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:54.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:54.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:54.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2534104565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:54.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2547162308' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:18:54.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2534104565' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:18:54.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2547162308' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-14T23:18:54.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:54.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:54.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:54.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:54.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-14T23:18:54.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:54.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:54.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:54.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:54.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:54.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:54.689 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:54.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:54.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:54.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:54.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:54.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:54.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:54.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:55.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:55.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:55.112 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:54 smithi137 ceph-mon[90634]: pgmap v1515: 119 pgs: 119 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:55.113 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:54 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2547162308' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:18:55.113 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:54 smithi137 ceph-mon[90634]: osdmap e414: 8 total, 8 up, 7 in 2024-06-14T23:18:55.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:55.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:55.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:55.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:55.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:55.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:55.279 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:54 smithi086 ceph-mon[97913]: pgmap v1515: 119 pgs: 119 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:55.279 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2547162308' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:18:55.279 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:54 smithi086 ceph-mon[97913]: osdmap e414: 8 total, 8 up, 7 in 2024-06-14T23:18:55.279 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:54 smithi086 ceph-mon[102754]: pgmap v1515: 119 pgs: 119 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:55.279 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2547162308' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-14T23:18:55.280 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:54 smithi086 ceph-mon[102754]: osdmap e414: 8 total, 8 up, 7 in 2024-06-14T23:18:55.280 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:18:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:18:54] "GET /metrics HTTP/1.1" 200 34270 "" "Prometheus/2.43.0" 2024-06-14T23:18:55.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:55.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:55.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:55.475 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:55.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:55.510 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:55.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:55.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:55.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:55.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:55.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:55.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:55.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:55.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:55.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:55.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:55.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:55.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:55.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:56.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:56.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:56.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:56.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:56.215 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:56 smithi137 ceph-mon[90634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:18:56.216 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:56 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:18:56.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:56.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:56.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:56.309 DEBUG:teuthology.orchestra.run.smithi086:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_9 bench 90 write' 2024-06-14T23:18:56.312 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-14T23:18:56.312 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-14T23:18:56.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:56 smithi086 ceph-mon[97913]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:18:56.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:56 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:18:56.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:56 smithi086 ceph-mon[102754]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-14T23:18:56.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:56 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:18:56.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:56.357 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:hints = 1 2024-06-14T23:18:56.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-14T23:18:56.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object prefix: benchmark_data_smithi086_333440 2024-06-14T23:18:56.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:18:56.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 0 0 0 0 0 0 - 0 2024-06-14T23:18:56.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:56.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:56.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:56.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:56.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:56.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:56.547 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:18:56.548 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:18:56.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:18:56.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:56.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:56.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:56.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:56.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:56.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:56.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:18:56.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:56.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:56.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:18:57.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:18:57.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:57.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:18:57.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:57 smithi086 ceph-mon[97913]: pgmap v1517: 119 pgs: 119 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:57.105 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:57 smithi086 ceph-mon[102754]: pgmap v1517: 119 pgs: 119 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:57.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:57.230 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:57.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:57.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:57 smithi137 ceph-mon[90634]: pgmap v1517: 119 pgs: 119 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail 2024-06-14T23:18:57.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:57.357 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 1 16 3912 3896 243.487 243.5 0.00661384 0.00409373 2024-06-14T23:18:57.368 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:57.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:57.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:57.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:57.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T23:18:57.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:57.619 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:18:57.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:57.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:57.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:57.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:57.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:57.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:57.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:57.958 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:57.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:57.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:57.977 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:18:58.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:58.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:58.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:58.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:58.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:58.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:58.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:58.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:58.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:58.357 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 2 16 6844 6828 213.36 183.25 0.00117002 0.00465722 2024-06-14T23:18:58.409 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:18:58.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:58.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:58.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:58.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:58.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:58.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:58.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:58.674 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:58.674 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:58.674 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:58 smithi137 ceph-mon[90634]: pgmap v1518: 119 pgs: 119 active+clean; 139 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 16 MiB/s wr, 250 op/s 2024-06-14T23:18:58.674 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1112051978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:58.674 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:58 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:58.674 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:58.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:58.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:58.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:58.821 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:58.821 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:58.821 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[102754]: pgmap v1518: 119 pgs: 119 active+clean; 139 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 16 MiB/s wr, 250 op/s 2024-06-14T23:18:58.821 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1112051978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:58.821 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:58.821 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:58.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:58.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:18:58.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[97913]: pgmap v1518: 119 pgs: 119 active+clean; 139 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 16 MiB/s wr, 250 op/s 2024-06-14T23:18:58.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1112051978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:58.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:18:58.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:18:58.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:18:58.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:58.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:18:59.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:59.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:59.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:59.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:18:59.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:59.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:18:59.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:18:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:59.343 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:59.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:18:59.357 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 3 16 10017 10001 208.339 198.312 0.00549766 0.00479357 2024-06-14T23:18:59.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:18:59.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:59.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:59.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:18:59.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:18:59.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:18:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:59.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:18:59.629 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:59 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:59.630 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:59 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:18:59.630 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:18:59 smithi137 ceph-mon[90634]: osdmap e415: 8 total, 8 up, 7 in 2024-06-14T23:18:59.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:59.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:59.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:59.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:59.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:59 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:59.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:59 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:18:59.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:18:59 smithi086 ceph-mon[97913]: osdmap e415: 8 total, 8 up, 7 in 2024-06-14T23:18:59.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:59 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:18:59.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:59 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:18:59.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:18:59 smithi086 ceph-mon[102754]: osdmap e415: 8 total, 8 up, 7 in 2024-06-14T23:18:59.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:18:59.883 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:18:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:18:59.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:18:59.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:00.019 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:00.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:00.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:00.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:00.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:00.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:00.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:00.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:00.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 4 16 13177 13161 205.625 197.5 0.00329644 0.00485533 2024-06-14T23:19:00.366 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:19:00.366 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:19:00.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:00.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:00.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:00.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:00.463 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 2, 0] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:19:00.463 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:19:00.463 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [3, 4, 5, 6, 7, 2, 0] 2024-06-14T23:19:00.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd out 2 2024-06-14T23:19:00.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:00 smithi137 ceph-mon[90634]: pgmap v1520: 119 pgs: 119 active+clean; 266 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 33 MiB/s wr, 530 op/s 2024-06-14T23:19:00.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:00.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:00.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:00.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:00 smithi086 ceph-mon[97913]: pgmap v1520: 119 pgs: 119 active+clean; 266 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 33 MiB/s wr, 530 op/s 2024-06-14T23:19:00.698 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:00 smithi086 ceph-mon[102754]: pgmap v1520: 119 pgs: 119 active+clean; 266 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 33 MiB/s wr, 530 op/s 2024-06-14T23:19:00.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:00.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:00.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:00.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:00.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:00.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:00.949 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:00.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:00.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:01.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:01.123 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:01.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:01.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:01.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:01.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:01.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:01.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:01.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-14T23:19:01.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 5 16 16241 16225 202.797 191.5 0.00645978 0.00492464 2024-06-14T23:19:01.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:01.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:01.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:01.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:01.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:01.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:01.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:01.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:01.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:01.773 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:19:01.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:19:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:19:01.835 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:19:01.835 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:19:01.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:01.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:01.867 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:19:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:19:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:19:01.902 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:19:01.903 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2673920342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:01.903 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2673920342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:01.932 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:19:01.932 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:19:01.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:01.961 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:19:01.961 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:19:01.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:01.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:01.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:02.000 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:19:02.000 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:19:02.032 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:19:02.033 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:19:02.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:02.105 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2673920342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:02.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:02.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:02.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:02.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:02.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:02.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:02.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:02.357 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 6 16 19052 19036 198.276 175.688 0.00413916 0.0050384 2024-06-14T23:19:02.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:02.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:02.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:02.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:02.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:02.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:02.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:02.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:02.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:19:02.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:02.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:02.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:02.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:02.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:02.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:02.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:02 smithi137 ceph-mon[90634]: pgmap v1521: 119 pgs: 119 active+clean; 428 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 53 MiB/s wr, 855 op/s 2024-06-14T23:19:03.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:03.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:02 smithi086 ceph-mon[97913]: pgmap v1521: 119 pgs: 119 active+clean; 428 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 53 MiB/s wr, 855 op/s 2024-06-14T23:19:03.084 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:02 smithi086 ceph-mon[102754]: pgmap v1521: 119 pgs: 119 active+clean; 428 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 53 MiB/s wr, 855 op/s 2024-06-14T23:19:03.102 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:03.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:03.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:03.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:03.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:03.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:03.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 7 16 21836 21820 194.807 174 0.00667243 0.00512714 2024-06-14T23:19:03.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:03.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:03.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:03.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:03.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:03.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:03.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:03.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:03.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:03.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:03.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:03.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:03.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:03.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:04.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:04.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:04.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:04.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:04.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:04.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:04.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:04.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:04.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:04.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:04.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 8 16 24350 24334 190.095 157.125 0.00452464 0.00525513 2024-06-14T23:19:04.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:04.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:04.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:04.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:04.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:04.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:04.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:04.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:04.754 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:04.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:04.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:04.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:04.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:04.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:04.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:05.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:05.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:05.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:05.097 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:04 smithi137 ceph-mon[90634]: pgmap v1522: 119 pgs: 119 active+clean; 1.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-06-14T23:19:05.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:05.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:05.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:05.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:05.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:05.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:05.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:04 smithi086 ceph-mon[97913]: pgmap v1522: 119 pgs: 119 active+clean; 1.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-06-14T23:19:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:04 smithi086 ceph-mon[102754]: pgmap v1522: 119 pgs: 119 active+clean; 1.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-06-14T23:19:05.338 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:19:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:19:04] "GET /metrics HTTP/1.1" 200 36605 "" "Prometheus/2.43.0" 2024-06-14T23:19:05.348 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:19:05.348 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:19:05.349 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:19:05.349 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:19:05.349 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:19:05.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 9 16 26771 26755 185.784 151.312 0.00157161 0.00534112 2024-06-14T23:19:05.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:05.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:05.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:05.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:05.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:05.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:05.710 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:05.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:05.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:05.852 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:05.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:05.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:05.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:05.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:06.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:06.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:06.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:06.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:06.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:06.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-14T23:19:06.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 10 16 29223 29207 182.529 153.25 0.00191965 0.00546803 2024-06-14T23:19:06.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:06.398 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:06.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:06.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:06.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:06.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:06.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:06.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:06.758 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:19:06.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:06.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:06.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:06.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:06.955 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:06.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:07.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:07.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:07.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:07.181 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:07.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:07.242 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:07.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:06 smithi137 ceph-mon[90634]: pgmap v1523: 119 pgs: 119 active+clean; 1.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-06-14T23:19:07.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:06 smithi086 ceph-mon[97913]: pgmap v1523: 119 pgs: 119 active+clean; 1.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-06-14T23:19:07.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:06 smithi086 ceph-mon[102754]: pgmap v1523: 119 pgs: 119 active+clean; 1.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 136 MiB/s wr, 2.17k op/s 2024-06-14T23:19:07.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:07.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 11 16 31308 31292 177.781 130.312 0.00283846 0.00561871 2024-06-14T23:19:07.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:07.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:07.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:07.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:07.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:07.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:07.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:07.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:07.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:07.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:07.792 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:07.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:07.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:07.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:07.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:07.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:08.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:08.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:08.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:08.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:08.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:08.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:08.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:08.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:08.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:08.358 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 12 16 33216 33200 172.903 119.25 0.00483162 0.0057781 2024-06-14T23:19:08.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:08.457 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:08.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:08.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:08.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:08.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:08.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:08.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:08.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:08.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:08.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:08.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:09.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:09.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:19:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:09.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:09.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:09.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:08 smithi086 ceph-mon[102754]: pgmap v1524: 119 pgs: 119 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-06-14T23:19:09.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:08 smithi086 ceph-mon[97913]: pgmap v1524: 119 pgs: 119 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-06-14T23:19:09.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:09.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:09.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:09.241 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:08 smithi137 ceph-mon[90634]: pgmap v1524: 119 pgs: 119 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 164 MiB/s wr, 2.63k op/s 2024-06-14T23:19:09.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:09.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:09.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:09.359 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 13 16 35305 35289 169.638 130.562 0.0066723 0.00588878 2024-06-14T23:19:09.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:09.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:09.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:09.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:09.637 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:09.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:09.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:09 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:19:09.372+0000 7fe0161db640 -1 mon.a@0(leader).osd e415 definitely_dead 0 2024-06-14T23:19:09.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:09.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:129230 12.5 12:a066fbee:::benchmark_data_smithi086_261492_object12725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:46.019841+0000", 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 443.75055924399999, 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.31967097700000002, 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.777 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 129230 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019841+0000", 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019841+0000", 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019840+0000", 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019845+0000", 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4879999999999998e-06 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019857+0000", 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1355999999999999e-05 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.019862+0000", 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0019999999999997e-06 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.212112+0000", 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.192250002 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:19:09.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.214077+0000", 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019654690000000001 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.242751+0000", 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028673700999999999 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.242841+0000", 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0150999999999998e-05 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:19:09.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.242981+0000", 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014017500000000001 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.288886+0000", 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045905051000000002 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.333634+0000", 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.044748084 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.333679+0000", 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4746999999999997e-05 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.333735+0000", 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5760999999999997e-05 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.339512+0000", 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057774380000000002 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.783 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:129255 12.5 12:aa1ad356:::benchmark_data_smithi086_261492_object12750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:46.061829+0000", 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 443.70857112900001, 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28357850400000001, 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 129255 2024-06-14T23:19:09.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061829+0000", 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061829+0000", 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061828+0000", 2024-06-14T23:19:09.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061834+0000", 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2260000000000002e-06 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061846+0000", 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2514000000000001e-05 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.061852+0000", 2024-06-14T23:19:09.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.169e-06 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.214111+0000", 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.152259175 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.214693+0000", 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058183400000000004 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.262221+0000", 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047528473000000002 2024-06-14T23:19:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.262306+0000", 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5204000000000006e-05 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.262424+0000", 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118159 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.288893+0000", 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026468814 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.339567+0000", 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050673608000000002 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.339786+0000", 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021960999999999999 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.339882+0000", 2024-06-14T23:19:09.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5200999999999999e-05 2024-06-14T23:19:09.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:09.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.345408+0000", 2024-06-14T23:19:09.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055261850000000003 2024-06-14T23:19:09.791 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.791 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.791 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:129271 12.5 12:ab5a4303:::benchmark_data_smithi086_261492_object12766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:11:46.081893+0000", 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 443.688507388, 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26981149900000001, 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 129271 2024-06-14T23:19:09.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081893+0000", 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081893+0000", 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081892+0000", 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081897+0000", 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3229999999999999e-06 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081909+0000", 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2832e-05 2024-06-14T23:19:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.081923+0000", 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3682e-05 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.214723+0000", 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.132799429 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.215997+0000", 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012744060000000001 2024-06-14T23:19:09.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.285772+0000", 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.069775167999999999 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.285855+0000", 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2842000000000004e-05 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.285986+0000", 2024-06-14T23:19:09.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000130615 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.328223+0000", 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.042237784 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.345483+0000", 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017259535999999999 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.345522+0000", 2024-06-14T23:19:09.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8757000000000002e-05 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.345572+0000", 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0362000000000003e-05 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:11:46.351705+0000", 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061325850000000003 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:152612 12.5 12:a222237b:::benchmark_data_smithi086_261492_object36107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:12:20.906053+0000", 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 408.86434780899998, 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.24355275700000001, 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 152612 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906053+0000", 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906053+0000", 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906052+0000", 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906057+0000", 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8000000000000004e-06 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906069+0000", 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1222e-05 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:20.906074+0000", 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5509999999999999e-06 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.029226+0000", 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12315158800000001 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.032720+0000", 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034940399999999999 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.067028+0000", 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034307799 2024-06-14T23:19:09.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.067125+0000", 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7015000000000005e-05 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3", 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.067265+0000", 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014023400000000001 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.093412+0000", 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026147169000000001 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.145382+0000", 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.051969862999999998 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.145450+0000", 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8207000000000006e-05 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.145516+0000", 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5505000000000004e-05 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:12:21.149605+0000", 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040898130000000003 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:238522 13.f 13:f94cb559:::benchmark_data_smithi086_299538_object117684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-06-14T23:19:09.805 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:18:16.882075+0000", 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.888325539999997, 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.068122243999999998, 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 238522 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882075+0000", 2024-06-14T23:19:09.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882075+0000", 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882074+0000", 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882077+0000", 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.559e-06 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882083+0000", 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5439999999999998e-06 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882086+0000", 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3110000000000001e-06 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882955+0000", 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00086958599999999999 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.883056+0000", 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010033 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.883152+0000", 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6323999999999995e-05 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.889573+0000", 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006420879 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.892127+0000", 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025542519999999999 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.950109+0000", 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.057981932 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.950157+0000", 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8352000000000001e-05 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.950197+0000", 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9685999999999999e-05 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:238523 13.f 13:f365b67a:::benchmark_data_smithi086_299538_object117685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-06-14T23:19:09.811 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:18:16.882100+0000", 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.888300237000003, 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.068205979999999999, 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 238523 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882100+0000", 2024-06-14T23:19:09.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882100+0000", 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882100+0000", 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882102+0000", 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3800000000000001e-06 2024-06-14T23:19:09.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882108+0000", 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7939999999999999e-06 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.882112+0000", 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5379999999999998e-06 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.886339+0000", 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042272610000000004 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.886430+0000", 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0680999999999995e-05 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.886530+0000", 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100543 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.892153+0000", 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056225709999999998 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.899945+0000", 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077924120000000003 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.950234+0000", 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050288963999999998 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.950273+0000", 2024-06-14T23:19:09.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9072999999999997e-05 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.950306+0000", 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3086999999999997e-05 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:238535 13.f 13:f7763519:::benchmark_data_smithi086_299538_object117697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:18:16.895597+0000", 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.874803098999998, 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.058788276, 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.817 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 238535 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.895597+0000", 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.895597+0000", 2024-06-14T23:19:09.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.895596+0000", 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.895603+0000", 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0820000000000001e-06 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.895616+0000", 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2678e-05 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.895622+0000", 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9739999999999999e-06 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.895664+0000", 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2752e-05 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.895778+0000", 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114171 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.895890+0000", 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111907 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.899975+0000", 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040847369999999997 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.951679+0000", 2024-06-14T23:19:09.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.051704181000000002 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.954223+0000", 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025437580000000001 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.954348+0000", 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012496499999999999 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.954386+0000", 2024-06-14T23:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7669999999999997e-05 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46844.0:238530 13.7 e405/351 13:e9e33ce0:::benchmark_data_smithi086_299538_object117692:head v 405'15034, mlcod=405'15031)", 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:18:16.897297+0000", 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.873103172999997, 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.058664422000000001, 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897297+0000", 2024-06-14T23:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897297+0000", 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897296+0000", 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897323+0000", 2024-06-14T23:19:09.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6322999999999999e-05 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897333+0000", 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0709e-05 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897339+0000", 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3449999999999996e-06 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.899986+0000", 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026467830000000002 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.900002+0000", 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6033e-05 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.955876+0000", 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.055874839000000003 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.955962+0000", 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5351000000000003e-05 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:238536 13.f 13:fb056852:::benchmark_data_smithi086_299538_object117698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:18:16.897488+0000", 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.872912937000002, 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.058683240999999997, 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:19:09.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 238536 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897488+0000", 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897488+0000", 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897486+0000", 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897492+0000", 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6210000000000001e-06 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897503+0000", 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0577999999999999e-05 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.897507+0000", 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8060000000000002e-06 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.946189+0000", 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.048681467999999999 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.946285+0000", 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5848999999999997e-05 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-06-14T23:19:09.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.946814+0000", 2024-06-14T23:19:09.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052965200000000003 2024-06-14T23:19:09.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.955973+0000", 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091589380000000001 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.956003+0000", 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9924999999999999e-05 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.956090+0000", 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6939000000000002e-05 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.956138+0000", 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7978e-05 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.956171+0000", 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2703999999999997e-05 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46844.0:238537 13.0 e405/351 13:07255e26:::benchmark_data_smithi086_299538_object117699:head v 405'14993, mlcod=405'14991)", 2024-06-14T23:19:09.833 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:18:16.898858+0000", 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.871542794, 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.058002933, 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.898858+0000", 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.898858+0000", 2024-06-14T23:19:09.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.898857+0000", 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.898862+0000", 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8080000000000004e-06 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.898870+0000", 2024-06-14T23:19:09.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9270000000000005e-06 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.898874+0000", 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9280000000000003e-06 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.900731+0000", 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001857156 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.900748+0000", 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7289e-05 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.956766+0000", 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.056017847000000003 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.956861+0000", 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4398e-05 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.46844.0:238544 13.8 e405/351 13:11bd50f8:::benchmark_data_smithi086_299538_object117706:head v 405'14803, mlcod=405'14801)", 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:18:16.904418+0000", 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 52.865982162999998, 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.055106885000000001, 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.904418+0000", 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.904418+0000", 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.904417+0000", 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.904425+0000", 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0469999999999994e-06 2024-06-14T23:19:09.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.946158+0000", 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.04173296 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.946173+0000", 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4888e-05 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.946209+0000", 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5774e-05 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.946230+0000", 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1229999999999998e-05 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.959477+0000", 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013247108 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:18:16.959525+0000", 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8007000000000003e-05 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:30860 14.2 14:4ef0fdd6:::benchmark_data_smithi086_333440_object30859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:07.116869+0000", 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.6535316440000001, 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.052043081999999997, 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 30860 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.116869+0000", 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.116869+0000", 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.116868+0000", 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.116898+0000", 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0239e-05 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.116907+0000", 2024-06-14T23:19:09.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.679e-06 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.116913+0000", 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2610000000000001e-06 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.120572+0000", 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036592420000000001 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.120684+0000", 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011191100000000001 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.120811+0000", 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127179 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.123255+0000", 2024-06-14T23:19:09.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024436810000000001 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.123426+0000", 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000170936 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.168841+0000", 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045415367999999998 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.168880+0000", 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8850000000000002e-05 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.168912+0000", 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2005999999999998e-05 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:30868 14.2 14:49885197:::benchmark_data_smithi086_333440_object30867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:07.119567+0000", 2024-06-14T23:19:09.848 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.6508334950000001, 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.051058962999999999, 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 30868 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.119567+0000", 2024-06-14T23:19:09.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.119567+0000", 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.119565+0000", 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:19:09.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.119638+0000", 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2624000000000001e-05 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.119648+0000", 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0560000000000001e-05 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.119654+0000", 2024-06-14T23:19:09.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1089999999999997e-06 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.121896+0000", 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022425470000000001 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.122082+0000", 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000186231 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:19:09.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.122238+0000", 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015571899999999999 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.123317+0000", 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001079441 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.123515+0000", 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000198023 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.170579+0000", 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047063702999999998 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.170607+0000", 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7946999999999999e-05 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.170626+0000", 2024-06-14T23:19:09.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8816000000000001e-05 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:32362 14.2 14:458aae84:::benchmark_data_smithi086_333440_object32361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:07.853889+0000", 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9165117089999999, 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059823622999999999, 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:19:09.855 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 32362 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853889+0000", 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853889+0000", 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853887+0000", 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853929+0000", 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2323000000000001e-05 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853945+0000", 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5821e-05 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853951+0000", 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8900000000000004e-06 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854290+0000", 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033893299999999998 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854400+0000", 2024-06-14T23:19:09.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110005 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854538+0000", 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000138579 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.860238+0000", 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056992170000000003 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.862739+0000", 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025013499999999998 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.913635+0000", 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050895689000000001 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.913677+0000", 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2404999999999998e-05 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.913712+0000", 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5233e-05 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:32363 14.2 14:4c0307a3:::benchmark_data_smithi086_333440_object32362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:07.853963+0000", 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.91643765, 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059849859999999998, 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.861 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 32363 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853963+0000", 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853963+0000", 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853962+0000", 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.853994+0000", 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1625000000000003e-05 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854001+0000", 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6619999999999999e-06 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854004+0000", 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7019999999999999e-06 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.855926+0000", 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001922153 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.857233+0000", 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013063840000000001 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.857361+0000", 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000128181 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.862713+0000", 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053520809999999999 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.862746+0000", 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2907999999999997e-05 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.913743+0000", 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050997426999999998 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.913784+0000", 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0846999999999997e-05 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.913813+0000", 2024-06-14T23:19:09.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8419e-05 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:32365 14.4 14:2b8740d8:::benchmark_data_smithi086_333440_object32364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:07.854062+0000", 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.916338369, 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.057282601000000002, 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 32365 2024-06-14T23:19:09.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854062+0000", 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854062+0000", 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854062+0000", 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:09.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854090+0000", 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8708999999999999e-05 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854096+0000", 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.993e-06 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.854100+0000", 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4809999999999998e-06 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.855909+0000", 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001808748 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.856034+0000", 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012576300000000001 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.856168+0000", 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013400599999999999 2024-06-14T23:19:09.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.859301+0000", 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003132569 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.862203+0000", 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029023159999999998 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.872 INFO:teuthology.orchestra.run.smithi086.stderr:marked out osd.2. 2024-06-14T23:19:09.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.911076+0000", 2024-06-14T23:19:09.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.048872602000000001 2024-06-14T23:19:09.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.911307+0000", 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000230838 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.911345+0000", 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8000999999999998e-05 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:32366 14.2 14:4416ec5a:::benchmark_data_smithi086_333440_object32365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:07.857969+0000", 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9124313420000001, 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059473817999999998, 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 32366 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.857969+0000", 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.857969+0000", 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.857968+0000", 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.858011+0000", 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3377999999999997e-05 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.858020+0000", 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1810000000000007e-06 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.858026+0000", 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4450000000000004e-06 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.860260+0000", 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022346290000000001 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.860374+0000", 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113219 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.860505+0000", 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013189699999999999 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.862750+0000", 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002244561 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.863331+0000", 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058075900000000003 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.917355+0000", 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.054024537999999997 2024-06-14T23:19:09.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.917405+0000", 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9948999999999999e-05 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.917443+0000", 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7673000000000003e-05 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.879 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:32370 14.2 14:4a5c8430:::benchmark_data_smithi086_333440_object32369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:07.858031+0000", 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9123693149999998, 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059527038999999997, 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 32370 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.858031+0000", 2024-06-14T23:19:09.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.858031+0000", 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.858031+0000", 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.858070+0000", 2024-06-14T23:19:09.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9258000000000003e-05 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.858077+0000", 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9319999999999999e-06 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.858081+0000", 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8539999999999999e-06 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.861566+0000", 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003485546 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.861673+0000", 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010702200000000001 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.861798+0000", 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124895 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.862895+0000", 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010964880000000001 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.863470+0000", 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00057509300000000005 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.917486+0000", 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.054016119000000001 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.917528+0000", 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.176e-05 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.917558+0000", 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0521000000000001e-05 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:32392 14.2 14:420a0cb1:::benchmark_data_smithi086_333440_object32391:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:09.885 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:07.915434+0000", 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.8549669369999999, 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.110389847, 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 32392 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.915434+0000", 2024-06-14T23:19:09.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.915434+0000", 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.915432+0000", 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.915471+0000", 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8299e-05 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.915485+0000", 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4312e-05 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.915491+0000", 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0390000000000001e-06 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.915622+0000", 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013086200000000001 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.915731+0000", 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109467 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-06-14T23:19:09.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.915855+0000", 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012424300000000001 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.917793+0000", 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019375799999999999 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.924572+0000", 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067785969999999999 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:08.025722+0000", 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.101149986 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:08.025778+0000", 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6212000000000003e-05 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:08.025823+0000", 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5528000000000002e-05 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.891 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:32420 14.4 14:2da36289:::benchmark_data_smithi086_333440_object32419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:07.927475+0000", 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.8429257269999999, 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.102613141, 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 32420 2024-06-14T23:19:09.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.927475+0000", 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.927475+0000", 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.927473+0000", 2024-06-14T23:19:09.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.927543+0000", 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9208000000000006e-05 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.927553+0000", 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8730000000000002e-06 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.927558+0000", 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2639999999999999e-06 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.927593+0000", 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5576e-05 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.927675+0000", 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1787999999999997e-05 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.927754+0000", 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8719000000000003e-05 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.928840+0000", 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010856049999999999 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:07.929249+0000", 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040969999999999998 2024-06-14T23:19:09.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:08.029979+0000", 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.100729584 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:08.030052+0000", 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3502000000000002e-05 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:08.030088+0000", 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5596000000000003e-05 2024-06-14T23:19:09.897 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.898 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.898 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.898 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:19:09.898 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:19:09.898 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:19:09.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:09.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:09.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:10.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:10.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:10.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:10.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2358360513' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:19:10.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:09 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:19:10.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:09 smithi086 ceph-mon[97913]: Client client.admin marked osd.2 out, while it was still marked up 2024-06-14T23:19:10.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2358360513' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:19:10.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:09 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:19:10.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:09 smithi086 ceph-mon[102754]: Client client.admin marked osd.2 out, while it was still marked up 2024-06-14T23:19:10.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:10.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:10.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2358360513' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:19:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:09 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-14T23:19:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:09 smithi137 ceph-mon[90634]: Client client.admin marked osd.2 out, while it was still marked up 2024-06-14T23:19:10.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:10.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:10.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:10.359 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 14 16 36384 36368 162.337 67.4375 0.00278225 0.00595258 2024-06-14T23:19:10.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:10.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:10.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:10.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:10.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:10.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:10.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:10.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:10.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:10.704 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:10.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:10.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:10.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:10.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:10.909 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:10.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:10.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:11.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:11.033 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:11.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:11.126 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:10 smithi137 ceph-mon[90634]: pgmap v1525: 119 pgs: 119 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-06-14T23:19:11.126 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:10 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-06-14T23:19:11.126 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:10 smithi137 ceph-mon[90634]: osdmap e416: 8 total, 8 up, 6 in 2024-06-14T23:19:11.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T23:19:11.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:11.149 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:10 smithi086 ceph-mon[97913]: pgmap v1525: 119 pgs: 119 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-06-14T23:19:11.149 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:10 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-06-14T23:19:11.149 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:10 smithi086 ceph-mon[97913]: osdmap e416: 8 total, 8 up, 6 in 2024-06-14T23:19:11.149 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:10 smithi086 ceph-mon[102754]: pgmap v1525: 119 pgs: 119 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 159 MiB/s wr, 2.55k op/s 2024-06-14T23:19:11.150 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:10 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-06-14T23:19:11.150 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:10 smithi086 ceph-mon[102754]: osdmap e416: 8 total, 8 up, 6 in 2024-06-14T23:19:11.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:11.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:11.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:11.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:11.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:11.359 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 15 16 36384 36368 151.515 0 - 0.00595258 2024-06-14T23:19:11.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:11.403 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:11.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:11.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:11.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:11.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:11.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:11.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:11.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:11.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:11.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:11.773 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:11.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:11.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:11.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:11.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:11.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:12.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:12.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:12.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:12.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:12.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:12.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:12.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:11 smithi086 ceph-mon[97913]: osdmap e417: 8 total, 8 up, 6 in 2024-06-14T23:19:12.175 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:11 smithi086 ceph-mon[102754]: osdmap e417: 8 total, 8 up, 6 in 2024-06-14T23:19:12.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:11 smithi137 ceph-mon[90634]: osdmap e417: 8 total, 8 up, 6 in 2024-06-14T23:19:12.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:12.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:12.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:12.359 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 16 16 36502 36486 142.506 3.6875 0.00192698 0.0066402 2024-06-14T23:19:12.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:12.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:12.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:12.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:12.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:12.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:12.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:12.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:12.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:12.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:12.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:12.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:12.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:12.906 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:12.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:13.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:13.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:13.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:13.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[97913]: pgmap v1528: 119 pgs: 8 peering, 1 unknown, 110 active+clean; 2.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 5.1 KiB/s rd, 122 MiB/s wr, 1.96k op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:19:13.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 5 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-14T23:19:13.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[97913]: osdmap e418: 8 total, 8 up, 6 in 2024-06-14T23:19:13.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:13.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:13.033 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[102754]: pgmap v1528: 119 pgs: 8 peering, 1 unknown, 110 active+clean; 2.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 5.1 KiB/s rd, 122 MiB/s wr, 1.96k op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:19:13.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 5 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-14T23:19:13.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[102754]: osdmap e418: 8 total, 8 up, 6 in 2024-06-14T23:19:13.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:13.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:13.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:13.147 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:12 smithi137 ceph-mon[90634]: pgmap v1528: 119 pgs: 8 peering, 1 unknown, 110 active+clean; 2.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 5.1 KiB/s rd, 122 MiB/s wr, 1.96k op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:19:13.147 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:12 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 5 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-14T23:19:13.147 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:12 smithi137 ceph-mon[90634]: osdmap e418: 8 total, 8 up, 6 in 2024-06-14T23:19:13.147 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:13.148 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:13.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:13.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:13.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:13.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:13.359 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 17 16 36502 36486 134.123 0 - 0.0066402 2024-06-14T23:19:13.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:13.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:13.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:13.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:13.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:13.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:13.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:13.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:13.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:13.738 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:13.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:13.772 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:13.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:13.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:13.889 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:19:13.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:13.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:14.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:14.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:13 smithi086 ceph-mon[97913]: osdmap e419: 8 total, 8 up, 6 in 2024-06-14T23:19:14.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:13 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1094951590' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:19:14.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:13 smithi086 ceph-mon[102754]: osdmap e419: 8 total, 8 up, 6 in 2024-06-14T23:19:14.020 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:13 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1094951590' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:19:14.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:14.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:14.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:14.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:14.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:14.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:13 smithi137 ceph-mon[90634]: osdmap e419: 8 total, 8 up, 6 in 2024-06-14T23:19:14.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:13 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1094951590' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:19:14.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:14.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:14.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:14.359 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 18 16 36502 36486 126.672 0 - 0.0066402 2024-06-14T23:19:14.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:14.394 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:14.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:14.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:14.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:14.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:14.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:14.670 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:14.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:14.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:14.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:14.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:14.899 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:19:14.899 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:19:14.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:14.930 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:19:14.930 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:19:14.947 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:14.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:14.959 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:19:14.959 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:19:14.987 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:19:14.988 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:19:15.020 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:19:15.021 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:19:15.046 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:19:15.046 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:19:15.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:15.077 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:19:15.077 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:19:15.080 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:15.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:15.103 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:19:15.104 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:19:15.120 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[97913]: pgmap v1531: 119 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+remapped, 10 peering, 1 unknown, 102 active+clean; 2.1 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 20 KiB/s rd, 57 MiB/s wr, 934 op/s; 4638/105183 objects degraded (4.409%); 39 KiB/s, 0 objects/s recovering 2024-06-14T23:19:15.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:19:15.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1094951590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:19:15.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[97913]: osdmap e420: 8 total, 8 up, 6 in 2024-06-14T23:19:15.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[97913]: Health check failed: Degraded data redundancy: 4638/105183 objects degraded (4.409%), 3 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:15.121 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3837833708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:15.121 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:19:14] "GET /metrics HTTP/1.1" 200 36689 "" "Prometheus/2.43.0" 2024-06-14T23:19:15.121 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[102754]: pgmap v1531: 119 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+remapped, 10 peering, 1 unknown, 102 active+clean; 2.1 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 20 KiB/s rd, 57 MiB/s wr, 934 op/s; 4638/105183 objects degraded (4.409%); 39 KiB/s, 0 objects/s recovering 2024-06-14T23:19:15.121 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:19:15.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1094951590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:19:15.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[102754]: osdmap e420: 8 total, 8 up, 6 in 2024-06-14T23:19:15.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[102754]: Health check failed: Degraded data redundancy: 4638/105183 objects degraded (4.409%), 3 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:15.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:14 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3837833708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:15.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:15.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:15.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:15.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:14 smithi137 ceph-mon[90634]: pgmap v1531: 119 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+remapped, 10 peering, 1 unknown, 102 active+clean; 2.1 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 20 KiB/s rd, 57 MiB/s wr, 934 op/s; 4638/105183 objects degraded (4.409%); 39 KiB/s, 0 objects/s recovering 2024-06-14T23:19:15.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:14 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:19:15.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1094951590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:19:15.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:14 smithi137 ceph-mon[90634]: osdmap e420: 8 total, 8 up, 6 in 2024-06-14T23:19:15.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:14 smithi137 ceph-mon[90634]: Health check failed: Degraded data redundancy: 4638/105183 objects degraded (4.409%), 3 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:15.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:14 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3837833708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:15.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:15.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:15.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:15.360 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 19 16 37787 37771 124.232 26.7708 0.0108629 0.00804405 2024-06-14T23:19:15.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:15.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:15.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:15.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:15.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:15.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:15.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:15.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:15.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:15.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:15.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:15.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:16.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:16.054 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:16.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:16.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:15 smithi086 ceph-mon[97913]: osdmap e421: 8 total, 8 up, 6 in 2024-06-14T23:19:16.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:15 smithi086 ceph-mon[102754]: osdmap e421: 8 total, 8 up, 6 in 2024-06-14T23:19:16.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:16.195 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:16.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:16.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:15 smithi137 ceph-mon[90634]: osdmap e421: 8 total, 8 up, 6 in 2024-06-14T23:19:16.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:16.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:16.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:16.361 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:19:16.357719+0000 min lat: 0.00111701 max lat: 4.60763 avg lat: 0.00814892 2024-06-14T23:19:16.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:19:16.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 20 16 39261 39245 122.626 92.125 0.0073957 0.00814892 2024-06-14T23:19:16.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:16.477 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:16.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:16.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:16.615 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:16.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:16.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:16.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:16.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:16.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:16.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:16.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:16.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:16.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:16 smithi086 ceph-mon[97913]: pgmap v1534: 119 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+remapped, 3 peering, 105 active+clean; 2.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 28 KiB/s rd, 63 MiB/s wr, 1.02k op/s; 13623/111006 objects degraded (12.272%); 1692/111006 objects misplaced (1.524%); 8.6 MiB/s, 2 keys/s, 137 objects/s recovering 2024-06-14T23:19:16.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:16 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 4 pgs peering) 2024-06-14T23:19:16.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:16 smithi086 ceph-mon[97913]: osdmap e422: 8 total, 8 up, 6 in 2024-06-14T23:19:16.987 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:16 smithi086 ceph-mon[102754]: pgmap v1534: 119 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+remapped, 3 peering, 105 active+clean; 2.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 28 KiB/s rd, 63 MiB/s wr, 1.02k op/s; 13623/111006 objects degraded (12.272%); 1692/111006 objects misplaced (1.524%); 8.6 MiB/s, 2 keys/s, 137 objects/s recovering 2024-06-14T23:19:16.987 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:16 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 4 pgs peering) 2024-06-14T23:19:16.987 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:16 smithi086 ceph-mon[102754]: osdmap e422: 8 total, 8 up, 6 in 2024-06-14T23:19:17.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:17.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:17.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:17.032 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:19:17.032 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:19:17.032 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-14T23:19:17.032 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-06-14T23:19:17.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 5 0 2024-06-14T23:19:17.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:17.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:17.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:17.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:16 smithi137 ceph-mon[90634]: pgmap v1534: 119 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+remapped, 3 peering, 105 active+clean; 2.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 28 KiB/s rd, 63 MiB/s wr, 1.02k op/s; 13623/111006 objects degraded (12.272%); 1692/111006 objects misplaced (1.524%); 8.6 MiB/s, 2 keys/s, 137 objects/s recovering 2024-06-14T23:19:17.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:16 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 4 pgs peering) 2024-06-14T23:19:17.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:16 smithi137 ceph-mon[90634]: osdmap e422: 8 total, 8 up, 6 in 2024-06-14T23:19:17.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:17.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:17.359 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 21 16 40712 40696 121.105 90.6875 0.00510518 0.00825114 2024-06-14T23:19:17.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:17.419 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:17.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:17.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:17.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:17.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:17.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:17.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:17.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:17.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:17.799 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:19:17.799 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:19:17.810 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:17.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:17.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:17.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:17.970 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:17.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:18.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:18.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:18.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:18.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:19:18.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:18.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:18.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:18.360 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 22 16 42175 42159 119.756 91.4375 0.00470636 0.00834433 2024-06-14T23:19:18.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:18.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:18.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:18.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:18.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:18.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:18.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:18.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:18.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-14T23:19:18.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:18.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:18.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:18.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:18.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:18.845 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:18.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:18.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:18.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:18.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:19.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:19.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:19.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:19.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:19:19.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:19.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:19.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:18 smithi137 ceph-mon[90634]: pgmap v1536: 119 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 peering, 108 active+clean; 2.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 14 KiB/s rd, 45 MiB/s wr, 727 op/s; 13623/113799 objects degraded (11.971%); 1692/113799 objects misplaced (1.487%); 7.0 MiB/s, 2 keys/s, 111 objects/s recovering 2024-06-14T23:19:19.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:18 smithi086 ceph-mon[97913]: pgmap v1536: 119 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 peering, 108 active+clean; 2.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 14 KiB/s rd, 45 MiB/s wr, 727 op/s; 13623/113799 objects degraded (11.971%); 1692/113799 objects misplaced (1.487%); 7.0 MiB/s, 2 keys/s, 111 objects/s recovering 2024-06-14T23:19:19.251 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:18 smithi086 ceph-mon[102754]: pgmap v1536: 119 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 peering, 108 active+clean; 2.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 14 KiB/s rd, 45 MiB/s wr, 727 op/s; 13623/113799 objects degraded (11.971%); 1692/113799 objects misplaced (1.487%); 7.0 MiB/s, 2 keys/s, 111 objects/s recovering 2024-06-14T23:19:19.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:19.360 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 23 16 43601 43585 118.424 89.125 0.0118226 0.00843729 2024-06-14T23:19:19.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:19.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:19.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:19.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:19.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:19.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:19.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:19.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:19.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:19.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:19.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:19.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:19.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:19.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:20.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:20.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:20.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:20.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:20.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:20.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:20.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:19 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 13621/119811 objects degraded (11.369%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:20.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:19 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 13621/119811 objects degraded (11.369%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:20.289 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:19 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 13621/119811 objects degraded (11.369%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:20.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:20.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:20.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:20.360 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 24 16 44968 44952 117.049 85.4375 0.00867161 0.00853695 2024-06-14T23:19:20.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:20.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:20.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:20.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:20.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:20.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:20.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:20.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:20.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:20.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:20.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:20.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:20.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:21.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:21.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:21.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:20 smithi086 ceph-mon[97913]: pgmap v1537: 119 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 1 peering, 110 active+clean; 2.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 12 KiB/s rd, 51 MiB/s wr, 821 op/s; 13621/119811 objects degraded (11.369%); 1692/119811 objects misplaced (1.412%); 5.8 MiB/s, 1 keys/s, 92 objects/s recovering 2024-06-14T23:19:21.006 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:20 smithi086 ceph-mon[102754]: pgmap v1537: 119 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 1 peering, 110 active+clean; 2.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 12 KiB/s rd, 51 MiB/s wr, 821 op/s; 13621/119811 objects degraded (11.369%); 1692/119811 objects misplaced (1.412%); 5.8 MiB/s, 1 keys/s, 92 objects/s recovering 2024-06-14T23:19:21.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:21.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:21.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:21.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:21.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:21.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:20 smithi137 ceph-mon[90634]: pgmap v1537: 119 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 1 peering, 110 active+clean; 2.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 12 KiB/s rd, 51 MiB/s wr, 821 op/s; 13621/119811 objects degraded (11.369%); 1692/119811 objects misplaced (1.412%); 5.8 MiB/s, 1 keys/s, 92 objects/s recovering 2024-06-14T23:19:21.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:21.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:21.360 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 25 16 46344 46328 115.807 86 0.0119552 0.00862806 2024-06-14T23:19:21.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:21.403 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:21.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:21.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:21.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:21.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:21.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:21.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:21.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:21.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:21.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:21.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:21.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:21.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:21.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:22.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:22.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:22.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:22.147 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:22.148 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:22.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:22.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:22.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:22.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:22.360 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 26 16 47670 47654 114.54 82.875 0.0115652 0.00872544 2024-06-14T23:19:22.376 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:22.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:22.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:22.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:22.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:22.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:22.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:22.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:22.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:22.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:22.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:22.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:22.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:22.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:22.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:22.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:22.978 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.5 primary-affinity to 0 (802) 2024-06-14T23:19:23.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:23.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:23.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:23.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:23.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:23.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:23.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:22 smithi137 ceph-mon[90634]: pgmap v1538: 119 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 2.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 8.8 KiB/s rd, 63 MiB/s wr, 1.02k op/s; 13621/129282 objects degraded (10.536%); 1377/129282 objects misplaced (1.065%); 6.8 MiB/s, 1 keys/s, 109 objects/s recovering 2024-06-14T23:19:23.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3491461804' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-06-14T23:19:23.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:22 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-06-14T23:19:23.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:23.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:23.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:23.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:22 smithi086 ceph-mon[97913]: pgmap v1538: 119 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 2.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 8.8 KiB/s rd, 63 MiB/s wr, 1.02k op/s; 13621/129282 objects degraded (10.536%); 1377/129282 objects misplaced (1.065%); 6.8 MiB/s, 1 keys/s, 109 objects/s recovering 2024-06-14T23:19:23.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3491461804' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-06-14T23:19:23.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:22 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-06-14T23:19:23.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:22 smithi086 ceph-mon[102754]: pgmap v1538: 119 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 2.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 8.8 KiB/s rd, 63 MiB/s wr, 1.02k op/s; 13621/129282 objects degraded (10.536%); 1377/129282 objects misplaced (1.065%); 6.8 MiB/s, 1 keys/s, 109 objects/s recovering 2024-06-14T23:19:23.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3491461804' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-06-14T23:19:23.334 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:22 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-06-14T23:19:23.360 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 27 16 48654 48638 112.575 61.5 0.00252507 0.00877307 2024-06-14T23:19:23.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:23.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:23.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:23.466 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:19:23.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:23.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:23.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:23.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:23.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:23.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:23.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:23.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:23.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:23.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:23.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:24.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:24.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:24.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:24.174 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:24.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:23 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-06-14T23:19:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:23 smithi137 ceph-mon[90634]: osdmap e423: 8 total, 8 up, 6 in 2024-06-14T23:19:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1448165029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:19:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:23 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:19:24.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1448165029' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:19:24.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:23 smithi137 ceph-mon[90634]: osdmap e424: 8 total, 8 up, 6 in 2024-06-14T23:19:24.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:24.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:24.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:24.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:24.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-06-14T23:19:24.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[97913]: osdmap e423: 8 total, 8 up, 6 in 2024-06-14T23:19:24.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:24.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1448165029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:19:24.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:19:24.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1448165029' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:19:24.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[97913]: osdmap e424: 8 total, 8 up, 6 in 2024-06-14T23:19:24.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-06-14T23:19:24.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[102754]: osdmap e423: 8 total, 8 up, 6 in 2024-06-14T23:19:24.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:24.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1448165029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:19:24.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:19:24.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1448165029' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:19:24.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:23 smithi086 ceph-mon[102754]: osdmap e424: 8 total, 8 up, 6 in 2024-06-14T23:19:24.360 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 28 16 48654 48638 108.555 0 - 0.00877307 2024-06-14T23:19:24.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:24.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:24.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:24.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:24.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:24.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:24.691 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:24.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:24.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:24.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:24.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:24.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:24.909 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:19:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:19:24] "GET /metrics HTTP/1.1" 200 36689 "" "Prometheus/2.43.0" 2024-06-14T23:19:24.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:24.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:24.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:25.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:25.114 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:25.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:25.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:25.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:24 smithi137 ceph-mon[90634]: pgmap v1541: 119 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 2.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 13621/136818 objects degraded (9.956%); 1377/136818 objects misplaced (1.006%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:19:25.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:24 smithi137 ceph-mon[90634]: osdmap e425: 8 total, 8 up, 6 in 2024-06-14T23:19:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:25.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:25.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:25.278 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:24 smithi086 ceph-mon[97913]: pgmap v1541: 119 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 2.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 13621/136818 objects degraded (9.956%); 1377/136818 objects misplaced (1.006%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:19:25.279 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:24 smithi086 ceph-mon[97913]: osdmap e425: 8 total, 8 up, 6 in 2024-06-14T23:19:25.279 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:24 smithi086 ceph-mon[102754]: pgmap v1541: 119 pgs: 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 2.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 13621/136818 objects degraded (9.956%); 1377/136818 objects misplaced (1.006%); 2.5 MiB/s, 39 objects/s recovering 2024-06-14T23:19:25.279 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:24 smithi086 ceph-mon[102754]: osdmap e425: 8 total, 8 up, 6 in 2024-06-14T23:19:25.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:25.360 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 29 16 49744 49728 107.161 34.0625 0.00786011 0.00932555 2024-06-14T23:19:25.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:25.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:25.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:25.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:25.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:25.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:25.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:25.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:25.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:25.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:25.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:25.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:25.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:25.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:26.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:26.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:26.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:26.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:26.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:26.217 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:26.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:26.267 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:26 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 13621/146370 objects degraded (9.306%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:26.267 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:26 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 13621/146370 objects degraded (9.306%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:26.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:26.361 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 30 16 51056 51040 106.32 82 0.0108919 0.00939864 2024-06-14T23:19:26.364 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:26.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:26.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:26.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:26 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 13621/146370 objects degraded (9.306%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:26.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:26.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:26.632 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:26.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:26.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:26.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:26.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:26.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:26.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:26.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:26.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:27.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:27.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:27.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:27.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:27.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:27.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:27.269 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:27 smithi086 ceph-mon[97913]: pgmap v1543: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 92 MiB/s wr, 1.48k op/s; 13621/146370 objects degraded (9.306%); 703/146370 objects misplaced (0.480%); 10 MiB/s, 164 objects/s recovering 2024-06-14T23:19:27.270 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:27 smithi086 ceph-mon[102754]: pgmap v1543: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 92 MiB/s wr, 1.48k op/s; 13621/146370 objects degraded (9.306%); 703/146370 objects misplaced (0.480%); 10 MiB/s, 164 objects/s recovering 2024-06-14T23:19:27.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:27.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:27.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:27.361 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 31 16 52354 52338 105.507 81.125 0.00784643 0.0094719 2024-06-14T23:19:27.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:27.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:27.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:27 smithi137 ceph-mon[90634]: pgmap v1543: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 92 MiB/s wr, 1.48k op/s; 13621/146370 objects degraded (9.306%); 703/146370 objects misplaced (0.480%); 10 MiB/s, 164 objects/s recovering 2024-06-14T23:19:27.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:27.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:27.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:27.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:27.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:27.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:27.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:27.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:27.908 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:19:27.909 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:19:27.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:28.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:28.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:28.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:28.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:28.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:28.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:28.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:28.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:28.361 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 32 16 53661 53645 104.763 81.6875 0.0182813 0.00953928 2024-06-14T23:19:28.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:28.439 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:28.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:28.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:28.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:28.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:28.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:28.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:28.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:28 smithi086 ceph-mon[97913]: pgmap v1544: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 10 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 13621/149904 objects degraded (9.086%); 703/149904 objects misplaced (0.469%); 7.0 MiB/s, 112 objects/s recovering 2024-06-14T23:19:28.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:28.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:28.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:28 smithi086 ceph-mon[102754]: pgmap v1544: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 10 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 13621/149904 objects degraded (9.086%); 703/149904 objects misplaced (0.469%); 7.0 MiB/s, 112 objects/s recovering 2024-06-14T23:19:28.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:28.720 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:28.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:28.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:28.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:28 smithi137 ceph-mon[90634]: pgmap v1544: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 10 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 13621/149904 objects degraded (9.086%); 703/149904 objects misplaced (0.469%); 7.0 MiB/s, 112 objects/s recovering 2024-06-14T23:19:28.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-14T23:19:28.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:28.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:28.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:28.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:28.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:29.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:29.128 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:29.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:29.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:29.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:29.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:29.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:29.361 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 33 16 54838 54822 103.817 73.5625 0.00542062 0.00962679 2024-06-14T23:19:29.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:29.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:29.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:29.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:29.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:29.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:29.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:29.676 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:19:29.676 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:19:29.676 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-14T23:19:29.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:19:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:29.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:29.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:29.700 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:29 smithi086 ceph-mon[97913]: 1.9 scrub starts 2024-06-14T23:19:29.700 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:29 smithi086 ceph-mon[102754]: 1.9 scrub starts 2024-06-14T23:19:29.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:29 smithi137 ceph-mon[90634]: 1.9 scrub starts 2024-06-14T23:19:29.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:29.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:19:29.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:29.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:29.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:29.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:29.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:30.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:30.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:30.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:30.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:30.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:30.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:30.361 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 34 16 55968 55952 102.841 70.625 0.0160387 0.00971845 2024-06-14T23:19:30.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:30.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:30.452 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:30 smithi086 ceph-mon[97913]: pgmap v1545: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 9.0 KiB/s rd, 59 MiB/s wr, 956 op/s; 13621/154113 objects degraded (8.838%); 703/154113 objects misplaced (0.456%); 6.2 MiB/s, 98 objects/s recovering 2024-06-14T23:19:30.452 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:30 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1210223243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:30.452 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:30 smithi086 ceph-mon[102754]: pgmap v1545: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 9.0 KiB/s rd, 59 MiB/s wr, 956 op/s; 13621/154113 objects degraded (8.838%); 703/154113 objects misplaced (0.456%); 6.2 MiB/s, 98 objects/s recovering 2024-06-14T23:19:30.452 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:30 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1210223243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:30.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:30.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:30.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:30.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:30.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:30.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:30.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:30.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:30 smithi137 ceph-mon[90634]: pgmap v1545: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 9.0 KiB/s rd, 59 MiB/s wr, 956 op/s; 13621/154113 objects degraded (8.838%); 703/154113 objects misplaced (0.456%); 6.2 MiB/s, 98 objects/s recovering 2024-06-14T23:19:30.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:30 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1210223243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:30.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:30.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:30.889 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:30.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:30.991 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:31.018 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:31.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:31.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:31.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:31.204 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:19:31.204 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:19:31.204 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:19:31.204 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:19:31.204 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:19:31.204 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:19:31.204 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:19:31.205 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:19:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:19:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:19:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:19:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:19:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:19:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:19:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:19:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:19:31.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:31.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:31.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:31.361 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 35 16 57197 57181 102.097 76.8125 0.0081589 0.00978951 2024-06-14T23:19:31.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:31.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:31.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:31.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:31.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:31.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:31.684 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:31.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:31.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:31.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:31.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:31.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:31.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:31.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:32.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:32.074 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:32.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:32.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:32.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:32.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:32.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:32.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:31 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 13621/163527 objects degraded (8.330%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:32.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:32.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:31 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 13621/163527 objects degraded (8.330%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:32.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:31 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 13621/163527 objects degraded (8.330%), 7 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:32.353 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:32.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:32.361 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 36 16 58283 58267 101.146 67.875 0.0104164 0.00988041 2024-06-14T23:19:32.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:32.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:32.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:32.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:32.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:32.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:32.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:32.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:32.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:32.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:32.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:32.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:32.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:32.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:33.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:33.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:33.054 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:32 smithi137 ceph-mon[90634]: pgmap v1546: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 7.4 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 13621/163527 objects degraded (8.330%); 256/163527 objects misplaced (0.157%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:19:33.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:33.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:33.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:33.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:33.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:33.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:33.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:32 smithi086 ceph-mon[97913]: pgmap v1546: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 7.4 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 13621/163527 objects degraded (8.330%); 256/163527 objects misplaced (0.157%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:19:33.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:32 smithi086 ceph-mon[102754]: pgmap v1546: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 7.4 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 13621/163527 objects degraded (8.330%); 256/163527 objects misplaced (0.157%); 8.4 MiB/s, 133 objects/s recovering 2024-06-14T23:19:33.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 37 16 59162 59146 99.897 54.9375 0.0189339 0.0100037 2024-06-14T23:19:33.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:33.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:33.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:33.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:33.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:33.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:33.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:33.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:33.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:33.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:33.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:33.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:33.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:34.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:34.071 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:34.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:34.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:34.197 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:34.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:34.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:34.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:34.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:34.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 38 16 59988 59972 98.6267 51.625 0.024003 0.010068 2024-06-14T23:19:34.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:34.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:34.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:34.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:34.570 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:34.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:34.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:34.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:34.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:34.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:34.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:34.816 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:34.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:34.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:34.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:34.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:35.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:35.071 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:35.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:35.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:35.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:35.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:35.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:34 smithi137 ceph-mon[90634]: pgmap v1547: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.5 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 6.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 13621/170196 objects degraded (8.003%); 256/170196 objects misplaced (0.150%); 7.0 MiB/s, 112 objects/s recovering 2024-06-14T23:19:35.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:34 smithi137 ceph-mon[90634]: osdmap e426: 8 total, 8 up, 6 in 2024-06-14T23:19:35.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:35.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:34 smithi086 ceph-mon[97913]: pgmap v1547: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.5 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 6.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 13621/170196 objects degraded (8.003%); 256/170196 objects misplaced (0.150%); 7.0 MiB/s, 112 objects/s recovering 2024-06-14T23:19:35.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:34 smithi086 ceph-mon[97913]: osdmap e426: 8 total, 8 up, 6 in 2024-06-14T23:19:35.338 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:19:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:19:34] "GET /metrics HTTP/1.1" 200 36706 "" "Prometheus/2.43.0" 2024-06-14T23:19:35.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:34 smithi086 ceph-mon[102754]: pgmap v1547: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.5 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 6.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 13621/170196 objects degraded (8.003%); 256/170196 objects misplaced (0.150%); 7.0 MiB/s, 112 objects/s recovering 2024-06-14T23:19:35.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:34 smithi086 ceph-mon[102754]: osdmap e426: 8 total, 8 up, 6 in 2024-06-14T23:19:35.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:35.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:35.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 39 16 60442 60426 96.8253 28.375 0.0225472 0.0103224 2024-06-14T23:19:35.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:35.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:35.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:35.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:35.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:35.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:35.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:35.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:35.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:35.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:35.901 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:35.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:36.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:36.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:36.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:36.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:36.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:36.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:36.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:35 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:19:36.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:35 smithi137 ceph-mon[90634]: osdmap e427: 8 total, 8 up, 6 in 2024-06-14T23:19:36.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:36.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:36.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:35 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:19:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:35 smithi086 ceph-mon[97913]: osdmap e427: 8 total, 8 up, 6 in 2024-06-14T23:19:36.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:35 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:19:36.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:35 smithi086 ceph-mon[102754]: osdmap e427: 8 total, 8 up, 6 in 2024-06-14T23:19:36.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:19:36.360005+0000 min lat: 0.00111701 max lat: 4.60763 avg lat: 0.0103988 2024-06-14T23:19:36.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:19:36.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 40 16 61539 61523 96.1186 68.5625 0.0104048 0.0103988 2024-06-14T23:19:36.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:36.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:36.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:36.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:36.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:36.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:36.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:36.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:36.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:36.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:36.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:36.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:36.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:37.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:37.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:37.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:37.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:37.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:36 smithi137 ceph-mon[90634]: 1.11 scrub starts 2024-06-14T23:19:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:36 smithi137 ceph-mon[90634]: pgmap v1550: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 75 MiB/s wr, 1.20k op/s; 11279/178722 objects degraded (6.311%); 2203/178722 objects misplaced (1.233%); 6.6 MiB/s, 105 objects/s recovering 2024-06-14T23:19:37.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:37.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:37.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:36 smithi086 ceph-mon[97913]: 1.11 scrub starts 2024-06-14T23:19:37.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:36 smithi086 ceph-mon[97913]: pgmap v1550: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 75 MiB/s wr, 1.20k op/s; 11279/178722 objects degraded (6.311%); 2203/178722 objects misplaced (1.233%); 6.6 MiB/s, 105 objects/s recovering 2024-06-14T23:19:37.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:36 smithi086 ceph-mon[102754]: 1.11 scrub starts 2024-06-14T23:19:37.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:36 smithi086 ceph-mon[102754]: pgmap v1550: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 75 MiB/s wr, 1.20k op/s; 11279/178722 objects degraded (6.311%); 2203/178722 objects misplaced (1.233%); 6.6 MiB/s, 105 objects/s recovering 2024-06-14T23:19:37.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 41 16 62586 62570 95.3702 65.4375 0.00708344 0.0104797 2024-06-14T23:19:37.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:37.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:37.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:37.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:37.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:37.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:37.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:37.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:37.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:37.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:37.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:37.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:37.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:37.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:38.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:38.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:38.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:38.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:38.233 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:38.241 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:38.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:38.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:37 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:19:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:38.283 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:38.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:37 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:19:38.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:37 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:19:38.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:38.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 42 16 63663 63647 94.702 67.3125 0.00944227 0.0105546 2024-06-14T23:19:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:38.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:38.482 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:38.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:38.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:38.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:38.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:38.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:38.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:38.807 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:38.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:38.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:38.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:38.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:39.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:39.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:39.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:39.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:39.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:39.200 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:39.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:39.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:38 smithi137 ceph-mon[90634]: pgmap v1551: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 73 MiB/s wr, 1.16k op/s; 8892/182037 objects degraded (4.885%); 4368/182037 objects misplaced (2.400%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:19:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8892/182037 objects degraded (4.885%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:39.270 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:38 smithi086 ceph-mon[97913]: pgmap v1551: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 73 MiB/s wr, 1.16k op/s; 8892/182037 objects degraded (4.885%); 4368/182037 objects misplaced (2.400%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:19:39.270 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8892/182037 objects degraded (4.885%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:39.270 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:38 smithi086 ceph-mon[102754]: pgmap v1551: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 73 MiB/s wr, 1.16k op/s; 8892/182037 objects degraded (4.885%); 4368/182037 objects misplaced (2.400%); 8.3 MiB/s, 132 objects/s recovering 2024-06-14T23:19:39.270 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8892/182037 objects degraded (4.885%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:39.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:39.343 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:39.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:39.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 43 16 64751 64735 94.0809 68 0.0101595 0.0106234 2024-06-14T23:19:39.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:39.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:39.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:39.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:39.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:39.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:39.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:39.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:39.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:39.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:39.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:39.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:40.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:40.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:40.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:40.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:40.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:40.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:40.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:19:40.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:40.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:40.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:40.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 44 16 65857 65841 93.5136 69.125 0.0130308 0.0106877 2024-06-14T23:19:40.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:40.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:40.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:40.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:40.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:40.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:40.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:40.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:40.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:40.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:40.852 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:40.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:40.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:40.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:40.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:41.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:41.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:41.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:41.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:41.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:40 smithi137 ceph-mon[90634]: 1.11 scrub starts 2024-06-14T23:19:41.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:40 smithi137 ceph-mon[90634]: pgmap v1552: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 57 MiB/s wr, 913 op/s; 8892/185454 objects degraded (4.795%); 4368/185454 objects misplaced (2.355%); 4.8 MiB/s, 77 objects/s recovering 2024-06-14T23:19:41.280 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:41.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:41.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:41.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:40 smithi086 ceph-mon[97913]: 1.11 scrub starts 2024-06-14T23:19:41.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:40 smithi086 ceph-mon[97913]: pgmap v1552: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 57 MiB/s wr, 913 op/s; 8892/185454 objects degraded (4.795%); 4368/185454 objects misplaced (2.355%); 4.8 MiB/s, 77 objects/s recovering 2024-06-14T23:19:41.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:40 smithi086 ceph-mon[102754]: 1.11 scrub starts 2024-06-14T23:19:41.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:40 smithi086 ceph-mon[102754]: pgmap v1552: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 3.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 57 MiB/s wr, 913 op/s; 8892/185454 objects degraded (4.795%); 4368/185454 objects misplaced (2.355%); 4.8 MiB/s, 77 objects/s recovering 2024-06-14T23:19:41.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 45 16 66945 66929 92.9465 68 0.0135452 0.0107536 2024-06-14T23:19:41.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:41.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:41.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:41.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:41.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:41.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:41.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:41.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:41.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:41.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:41.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:41.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:41.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:41.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:41.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:41.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:42.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:42.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:42.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:42.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:42.234 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:42.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:42.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:42.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 46 16 68046 68030 92.4218 68.8125 0.0180331 0.0108148 2024-06-14T23:19:42.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:42.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:42.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:42.504 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:42.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:42.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:42.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:42.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221574 13.a 13.92058aba (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.064649+0000", 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.58800566799999, 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0051272010000001, 2024-06-14T23:19:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221574 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064649+0000", 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064649+0000", 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064647+0000", 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064653+0000", 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8579999999999996e-06 2024-06-14T23:19:42.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064668+0000", 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4479e-05 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064677+0000", 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6819999999999998e-06 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.071107+0000", 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064300779999999997 2024-06-14T23:19:42.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.071115+0000", 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3560000000000004e-06 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069766+0000", 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99865115400000004 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069776+0000", 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0114e-05 2024-06-14T23:19:42.664 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221579 13.a 13.308535ea (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.064688+0000", 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.58796691000001, 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.005117898, 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221579 2024-06-14T23:19:42.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064688+0000", 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064688+0000", 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064687+0000", 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064691+0000", 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4580000000000002e-06 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064697+0000", 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.922e-06 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.064703+0000", 2024-06-14T23:19:42.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3689999999999998e-06 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.071132+0000", 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0064291720000000004 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.071136+0000", 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.121e-06 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069798+0000", 2024-06-14T23:19:42.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99866206599999996 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069806+0000", 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3089999999999998e-06 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221610 13.a 13.3aeabdda (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.087779+0000", 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.56487599099999, 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98205399999999998, 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.669 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221610 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.087779+0000", 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.087779+0000", 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.087778+0000", 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.087782+0000", 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0199999999999996e-06 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.087790+0000", 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0450000000000001e-06 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.087796+0000", 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5389999999999998e-06 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.087872+0000", 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6229000000000005e-05 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.087887+0000", 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4995e-05 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069825+0000", 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98193833799999997 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069833+0000", 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.374e-06 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221617 13.a 13.7b6df2a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.097540+0000", 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.555114446, 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97231949799999995, 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221617 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.674 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.097540+0000", 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.097540+0000", 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.097539+0000", 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:42.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.097545+0000", 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3640000000000004e-06 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.097557+0000", 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1591e-05 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.097564+0000", 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7400000000000004e-06 2024-06-14T23:19:42.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.097605+0000", 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0553999999999998e-05 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.097612+0000", 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8499999999999996e-06 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069853+0000", 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97224072399999995 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069860+0000", 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0520000000000004e-06 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221642 13.a 13.8182c75a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.118887+0000", 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.53376740100001, 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95099966499999999, 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.678 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221642 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.118887+0000", 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.118887+0000", 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.118886+0000", 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.118890+0000", 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1649999999999999e-06 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.118899+0000", 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0189999999999995e-06 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.118905+0000", 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3260000000000003e-06 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.119020+0000", 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114764 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.119027+0000", 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.498e-06 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069879+0000", 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95085180899999999 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069887+0000", 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8960000000000003e-06 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221666 13.a 13.5277165a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.146376+0000", 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.50627849999999, 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92373563299999994, 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221666 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.146376+0000", 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.146376+0000", 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.146375+0000", 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.146382+0000", 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0840000000000003e-06 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.146397+0000", 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5333e-05 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.146405+0000", 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8760000000000005e-06 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.146452+0000", 2024-06-14T23:19:42.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6726e-05 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.146459+0000", 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5360000000000004e-06 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.069907+0000", 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92344724700000003 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070112+0000", 2024-06-14T23:19:42.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020514400000000001 2024-06-14T23:19:42.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.687 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.687 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221678 13.a 13.fe0f58ba (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.155029+0000", 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.497625767, 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.91512337300000002, 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221678 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.155029+0000", 2024-06-14T23:19:42.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.155029+0000", 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.155028+0000", 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.155031+0000", 2024-06-14T23:19:42.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.988e-06 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.155038+0000", 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8229999999999997e-06 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.155043+0000", 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7169999999999999e-06 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.155087+0000", 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4461e-05 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.155094+0000", 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9460000000000001e-06 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070142+0000", 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.91504745799999998 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070152+0000", 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0431999999999999e-05 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221688 13.a 13.dda33c2a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.166994+0000", 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.485660682, 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.90324032200000004, 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.692 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221688 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.166994+0000", 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.166994+0000", 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.166992+0000", 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.166999+0000", 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1230000000000002e-06 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167020+0000", 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0166999999999999e-05 2024-06-14T23:19:42.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167028+0000", 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.225e-06 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167109+0000", 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1638999999999995e-05 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167118+0000", 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8279999999999992e-06 2024-06-14T23:19:42.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070173+0000", 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.90305499199999995 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070234+0000", 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0924999999999997e-05 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.696 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221689 13.a 13.abb36e9a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.167041+0000", 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.485613921, 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.90322725999999998, 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221689 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167041+0000", 2024-06-14T23:19:42.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167041+0000", 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167040+0000", 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167064+0000", 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3730000000000001e-05 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167073+0000", 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8589999999999994e-06 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167079+0000", 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3110000000000002e-06 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167151+0000", 2024-06-14T23:19:42.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1687000000000007e-05 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167157+0000", 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0220000000000001e-06 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070260+0000", 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.90310349099999998 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070268+0000", 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6109999999999996e-06 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221691 13.a 13.b09e8aaa (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.167218+0000", 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.4854366, 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.90307950100000001, 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.701 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221691 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167218+0000", 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167218+0000", 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.702 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167217+0000", 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167222+0000", 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7979999999999996e-06 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167315+0000", 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3399999999999993e-05 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167323+0000", 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5800000000000003e-06 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167359+0000", 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6508999999999999e-05 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.167366+0000", 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8510000000000001e-06 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070290+0000", 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.90292346000000001 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070297+0000", 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6380000000000004e-06 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221694 13.a 13.2057fc7a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.170362+0000", 2024-06-14T23:19:42.705 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.48229208799999, 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.89996268899999998, 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221694 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.170362+0000", 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.170362+0000", 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.170361+0000", 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.170370+0000", 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0759999999999992e-06 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.170389+0000", 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8839e-05 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.170398+0000", 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8529999999999998e-06 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.170455+0000", 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7117999999999998e-05 2024-06-14T23:19:42.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.170466+0000", 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1596999999999999e-05 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070318+0000", 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.89985146500000002 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070325+0000", 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3780000000000003e-06 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.709 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221705 13.a 13.41cac42a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.181135+0000", 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.47151922899999, 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.88921895100000004, 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221705 2024-06-14T23:19:42.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.181135+0000", 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.181135+0000", 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.711 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.181134+0000", 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.181139+0000", 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2729999999999996e-06 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.181155+0000", 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5790999999999998e-05 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.712 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.181163+0000", 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4100000000000008e-06 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.181214+0000", 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1118000000000001e-05 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.181224+0000", 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5070000000000006e-06 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.713 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070347+0000", 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.88912325599999997 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070354+0000", 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4340000000000003e-06 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.714 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221714 13.a 13.fd218b8a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.190673+0000", 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.461981288, 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.87971035500000005, 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221714 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.190673+0000", 2024-06-14T23:19:42.715 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.190673+0000", 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.190672+0000", 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.190677+0000", 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3549999999999998e-06 2024-06-14T23:19:42.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.190684+0000", 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4499999999999998e-06 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.190688+0000", 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.092e-06 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.190728+0000", 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9932000000000003e-05 2024-06-14T23:19:42.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.190737+0000", 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.986e-06 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070376+0000", 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.87963867799999995 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070384+0000", 2024-06-14T23:19:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7510000000000008e-06 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221720 13.a 13.e2e0a09a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.200112+0000", 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.452542673, 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.87029934200000003, 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.719 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221720 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.200112+0000", 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.200112+0000", 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.200111+0000", 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.200115+0000", 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7910000000000004e-06 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.200126+0000", 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0389e-05 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.721 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.200132+0000", 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9309999999999996e-06 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.200175+0000", 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2928e-05 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.200180+0000", 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4199999999999998e-06 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.722 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070403+0000", 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.87022316200000005 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070411+0000", 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8830000000000006e-06 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221728 13.a 13.361e8bda (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.213993+0000", 2024-06-14T23:19:42.723 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.438661932, 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.856451187, 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221728 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.213993+0000", 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.213993+0000", 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.213992+0000", 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.213996+0000", 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5480000000000003e-06 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.214009+0000", 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2332e-05 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.214015+0000", 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4439999999999996e-06 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.214055+0000", 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0145000000000003e-05 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.214061+0000", 2024-06-14T23:19:42.727 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1310000000000003e-06 2024-06-14T23:19:42.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.727 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.727 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070436+0000", 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.85637450699999995 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070444+0000", 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8709999999999997e-06 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.728 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.46844.0:221754 13.a 13.13f167fa (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:17:57.301828+0000", 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.350826791, 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.76864288400000003, 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.46844", 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 221754 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.729 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.301828+0000", 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.301828+0000", 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.301827+0000", 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.730 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.301833+0000", 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5529999999999998e-06 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.301845+0000", 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2326e-05 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.301851+0000", 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7339999999999996e-06 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.731 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.301895+0000", 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4175999999999999e-05 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:57.301903+0000", 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3440000000000003e-06 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070463+0000", 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.76856038999999998 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:17:58.070471+0000", 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5360000000000004e-06 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.37287.0:32396 14.8 14:188cc8b8:::benchmark_data_smithi086_333440_object32395:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:19:07.915692+0000", 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 34.736962378000001, 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.11274967299999999, 2024-06-14T23:19:42.733 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.37287", 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32396 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.915692+0000", 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.915692+0000", 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.915691+0000", 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.915714+0000", 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2773999999999999e-05 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.735 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.915726+0000", 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1718e-05 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.915746+0000", 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0536000000000001e-05 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.915789+0000", 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2839000000000001e-05 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.736 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.916560+0000", 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00077109599999999998 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3", 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.916680+0000", 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120238 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.918139+0000", 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001458336 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.918982+0000", 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00084373 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.028358+0000", 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.109375635 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.028406+0000", 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7574000000000002e-05 2024-06-14T23:19:42.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.028442+0000", 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6239000000000003e-05 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.37287.0:32404 14.8 14:141e21a6:::benchmark_data_smithi086_333440_object32403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:19:07.919199+0000", 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 34.733455188000001, 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.110986733, 2024-06-14T23:19:42.739 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.37287", 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32404 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.919199+0000", 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.740 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.919199+0000", 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.919198+0000", 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.919221+0000", 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2189999999999999e-05 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.741 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.919233+0000", 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2498e-05 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.919239+0000", 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2299999999999996e-06 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.919284+0000", 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4790000000000003e-05 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.919370+0000", 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6199000000000006e-05 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3", 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.919453+0000", 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2937999999999998e-05 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.920575+0000", 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011221409999999999 2024-06-14T23:19:42.743 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.921005+0000", 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00042956 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.030078+0000", 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10907328600000001 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.030152+0000", 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3912000000000007e-05 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.030186+0000", 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3835999999999999e-05 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.37287.0:32408 14.6 14:67b2bcfa:::benchmark_data_smithi086_333440_object32407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:19:07.920045+0000", 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 34.732609101000001, 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.108513175, 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:42.745 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.37287", 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32408 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.920045+0000", 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.920045+0000", 2024-06-14T23:19:42.746 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.920045+0000", 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.920066+0000", 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1841e-05 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.747 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.920077+0000", 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0558e-05 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.920083+0000", 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3389999999999998e-06 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.921204+0000", 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001120221 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.748 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.921311+0000", 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010779 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3", 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.921452+0000", 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014075399999999999 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.923784+0000", 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023317059999999998 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:19:42.749 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.923789+0000", 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6909999999999996e-06 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.028493+0000", 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.104704293 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.028529+0000", 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5788999999999999e-05 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.750 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.028559+0000", 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9986000000000001e-05 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.37287.0:32411 14.6 14:6d46c26d:::benchmark_data_smithi086_333440_object32410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:19:07.922697+0000", 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 34.729957532999997, 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.107604326, 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.37287", 2024-06-14T23:19:42.751 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 32411 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.922697+0000", 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.922697+0000", 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.752 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.922696+0000", 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.922719+0000", 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2535999999999999e-05 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.922726+0000", 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.182e-06 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:19:42.753 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.922732+0000", 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.485e-06 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.922787+0000", 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5473999999999998e-05 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.922879+0000", 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1638000000000006e-05 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3", 2024-06-14T23:19:42.754 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.922983+0000", 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103999 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.923925+0000", 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00094245799999999997 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:07.924118+0000", 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019300400000000001 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.755 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.030239+0000", 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10612078699999999 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.030273+0000", 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4060000000000003e-05 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:19:08.030301+0000", 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.83e-05 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.756 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.757 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:19:42.757 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:19:42.757 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:19:42.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:42.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:42.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:42.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:42.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:42.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:42.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:43.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:43.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:43.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:43.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:42 smithi086 ceph-mon[97913]: pgmap v1553: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 3.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 60 MiB/s wr, 958 op/s; 8892/193215 objects degraded (4.602%); 4042/193215 objects misplaced (2.092%); 7.4 MiB/s, 117 objects/s recovering 2024-06-14T23:19:43.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:43.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:19:43.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:42 smithi086 ceph-mon[102754]: pgmap v1553: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 3.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 60 MiB/s wr, 958 op/s; 8892/193215 objects degraded (4.602%); 4042/193215 objects misplaced (2.092%); 7.4 MiB/s, 117 objects/s recovering 2024-06-14T23:19:43.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:43.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:19:43.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:43.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:43.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:43.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:42 smithi137 ceph-mon[90634]: pgmap v1553: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 3.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 60 MiB/s wr, 958 op/s; 8892/193215 objects degraded (4.602%); 4042/193215 objects misplaced (2.092%); 7.4 MiB/s, 117 objects/s recovering 2024-06-14T23:19:43.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:43.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:19:43.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:43.362 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 47 16 69033 69017 91.7677 61.6875 0.0361334 0.0108906 2024-06-14T23:19:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:43.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:43.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:43.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:43.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:43.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:43.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:43.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:43.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:43.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:43.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:43.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:43.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:43.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:43.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:43.989 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:43 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:19:44.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:44.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:44.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:44.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8892/193215 objects degraded (4.602%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:44.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:43 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:19:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8892/193215 objects degraded (4.602%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:44.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:43 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:19:44.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8892/193215 objects degraded (4.602%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:44.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:44.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 48 16 70025 70009 91.1475 62 0.0152214 0.0109635 2024-06-14T23:19:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:44.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:44.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:44.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:44.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:44.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:44.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:44.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:44.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:44.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:44.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:44.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:44.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:44.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:44.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:44.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:45.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:45.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:45.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:45.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:44 smithi086 ceph-mon[97913]: pgmap v1554: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 62 MiB/s wr, 990 op/s; 8892/199590 objects degraded (4.455%); 3852/199590 objects misplaced (1.930%); 7.2 MiB/s, 114 objects/s recovering 2024-06-14T23:19:45.236 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:19:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:19:44] "GET /metrics HTTP/1.1" 200 36711 "" "Prometheus/2.43.0" 2024-06-14T23:19:45.236 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:44 smithi086 ceph-mon[102754]: pgmap v1554: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 62 MiB/s wr, 990 op/s; 8892/199590 objects degraded (4.455%); 3852/199590 objects misplaced (1.930%); 7.2 MiB/s, 114 objects/s recovering 2024-06-14T23:19:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:44 smithi137 ceph-mon[90634]: pgmap v1554: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 62 MiB/s wr, 990 op/s; 8892/199590 objects degraded (4.455%); 3852/199590 objects misplaced (1.930%); 7.2 MiB/s, 114 objects/s recovering 2024-06-14T23:19:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:45.273 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:45.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:45.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 49 16 71014 70998 90.5487 61.8125 0.0222482 0.0110389 2024-06-14T23:19:45.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:45.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:45.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:45.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:45.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:45.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:45.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:45.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:45.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:45.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:19:45.811 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:19:45.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:45.847 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:45.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:45.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:45.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:45.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:46.018 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:19:46.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:46.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1472176028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:19:46.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:46 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:19:46.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1472176028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:19:46.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:46 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:19:46.104 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:46.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:46.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:46.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:46.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:46.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-14T23:19:46.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:46.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 50 16 71987 71971 89.9539 60.8125 0.0354898 0.0111101 2024-06-14T23:19:46.379 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1472176028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:19:46.379 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:46 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:19:46.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:46.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:46.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:46.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:46.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:46.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:46.677 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:46.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:46.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:46.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:46.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:46.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:46.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:46.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:46.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:46.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:47.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:47.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[97913]: from='client.37344 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:19:47.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[97913]: pgmap v1555: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s; 8892/209271 objects degraded (4.249%); 3512/209271 objects misplaced (1.678%); 7.7 MiB/s, 122 objects/s recovering 2024-06-14T23:19:47.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:19:47.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:19:47.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[97913]: osdmap e428: 8 total, 8 up, 6 in 2024-06-14T23:19:47.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[102754]: from='client.37344 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:19:47.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[102754]: pgmap v1555: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s; 8892/209271 objects degraded (4.249%); 3512/209271 objects misplaced (1.678%); 7.7 MiB/s, 122 objects/s recovering 2024-06-14T23:19:47.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:19:47.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:19:47.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:47 smithi086 ceph-mon[102754]: osdmap e428: 8 total, 8 up, 6 in 2024-06-14T23:19:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:47.113 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:47.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:47.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:47.245 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:47.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:47.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:47.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 51 16 72904 72888 89.3139 57.3125 0.00806575 0.0111912 2024-06-14T23:19:47.394 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:47.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:47.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:47.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:47 smithi137 ceph-mon[90634]: from='client.37344 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:19:47.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:47 smithi137 ceph-mon[90634]: pgmap v1555: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s; 8892/209271 objects degraded (4.249%); 3512/209271 objects misplaced (1.678%); 7.7 MiB/s, 122 objects/s recovering 2024-06-14T23:19:47.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:47 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:19:47.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:47 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:19:47.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:47 smithi137 ceph-mon[90634]: osdmap e428: 8 total, 8 up, 6 in 2024-06-14T23:19:47.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:47.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:47.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:47.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:47.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:47.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:47.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:47.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:47.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:47.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:47.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:48.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:48.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:48.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:48.211 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1554,"stamp":"2024-06-14T23:19:43.833681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4359271773,"num_objects":66530,"num_object_clones":0,"num_object_copies":199590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8892,"num_objects_misplaced":3852,"num_objects_unfound":0,"num_objects_dirty":66530,"num_whiteouts":0,"num_read":114,"num_read_kb":214,"num_write":66622,"num_write_kb":4256978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3150,"num_bytes_recovered":205217817,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71554,"ondisk_log_size":71554,"up":357,"acting":352,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19638068,"kb_used_data":12045124,"kb_used_omap":23,"kb_used_meta":7592872,"kb_avail":542832844,"statfs":{"total":575970213888,"available":555860832256,"internally_reserved":0,"allocated":12334206976,"data_stored":12321133134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23558,"internal_metadata":7775101946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,10,11,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":642121728,"num_objects":9798,"num_object_clones":0,"num_object_copies":29394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4729,"num_objects_misplaced":3596,"num_objects_unfound":0,"num_objects_dirty":9798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9798,"num_write_kb":627072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1133,"num_bytes_recovered":74252288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.890723"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":826,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946474+0000","last_change":"2024-06-14T23:19:13.894745+0000","last_active":"2024-06-14T23:19:33.946474+0000","last_peered":"2024-06-14T23:19:33.946474+0000","last_clean":"2024-06-14T23:19:33.946474+0000","last_became_active":"2024-06-14T23:19:13.894348+0000","last_became_peered":"2024-06-14T23:19:13.894348+0000","last_unstale":"2024-06-14T23:19:33.946474+0000","last_undegraded":"2024-06-14T23:19:33.946474+0000","last_fullsized":"2024-06-14T23:19:33.946474+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:29:45.039008+0000","scrub_duration":0.0035785970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":26,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.958788+0000","last_change":"2024-06-14T23:19:10.893345+0000","last_active":"2024-06-14T23:19:33.958788+0000","last_peered":"2024-06-14T23:19:33.958788+0000","last_clean":"2024-06-14T23:19:33.958788+0000","last_became_active":"2024-06-14T23:19:10.880062+0000","last_became_peered":"2024-06-14T23:19:10.880062+0000","last_unstale":"2024-06-14T23:19:33.958788+0000","last_undegraded":"2024-06-14T23:19:33.958788+0000","last_fullsized":"2024-06-14T23:19:33.958788+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:18.063352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":414,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.953321+0000","last_change":"2024-06-14T23:10:26.045674+0000","last_active":"2024-06-14T23:19:33.953321+0000","last_peered":"2024-06-14T23:19:33.953321+0000","last_clean":"2024-06-14T23:19:33.953321+0000","last_became_active":"2024-06-14T23:10:26.045329+0000","last_became_peered":"2024-06-14T23:10:26.045329+0000","last_unstale":"2024-06-14T23:19:33.953321+0000","last_undegraded":"2024-06-14T23:19:33.953321+0000","last_fullsized":"2024-06-14T23:19:33.953321+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:53:12.924895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":20,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.950887+0000","last_change":"2024-06-14T23:19:10.885669+0000","last_active":"2024-06-14T23:19:33.950887+0000","last_peered":"2024-06-14T23:19:33.950887+0000","last_clean":"2024-06-14T23:19:33.950887+0000","last_became_active":"2024-06-14T23:19:10.885155+0000","last_became_peered":"2024-06-14T23:19:10.885155+0000","last_unstale":"2024-06-14T23:19:33.950887+0000","last_undegraded":"2024-06-14T23:19:33.950887+0000","last_fullsized":"2024-06-14T23:19:33.950887+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:50:16.711912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":185,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946930+0000","last_change":"2024-06-14T23:19:23.472287+0000","last_active":"2024-06-14T23:19:33.946930+0000","last_peered":"2024-06-14T23:19:33.946930+0000","last_clean":"2024-06-14T23:19:33.946930+0000","last_became_active":"2024-06-14T23:19:23.472025+0000","last_became_peered":"2024-06-14T23:19:23.472025+0000","last_unstale":"2024-06-14T23:19:33.946930+0000","last_undegraded":"2024-06-14T23:19:33.946930+0000","last_fullsized":"2024-06-14T23:19:33.946930+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:58.486967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":660,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951341+0000","last_change":"2024-06-14T23:19:23.472062+0000","last_active":"2024-06-14T23:19:33.951341+0000","last_peered":"2024-06-14T23:19:33.951341+0000","last_clean":"2024-06-14T23:19:33.951341+0000","last_became_active":"2024-06-14T23:19:23.471190+0000","last_became_peered":"2024-06-14T23:19:23.471190+0000","last_unstale":"2024-06-14T23:19:33.951341+0000","last_undegraded":"2024-06-14T23:19:33.951341+0000","last_fullsized":"2024-06-14T23:19:33.951341+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:14:16.297860+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:14:16.297860+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:50:51.694519+0000","scrub_duration":0.0039154719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":58,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959957+0000","last_change":"2024-06-14T23:17:36.973694+0000","last_active":"2024-06-14T23:19:33.959957+0000","last_peered":"2024-06-14T23:19:33.959957+0000","last_clean":"2024-06-14T23:19:33.959957+0000","last_became_active":"2024-06-14T23:17:36.973328+0000","last_became_peered":"2024-06-14T23:17:36.973328+0000","last_unstale":"2024-06-14T23:19:33.959957+0000","last_undegraded":"2024-06-14T23:19:33.959957+0000","last_fullsized":"2024-06-14T23:19:33.959957+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:42:16.457436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":415,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.956021+0000","last_change":"2024-06-14T23:17:22.897703+0000","last_active":"2024-06-14T23:19:33.956021+0000","last_peered":"2024-06-14T23:19:33.956021+0000","last_clean":"2024-06-14T23:19:33.956021+0000","last_became_active":"2024-06-14T23:17:22.896776+0000","last_became_peered":"2024-06-14T23:17:22.896776+0000","last_unstale":"2024-06-14T23:19:33.956021+0000","last_undegraded":"2024-06-14T23:19:33.956021+0000","last_fullsized":"2024-06-14T23:19:33.956021+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:30:54.965205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":581,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.947331+0000","last_change":"2024-06-14T23:09:51.419615+0000","last_active":"2024-06-14T23:19:33.947331+0000","last_peered":"2024-06-14T23:19:33.947331+0000","last_clean":"2024-06-14T23:19:33.947331+0000","last_became_active":"2024-06-14T23:09:23.060746+0000","last_became_peered":"2024-06-14T23:09:23.060746+0000","last_unstale":"2024-06-14T23:19:33.947331+0000","last_undegraded":"2024-06-14T23:19:33.947331+0000","last_fullsized":"2024-06-14T23:19:33.947331+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:25:36.984276+0000","scrub_duration":0.0034859700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":404,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.964111+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:19:33.964111+0000","last_peered":"2024-06-14T23:19:33.964111+0000","last_clean":"2024-06-14T23:19:33.964111+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:19:33.964111+0000","last_undegraded":"2024-06-14T23:19:33.964111+0000","last_fullsized":"2024-06-14T23:19:33.964111+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":425,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946460+0000","last_change":"2024-06-14T23:19:10.888044+0000","last_active":"2024-06-14T23:19:33.946460+0000","last_peered":"2024-06-14T23:19:33.946460+0000","last_clean":"2024-06-14T23:19:33.946460+0000","last_became_active":"2024-06-14T23:19:10.886957+0000","last_became_peered":"2024-06-14T23:19:10.886957+0000","last_unstale":"2024-06-14T23:19:33.946460+0000","last_undegraded":"2024-06-14T23:19:33.946460+0000","last_fullsized":"2024-06-14T23:19:33.946460+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:05:56.361425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":52,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.955393+0000","last_change":"2024-06-14T23:17:22.888832+0000","last_active":"2024-06-14T23:19:33.955393+0000","last_peered":"2024-06-14T23:19:33.955393+0000","last_clean":"2024-06-14T23:19:33.955393+0000","last_became_active":"2024-06-14T23:17:22.888525+0000","last_became_peered":"2024-06-14T23:17:22.888525+0000","last_unstale":"2024-06-14T23:19:33.955393+0000","last_undegraded":"2024-06-14T23:19:33.955393+0000","last_fullsized":"2024-06-14T23:19:33.955393+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:36.414213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":179,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.947244+0000","last_change":"2024-06-14T23:19:10.886925+0000","last_active":"2024-06-14T23:19:33.947244+0000","last_peered":"2024-06-14T23:19:33.947244+0000","last_clean":"2024-06-14T23:19:33.947244+0000","last_became_active":"2024-06-14T23:19:10.886481+0000","last_became_peered":"2024-06-14T23:19:10.886481+0000","last_unstale":"2024-06-14T23:19:33.947244+0000","last_undegraded":"2024-06-14T23:19:33.947244+0000","last_fullsized":"2024-06-14T23:19:33.947244+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:04:39.769981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":750,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.960784+0000","last_change":"2024-06-14T23:17:25.892601+0000","last_active":"2024-06-14T23:19:33.960784+0000","last_peered":"2024-06-14T23:19:33.960784+0000","last_clean":"2024-06-14T23:19:33.960784+0000","last_became_active":"2024-06-14T23:17:25.891955+0000","last_became_peered":"2024-06-14T23:17:25.891955+0000","last_unstale":"2024-06-14T23:19:33.960784+0000","last_undegraded":"2024-06-14T23:19:33.960784+0000","last_fullsized":"2024-06-14T23:19:33.960784+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:39:49.399709+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":406,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.956103+0000","last_change":"2024-06-14T23:19:10.884119+0000","last_active":"2024-06-14T23:19:33.956103+0000","last_peered":"2024-06-14T23:19:33.956103+0000","last_clean":"2024-06-14T23:19:33.956103+0000","last_became_active":"2024-06-14T23:19:10.883779+0000","last_became_peered":"2024-06-14T23:19:10.883779+0000","last_unstale":"2024-06-14T23:19:33.956103+0000","last_undegraded":"2024-06-14T23:19:33.956103+0000","last_fullsized":"2024-06-14T23:19:33.956103+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:03:17.903757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":219,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946638+0000","last_change":"2024-06-14T23:15:09.028225+0000","last_active":"2024-06-14T23:19:33.946638+0000","last_peered":"2024-06-14T23:19:33.946638+0000","last_clean":"2024-06-14T23:19:33.946638+0000","last_became_active":"2024-06-14T23:15:09.027806+0000","last_became_peered":"2024-06-14T23:15:09.027806+0000","last_unstale":"2024-06-14T23:19:33.946638+0000","last_undegraded":"2024-06-14T23:19:33.946638+0000","last_fullsized":"2024-06-14T23:19:33.946638+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:00:43.248153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":843,"reported_epoch":426,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:19:33.963768+0000","last_change":"2024-06-14T23:19:14.432117+0000","last_active":"2024-06-14T23:19:33.963768+0000","last_peered":"2024-06-14T23:19:33.963768+0000","last_clean":"2024-06-14T23:18:58.404718+0000","last_became_active":"2024-06-14T23:19:13.724529+0000","last_became_peered":"2024-06-14T23:19:13.724529+0000","last_unstale":"2024-06-14T23:19:33.963768+0000","last_undegraded":"2024-06-14T23:19:11.876815+0000","last_fullsized":"2024-06-14T23:19:11.876291+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:41:33.509018+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":226,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946615+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:19:33.946615+0000","last_peered":"2024-06-14T23:19:33.946615+0000","last_clean":"2024-06-14T23:19:33.946615+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:19:33.946615+0000","last_undegraded":"2024-06-14T23:19:33.946615+0000","last_fullsized":"2024-06-14T23:19:33.946615+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":298,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.947274+0000","last_change":"2024-06-14T23:17:22.890735+0000","last_active":"2024-06-14T23:19:33.947274+0000","last_peered":"2024-06-14T23:19:33.947274+0000","last_clean":"2024-06-14T23:19:33.947274+0000","last_became_active":"2024-06-14T23:17:22.890377+0000","last_became_peered":"2024-06-14T23:17:22.890377+0000","last_unstale":"2024-06-14T23:19:33.947274+0000","last_undegraded":"2024-06-14T23:19:33.947274+0000","last_fullsized":"2024-06-14T23:19:33.947274+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:54.873390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":801,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.963507+0000","last_change":"2024-06-14T23:17:25.906387+0000","last_active":"2024-06-14T23:19:33.963507+0000","last_peered":"2024-06-14T23:19:33.963507+0000","last_clean":"2024-06-14T23:19:33.963507+0000","last_became_active":"2024-06-14T23:17:25.906076+0000","last_became_peered":"2024-06-14T23:17:25.906076+0000","last_unstale":"2024-06-14T23:19:33.963507+0000","last_undegraded":"2024-06-14T23:19:33.963507+0000","last_fullsized":"2024-06-14T23:19:33.963507+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:11:42.673437+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":180,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946344+0000","last_change":"2024-06-14T23:19:10.886218+0000","last_active":"2024-06-14T23:19:33.946344+0000","last_peered":"2024-06-14T23:19:33.946344+0000","last_clean":"2024-06-14T23:19:33.946344+0000","last_became_active":"2024-06-14T23:19:10.885627+0000","last_became_peered":"2024-06-14T23:19:10.885627+0000","last_unstale":"2024-06-14T23:19:33.946344+0000","last_undegraded":"2024-06-14T23:19:33.946344+0000","last_fullsized":"2024-06-14T23:19:33.946344+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:01.053922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":388,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959272+0000","last_change":"2024-06-14T23:17:22.900785+0000","last_active":"2024-06-14T23:19:33.959272+0000","last_peered":"2024-06-14T23:19:33.959272+0000","last_clean":"2024-06-14T23:19:33.959272+0000","last_became_active":"2024-06-14T23:17:22.900478+0000","last_became_peered":"2024-06-14T23:17:22.900478+0000","last_unstale":"2024-06-14T23:19:33.959272+0000","last_undegraded":"2024-06-14T23:19:33.959272+0000","last_fullsized":"2024-06-14T23:19:33.959272+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:36:48.160082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":710,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.955730+0000","last_change":"2024-06-14T23:17:36.971150+0000","last_active":"2024-06-14T23:19:33.955730+0000","last_peered":"2024-06-14T23:19:33.955730+0000","last_clean":"2024-06-14T23:19:33.955730+0000","last_became_active":"2024-06-14T23:17:36.970756+0000","last_became_peered":"2024-06-14T23:17:36.970756+0000","last_unstale":"2024-06-14T23:19:33.955730+0000","last_undegraded":"2024-06-14T23:19:33.955730+0000","last_fullsized":"2024-06-14T23:19:33.955730+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:32:06.129704+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":412,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946256+0000","last_change":"2024-06-14T23:10:26.048074+0000","last_active":"2024-06-14T23:19:33.946256+0000","last_peered":"2024-06-14T23:19:33.946256+0000","last_clean":"2024-06-14T23:19:33.946256+0000","last_became_active":"2024-06-14T23:10:26.047270+0000","last_became_peered":"2024-06-14T23:10:26.047270+0000","last_unstale":"2024-06-14T23:19:33.946256+0000","last_undegraded":"2024-06-14T23:19:33.946256+0000","last_fullsized":"2024-06-14T23:19:33.946256+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:15.121432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":249,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951060+0000","last_change":"2024-06-14T23:19:23.470327+0000","last_active":"2024-06-14T23:19:33.951060+0000","last_peered":"2024-06-14T23:19:33.951060+0000","last_clean":"2024-06-14T23:19:33.951060+0000","last_became_active":"2024-06-14T23:19:23.470050+0000","last_became_peered":"2024-06-14T23:19:23.470050+0000","last_unstale":"2024-06-14T23:19:33.951060+0000","last_undegraded":"2024-06-14T23:19:33.951060+0000","last_fullsized":"2024-06-14T23:19:33.951060+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:48:40.284598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":733,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951890+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:19:33.951890+0000","last_peered":"2024-06-14T23:19:33.951890+0000","last_clean":"2024-06-14T23:19:33.951890+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:19:33.951890+0000","last_undegraded":"2024-06-14T23:19:33.951890+0000","last_fullsized":"2024-06-14T23:19:33.951890+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":295,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946330+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:19:33.946330+0000","last_peered":"2024-06-14T23:19:33.946330+0000","last_clean":"2024-06-14T23:19:33.946330+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:19:33.946330+0000","last_undegraded":"2024-06-14T23:19:33.946330+0000","last_fullsized":"2024-06-14T23:19:33.946330+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":287,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946736+0000","last_change":"2024-06-14T23:19:23.473074+0000","last_active":"2024-06-14T23:19:33.946736+0000","last_peered":"2024-06-14T23:19:33.946736+0000","last_clean":"2024-06-14T23:19:33.946736+0000","last_became_active":"2024-06-14T23:19:23.472703+0000","last_became_peered":"2024-06-14T23:19:23.472703+0000","last_unstale":"2024-06-14T23:19:33.946736+0000","last_undegraded":"2024-06-14T23:19:33.946736+0000","last_fullsized":"2024-06-14T23:19:33.946736+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:03:20.334417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":531,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951223+0000","last_change":"2024-06-14T23:19:13.896251+0000","last_active":"2024-06-14T23:19:33.951223+0000","last_peered":"2024-06-14T23:19:33.951223+0000","last_clean":"2024-06-14T23:19:33.951223+0000","last_became_active":"2024-06-14T23:19:13.895814+0000","last_became_peered":"2024-06-14T23:19:13.895814+0000","last_unstale":"2024-06-14T23:19:33.951223+0000","last_undegraded":"2024-06-14T23:19:33.951223+0000","last_fullsized":"2024-06-14T23:19:33.951223+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:17.053258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:17.053258+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":28,"num_write":7,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":313,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946789+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:19:33.946789+0000","last_peered":"2024-06-14T23:19:33.946789+0000","last_clean":"2024-06-14T23:19:33.946789+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:19:33.946789+0000","last_undegraded":"2024-06-14T23:19:33.946789+0000","last_fullsized":"2024-06-14T23:19:33.946789+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":460,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959629+0000","last_change":"2024-06-14T23:19:10.884867+0000","last_active":"2024-06-14T23:19:33.959629+0000","last_peered":"2024-06-14T23:19:33.959629+0000","last_clean":"2024-06-14T23:19:33.959629+0000","last_became_active":"2024-06-14T23:19:10.884115+0000","last_became_peered":"2024-06-14T23:19:10.884115+0000","last_unstale":"2024-06-14T23:19:33.959629+0000","last_undegraded":"2024-06-14T23:19:33.959629+0000","last_fullsized":"2024-06-14T23:19:33.959629+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:25:03.294186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":570,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.958879+0000","last_change":"2024-06-14T23:19:23.473395+0000","last_active":"2024-06-14T23:19:33.958879+0000","last_peered":"2024-06-14T23:19:33.958879+0000","last_clean":"2024-06-14T23:19:33.958879+0000","last_became_active":"2024-06-14T23:19:23.472999+0000","last_became_peered":"2024-06-14T23:19:23.472999+0000","last_unstale":"2024-06-14T23:19:33.958879+0000","last_undegraded":"2024-06-14T23:19:33.958879+0000","last_fullsized":"2024-06-14T23:19:33.958879+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":126,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951511+0000","last_change":"2024-06-14T23:15:09.030189+0000","last_active":"2024-06-14T23:19:33.951511+0000","last_peered":"2024-06-14T23:19:33.951511+0000","last_clean":"2024-06-14T23:19:33.951511+0000","last_became_active":"2024-06-14T23:15:09.029757+0000","last_became_peered":"2024-06-14T23:15:09.029757+0000","last_unstale":"2024-06-14T23:19:33.951511+0000","last_undegraded":"2024-06-14T23:19:33.951511+0000","last_fullsized":"2024-06-14T23:19:33.951511+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:58:48.305434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":368,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.960430+0000","last_change":"2024-06-14T23:15:09.044391+0000","last_active":"2024-06-14T23:19:33.960430+0000","last_peered":"2024-06-14T23:19:33.960430+0000","last_clean":"2024-06-14T23:19:33.960430+0000","last_became_active":"2024-06-14T23:15:09.043763+0000","last_became_peered":"2024-06-14T23:15:09.043763+0000","last_unstale":"2024-06-14T23:19:33.960430+0000","last_undegraded":"2024-06-14T23:19:33.960430+0000","last_fullsized":"2024-06-14T23:19:33.960430+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:46:25.597564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":471,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951356+0000","last_change":"2024-06-14T23:17:53.634166+0000","last_active":"2024-06-14T23:19:33.951356+0000","last_peered":"2024-06-14T23:19:33.951356+0000","last_clean":"2024-06-14T23:19:33.951356+0000","last_became_active":"2024-06-14T23:17:25.900117+0000","last_became_peered":"2024-06-14T23:17:25.900117+0000","last_unstale":"2024-06-14T23:19:33.951356+0000","last_undegraded":"2024-06-14T23:19:33.951356+0000","last_fullsized":"2024-06-14T23:19:33.951356+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.634078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.634078+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:52:58.337328+0000","scrub_duration":0.00160432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.b","version":"427'4100","reported_seq":5063,"reported_epoch":427,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-14T23:19:39.872819+0000","last_change":"2024-06-14T23:19:33.488265+0000","last_active":"2024-06-14T23:19:39.872819+0000","last_peered":"2024-06-14T23:19:39.872819+0000","last_clean":"2024-06-14T23:19:09.838265+0000","last_became_active":"2024-06-14T23:19:11.967481+0000","last_became_peered":"2024-06-14T23:19:11.967481+0000","last_unstale":"2024-06-14T23:19:39.872819+0000","last_undegraded":"2024-06-14T23:19:39.872819+0000","last_fullsized":"2024-06-14T23:19:11.875848+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4100,"log_dups_size":0,"ondisk_log_size":4100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:11:31.974224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268697600,"num_objects":4100,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1877,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4100,"num_write_kb":262400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":30474240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":1877},{"shards":"0,6","objects":2223}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":244,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.960981+0000","last_change":"2024-06-14T23:19:23.895377+0000","last_active":"2024-06-14T23:19:33.960981+0000","last_peered":"2024-06-14T23:19:33.960981+0000","last_clean":"2024-06-14T23:19:33.960981+0000","last_became_active":"2024-06-14T23:19:23.895084+0000","last_became_peered":"2024-06-14T23:19:23.895084+0000","last_unstale":"2024-06-14T23:19:33.960981+0000","last_undegraded":"2024-06-14T23:19:33.960981+0000","last_fullsized":"2024-06-14T23:19:33.960981+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:15:21.807921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":409,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.958970+0000","last_change":"2024-06-14T23:19:10.883009+0000","last_active":"2024-06-14T23:19:33.958970+0000","last_peered":"2024-06-14T23:19:33.958970+0000","last_clean":"2024-06-14T23:19:33.958970+0000","last_became_active":"2024-06-14T23:19:10.882611+0000","last_became_peered":"2024-06-14T23:19:10.882611+0000","last_unstale":"2024-06-14T23:19:33.958970+0000","last_undegraded":"2024-06-14T23:19:33.958970+0000","last_fullsized":"2024-06-14T23:19:33.958970+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:22:34.079775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":125,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.950928+0000","last_change":"2024-06-14T23:17:22.914912+0000","last_active":"2024-06-14T23:19:33.950928+0000","last_peered":"2024-06-14T23:19:33.950928+0000","last_clean":"2024-06-14T23:19:33.950928+0000","last_became_active":"2024-06-14T23:17:22.896173+0000","last_became_peered":"2024-06-14T23:17:22.896173+0000","last_unstale":"2024-06-14T23:19:33.950928+0000","last_undegraded":"2024-06-14T23:19:33.950928+0000","last_fullsized":"2024-06-14T23:19:33.950928+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:58:10.727766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":755,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946777+0000","last_change":"2024-06-14T23:07:10.229317+0000","last_active":"2024-06-14T23:19:33.946777+0000","last_peered":"2024-06-14T23:19:33.946777+0000","last_clean":"2024-06-14T23:19:33.946777+0000","last_became_active":"2024-06-14T23:06:52.050489+0000","last_became_peered":"2024-06-14T23:06:52.050489+0000","last_unstale":"2024-06-14T23:19:33.946777+0000","last_undegraded":"2024-06-14T23:19:33.946777+0000","last_fullsized":"2024-06-14T23:19:33.946777+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:07:47.958642+0000","scrub_duration":0.014545446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"427'4302","reported_seq":4326,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:43.755484+0000","last_change":"2024-06-14T23:18:48.033782+0000","last_active":"2024-06-14T23:19:43.755484+0000","last_peered":"2024-06-14T23:19:43.755484+0000","last_clean":"2024-06-14T23:19:43.755484+0000","last_became_active":"2024-06-14T23:18:48.033507+0000","last_became_peered":"2024-06-14T23:18:48.033507+0000","last_unstale":"2024-06-14T23:19:43.755484+0000","last_undegraded":"2024-06-14T23:19:43.755484+0000","last_fullsized":"2024-06-14T23:19:43.755484+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4302,"log_dups_size":0,"ondisk_log_size":4302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T11:17:15.128767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281935872,"num_objects":4302,"num_object_clones":0,"num_object_copies":12906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":275328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":352,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946260+0000","last_change":"2024-06-14T23:17:22.889183+0000","last_active":"2024-06-14T23:19:33.946260+0000","last_peered":"2024-06-14T23:19:33.946260+0000","last_clean":"2024-06-14T23:19:33.946260+0000","last_became_active":"2024-06-14T23:17:22.888268+0000","last_became_peered":"2024-06-14T23:17:22.888268+0000","last_unstale":"2024-06-14T23:19:33.946260+0000","last_undegraded":"2024-06-14T23:19:33.946260+0000","last_fullsized":"2024-06-14T23:19:33.946260+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:16:12.669660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":345,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946767+0000","last_change":"2024-06-14T23:15:09.029369+0000","last_active":"2024-06-14T23:19:33.946767+0000","last_peered":"2024-06-14T23:19:33.946767+0000","last_clean":"2024-06-14T23:19:33.946767+0000","last_became_active":"2024-06-14T23:15:09.029112+0000","last_became_peered":"2024-06-14T23:15:09.029112+0000","last_unstale":"2024-06-14T23:19:33.946767+0000","last_undegraded":"2024-06-14T23:19:33.946767+0000","last_fullsized":"2024-06-14T23:19:33.946767+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:39:40.009379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":247,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:34.956984+0000","last_change":"2024-06-14T23:17:22.899160+0000","last_active":"2024-06-14T23:19:34.956984+0000","last_peered":"2024-06-14T23:19:34.956984+0000","last_clean":"2024-06-14T23:19:34.956984+0000","last_became_active":"2024-06-14T23:17:22.892666+0000","last_became_peered":"2024-06-14T23:17:22.892666+0000","last_unstale":"2024-06-14T23:19:34.956984+0000","last_undegraded":"2024-06-14T23:19:34.956984+0000","last_fullsized":"2024-06-14T23:19:34.956984+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:21:27.960103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":531,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959880+0000","last_change":"2024-06-14T23:19:13.898135+0000","last_active":"2024-06-14T23:19:33.959880+0000","last_peered":"2024-06-14T23:19:33.959880+0000","last_clean":"2024-06-14T23:19:33.959880+0000","last_became_active":"2024-06-14T23:19:13.897759+0000","last_became_peered":"2024-06-14T23:19:13.897759+0000","last_unstale":"2024-06-14T23:19:33.959880+0000","last_undegraded":"2024-06-14T23:19:33.959880+0000","last_fullsized":"2024-06-14T23:19:33.959880+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:51.748589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:51.748589+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:50:54.671516+0000","scrub_duration":0.0036050689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.9","version":"427'4160","reported_seq":4185,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:40.693661+0000","last_change":"2024-06-14T23:18:48.032161+0000","last_active":"2024-06-14T23:19:40.693661+0000","last_peered":"2024-06-14T23:19:40.693661+0000","last_clean":"2024-06-14T23:19:40.693661+0000","last_became_active":"2024-06-14T23:18:48.031847+0000","last_became_peered":"2024-06-14T23:18:48.031847+0000","last_unstale":"2024-06-14T23:19:40.693661+0000","last_undegraded":"2024-06-14T23:19:40.693661+0000","last_fullsized":"2024-06-14T23:19:40.693661+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:18:59.248622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":4160,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":223,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.955245+0000","last_change":"2024-06-14T23:19:23.471579+0000","last_active":"2024-06-14T23:19:33.955245+0000","last_peered":"2024-06-14T23:19:33.955245+0000","last_clean":"2024-06-14T23:19:33.955245+0000","last_became_active":"2024-06-14T23:19:23.471172+0000","last_became_peered":"2024-06-14T23:19:23.471172+0000","last_unstale":"2024-06-14T23:19:33.955245+0000","last_undegraded":"2024-06-14T23:19:33.955245+0000","last_fullsized":"2024-06-14T23:19:33.955245+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:26.499184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":422,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951972+0000","last_change":"2024-06-14T23:17:22.890776+0000","last_active":"2024-06-14T23:19:33.951972+0000","last_peered":"2024-06-14T23:19:33.951972+0000","last_clean":"2024-06-14T23:19:33.951972+0000","last_became_active":"2024-06-14T23:17:22.890463+0000","last_became_peered":"2024-06-14T23:17:22.890463+0000","last_unstale":"2024-06-14T23:19:33.951972+0000","last_undegraded":"2024-06-14T23:19:33.951972+0000","last_fullsized":"2024-06-14T23:19:33.951972+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:45:42.223422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":315,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959286+0000","last_change":"2024-06-14T23:19:23.473025+0000","last_active":"2024-06-14T23:19:33.959286+0000","last_peered":"2024-06-14T23:19:33.959286+0000","last_clean":"2024-06-14T23:19:33.959286+0000","last_became_active":"2024-06-14T23:19:23.471723+0000","last_became_peered":"2024-06-14T23:19:23.471723+0000","last_unstale":"2024-06-14T23:19:33.959286+0000","last_undegraded":"2024-06-14T23:19:33.959286+0000","last_fullsized":"2024-06-14T23:19:33.959286+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:58:40.116389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":746,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.964430+0000","last_change":"2024-06-14T23:17:25.892084+0000","last_active":"2024-06-14T23:19:33.964430+0000","last_peered":"2024-06-14T23:19:33.964430+0000","last_clean":"2024-06-14T23:19:33.964430+0000","last_became_active":"2024-06-14T23:17:25.891748+0000","last_became_peered":"2024-06-14T23:17:25.891748+0000","last_unstale":"2024-06-14T23:19:33.964430+0000","last_undegraded":"2024-06-14T23:19:33.964430+0000","last_fullsized":"2024-06-14T23:19:33.964430+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:13:20.992100+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"427'3974","reported_seq":4013,"reported_epoch":427,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:19:39.776830+0000","last_change":"2024-06-14T23:19:24.497488+0000","last_active":"2024-06-14T23:19:39.776830+0000","last_peered":"2024-06-14T23:19:39.776830+0000","last_clean":"2024-06-14T23:19:09.846924+0000","last_became_active":"2024-06-14T23:19:24.477518+0000","last_became_peered":"2024-06-14T23:19:24.477518+0000","last_unstale":"2024-06-14T23:19:39.776830+0000","last_undegraded":"2024-06-14T23:19:24.471298+0000","last_fullsized":"2024-06-14T23:19:39.776830+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":3974,"log_dups_size":0,"ondisk_log_size":3974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:13:26.932917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":254336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":2215},{"shards":"0,3,5","objects":1759}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":223,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.965882+0000","last_change":"2024-06-14T23:17:22.905260+0000","last_active":"2024-06-14T23:19:33.965882+0000","last_peered":"2024-06-14T23:19:33.965882+0000","last_clean":"2024-06-14T23:19:33.965882+0000","last_became_active":"2024-06-14T23:17:22.903093+0000","last_became_peered":"2024-06-14T23:17:22.903093+0000","last_unstale":"2024-06-14T23:19:33.965882+0000","last_undegraded":"2024-06-14T23:19:33.965882+0000","last_fullsized":"2024-06-14T23:19:33.965882+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:57.000855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":130,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.950900+0000","last_change":"2024-06-14T23:19:10.883324+0000","last_active":"2024-06-14T23:19:33.950900+0000","last_peered":"2024-06-14T23:19:33.950900+0000","last_clean":"2024-06-14T23:19:33.950900+0000","last_became_active":"2024-06-14T23:19:10.882951+0000","last_became_peered":"2024-06-14T23:19:10.882951+0000","last_unstale":"2024-06-14T23:19:33.950900+0000","last_undegraded":"2024-06-14T23:19:33.950900+0000","last_fullsized":"2024-06-14T23:19:33.950900+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:58:10.955982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":448,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946868+0000","last_change":"2024-06-14T23:19:23.471777+0000","last_active":"2024-06-14T23:19:33.946868+0000","last_peered":"2024-06-14T23:19:33.946868+0000","last_clean":"2024-06-14T23:19:33.946868+0000","last_became_active":"2024-06-14T23:19:23.471478+0000","last_became_peered":"2024-06-14T23:19:23.471478+0000","last_unstale":"2024-06-14T23:19:33.946868+0000","last_undegraded":"2024-06-14T23:19:33.946868+0000","last_fullsized":"2024-06-14T23:19:33.946868+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:12.157654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":706,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959118+0000","last_change":"2024-06-14T23:09:53.304850+0000","last_active":"2024-06-14T23:19:33.959118+0000","last_peered":"2024-06-14T23:19:33.959118+0000","last_clean":"2024-06-14T23:19:33.959118+0000","last_became_active":"2024-06-14T23:09:12.037127+0000","last_became_peered":"2024-06-14T23:09:12.037127+0000","last_unstale":"2024-06-14T23:19:33.959118+0000","last_undegraded":"2024-06-14T23:19:33.959118+0000","last_fullsized":"2024-06-14T23:19:33.959118+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:06.191289+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"427'4316","reported_seq":5172,"reported_epoch":427,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-14T23:19:42.091114+0000","last_change":"2024-06-14T23:19:33.494211+0000","last_active":"2024-06-14T23:19:42.091114+0000","last_peered":"2024-06-14T23:19:42.091114+0000","last_clean":"2024-06-14T23:19:09.849497+0000","last_became_active":"2024-06-14T23:19:11.970122+0000","last_became_peered":"2024-06-14T23:19:11.970122+0000","last_unstale":"2024-06-14T23:19:42.091114+0000","last_undegraded":"2024-06-14T23:19:42.091114+0000","last_fullsized":"2024-06-14T23:19:11.876030+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:41:47.829038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282853376,"num_objects":4316,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1975,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4316,"num_write_kb":276224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":27000832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":1975},{"shards":"4,6","objects":2341}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":287,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946863+0000","last_change":"2024-06-14T23:19:10.886748+0000","last_active":"2024-06-14T23:19:33.946863+0000","last_peered":"2024-06-14T23:19:33.946863+0000","last_clean":"2024-06-14T23:19:33.946863+0000","last_became_active":"2024-06-14T23:19:10.886101+0000","last_became_peered":"2024-06-14T23:19:10.886101+0000","last_unstale":"2024-06-14T23:19:33.946863+0000","last_undegraded":"2024-06-14T23:19:33.946863+0000","last_fullsized":"2024-06-14T23:19:33.946863+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:22:59.159253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":389,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.960362+0000","last_change":"2024-06-14T23:17:22.896004+0000","last_active":"2024-06-14T23:19:33.960362+0000","last_peered":"2024-06-14T23:19:33.960362+0000","last_clean":"2024-06-14T23:19:33.960362+0000","last_became_active":"2024-06-14T23:17:22.895677+0000","last_became_peered":"2024-06-14T23:17:22.895677+0000","last_unstale":"2024-06-14T23:19:33.960362+0000","last_undegraded":"2024-06-14T23:19:33.960362+0000","last_fullsized":"2024-06-14T23:19:33.960362+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:07:44.738249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":426,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959271+0000","last_change":"2024-06-14T23:17:36.972069+0000","last_active":"2024-06-14T23:19:33.959271+0000","last_peered":"2024-06-14T23:19:33.959271+0000","last_clean":"2024-06-14T23:19:33.959271+0000","last_became_active":"2024-06-14T23:17:36.971783+0000","last_became_peered":"2024-06-14T23:17:36.971783+0000","last_unstale":"2024-06-14T23:19:33.959271+0000","last_undegraded":"2024-06-14T23:19:33.959271+0000","last_fullsized":"2024-06-14T23:19:33.959271+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:04:41.244348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":836,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.964743+0000","last_change":"2024-06-14T23:19:13.894900+0000","last_active":"2024-06-14T23:19:33.964743+0000","last_peered":"2024-06-14T23:19:33.964743+0000","last_clean":"2024-06-14T23:19:33.964743+0000","last_became_active":"2024-06-14T23:19:13.894578+0000","last_became_peered":"2024-06-14T23:19:13.894578+0000","last_unstale":"2024-06-14T23:19:33.964743+0000","last_undegraded":"2024-06-14T23:19:33.964743+0000","last_fullsized":"2024-06-14T23:19:33.964743+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.522124+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:17:56.289947+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"427'4267","reported_seq":4291,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:42.060824+0000","last_change":"2024-06-14T23:18:48.032086+0000","last_active":"2024-06-14T23:19:42.060824+0000","last_peered":"2024-06-14T23:19:42.060824+0000","last_clean":"2024-06-14T23:19:42.060824+0000","last_became_active":"2024-06-14T23:18:48.031779+0000","last_became_peered":"2024-06-14T23:18:48.031779+0000","last_unstale":"2024-06-14T23:19:42.060824+0000","last_undegraded":"2024-06-14T23:19:42.060824+0000","last_fullsized":"2024-06-14T23:19:42.060824+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:21.593686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279642112,"num_objects":4267,"num_object_clones":0,"num_object_copies":12801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4267,"num_write_kb":273088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":397,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.964017+0000","last_change":"2024-06-14T23:17:22.902652+0000","last_active":"2024-06-14T23:19:33.964017+0000","last_peered":"2024-06-14T23:19:33.964017+0000","last_clean":"2024-06-14T23:19:33.964017+0000","last_became_active":"2024-06-14T23:17:22.902142+0000","last_became_peered":"2024-06-14T23:17:22.902142+0000","last_unstale":"2024-06-14T23:19:33.964017+0000","last_undegraded":"2024-06-14T23:19:33.964017+0000","last_fullsized":"2024-06-14T23:19:33.964017+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:45.185030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":408,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959060+0000","last_change":"2024-06-14T23:19:10.884690+0000","last_active":"2024-06-14T23:19:33.959060+0000","last_peered":"2024-06-14T23:19:33.959060+0000","last_clean":"2024-06-14T23:19:33.959060+0000","last_became_active":"2024-06-14T23:19:10.884288+0000","last_became_peered":"2024-06-14T23:19:10.884288+0000","last_unstale":"2024-06-14T23:19:33.959060+0000","last_undegraded":"2024-06-14T23:19:33.959060+0000","last_fullsized":"2024-06-14T23:19:33.959060+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:22:25.034254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":211,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946846+0000","last_change":"2024-06-14T23:19:10.884847+0000","last_active":"2024-06-14T23:19:33.946846+0000","last_peered":"2024-06-14T23:19:33.946846+0000","last_clean":"2024-06-14T23:19:33.946846+0000","last_became_active":"2024-06-14T23:19:10.884166+0000","last_became_peered":"2024-06-14T23:19:10.884166+0000","last_unstale":"2024-06-14T23:19:33.946846+0000","last_undegraded":"2024-06-14T23:19:33.946846+0000","last_fullsized":"2024-06-14T23:19:33.946846+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:44:12.618973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":564,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.952470+0000","last_change":"2024-06-14T23:17:52.317490+0000","last_active":"2024-06-14T23:19:33.952470+0000","last_peered":"2024-06-14T23:19:33.952470+0000","last_clean":"2024-06-14T23:19:33.952470+0000","last_became_active":"2024-06-14T23:17:25.912414+0000","last_became_peered":"2024-06-14T23:17:25.912414+0000","last_unstale":"2024-06-14T23:19:33.952470+0000","last_undegraded":"2024-06-14T23:19:33.952470+0000","last_fullsized":"2024-06-14T23:19:33.952470+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:52.317329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:52.317329+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:25:14.327597+0000","scrub_duration":0.0087017850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"427'4101","reported_seq":4130,"reported_epoch":427,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:19:39.878511+0000","last_change":"2024-06-14T23:19:11.983461+0000","last_active":"2024-06-14T23:19:39.878511+0000","last_peered":"2024-06-14T23:19:39.878511+0000","last_clean":"2024-06-14T23:19:09.848418+0000","last_became_active":"2024-06-14T23:19:11.978463+0000","last_became_peered":"2024-06-14T23:19:11.978463+0000","last_unstale":"2024-06-14T23:19:39.878511+0000","last_undegraded":"2024-06-14T23:19:11.899754+0000","last_fullsized":"2024-06-14T23:19:11.875629+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:21:12.903820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268763136,"num_objects":4101,"num_object_clones":0,"num_object_copies":12303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4101,"num_write_kb":262464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[0,4],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2,4","objects":2258},{"shards":"0,4","objects":1843}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":337,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951112+0000","last_change":"2024-06-14T23:19:23.470574+0000","last_active":"2024-06-14T23:19:33.951112+0000","last_peered":"2024-06-14T23:19:33.951112+0000","last_clean":"2024-06-14T23:19:33.951112+0000","last_became_active":"2024-06-14T23:19:23.470021+0000","last_became_peered":"2024-06-14T23:19:23.470021+0000","last_unstale":"2024-06-14T23:19:33.951112+0000","last_undegraded":"2024-06-14T23:19:33.951112+0000","last_fullsized":"2024-06-14T23:19:33.951112+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:34:44.026259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":229,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946435+0000","last_change":"2024-06-14T23:17:22.894491+0000","last_active":"2024-06-14T23:19:33.946435+0000","last_peered":"2024-06-14T23:19:33.946435+0000","last_clean":"2024-06-14T23:19:33.946435+0000","last_became_active":"2024-06-14T23:17:22.894162+0000","last_became_peered":"2024-06-14T23:17:22.894162+0000","last_unstale":"2024-06-14T23:19:33.946435+0000","last_undegraded":"2024-06-14T23:19:33.946435+0000","last_fullsized":"2024-06-14T23:19:33.946435+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:48.949010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":404,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946163+0000","last_change":"2024-06-14T23:17:22.888817+0000","last_active":"2024-06-14T23:19:33.946163+0000","last_peered":"2024-06-14T23:19:33.946163+0000","last_clean":"2024-06-14T23:19:33.946163+0000","last_became_active":"2024-06-14T23:17:22.888355+0000","last_became_peered":"2024-06-14T23:17:22.888355+0000","last_unstale":"2024-06-14T23:19:33.946163+0000","last_undegraded":"2024-06-14T23:19:33.946163+0000","last_fullsized":"2024-06-14T23:19:33.946163+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:10:02.306355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":456,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.960318+0000","last_change":"2024-06-14T23:17:53.735170+0000","last_active":"2024-06-14T23:19:33.960318+0000","last_peered":"2024-06-14T23:19:33.960318+0000","last_clean":"2024-06-14T23:19:33.960318+0000","last_became_active":"2024-06-14T23:17:36.972022+0000","last_became_peered":"2024-06-14T23:17:36.972022+0000","last_unstale":"2024-06-14T23:19:33.960318+0000","last_undegraded":"2024-06-14T23:19:33.960318+0000","last_fullsized":"2024-06-14T23:19:33.960318+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.735114+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:17:53.735114+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.735114+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:44:04.045839+0000","scrub_duration":0.0011940379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.f","version":"427'4012","reported_seq":4035,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:39.052204+0000","last_change":"2024-06-14T23:18:47.964866+0000","last_active":"2024-06-14T23:19:39.052204+0000","last_peered":"2024-06-14T23:19:39.052204+0000","last_clean":"2024-06-14T23:19:39.052204+0000","last_became_active":"2024-06-14T23:18:47.964411+0000","last_became_peered":"2024-06-14T23:18:47.964411+0000","last_unstale":"2024-06-14T23:19:39.052204+0000","last_undegraded":"2024-06-14T23:19:39.052204+0000","last_fullsized":"2024-06-14T23:19:39.052204+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4012,"log_dups_size":0,"ondisk_log_size":4012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:47:20.857507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262930432,"num_objects":4012,"num_object_clones":0,"num_object_copies":12036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":256768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":342,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.960085+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:19:33.960085+0000","last_peered":"2024-06-14T23:19:33.960085+0000","last_clean":"2024-06-14T23:19:33.960085+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:19:33.960085+0000","last_undegraded":"2024-06-14T23:19:33.960085+0000","last_fullsized":"2024-06-14T23:19:33.960085+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":201,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959595+0000","last_change":"2024-06-14T23:19:23.476620+0000","last_active":"2024-06-14T23:19:33.959595+0000","last_peered":"2024-06-14T23:19:33.959595+0000","last_clean":"2024-06-14T23:19:33.959595+0000","last_became_active":"2024-06-14T23:19:23.476367+0000","last_became_peered":"2024-06-14T23:19:23.476367+0000","last_unstale":"2024-06-14T23:19:33.959595+0000","last_undegraded":"2024-06-14T23:19:33.959595+0000","last_fullsized":"2024-06-14T23:19:33.959595+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:38:03.590285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":436,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951155+0000","last_change":"2024-06-14T23:19:23.471979+0000","last_active":"2024-06-14T23:19:33.951155+0000","last_peered":"2024-06-14T23:19:33.951155+0000","last_clean":"2024-06-14T23:19:33.951155+0000","last_became_active":"2024-06-14T23:19:23.471490+0000","last_became_peered":"2024-06-14T23:19:23.471490+0000","last_unstale":"2024-06-14T23:19:33.951155+0000","last_undegraded":"2024-06-14T23:19:33.951155+0000","last_fullsized":"2024-06-14T23:19:33.951155+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:09:16.712112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":671,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.950506+0000","last_change":"2024-06-14T23:17:53.277122+0000","last_active":"2024-06-14T23:19:33.950506+0000","last_peered":"2024-06-14T23:19:33.950506+0000","last_clean":"2024-06-14T23:19:33.950506+0000","last_became_active":"2024-06-14T23:17:25.909540+0000","last_became_peered":"2024-06-14T23:17:25.909540+0000","last_unstale":"2024-06-14T23:19:33.950506+0000","last_undegraded":"2024-06-14T23:19:33.950506+0000","last_fullsized":"2024-06-14T23:19:33.950506+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.277028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.277028+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:37:17.362500+0000","scrub_duration":0.002049375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"427'4086","reported_seq":4110,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:39.034804+0000","last_change":"2024-06-14T23:18:48.033959+0000","last_active":"2024-06-14T23:19:39.034804+0000","last_peered":"2024-06-14T23:19:39.034804+0000","last_clean":"2024-06-14T23:19:39.034804+0000","last_became_active":"2024-06-14T23:18:48.033347+0000","last_became_peered":"2024-06-14T23:18:48.033347+0000","last_unstale":"2024-06-14T23:19:39.034804+0000","last_undegraded":"2024-06-14T23:19:39.034804+0000","last_fullsized":"2024-06-14T23:19:39.034804+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4086,"log_dups_size":0,"ondisk_log_size":4086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:44:39.633537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267780096,"num_objects":4086,"num_object_clones":0,"num_object_copies":12258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":261504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":228,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.947477+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:19:33.947477+0000","last_peered":"2024-06-14T23:19:33.947477+0000","last_clean":"2024-06-14T23:19:33.947477+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:19:33.947477+0000","last_undegraded":"2024-06-14T23:19:33.947477+0000","last_fullsized":"2024-06-14T23:19:33.947477+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":399,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946426+0000","last_change":"2024-06-14T23:10:26.049132+0000","last_active":"2024-06-14T23:19:33.946426+0000","last_peered":"2024-06-14T23:19:33.946426+0000","last_clean":"2024-06-14T23:19:33.946426+0000","last_became_active":"2024-06-14T23:10:26.048640+0000","last_became_peered":"2024-06-14T23:10:26.048640+0000","last_unstale":"2024-06-14T23:19:33.946426+0000","last_undegraded":"2024-06-14T23:19:33.946426+0000","last_fullsized":"2024-06-14T23:19:33.946426+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:43:11.231024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":432,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959430+0000","last_change":"2024-06-14T23:19:13.897316+0000","last_active":"2024-06-14T23:19:33.959430+0000","last_peered":"2024-06-14T23:19:33.959430+0000","last_clean":"2024-06-14T23:19:33.959430+0000","last_became_active":"2024-06-14T23:19:13.896978+0000","last_became_peered":"2024-06-14T23:19:13.896978+0000","last_unstale":"2024-06-14T23:19:33.959430+0000","last_undegraded":"2024-06-14T23:19:33.959430+0000","last_fullsized":"2024-06-14T23:19:33.959430+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:43.479469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:43.479469+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"427'4289","reported_seq":4313,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:42.046328+0000","last_change":"2024-06-14T23:18:47.964784+0000","last_active":"2024-06-14T23:19:42.046328+0000","last_peered":"2024-06-14T23:19:42.046328+0000","last_clean":"2024-06-14T23:19:42.046328+0000","last_became_active":"2024-06-14T23:18:47.964479+0000","last_became_peered":"2024-06-14T23:18:47.964479+0000","last_unstale":"2024-06-14T23:19:42.046328+0000","last_undegraded":"2024-06-14T23:19:42.046328+0000","last_fullsized":"2024-06-14T23:19:42.046328+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:57.833878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281083904,"num_objects":4289,"num_object_clones":0,"num_object_copies":12867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4289,"num_write_kb":274496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":414,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.954093+0000","last_change":"2024-06-14T23:10:26.050654+0000","last_active":"2024-06-14T23:19:33.954093+0000","last_peered":"2024-06-14T23:19:33.954093+0000","last_clean":"2024-06-14T23:19:33.954093+0000","last_became_active":"2024-06-14T23:10:26.050149+0000","last_became_peered":"2024-06-14T23:10:26.050149+0000","last_unstale":"2024-06-14T23:19:33.954093+0000","last_undegraded":"2024-06-14T23:19:33.954093+0000","last_fullsized":"2024-06-14T23:19:33.954093+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:10:05.434865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":337,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.950534+0000","last_change":"2024-06-14T23:19:10.885458+0000","last_active":"2024-06-14T23:19:33.950534+0000","last_peered":"2024-06-14T23:19:33.950534+0000","last_clean":"2024-06-14T23:19:33.950534+0000","last_became_active":"2024-06-14T23:19:10.885144+0000","last_became_peered":"2024-06-14T23:19:10.885144+0000","last_unstale":"2024-06-14T23:19:33.950534+0000","last_undegraded":"2024-06-14T23:19:33.950534+0000","last_fullsized":"2024-06-14T23:19:33.950534+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:14:13.875726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":816,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.960401+0000","last_change":"2024-06-14T23:19:23.473100+0000","last_active":"2024-06-14T23:19:33.960401+0000","last_peered":"2024-06-14T23:19:33.960401+0000","last_clean":"2024-06-14T23:19:33.960401+0000","last_became_active":"2024-06-14T23:19:23.472322+0000","last_became_peered":"2024-06-14T23:19:23.472322+0000","last_unstale":"2024-06-14T23:19:33.960401+0000","last_undegraded":"2024-06-14T23:19:33.960401+0000","last_fullsized":"2024-06-14T23:19:33.960401+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:08:54.694628+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"427'4343","reported_seq":4366,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:43.781355+0000","last_change":"2024-06-14T23:18:48.032106+0000","last_active":"2024-06-14T23:19:43.781355+0000","last_peered":"2024-06-14T23:19:43.781355+0000","last_clean":"2024-06-14T23:19:43.781355+0000","last_became_active":"2024-06-14T23:18:48.031823+0000","last_became_peered":"2024-06-14T23:18:48.031823+0000","last_unstale":"2024-06-14T23:19:43.781355+0000","last_undegraded":"2024-06-14T23:19:43.781355+0000","last_fullsized":"2024-06-14T23:19:43.781355+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:11:19.113099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284622848,"num_objects":4343,"num_object_clones":0,"num_object_copies":13029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":277952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":356,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959817+0000","last_change":"2024-06-14T23:19:10.881971+0000","last_active":"2024-06-14T23:19:33.959817+0000","last_peered":"2024-06-14T23:19:33.959817+0000","last_clean":"2024-06-14T23:19:33.959817+0000","last_became_active":"2024-06-14T23:19:10.881225+0000","last_became_peered":"2024-06-14T23:19:10.881225+0000","last_unstale":"2024-06-14T23:19:33.959817+0000","last_undegraded":"2024-06-14T23:19:33.959817+0000","last_fullsized":"2024-06-14T23:19:33.959817+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:03:18.706381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":279,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959670+0000","last_change":"2024-06-14T23:15:09.037746+0000","last_active":"2024-06-14T23:19:33.959670+0000","last_peered":"2024-06-14T23:19:33.959670+0000","last_clean":"2024-06-14T23:19:33.959670+0000","last_became_active":"2024-06-14T23:15:09.037427+0000","last_became_peered":"2024-06-14T23:15:09.037427+0000","last_unstale":"2024-06-14T23:19:33.959670+0000","last_undegraded":"2024-06-14T23:19:33.959670+0000","last_fullsized":"2024-06-14T23:19:33.959670+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:40:34.511973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":490,"reported_epoch":427,"state":"active+clean+scrubbing","last_fresh":"2024-06-14T23:19:34.956064+0000","last_change":"2024-06-14T23:19:36.693155+0000","last_active":"2024-06-14T23:19:34.956064+0000","last_peered":"2024-06-14T23:19:34.956064+0000","last_clean":"2024-06-14T23:19:34.956064+0000","last_became_active":"2024-06-14T23:19:15.937037+0000","last_became_peered":"2024-06-14T23:19:15.937037+0000","last_unstale":"2024-06-14T23:19:34.956064+0000","last_undegraded":"2024-06-14T23:19:34.956064+0000","last_fullsized":"2024-06-14T23:19:34.956064+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:44.494532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:44.494532+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.6","version":"427'3978","reported_seq":4008,"reported_epoch":427,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-14T23:19:39.846991+0000","last_change":"2024-06-14T23:19:11.988887+0000","last_active":"2024-06-14T23:19:39.846991+0000","last_peered":"2024-06-14T23:19:39.846991+0000","last_clean":"2024-06-14T23:19:09.855587+0000","last_became_active":"2024-06-14T23:19:11.983781+0000","last_became_peered":"2024-06-14T23:19:11.983781+0000","last_unstale":"2024-06-14T23:19:39.846991+0000","last_undegraded":"2024-06-14T23:19:11.895844+0000","last_fullsized":"2024-06-14T23:19:11.875668+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:57:07.232321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260702208,"num_objects":3978,"num_object_clones":0,"num_object_copies":11934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3978,"num_write_kb":254592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[0,3],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":2170},{"shards":"0,3","objects":1808}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":317,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.964066+0000","last_change":"2024-06-14T23:19:23.895155+0000","last_active":"2024-06-14T23:19:33.964066+0000","last_peered":"2024-06-14T23:19:33.964066+0000","last_clean":"2024-06-14T23:19:33.964066+0000","last_became_active":"2024-06-14T23:19:23.894798+0000","last_became_peered":"2024-06-14T23:19:23.894798+0000","last_unstale":"2024-06-14T23:19:33.964066+0000","last_undegraded":"2024-06-14T23:19:33.964066+0000","last_fullsized":"2024-06-14T23:19:33.964066+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:42:35.298855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":191,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:34.957158+0000","last_change":"2024-06-14T23:17:22.896793+0000","last_active":"2024-06-14T23:19:34.957158+0000","last_peered":"2024-06-14T23:19:34.957158+0000","last_clean":"2024-06-14T23:19:34.957158+0000","last_became_active":"2024-06-14T23:17:22.891352+0000","last_became_peered":"2024-06-14T23:17:22.891352+0000","last_unstale":"2024-06-14T23:19:34.957158+0000","last_undegraded":"2024-06-14T23:19:34.957158+0000","last_fullsized":"2024-06-14T23:19:34.957158+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:13:32.350680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":534,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.950416+0000","last_change":"2024-06-14T23:19:28.288626+0000","last_active":"2024-06-14T23:19:33.950416+0000","last_peered":"2024-06-14T23:19:33.950416+0000","last_clean":"2024-06-14T23:19:33.950416+0000","last_became_active":"2024-06-14T23:19:13.892549+0000","last_became_peered":"2024-06-14T23:19:13.892549+0000","last_unstale":"2024-06-14T23:19:33.950416+0000","last_undegraded":"2024-06-14T23:19:33.950416+0000","last_fullsized":"2024-06-14T23:19:33.950416+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:14.052631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:14.052631+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":42,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"427'4093","reported_seq":8620,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:40.749004+0000","last_change":"2024-06-14T23:19:34.962615+0000","last_active":"2024-06-14T23:19:40.749004+0000","last_peered":"2024-06-14T23:19:40.749004+0000","last_clean":"2024-06-14T23:19:40.749004+0000","last_became_active":"2024-06-14T23:19:34.959611+0000","last_became_peered":"2024-06-14T23:19:34.959611+0000","last_unstale":"2024-06-14T23:19:40.749004+0000","last_undegraded":"2024-06-14T23:19:40.749004+0000","last_fullsized":"2024-06-14T23:19:40.749004+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4093,"log_dups_size":0,"ondisk_log_size":4093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:57:40.010560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268238848,"num_objects":4093,"num_object_clones":0,"num_object_copies":12279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":261952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2242,"num_bytes_recovered":146931712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":215,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.955471+0000","last_change":"2024-06-14T23:19:10.883077+0000","last_active":"2024-06-14T23:19:33.955471+0000","last_peered":"2024-06-14T23:19:33.955471+0000","last_clean":"2024-06-14T23:19:33.955471+0000","last_became_active":"2024-06-14T23:19:10.882516+0000","last_became_peered":"2024-06-14T23:19:10.882516+0000","last_unstale":"2024-06-14T23:19:33.955471+0000","last_undegraded":"2024-06-14T23:19:33.955471+0000","last_fullsized":"2024-06-14T23:19:33.955471+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:03.107842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":288,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.947265+0000","last_change":"2024-06-14T23:17:36.971770+0000","last_active":"2024-06-14T23:19:33.947265+0000","last_peered":"2024-06-14T23:19:33.947265+0000","last_clean":"2024-06-14T23:19:33.947265+0000","last_became_active":"2024-06-14T23:17:36.971467+0000","last_became_peered":"2024-06-14T23:17:36.971467+0000","last_unstale":"2024-06-14T23:19:33.947265+0000","last_undegraded":"2024-06-14T23:19:33.947265+0000","last_fullsized":"2024-06-14T23:19:33.947265+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:05:21.955007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":548,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.950819+0000","last_change":"2024-06-14T23:15:12.943908+0000","last_active":"2024-06-14T23:19:33.950819+0000","last_peered":"2024-06-14T23:19:33.950819+0000","last_clean":"2024-06-14T23:19:33.950819+0000","last_became_active":"2024-06-14T23:15:09.031629+0000","last_became_peered":"2024-06-14T23:15:09.031629+0000","last_unstale":"2024-06-14T23:19:33.950819+0000","last_undegraded":"2024-06-14T23:19:33.950819+0000","last_fullsized":"2024-06-14T23:19:33.950819+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:12.943827+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:15:12.943827+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:48:02.155745+0000","scrub_duration":0.0029576450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.0","version":"427'4015","reported_seq":4052,"reported_epoch":427,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-14T23:19:39.847010+0000","last_change":"2024-06-14T23:19:24.498302+0000","last_active":"2024-06-14T23:19:39.847010+0000","last_peered":"2024-06-14T23:19:39.847010+0000","last_clean":"2024-06-14T23:19:09.847157+0000","last_became_active":"2024-06-14T23:19:24.476932+0000","last_became_peered":"2024-06-14T23:19:24.476932+0000","last_unstale":"2024-06-14T23:19:39.847010+0000","last_undegraded":"2024-06-14T23:19:24.470764+0000","last_fullsized":"2024-06-14T23:19:39.847010+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:31:34.966461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263127040,"num_objects":4015,"num_object_clones":0,"num_object_copies":12045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4015,"num_write_kb":256960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,2,5","objects":2248},{"shards":"0,5,6","objects":1767}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":378,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959513+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:19:33.959513+0000","last_peered":"2024-06-14T23:19:33.959513+0000","last_clean":"2024-06-14T23:19:33.959513+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:19:33.959513+0000","last_undegraded":"2024-06-14T23:19:33.959513+0000","last_fullsized":"2024-06-14T23:19:33.959513+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":27,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.960026+0000","last_change":"2024-06-14T23:19:23.476782+0000","last_active":"2024-06-14T23:19:33.960026+0000","last_peered":"2024-06-14T23:19:33.960026+0000","last_clean":"2024-06-14T23:19:33.960026+0000","last_became_active":"2024-06-14T23:19:23.476292+0000","last_became_peered":"2024-06-14T23:19:23.476292+0000","last_unstale":"2024-06-14T23:19:33.960026+0000","last_undegraded":"2024-06-14T23:19:33.960026+0000","last_fullsized":"2024-06-14T23:19:33.960026+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:42:11.010204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":706,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959201+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:19:33.959201+0000","last_peered":"2024-06-14T23:19:33.959201+0000","last_clean":"2024-06-14T23:19:33.959201+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:19:33.959201+0000","last_undegraded":"2024-06-14T23:19:33.959201+0000","last_fullsized":"2024-06-14T23:19:33.959201+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"427'4148","reported_seq":4173,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:40.709065+0000","last_change":"2024-06-14T23:18:48.032366+0000","last_active":"2024-06-14T23:19:40.709065+0000","last_peered":"2024-06-14T23:19:40.709065+0000","last_clean":"2024-06-14T23:19:40.709065+0000","last_became_active":"2024-06-14T23:18:48.031974+0000","last_became_peered":"2024-06-14T23:18:48.031974+0000","last_unstale":"2024-06-14T23:19:40.709065+0000","last_undegraded":"2024-06-14T23:19:40.709065+0000","last_fullsized":"2024-06-14T23:19:40.709065+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:15:09.427667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271843328,"num_objects":4148,"num_object_clones":0,"num_object_copies":12444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":265472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":369,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946513+0000","last_change":"2024-06-14T23:15:09.030332+0000","last_active":"2024-06-14T23:19:33.946513+0000","last_peered":"2024-06-14T23:19:33.946513+0000","last_clean":"2024-06-14T23:19:33.946513+0000","last_became_active":"2024-06-14T23:15:09.030035+0000","last_became_peered":"2024-06-14T23:15:09.030035+0000","last_unstale":"2024-06-14T23:19:33.946513+0000","last_undegraded":"2024-06-14T23:19:33.946513+0000","last_fullsized":"2024-06-14T23:19:33.946513+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:14:00.226413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":406,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951088+0000","last_change":"2024-06-14T23:17:22.916573+0000","last_active":"2024-06-14T23:19:33.951088+0000","last_peered":"2024-06-14T23:19:33.951088+0000","last_clean":"2024-06-14T23:19:33.951088+0000","last_became_active":"2024-06-14T23:17:22.897253+0000","last_became_peered":"2024-06-14T23:17:22.897253+0000","last_unstale":"2024-06-14T23:19:33.951088+0000","last_undegraded":"2024-06-14T23:19:33.951088+0000","last_fullsized":"2024-06-14T23:19:33.951088+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:33:59.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":858,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946779+0000","last_change":"2024-06-14T23:17:25.896480+0000","last_active":"2024-06-14T23:19:33.946779+0000","last_peered":"2024-06-14T23:19:33.946779+0000","last_clean":"2024-06-14T23:19:33.946779+0000","last_became_active":"2024-06-14T23:17:25.896062+0000","last_became_peered":"2024-06-14T23:17:25.896062+0000","last_unstale":"2024-06-14T23:19:33.946779+0000","last_undegraded":"2024-06-14T23:19:33.946779+0000","last_fullsized":"2024-06-14T23:19:33.946779+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:15.822461+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"427'4322","reported_seq":4346,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-06-14T23:19:42.058517+0000","last_change":"2024-06-14T23:18:47.965214+0000","last_active":"2024-06-14T23:19:42.058517+0000","last_peered":"2024-06-14T23:19:42.058517+0000","last_clean":"2024-06-14T23:19:42.058517+0000","last_became_active":"2024-06-14T23:18:47.964921+0000","last_became_peered":"2024-06-14T23:18:47.964921+0000","last_unstale":"2024-06-14T23:19:42.058517+0000","last_undegraded":"2024-06-14T23:19:42.058517+0000","last_fullsized":"2024-06-14T23:19:42.058517+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4322,"log_dups_size":0,"ondisk_log_size":4322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:04.291087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283246592,"num_objects":4322,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4322,"num_write_kb":276608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":240,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959445+0000","last_change":"2024-06-14T23:19:23.479101+0000","last_active":"2024-06-14T23:19:33.959445+0000","last_peered":"2024-06-14T23:19:33.959445+0000","last_clean":"2024-06-14T23:19:33.959445+0000","last_became_active":"2024-06-14T23:19:23.478364+0000","last_became_peered":"2024-06-14T23:19:23.478364+0000","last_unstale":"2024-06-14T23:19:33.959445+0000","last_undegraded":"2024-06-14T23:19:33.959445+0000","last_fullsized":"2024-06-14T23:19:33.959445+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:57:30.628738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":419,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.954927+0000","last_change":"2024-06-14T23:17:22.904229+0000","last_active":"2024-06-14T23:19:33.954927+0000","last_peered":"2024-06-14T23:19:33.954927+0000","last_clean":"2024-06-14T23:19:33.954927+0000","last_became_active":"2024-06-14T23:17:22.896077+0000","last_became_peered":"2024-06-14T23:17:22.896077+0000","last_unstale":"2024-06-14T23:19:33.954927+0000","last_undegraded":"2024-06-14T23:19:33.954927+0000","last_fullsized":"2024-06-14T23:19:33.954927+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:05.630800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":800,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959646+0000","last_change":"2024-06-14T23:19:13.898381+0000","last_active":"2024-06-14T23:19:33.959646+0000","last_peered":"2024-06-14T23:19:33.959646+0000","last_clean":"2024-06-14T23:19:33.959646+0000","last_became_active":"2024-06-14T23:19:13.897735+0000","last_became_peered":"2024-06-14T23:19:13.897735+0000","last_unstale":"2024-06-14T23:19:33.959646+0000","last_undegraded":"2024-06-14T23:19:33.959646+0000","last_fullsized":"2024-06-14T23:19:33.959646+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:17:35.215870+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":433,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.958905+0000","last_change":"2024-06-14T23:17:22.910264+0000","last_active":"2024-06-14T23:19:33.958905+0000","last_peered":"2024-06-14T23:19:33.958905+0000","last_clean":"2024-06-14T23:19:33.958905+0000","last_became_active":"2024-06-14T23:17:22.900551+0000","last_became_peered":"2024-06-14T23:17:22.900551+0000","last_unstale":"2024-06-14T23:19:33.958905+0000","last_undegraded":"2024-06-14T23:19:33.958905+0000","last_fullsized":"2024-06-14T23:19:33.958905+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:01.672829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":421,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946858+0000","last_change":"2024-06-14T23:19:10.885799+0000","last_active":"2024-06-14T23:19:33.946858+0000","last_peered":"2024-06-14T23:19:33.946858+0000","last_clean":"2024-06-14T23:19:33.946858+0000","last_became_active":"2024-06-14T23:19:10.885481+0000","last_became_peered":"2024-06-14T23:19:10.885481+0000","last_unstale":"2024-06-14T23:19:33.946858+0000","last_undegraded":"2024-06-14T23:19:33.946858+0000","last_fullsized":"2024-06-14T23:19:33.946858+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:51:22.042956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":577,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959179+0000","last_change":"2024-06-14T23:19:23.473442+0000","last_active":"2024-06-14T23:19:33.959179+0000","last_peered":"2024-06-14T23:19:33.959179+0000","last_clean":"2024-06-14T23:19:33.959179+0000","last_became_active":"2024-06-14T23:19:23.472785+0000","last_became_peered":"2024-06-14T23:19:23.472785+0000","last_unstale":"2024-06-14T23:19:33.959179+0000","last_undegraded":"2024-06-14T23:19:33.959179+0000","last_fullsized":"2024-06-14T23:19:33.959179+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001647253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":228,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.960910+0000","last_change":"2024-06-14T23:15:09.039681+0000","last_active":"2024-06-14T23:19:33.960910+0000","last_peered":"2024-06-14T23:19:33.960910+0000","last_clean":"2024-06-14T23:19:33.960910+0000","last_became_active":"2024-06-14T23:15:09.039180+0000","last_became_peered":"2024-06-14T23:15:09.039180+0000","last_unstale":"2024-06-14T23:19:33.960910+0000","last_undegraded":"2024-06-14T23:19:33.960910+0000","last_fullsized":"2024-06-14T23:19:33.960910+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:54.312788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":302,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.947345+0000","last_change":"2024-06-14T23:17:22.891259+0000","last_active":"2024-06-14T23:19:33.947345+0000","last_peered":"2024-06-14T23:19:33.947345+0000","last_clean":"2024-06-14T23:19:33.947345+0000","last_became_active":"2024-06-14T23:17:22.890498+0000","last_became_peered":"2024-06-14T23:17:22.890498+0000","last_unstale":"2024-06-14T23:19:33.947345+0000","last_undegraded":"2024-06-14T23:19:33.947345+0000","last_fullsized":"2024-06-14T23:19:33.947345+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:10:37.941550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":768,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946383+0000","last_change":"2024-06-14T23:17:25.894661+0000","last_active":"2024-06-14T23:19:33.946383+0000","last_peered":"2024-06-14T23:19:33.946383+0000","last_clean":"2024-06-14T23:19:33.946383+0000","last_became_active":"2024-06-14T23:17:25.893869+0000","last_became_peered":"2024-06-14T23:17:25.893869+0000","last_unstale":"2024-06-14T23:19:33.946383+0000","last_undegraded":"2024-06-14T23:19:33.946383+0000","last_fullsized":"2024-06-14T23:19:33.946383+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:17:44.833265+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":52,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.950651+0000","last_change":"2024-06-14T23:17:22.887257+0000","last_active":"2024-06-14T23:19:33.950651+0000","last_peered":"2024-06-14T23:19:33.950651+0000","last_clean":"2024-06-14T23:19:33.950651+0000","last_became_active":"2024-06-14T23:17:22.886972+0000","last_became_peered":"2024-06-14T23:17:22.886972+0000","last_unstale":"2024-06-14T23:19:33.950651+0000","last_undegraded":"2024-06-14T23:19:33.950651+0000","last_fullsized":"2024-06-14T23:19:33.950651+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:42:02.720189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":105,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.946922+0000","last_change":"2024-06-14T23:19:23.471655+0000","last_active":"2024-06-14T23:19:33.946922+0000","last_peered":"2024-06-14T23:19:33.946922+0000","last_clean":"2024-06-14T23:19:33.946922+0000","last_became_active":"2024-06-14T23:19:23.470993+0000","last_became_peered":"2024-06-14T23:19:23.470993+0000","last_unstale":"2024-06-14T23:19:33.946922+0000","last_undegraded":"2024-06-14T23:19:33.946922+0000","last_fullsized":"2024-06-14T23:19:33.946922+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:57:59.933536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":415,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.959003+0000","last_change":"2024-06-14T23:19:23.479406+0000","last_active":"2024-06-14T23:19:33.959003+0000","last_peered":"2024-06-14T23:19:33.959003+0000","last_clean":"2024-06-14T23:19:33.959003+0000","last_became_active":"2024-06-14T23:19:23.478774+0000","last_became_peered":"2024-06-14T23:19:23.478774+0000","last_unstale":"2024-06-14T23:19:33.959003+0000","last_undegraded":"2024-06-14T23:19:33.959003+0000","last_fullsized":"2024-06-14T23:19:33.959003+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":455,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.951085+0000","last_change":"2024-06-14T23:19:10.893082+0000","last_active":"2024-06-14T23:19:33.951085+0000","last_peered":"2024-06-14T23:19:33.951085+0000","last_clean":"2024-06-14T23:19:33.951085+0000","last_became_active":"2024-06-14T23:19:10.888779+0000","last_became_peered":"2024-06-14T23:19:10.888779+0000","last_unstale":"2024-06-14T23:19:33.951085+0000","last_undegraded":"2024-06-14T23:19:33.951085+0000","last_fullsized":"2024-06-14T23:19:33.951085+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:10:49.227729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":165,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-06-14T23:19:33.947198+0000","last_change":"2024-06-14T23:19:23.471698+0000","last_active":"2024-06-14T23:19:33.947198+0000","last_peered":"2024-06-14T23:19:33.947198+0000","last_clean":"2024-06-14T23:19:33.947198+0000","last_became_active":"2024-06-14T23:19:23.470762+0000","last_became_peered":"2024-06-14T23:19:23.470762+0000","last_unstale":"2024-06-14T23:19:33.947198+0000","last_undegraded":"2024-06-14T23:19:33.947198+0000","last_fullsized":"2024-06-14T23:19:33.947198+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:09:09.308256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":627,"reported_epoch":427,"state":"active+clean+scrubbing","last_fresh":"2024-06-14T23:19:35.677381+0000","last_change":"2024-06-14T23:19:35.677381+0000","last_active":"2024-06-14T23:19:35.677381+0000","last_peered":"2024-06-14T23:19:35.677381+0000","last_clean":"2024-06-14T23:19:35.677381+0000","last_became_active":"2024-06-14T23:19:23.472051+0000","last_became_peered":"2024-06-14T23:19:23.472051+0000","last_unstale":"2024-06-14T23:19:35.677381+0000","last_undegraded":"2024-06-14T23:19:35.677381+0000","last_fullsized":"2024-06-14T23:19:35.677381+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013568390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1878,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":17252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":14,"num_pg":16,"stat_sum":{"num_bytes":4358537216,"num_objects":66506,"num_object_clones":0,"num_object_copies":199518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8891,"num_objects_misplaced":3852,"num_objects_unfound":0,"num_objects_dirty":66506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66506,"num_write_kb":4256384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3120,"num_bytes_recovered":204472320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13286899712,"data_stored":13286899712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66506,"ondisk_log_size":66506,"up":48,"acting":44,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":729015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":214,"num_write":55,"num_write_kb":565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":743619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":92,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234547,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2401712,"kb_used_data":898920,"kb_used_omap":1,"kb_used_meta":1502782,"kb_avail":91343440,"statfs":{"total":95995035648,"available":93535682560,"internally_reserved":0,"allocated":920494080,"data_stored":918299822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1538849112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:18:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.99199999999999999,"15min":0.96899999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.214},"max":{"1min":0.68700000000000006,"5min":12.263999999999999,"15min":12.263999999999999},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.0860000000000001,"15min":1.081},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.224},"max":{"1min":5.6719999999999997,"5min":5.6719999999999997,"15min":6.5149999999999997},"last":3.8889999999999998}]},{"osd":1,"last update":"Fri Jun 14 23:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":1.109,"15min":1.2},"min":{"1min":0.39100000000000001,"5min":0.25600000000000001,"15min":0.22600000000000001},"max":{"1min":1.4770000000000001,"5min":5.7309999999999999,"15min":12.010999999999999},"last":3.98},{"interface":"front","average":{"1min":0.50900000000000001,"5min":1.0720000000000001,"15min":1.2689999999999999},"min":{"1min":0.32500000000000001,"5min":0.27700000000000002,"15min":0.27500000000000002},"max":{"1min":0.85799999999999998,"5min":12.295,"15min":13.531000000000001},"last":1.3899999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.0029999999999999,"15min":1.2629999999999999},"min":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.0800000000000001,"5min":6.0819999999999999,"15min":10.539},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":1.0580000000000001,"15min":1.2430000000000001},"min":{"1min":0.41099999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0299999999999998,"5min":6.1529999999999996,"15min":10.56},"last":0.46300000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:18:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.0049999999999999,"15min":1.343},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.28299999999999997},"max":{"1min":2.5760000000000001,"5min":6.1130000000000004,"15min":20.645},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.294,"5min":1.1220000000000001,"15min":1.4650000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.28299999999999997},"max":{"1min":5.7080000000000002,"5min":5.7080000000000002,"15min":22.042999999999999},"last":3.9489999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":1.528,"15min":1.1899999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.23100000000000001},"max":{"1min":4.0099999999999998,"5min":6.1539999999999999,"15min":6.1539999999999999},"last":4.0099999999999998},{"interface":"front","average":{"1min":0.75,"5min":1.3360000000000001,"15min":1.165},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":2.2669999999999999,"5min":6.2199999999999998,"15min":10.499000000000001},"last":0.44}]},{"osd":5,"last update":"Fri Jun 14 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.5629999999999999,"15min":1.1579999999999999},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.23699999999999999},"max":{"1min":5.1349999999999998,"5min":12.236000000000001,"15min":12.236000000000001},"last":5.1349999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.923,"15min":1.357},"min":{"1min":0.35899999999999999,"5min":0.29299999999999998,"15min":0.24399999999999999},"max":{"1min":7.1950000000000003,"5min":12.842000000000001,"15min":12.842000000000001},"last":7.1950000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":1.5880000000000001,"15min":1.599},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.198,"5min":12.317,"15min":12.317},"last":1.329},{"interface":"front","average":{"1min":0.83899999999999997,"5min":1.639,"15min":1.5780000000000001},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.7759999999999998,"5min":10.050000000000001,"15min":10.087999999999999},"last":6.9509999999999996}]}]},{"osd":6,"up_from":272,"seq":1168231104654,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2977316,"kb_used_data":2012316,"kb_used_omap":7,"kb_used_meta":964984,"kb_avail":90767836,"statfs":{"total":95995035648,"available":92946264064,"internally_reserved":0,"allocated":2060611584,"data_stored":2058429205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":988144058},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":1.218,"15min":1.327},"min":{"1min":0.33700000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.3540000000000001,"5min":6.109,"15min":6.6440000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":1.393,"15min":1.425},"min":{"1min":0.40799999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8079999999999998,"5min":14.148999999999999,"15min":14.148999999999999},"last":0.37}]},{"osd":1,"last update":"Fri Jun 14 23:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.109,"15min":1.538},"min":{"1min":0.31,"5min":0.27200000000000002,"15min":0.25800000000000001},"max":{"1min":2.0649999999999999,"5min":5.6710000000000003,"15min":8.173},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":1.3700000000000001,"15min":1.5660000000000001},"min":{"1min":0.39700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":3.641,"5min":14.086,"15min":14.086},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.5049999999999999,"15min":1.2450000000000001},"min":{"1min":0.35999999999999999,"5min":0.249,"15min":0.249},"max":{"1min":7.6909999999999998,"5min":29.940999999999999,"15min":29.940999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.266,"5min":1.544,"15min":1.3149999999999999},"min":{"1min":0.33100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":8.0679999999999996,"5min":30.135999999999999,"15min":30.135999999999999},"last":0.5}]},{"osd":3,"last update":"Fri Jun 14 23:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.2989999999999999,"15min":1.4510000000000001},"min":{"1min":0.373,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.482,"5min":6.0839999999999996,"15min":6.7089999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.2,"15min":1.6259999999999999},"min":{"1min":0.36399999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.0030000000000001,"5min":6.1449999999999996,"15min":8.4740000000000002},"last":0.51900000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.413,"15min":1.4279999999999999},"min":{"1min":0.27600000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.627,"5min":6.6299999999999999,"15min":11.010999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.78,"15min":1.4059999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.22900000000000001},"max":{"1min":3.8359999999999999,"5min":14.199999999999999,"15min":14.199999999999999},"last":5.9779999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":2.403,"15min":1.619},"min":{"1min":0.308,"5min":0.29999999999999999,"15min":0.28199999999999997},"max":{"1min":3.7730000000000001,"5min":26.172000000000001,"15min":26.172000000000001},"last":2.3090000000000002},{"interface":"front","average":{"1min":0.754,"5min":1.976,"15min":1.675},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.141,"5min":25.946999999999999,"15min":25.946999999999999},"last":1.171}]},{"osd":7,"last update":"Fri Jun 14 23:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.617,"15min":1.712},"min":{"1min":0.32100000000000001,"5min":0.29999999999999999,"15min":0.29399999999999998},"max":{"1min":3.1139999999999999,"5min":6.9829999999999997,"15min":10.016999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.849,"15min":1.6759999999999999},"min":{"1min":0.34799999999999998,"5min":0.32700000000000001,"15min":0.26000000000000001},"max":{"1min":2.105,"5min":10.906000000000001,"15min":10.906000000000001},"last":0.45400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757823,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3554648,"kb_used_data":2074960,"kb_used_omap":1,"kb_used_meta":1479678,"kb_avail":90190504,"statfs":{"total":95995035648,"available":92355076096,"internally_reserved":0,"allocated":2124759040,"data_stored":2122584703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1515190616},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.5229999999999999,"15min":1.115},"min":{"1min":0.29999999999999999,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":1.9630000000000001,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":1.6399999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.3420000000000001,"15min":1.1319999999999999},"min":{"1min":0.28999999999999998,"5min":0.23100000000000001,"15min":0.219},"max":{"1min":2,"5min":13.398999999999999,"15min":13.398999999999999},"last":1.599}]},{"osd":1,"last update":"Fri Jun 14 23:19:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.2030000000000001,"15min":1.03},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.245},"max":{"1min":2.0910000000000002,"5min":7.6219999999999999,"15min":7.6219999999999999},"last":1.5820000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.573,"15min":1.177},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":1.671,"5min":8.4480000000000004,"15min":8.4480000000000004},"last":1.591}]},{"osd":2,"last update":"Fri Jun 14 23:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.671,"15min":1.0189999999999999},"min":{"1min":0.314,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.073,"5min":8.3989999999999991,"15min":8.3989999999999991},"last":1.617},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.72,"15min":1.0409999999999999},"min":{"1min":0.315,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.1320000000000001,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.665}]},{"osd":3,"last update":"Fri Jun 14 23:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.7050000000000001,"15min":1.3560000000000001},"min":{"1min":0.27800000000000002,"5min":0.25,"15min":0.249},"max":{"1min":3.4510000000000001,"5min":21.815000000000001,"15min":21.815000000000001},"last":1.4790000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.9870000000000001,"15min":1.4590000000000001},"min":{"1min":0.33900000000000002,"5min":0.313,"15min":0.28899999999999998},"max":{"1min":2.9199999999999999,"5min":21.797000000000001,"15min":21.797000000000001},"last":1.554}]},{"osd":4,"last update":"Fri Jun 14 23:19:36 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.849,"15min":1.1499999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.23599999999999999},"max":{"1min":7.75,"5min":8.3109999999999999,"15min":8.3109999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.119,"5min":2.2599999999999998,"15min":1.327},"min":{"1min":0.312,"5min":0.28699999999999998,"15min":0.253},"max":{"1min":4.3010000000000002,"5min":13.443,"15min":13.443},"last":1.524}]},{"osd":6,"last update":"Fri Jun 14 23:19:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.944,"15min":1.55},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.26100000000000001},"max":{"1min":7.8600000000000003,"5min":10.893000000000001,"15min":30.934000000000001},"last":2.1589999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":2.2989999999999999,"15min":1.8280000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.26100000000000001},"max":{"1min":4.9299999999999997,"5min":13.465999999999999,"15min":31.059999999999999},"last":1.6279999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":2.3140000000000001,"15min":1.546},"min":{"1min":0.33000000000000002,"5min":0.32400000000000001,"15min":0.25},"max":{"1min":3.5659999999999998,"5min":9.7040000000000006,"15min":26.981999999999999},"last":2.589},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.994,"15min":1.506},"min":{"1min":0.36899999999999999,"5min":0.36699999999999999,"15min":0.26400000000000001},"max":{"1min":4.1600000000000001,"5min":10.925000000000001,"15min":23.326000000000001},"last":0.58899999999999997}]}]},{"osd":4,"up_from":30,"seq":128849019459,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3971620,"kb_used_data":2483484,"kb_used_omap":1,"kb_used_meta":1488126,"kb_avail":89773532,"statfs":{"total":95995035648,"available":91928096768,"internally_reserved":0,"allocated":2543087616,"data_stored":2540904767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1523841368},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.014,"15min":0.96999999999999997},"min":{"1min":0.22600000000000001,"5min":0.221,"15min":0.217},"max":{"1min":5.1580000000000004,"5min":5.1580000000000004,"15min":6.875},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.0900000000000001,"15min":0.996},"min":{"1min":0.26800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.0390000000000001,"5min":5.5979999999999999,"15min":7.4900000000000002},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Jun 14 23:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":1.014,"15min":1.1120000000000001},"min":{"1min":0.309,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":5.8730000000000002,"5min":6.6790000000000003,"15min":42.656999999999996},"last":2.4670000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":1.0349999999999999,"15min":1.1259999999999999},"min":{"1min":0.35899999999999999,"5min":0.28399999999999997,"15min":0.249},"max":{"1min":5.2030000000000003,"5min":5.2030000000000003,"15min":42.192999999999998},"last":0.66100000000000003}]},{"osd":2,"last update":"Fri Jun 14 23:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.151,"15min":0.93600000000000005},"min":{"1min":0.33200000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.9769999999999999,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":2.5270000000000001},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.2110000000000001,"15min":0.999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.956,"5min":6.6550000000000002,"15min":6.6550000000000002},"last":2.5499999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:19:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.1870000000000001,"15min":1.25},"min":{"1min":0.30199999999999999,"5min":0.27500000000000002,"15min":0.25},"max":{"1min":3,"5min":5.5700000000000003,"15min":44.389000000000003},"last":0.498},{"interface":"front","average":{"1min":1.022,"5min":1.0980000000000001,"15min":1.3520000000000001},"min":{"1min":0.34799999999999998,"5min":0.29599999999999999,"15min":0.29099999999999998},"max":{"1min":5.2329999999999997,"5min":5.2329999999999997,"15min":46.101999999999997},"last":0.58599999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.7509999999999999,"15min":1.204},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.20300000000000001},"max":{"1min":5.3559999999999999,"5min":8.173,"15min":12.315},"last":2.5600000000000001},{"interface":"front","average":{"1min":1.6439999999999999,"5min":1.7210000000000001,"15min":1.196},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":6.5599999999999996,"5min":7.7789999999999999,"15min":10.959},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.6519999999999999,"15min":1.7190000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.25800000000000001},"max":{"1min":6.9109999999999996,"5min":6.9109999999999996,"15min":122.88500000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.6930000000000001,"15min":1.901},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.25700000000000001},"max":{"1min":2.9769999999999999,"5min":7.2439999999999998,"15min":122.21299999999999},"last":0.60499999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":2.0800000000000001,"15min":1.4039999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":6.7910000000000004,"5min":11.430999999999999,"15min":11.430999999999999},"last":2.5019999999999998},{"interface":"front","average":{"1min":1.298,"5min":1.8300000000000001,"15min":1.337},"min":{"1min":0.376,"5min":0.36299999999999999,"15min":0.27300000000000002},"max":{"1min":6.8470000000000004,"5min":7.2930000000000001,"15min":8.5340000000000007},"last":0.55600000000000005}]}]},{"osd":3,"up_from":138,"seq":592705487157,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3504660,"kb_used_data":2375820,"kb_used_omap":4,"kb_used_meta":1128827,"kb_avail":90240492,"statfs":{"total":95995035648,"available":92406263808,"internally_reserved":0,"allocated":2432839680,"data_stored":2430668668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":1155919326},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.32100000000000001,"15min":0.91000000000000003},"min":{"1min":0.19600000000000001,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":1.325,"5min":1.325,"15min":25.190000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.39000000000000001,"15min":0.96299999999999997},"min":{"1min":0.221,"5min":0.215,"15min":0.17799999999999999},"max":{"1min":2.391,"5min":2.391,"15min":25.652000000000001},"last":0.23599999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.40000000000000002,"15min":0.98699999999999999},"min":{"1min":0.246,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":25.754000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.45200000000000001,"15min":1.1519999999999999},"min":{"1min":0.26500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":2.4710000000000001,"5min":2.4710000000000001,"15min":27.992999999999999},"last":0.372}]},{"osd":2,"last update":"Fri Jun 14 23:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.53200000000000003,"15min":0.754},"min":{"1min":0.22600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":5.9189999999999996,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.65100000000000002},"min":{"1min":0.31900000000000001,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":1.3240000000000001,"5min":2.2909999999999999,"15min":3.5550000000000002},"last":0.34399999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:18:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":1.036,"15min":1.7350000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":4.8129999999999997,"5min":6.0490000000000004,"15min":73.263000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":1.075,"15min":1.627},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":3.3170000000000002,"5min":5.7130000000000001,"15min":73.5},"last":2.2629999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:18:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.3520000000000001,"15min":1.7},"min":{"1min":0.39200000000000002,"5min":0.29999999999999999,"15min":0.28699999999999998},"max":{"1min":4.8689999999999998,"5min":12.718999999999999,"15min":81.438999999999993},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.3460000000000001,"15min":1.7310000000000001},"min":{"1min":0.379,"5min":0.318,"15min":0.29299999999999998},"max":{"1min":6.0730000000000004,"5min":12.706,"15min":81.626999999999995},"last":0.42899999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":1.474,"15min":1.6339999999999999},"min":{"1min":0.41099999999999998,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":2.46,"5min":15.458,"15min":25.869},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.2769999999999999,"15min":1.466},"min":{"1min":0.35599999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.407,"5min":15.699,"15min":25.143000000000001},"last":6.125}]},{"osd":7,"last update":"Fri Jun 14 23:18:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.0449999999999999,"15min":1.456},"min":{"1min":0.42999999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.8230000000000004,"5min":4.8230000000000004,"15min":44.887999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.753,"5min":1.0960000000000001,"15min":1.462},"min":{"1min":0.42199999999999999,"5min":0.38600000000000001,"15min":0.33800000000000002},"max":{"1min":1.861,"5min":4.1920000000000002,"15min":45.835000000000001},"last":0.438}]}]},{"osd":2,"up_from":0,"seq":1382979469413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":498216206702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":91,"seq":390842024356,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3228112,"kb_used_data":2199624,"kb_used_omap":5,"kb_used_meta":1028474,"kb_avail":90517040,"statfs":{"total":95995035648,"available":92689448960,"internally_reserved":0,"allocated":2252414976,"data_stored":2250245969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":1053157466},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.371,"15min":0.88300000000000001},"min":{"1min":0.253,"5min":0.20300000000000001,"15min":0.17499999999999999},"max":{"1min":0.71199999999999997,"5min":2.5099999999999998,"15min":34.448},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.373,"15min":0.873},"min":{"1min":0.23000000000000001,"5min":0.19500000000000001,"15min":0.16900000000000001},"max":{"1min":1.248,"5min":1.248,"15min":34.354999999999997},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41699999999999998,"15min":0.56000000000000005},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.20899999999999999},"max":{"1min":0.90000000000000002,"5min":2.6549999999999998,"15min":4.0659999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.44400000000000001,"15min":0.505},"min":{"1min":0.26000000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.4139999999999999,"5min":2.657,"15min":3.2509999999999999},"last":0.91000000000000003}]},{"osd":3,"last update":"Fri Jun 14 23:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.42899999999999999,"15min":1.073},"min":{"1min":0.27800000000000002,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":1.726,"5min":1.726,"15min":35.094999999999999},"last":0.504},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.47099999999999997,"15min":1.1539999999999999},"min":{"1min":0.29499999999999998,"5min":0.217,"15min":0.217},"max":{"1min":2.6059999999999999,"5min":2.6059999999999999,"15min":38.444000000000003},"last":0.97099999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:19:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.165,"15min":1.2230000000000001},"min":{"1min":0.379,"5min":0.32200000000000001,"15min":0.27000000000000002},"max":{"1min":2.5659999999999998,"5min":6.556,"15min":24.170999999999999},"last":1.5389999999999999},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.181,"15min":1.137},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.26200000000000001},"max":{"1min":4.7359999999999998,"5min":5.0439999999999996,"15min":24.204000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:19:00 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.5820000000000001,"15min":1.224},"min":{"1min":0.38500000000000001,"5min":0.28599999999999998,"15min":0.28199999999999997},"max":{"1min":4.4249999999999998,"5min":45.274999999999999,"15min":45.274999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.571,"15min":1.1779999999999999},"min":{"1min":0.32600000000000001,"5min":0.315,"15min":0.315},"max":{"1min":2.9910000000000001,"5min":45.241999999999997,"15min":45.241999999999997},"last":1.008}]},{"osd":6,"last update":"Fri Jun 14 23:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.1439999999999999,"15min":1.3149999999999999},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.33200000000000002},"max":{"1min":2.7320000000000002,"5min":5.6689999999999996,"15min":24.600000000000001},"last":1.873},{"interface":"front","average":{"1min":0.873,"5min":1.1970000000000001,"15min":1.4670000000000001},"min":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.681,"5min":5.6879999999999997,"15min":24.594999999999999},"last":2.48}]},{"osd":7,"last update":"Fri Jun 14 23:19:00 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.048,"15min":1.2130000000000001},"min":{"1min":0.45400000000000001,"5min":0.35999999999999999,"15min":0.33600000000000002},"max":{"1min":5.0110000000000001,"5min":5.8879999999999999,"15min":24.131},"last":1.042},{"interface":"front","average":{"1min":0.83399999999999996,"5min":1.1599999999999999,"15min":1.3220000000000001},"min":{"1min":0.42899999999999999,"5min":0.313,"15min":0.313},"max":{"1min":3.0830000000000002,"5min":5.8289999999999997,"15min":24.257000000000001},"last":0.94599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2269118464,"data_stored":2269118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":892665856,"data_stored":892665856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2434269184,"data_stored":2434269184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2568290304,"data_stored":2568290304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2122448896,"data_stored":2122448896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2064515072,"data_stored":2064515072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":935591936,"data_stored":935591936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:19:48.218 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-14T23:19:48.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set .mgr pg_num 51 2024-06-14T23:19:48.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:48.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:48 smithi137 ceph-mon[90634]: 1.11 scrub starts 2024-06-14T23:19:48.250 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:48.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:48.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:48 smithi086 ceph-mon[102754]: 1.11 scrub starts 2024-06-14T23:19:48.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:48 smithi086 ceph-mon[97913]: 1.11 scrub starts 2024-06-14T23:19:48.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:48.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 52 16 73857 73841 88.7416 59.5625 0.0132451 0.0112626 2024-06-14T23:19:48.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:48.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:48.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:48.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:48.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:48.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:48.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:48.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:48.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:48.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:48.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:48.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:48.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:49.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:49.049 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:49.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:49.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:49.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:49.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:19:49.230 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:49.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:49.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:49.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:49.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:49 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:19:49.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:49 smithi086 ceph-mon[102754]: pgmap v1557: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s; 8892/213096 objects degraded (4.173%); 3334/213096 objects misplaced (1.565%); 6.5 MiB/s, 103 objects/s recovering 2024-06-14T23:19:49.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:49 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8892/213096 objects degraded (4.173%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:49.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:49 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:19:49.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:49 smithi086 ceph-mon[97913]: pgmap v1557: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s; 8892/213096 objects degraded (4.173%); 3334/213096 objects misplaced (1.565%); 6.5 MiB/s, 103 objects/s recovering 2024-06-14T23:19:49.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:49 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8892/213096 objects degraded (4.173%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:49.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 53 16 74820 74804 88.2028 60.1875 0.0122638 0.0113319 2024-06-14T23:19:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:49.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:49.397 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:49 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:19:49.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:49 smithi137 ceph-mon[90634]: pgmap v1557: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s; 8892/213096 objects degraded (4.173%); 3334/213096 objects misplaced (1.565%); 6.5 MiB/s, 103 objects/s recovering 2024-06-14T23:19:49.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:49 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8892/213096 objects degraded (4.173%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:49.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:49.500 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:49.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:49.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:49.633 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:49.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:49.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:49.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:49.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:49.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:49.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:49.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:50.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:50.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:50.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:50.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:50.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:50.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:50.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:50.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 54 16 75752 75736 87.648 58.25 0.0126285 0.0114027 2024-06-14T23:19:50.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:50.473 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:50.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:50.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:50.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:50.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:50.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:50.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:50.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:50.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:50.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:50.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:50.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:50.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:50.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:51.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:51.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:51.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:50 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:51.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:50 smithi086 ceph-mon[102754]: pgmap v1558: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 65 MiB/s wr, 1.04k op/s; 8892/216684 objects degraded (4.104%); 3334/216684 objects misplaced (1.539%); 6.5 MiB/s, 103 objects/s recovering 2024-06-14T23:19:51.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:50 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:51.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:50 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:51.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:50 smithi086 ceph-mon[97913]: pgmap v1558: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 65 MiB/s wr, 1.04k op/s; 8892/216684 objects degraded (4.104%); 3334/216684 objects misplaced (1.539%); 6.5 MiB/s, 103 objects/s recovering 2024-06-14T23:19:51.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:50 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:51.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:51.165 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:51.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:50 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:50 smithi137 ceph-mon[90634]: pgmap v1558: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 65 MiB/s wr, 1.04k op/s; 8892/216684 objects degraded (4.104%); 3334/216684 objects misplaced (1.539%); 6.5 MiB/s, 103 objects/s recovering 2024-06-14T23:19:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:50 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:51.251 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:19:51.251 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:19:51.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:51.280 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:19:51.280 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:19:51.308 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:19:51.308 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:19:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:51.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:51.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:51.334 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:19:51.334 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:19:51.363 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:19:51.363 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:19:51.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 55 16 76767 76751 87.2077 63.4375 0.00975117 0.0114606 2024-06-14T23:19:51.394 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:19:51.394 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:19:51.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:51.422 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:19:51.423 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:19:51.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:51.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:51.455 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:19:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:19:51.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:51.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:51.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:51.605 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:51.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:51.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:51.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:51.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:51.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:51.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:51.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:52.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:52.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:52.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:52.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:52.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:52.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:52.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:52.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:51 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2621249063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:52.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:52.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:52.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:51 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2621249063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:52.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:51 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2621249063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:19:52.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 56 16 77780 77764 86.7809 63.3125 0.0126058 0.0115182 2024-06-14T23:19:52.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:52.394 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:52.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:52.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:52.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:52.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:52.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:52.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:52.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:52.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:52.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:52.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:52.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:52.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:52.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:52.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:53.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:53.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:53.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:53.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:53.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:53.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:53.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:53.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:52 smithi086 ceph-mon[97913]: 1.11 scrub starts 2024-06-14T23:19:53.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:52 smithi086 ceph-mon[97913]: pgmap v1559: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 8892/223869 objects degraded (3.972%); 2996/223869 objects misplaced (1.338%); 6.5 MiB/s, 104 objects/s recovering 2024-06-14T23:19:53.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:52 smithi086 ceph-mon[102754]: 1.11 scrub starts 2024-06-14T23:19:53.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:52 smithi086 ceph-mon[102754]: pgmap v1559: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 8892/223869 objects degraded (3.972%); 2996/223869 objects misplaced (1.338%); 6.5 MiB/s, 104 objects/s recovering 2024-06-14T23:19:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:52 smithi137 ceph-mon[90634]: 1.11 scrub starts 2024-06-14T23:19:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:52 smithi137 ceph-mon[90634]: pgmap v1559: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 8892/223869 objects degraded (3.972%); 2996/223869 objects misplaced (1.338%); 6.5 MiB/s, 104 objects/s recovering 2024-06-14T23:19:53.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:53.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:53.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:53.363 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 57 16 78795 78779 86.3713 63.4375 0.0110654 0.0115723 2024-06-14T23:19:53.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:53.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:53.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:53.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:53.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:53.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:53.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:53.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:53.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:53.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:53.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:53.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:53.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:54.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:54.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:54.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:53 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8892/223869 objects degraded (3.972%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:54.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:53 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8892/223869 objects degraded (3.972%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:54.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:54.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:54.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:54.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:54.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:53 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8892/223869 objects degraded (3.972%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:54.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:54.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:54.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 58 16 79786 79770 85.9497 61.9375 0.01127 0.0116284 2024-06-14T23:19:54.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:54.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:54.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:54.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:54.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:54.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:54.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:54.697 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:54.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:54.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:54.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:54.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:54.863 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:19:54.863 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:19:54.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:55.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:55.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:55.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:54 smithi086 ceph-mon[97913]: pgmap v1560: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 62 MiB/s wr, 993 op/s; 8892/229398 objects degraded (3.876%); 2800/229398 objects misplaced (1.221%); 6.6 MiB/s, 105 objects/s recovering 2024-06-14T23:19:55.089 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:54 smithi086 ceph-mon[102754]: pgmap v1560: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 62 MiB/s wr, 993 op/s; 8892/229398 objects degraded (3.876%); 2800/229398 objects misplaced (1.221%); 6.6 MiB/s, 105 objects/s recovering 2024-06-14T23:19:55.089 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:19:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:19:54] "GET /metrics HTTP/1.1" 200 36711 "" "Prometheus/2.43.0" 2024-06-14T23:19:55.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:55.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:55.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:55.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:54 smithi137 ceph-mon[90634]: pgmap v1560: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 62 MiB/s wr, 993 op/s; 8892/229398 objects degraded (3.876%); 2800/229398 objects misplaced (1.221%); 6.6 MiB/s, 105 objects/s recovering 2024-06-14T23:19:55.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:55.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:55.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:55.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:55.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 59 16 80780 80764 85.5458 62.125 0.0130359 0.0116844 2024-06-14T23:19:55.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:55.437 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:55.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:55.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:55.579 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:55.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:55.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:19:55.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:55.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:19:55.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:19:55.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:55.852 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:55.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:55.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:55.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:55.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:56.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:56.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:56.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:56.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:56.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:56.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:56.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:19:56.361870+0000 min lat: 0.00111701 max lat: 4.60763 avg lat: 0.0117388 2024-06-14T23:19:56.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:19:56.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 60 16 81760 81744 85.1408 61.25 0.010823 0.0117388 2024-06-14T23:19:56.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:56.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:56.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:56.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:56.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:56.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:56.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:56.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:56.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:56.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:56.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-14T23:19:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:56.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:56.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:56.964 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:56.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:57.063 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:56 smithi086 ceph-mon[97913]: pgmap v1561: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 61 MiB/s wr, 978 op/s; 8892/238635 objects degraded (3.726%); 2432/238635 objects misplaced (1.019%); 6.7 MiB/s, 107 objects/s recovering 2024-06-14T23:19:57.063 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:56 smithi086 ceph-mon[102754]: pgmap v1561: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 61 MiB/s wr, 978 op/s; 8892/238635 objects degraded (3.726%); 2432/238635 objects misplaced (1.019%); 6.7 MiB/s, 107 objects/s recovering 2024-06-14T23:19:57.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:57.113 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:57.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:57.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:57.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:57.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:56 smithi137 ceph-mon[90634]: pgmap v1561: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 61 MiB/s wr, 978 op/s; 8892/238635 objects degraded (3.726%); 2432/238635 objects misplaced (1.019%); 6.7 MiB/s, 107 objects/s recovering 2024-06-14T23:19:57.261 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:57.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:57.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:57.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 61 16 82750 82734 84.7594 61.875 0.0197161 0.011793 2024-06-14T23:19:57.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:57.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:57.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:57.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:57.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:57.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:57.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:57.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:57.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:57.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:57.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:57.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:57.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:58.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:58.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:58.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:58.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:58.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:19:58.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:58.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:19:58.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:58.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:58.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:58.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 62 16 83715 83699 84.365 60.3125 0.0164408 0.0118479 2024-06-14T23:19:58.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:19:58.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:58.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:58.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:58.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:58.637 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:58.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:58.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:58.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:58 smithi137 ceph-mon[90634]: pgmap v1562: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 58 MiB/s wr, 933 op/s; 8892/242412 objects degraded (3.668%); 2198/242412 objects misplaced (0.907%); 6.9 MiB/s, 111 objects/s recovering 2024-06-14T23:19:58.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:58.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:58.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:58.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:58.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:58 smithi086 ceph-mon[97913]: pgmap v1562: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 58 MiB/s wr, 933 op/s; 8892/242412 objects degraded (3.668%); 2198/242412 objects misplaced (0.907%); 6.9 MiB/s, 111 objects/s recovering 2024-06-14T23:19:58.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:19:58.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:19:58.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:58 smithi086 ceph-mon[102754]: pgmap v1562: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 4.9 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 58 MiB/s wr, 933 op/s; 8892/242412 objects degraded (3.668%); 2198/242412 objects misplaced (0.907%); 6.9 MiB/s, 111 objects/s recovering 2024-06-14T23:19:58.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:19:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:58.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:58.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:19:59.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:19:59.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:59.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:19:59.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:59.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:59.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:59.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:59.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:59.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:59.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 63 16 84715 84699 84.0178 62.5 0.0107621 0.0118966 2024-06-14T23:19:59.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:19:59.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:59.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:19:59.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:19:59.605 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:19:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:59.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:19:59.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:19:59.731 INFO:teuthology.orchestra.run.smithi086.stderr:Error EINVAL: specified pg_num 51 < pg_num_max 32 2024-06-14T23:19:59.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:19:59 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8892/242412 objects degraded (3.668%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:59.732 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:19:59 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8892/242412 objects degraded (3.668%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:59.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:59.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:19:59.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:19:59 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8892/242412 objects degraded (3.668%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:19:59.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:19:59.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:19:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:19:59.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:19:59.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:00.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:00.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:00.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:00.158 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:00.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:00.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:00.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:00.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:00.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 64 16 85684 85668 83.6512 60.5625 0.0192337 0.0119494 2024-06-14T23:20:00.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:00.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:00.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:00.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:00.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:00.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:00.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:00.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:00.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:00.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3284171540' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-06-14T23:20:00.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: pgmap v1563: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 5.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 57 MiB/s wr, 919 op/s; 8892/246201 objects degraded (3.612%); 2198/246201 objects misplaced (0.893%); 5.9 MiB/s, 94 objects/s recovering 2024-06-14T23:20:00.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:20:00.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: Health detail: HEALTH_WARN Degraded data redundancy: 8892/242412 objects degraded (3.668%), 5 pgs degraded 2024-06-14T23:20:00.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: [WRN] PG_DEGRADED: Degraded data redundancy: 8892/242412 objects degraded (3.668%), 5 pgs degraded 2024-06-14T23:20:00.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: pg 1.18 is active+recovery_wait+undersized+degraded+remapped, acting [5,7] 2024-06-14T23:20:00.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: pg 14.0 is active+recovery_wait+degraded, acting [0,5,6] 2024-06-14T23:20:00.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: pg 14.6 is active+recovery_wait+undersized+degraded+remapped, acting [0,3] 2024-06-14T23:20:00.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: pg 14.8 is active+recovery_wait+degraded, acting [0,3,5] 2024-06-14T23:20:00.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:00 smithi137 ceph-mon[90634]: pg 14.c is active+recovery_wait+undersized+degraded+remapped, acting [0,4] 2024-06-14T23:20:00.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:00.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:00.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:00.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3284171540' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-06-14T23:20:00.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: pgmap v1563: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 5.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 57 MiB/s wr, 919 op/s; 8892/246201 objects degraded (3.612%); 2198/246201 objects misplaced (0.893%); 5.9 MiB/s, 94 objects/s recovering 2024-06-14T23:20:00.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: Health detail: HEALTH_WARN Degraded data redundancy: 8892/242412 objects degraded (3.668%), 5 pgs degraded 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: [WRN] PG_DEGRADED: Degraded data redundancy: 8892/242412 objects degraded (3.668%), 5 pgs degraded 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: pg 1.18 is active+recovery_wait+undersized+degraded+remapped, acting [5,7] 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: pg 14.0 is active+recovery_wait+degraded, acting [0,5,6] 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: pg 14.6 is active+recovery_wait+undersized+degraded+remapped, acting [0,3] 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: pg 14.8 is active+recovery_wait+degraded, acting [0,3,5] 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[97913]: pg 14.c is active+recovery_wait+undersized+degraded+remapped, acting [0,4] 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3284171540' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"}]: dispatch 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: pgmap v1563: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 5.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 57 MiB/s wr, 919 op/s; 8892/246201 objects degraded (3.612%); 2198/246201 objects misplaced (0.893%); 5.9 MiB/s, 94 objects/s recovering 2024-06-14T23:20:00.834 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:20:00.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: Health detail: HEALTH_WARN Degraded data redundancy: 8892/242412 objects degraded (3.668%), 5 pgs degraded 2024-06-14T23:20:00.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: [WRN] PG_DEGRADED: Degraded data redundancy: 8892/242412 objects degraded (3.668%), 5 pgs degraded 2024-06-14T23:20:00.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: pg 1.18 is active+recovery_wait+undersized+degraded+remapped, acting [5,7] 2024-06-14T23:20:00.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: pg 14.0 is active+recovery_wait+degraded, acting [0,5,6] 2024-06-14T23:20:00.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: pg 14.6 is active+recovery_wait+undersized+degraded+remapped, acting [0,3] 2024-06-14T23:20:00.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: pg 14.8 is active+recovery_wait+degraded, acting [0,3,5] 2024-06-14T23:20:00.835 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:00 smithi086 ceph-mon[102754]: pg 14.c is active+recovery_wait+undersized+degraded+remapped, acting [0,4] 2024-06-14T23:20:00.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:00.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:00.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:01.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:01.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:01.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:01.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:01.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:01.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:01.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:01.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:01.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 65 16 86630 86614 83.2738 59.125 0.00975413 0.0120025 2024-06-14T23:20:01.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:01.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:01.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:01.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:01.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:01.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:01.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:01.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:01.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:01.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:01.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:01.915 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:01.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:02.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:02.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:02.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:02.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:02.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:02.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:02.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:02.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:02.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:02.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:02.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:02.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 66 16 87628 87612 82.9571 62.375 0.0110181 0.0120489 2024-06-14T23:20:02.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:02.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:02.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:02.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:02.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:02.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:02.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:02.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:02.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:02.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:02.893 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:02.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:02.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:03.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:03.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:03.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:03.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:02 smithi086 ceph-mon[97913]: pgmap v1564: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 5.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 8892/253494 objects degraded (3.508%); 1789/253494 objects misplaced (0.706%); 8.0 MiB/s, 128 objects/s recovering 2024-06-14T23:20:03.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:02 smithi086 ceph-mon[102754]: pgmap v1564: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 5.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 8892/253494 objects degraded (3.508%); 1789/253494 objects misplaced (0.706%); 8.0 MiB/s, 128 objects/s recovering 2024-06-14T23:20:03.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:03.160 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:03.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:03.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:02 smithi137 ceph-mon[90634]: pgmap v1564: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 5.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 8892/253494 objects degraded (3.508%); 1789/253494 objects misplaced (0.706%); 8.0 MiB/s, 128 objects/s recovering 2024-06-14T23:20:03.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:03.303 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:03.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:03.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:03.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 67 16 88618 88602 82.6424 61.875 0.0195673 0.0120955 2024-06-14T23:20:03.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:03.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:03.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:03.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:03.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:03.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:03.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:03.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:03.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:03.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:03.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:03.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:03.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:04.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:04.104 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:04.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:04.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:04.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:04.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:04.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:03 smithi137 ceph-mon[90634]: 1.11 scrub starts 2024-06-14T23:20:04.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:03 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8892/253494 objects degraded (3.508%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:04.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:03 smithi086 ceph-mon[97913]: 1.11 scrub starts 2024-06-14T23:20:04.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:03 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8892/253494 objects degraded (3.508%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:04.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:03 smithi086 ceph-mon[102754]: 1.11 scrub starts 2024-06-14T23:20:04.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:03 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8892/253494 objects degraded (3.508%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:04.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:04.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 68 16 89574 89558 82.3057 59.75 0.00861379 0.0121443 2024-06-14T23:20:04.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:04.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:04.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:04.508 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:04.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:04.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:04.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:04.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:04.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:04.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:04.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:04.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:04.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:05.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:05.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:05.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:05.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:05.173 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:05.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:05.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:04 smithi137 ceph-mon[90634]: pgmap v1565: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 61 MiB/s wr, 979 op/s; 8892/259149 objects degraded (3.431%); 1563/259149 objects misplaced (0.603%); 7.5 MiB/s, 119 objects/s recovering 2024-06-14T23:20:05.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:05.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:05.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:04 smithi086 ceph-mon[97913]: pgmap v1565: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 61 MiB/s wr, 979 op/s; 8892/259149 objects degraded (3.431%); 1563/259149 objects misplaced (0.603%); 7.5 MiB/s, 119 objects/s recovering 2024-06-14T23:20:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:04 smithi086 ceph-mon[102754]: pgmap v1565: 119 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 61 MiB/s wr, 979 op/s; 8892/259149 objects degraded (3.431%); 1563/259149 objects misplaced (0.603%); 7.5 MiB/s, 119 objects/s recovering 2024-06-14T23:20:05.338 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:20:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:20:04] "GET /metrics HTTP/1.1" 200 36711 "" "Prometheus/2.43.0" 2024-06-14T23:20:05.364 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 69 16 90430 90414 81.8881 53.5 0.00345195 0.0121722 2024-06-14T23:20:05.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:05.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:05.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:05.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:05.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:05.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:05.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:05.673 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:05.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:05.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:05.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:05.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:05.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:05.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:05.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:05.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:06.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:06.069 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:06.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:06.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:05 smithi086 ceph-mon[97913]: 1.8 deep-scrub starts 2024-06-14T23:20:06.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:05 smithi086 ceph-mon[97913]: osdmap e429: 8 total, 8 up, 6 in 2024-06-14T23:20:06.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:05 smithi086 ceph-mon[102754]: 1.8 deep-scrub starts 2024-06-14T23:20:06.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:05 smithi086 ceph-mon[102754]: osdmap e429: 8 total, 8 up, 6 in 2024-06-14T23:20:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-14T23:20:06.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:06.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:06.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:06.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:05 smithi137 ceph-mon[90634]: 1.8 deep-scrub starts 2024-06-14T23:20:06.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:05 smithi137 ceph-mon[90634]: osdmap e429: 8 total, 8 up, 6 in 2024-06-14T23:20:06.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:06.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:06.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:06.365 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 70 16 90922 90906 81.1576 30.75 0.00774988 0.0123164 2024-06-14T23:20:06.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:06.482 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:06.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:06.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:06.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:06.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:06.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:06.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:06.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:06.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:06.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:06.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:06.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:07.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:07.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:07.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:07.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:07.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:06 smithi137 ceph-mon[90634]: pgmap v1567: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 5.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 61 MiB/s wr, 973 op/s; 8892/267861 objects degraded (3.320%); 1141/267861 objects misplaced (0.426%); 8.1 MiB/s, 129 objects/s recovering 2024-06-14T23:20:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:06 smithi137 ceph-mon[90634]: osdmap e430: 8 total, 8 up, 6 in 2024-06-14T23:20:07.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:06 smithi137 ceph-mon[90634]: 1.14 scrub starts 2024-06-14T23:20:07.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:07.315 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:07.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:07.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:06 smithi086 ceph-mon[97913]: pgmap v1567: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 5.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 61 MiB/s wr, 973 op/s; 8892/267861 objects degraded (3.320%); 1141/267861 objects misplaced (0.426%); 8.1 MiB/s, 129 objects/s recovering 2024-06-14T23:20:07.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:06 smithi086 ceph-mon[97913]: osdmap e430: 8 total, 8 up, 6 in 2024-06-14T23:20:07.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:06 smithi086 ceph-mon[97913]: 1.14 scrub starts 2024-06-14T23:20:07.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:06 smithi086 ceph-mon[102754]: pgmap v1567: 119 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 5.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 61 MiB/s wr, 973 op/s; 8892/267861 objects degraded (3.320%); 1141/267861 objects misplaced (0.426%); 8.1 MiB/s, 129 objects/s recovering 2024-06-14T23:20:07.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:06 smithi086 ceph-mon[102754]: osdmap e430: 8 total, 8 up, 6 in 2024-06-14T23:20:07.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:06 smithi086 ceph-mon[102754]: 1.14 scrub starts 2024-06-14T23:20:07.365 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 71 16 91944 91928 80.9141 63.875 0.00879328 0.0123519 2024-06-14T23:20:07.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:07.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:07.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:07.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:07.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:07.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:07.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:07.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:07.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:07.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:07.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:07.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:07.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:08.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:08.129 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:08.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:08.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:08.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:08.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:08.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:08.365 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 72 16 93010 92994 80.7156 66.625 0.023173 0.0123841 2024-06-14T23:20:08.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:08.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:08.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:08.522 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:08.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:08.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:08.632 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:08.667 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:08.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:08.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:08.801 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:08.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:08.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:08.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:08.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:08.943 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:20:09.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:09.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:09.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:09.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:09.179 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[102754]: pgmap v1569: 119 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s; 8892/271041 objects degraded (3.281%); 751/271041 objects misplaced (0.277%); 11 MiB/s, 180 objects/s recovering 2024-06-14T23:20:09.180 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[102754]: 1.11 scrub starts 2024-06-14T23:20:09.180 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3813514282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:20:09.180 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[102754]: 1.9 scrub starts 2024-06-14T23:20:09.180 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8892/271041 objects degraded (3.281%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:09.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[97913]: pgmap v1569: 119 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s; 8892/271041 objects degraded (3.281%); 751/271041 objects misplaced (0.277%); 11 MiB/s, 180 objects/s recovering 2024-06-14T23:20:09.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[97913]: 1.11 scrub starts 2024-06-14T23:20:09.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3813514282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:20:09.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[97913]: 1.9 scrub starts 2024-06-14T23:20:09.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:08 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8892/271041 objects degraded (3.281%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:09.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:09.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:09.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:08 smithi137 ceph-mon[90634]: pgmap v1569: 119 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s; 8892/271041 objects degraded (3.281%); 751/271041 objects misplaced (0.277%); 11 MiB/s, 180 objects/s recovering 2024-06-14T23:20:09.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:08 smithi137 ceph-mon[90634]: 1.11 scrub starts 2024-06-14T23:20:09.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:08 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3813514282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:20:09.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:08 smithi137 ceph-mon[90634]: 1.9 scrub starts 2024-06-14T23:20:09.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:08 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8892/271041 objects degraded (3.281%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:09.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:09.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:09.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:09.365 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 73 16 94039 94023 80.4908 64.3125 0.0277041 0.0124187 2024-06-14T23:20:09.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:09.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:09.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:09.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:09.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:09.603 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:09.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:09.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:09.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:09.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:09.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:09.873 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:09.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:09.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:10.018 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:10.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:10.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:10.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:10.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:10.192 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:09 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:20:10.192 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:09 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:20:10.192 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3813514282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:20:10.192 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:09 smithi086 ceph-mon[97913]: osdmap e431: 8 total, 8 up, 6 in 2024-06-14T23:20:10.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:09 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:20:10.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:09 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:20:10.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3813514282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:20:10.192 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:09 smithi086 ceph-mon[102754]: osdmap e431: 8 total, 8 up, 6 in 2024-06-14T23:20:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:09 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:20:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:09 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:20:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3813514282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:20:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:09 smithi137 ceph-mon[90634]: osdmap e431: 8 total, 8 up, 6 in 2024-06-14T23:20:10.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:10.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:10.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:10.365 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 74 16 95088 95072 80.289 65.5625 0.019412 0.0124495 2024-06-14T23:20:10.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:10.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:10.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:10.480 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:20:10.481 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:20:10.509 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:20:10.509 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:20:10.535 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:20:10.535 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:20:10.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:10.569 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:20:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:20:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:10.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:10.602 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:20:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:20:10.631 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:20:10.631 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:20:10.658 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:20:10.658 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:20:10.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:10.686 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:20:10.686 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:20:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:10.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:10.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:10.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:10.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:10.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:10.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:10.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:10.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:10.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:10.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:10 smithi086 ceph-mon[97913]: pgmap v1571: 119 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 53 MiB/s wr, 846 op/s; 8892/274389 objects degraded (3.241%); 751/274389 objects misplaced (0.274%); 8.5 MiB/s, 135 objects/s recovering 2024-06-14T23:20:10.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:10 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3356067779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:10.997 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:10 smithi086 ceph-mon[102754]: pgmap v1571: 119 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 53 MiB/s wr, 846 op/s; 8892/274389 objects degraded (3.241%); 751/274389 objects misplaced (0.274%); 8.5 MiB/s, 135 objects/s recovering 2024-06-14T23:20:10.997 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:10 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3356067779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:11.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:11.128 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:11.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:11.135 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:20:11.135 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:20:11.135 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.32160640678487873 2024-06-14T23:20:11.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 3 0.32160640678487873 2024-06-14T23:20:11.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:11.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:10 smithi137 ceph-mon[90634]: pgmap v1571: 119 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 53 MiB/s wr, 846 op/s; 8892/274389 objects degraded (3.241%); 751/274389 objects misplaced (0.274%); 8.5 MiB/s, 135 objects/s recovering 2024-06-14T23:20:11.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:10 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3356067779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:11.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:11.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:11.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:11.365 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 75 16 96138 96122 80.0934 65.625 0.0157265 0.0124804 2024-06-14T23:20:11.400 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:11.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:11.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:11.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:11.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:11.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:11.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:11.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:11.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:11.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:11.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:11.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:11.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:11.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:11.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:12.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:12.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:12.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:12.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:12.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:12.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:12.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:20:12.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:12.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:12.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:12.365 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 76 16 97155 97139 79.8758 63.5625 0.0252177 0.0125142 2024-06-14T23:20:12.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:12.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:12.471 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:12.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:12.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:12.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:12.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:12.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:12.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:12.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:12.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:12.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:12.873 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:12.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:12.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:12.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:13.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:13.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:13.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:13.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:13.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:13.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:13.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:12 smithi137 ceph-mon[90634]: pgmap v1572: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 62 MiB/s wr, 992 op/s; 8892/281451 objects degraded (3.159%); 751/281451 objects misplaced (0.267%); 7.3 MiB/s, 117 objects/s recovering 2024-06-14T23:20:13.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:13.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:13.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:13.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:13.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:12 smithi086 ceph-mon[102754]: pgmap v1572: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 62 MiB/s wr, 992 op/s; 8892/281451 objects degraded (3.159%); 751/281451 objects misplaced (0.267%); 7.3 MiB/s, 117 objects/s recovering 2024-06-14T23:20:13.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:13.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:13.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:12 smithi086 ceph-mon[97913]: pgmap v1572: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 62 MiB/s wr, 992 op/s; 8892/281451 objects degraded (3.159%); 751/281451 objects misplaced (0.267%); 7.3 MiB/s, 117 objects/s recovering 2024-06-14T23:20:13.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:13.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:13.365 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 77 15 98159 98144 79.6541 62.8125 0.0126075 0.0125488 2024-06-14T23:20:13.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:13.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:13.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:13.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:13.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:13.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:13.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:13.698 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:13.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:13.710 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:13.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:13.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:13.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:13.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:13.986 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:13.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:13.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:14.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:14.128 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:14.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:14.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:14.248 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8892/281451 objects degraded (3.159%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:14.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:14.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:14.283 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8892/281451 objects degraded (3.159%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:14.283 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8892/281451 objects degraded (3.159%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:14.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 78 16 99057 99041 79.3516 56.0625 0.0205728 0.0125959 2024-06-14T23:20:14.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:14.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:14.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:14.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:14.549 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:14.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:14.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:14.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:14.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:14.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:14.814 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:14.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:14.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:14.947 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:14.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:15.030 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:20:15.030 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:20:15.030 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-06-14T23:20:15.030 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-06-14T23:20:15.030 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:20:15.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:20:15.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:15.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:15.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:15.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:15.202 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:14 smithi137 ceph-mon[90634]: pgmap v1573: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 5.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 50 MiB/s wr, 805 op/s; 8892/287193 objects degraded (3.096%); 449/287193 objects misplaced (0.156%); 5.4 MiB/s, 86 objects/s recovering 2024-06-14T23:20:15.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:15.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:15.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:15.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:15.338 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:20:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:20:14] "GET /metrics HTTP/1.1" 200 36709 "" "Prometheus/2.43.0" 2024-06-14T23:20:15.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:14 smithi086 ceph-mon[97913]: pgmap v1573: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 5.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 50 MiB/s wr, 805 op/s; 8892/287193 objects degraded (3.096%); 449/287193 objects misplaced (0.156%); 5.4 MiB/s, 86 objects/s recovering 2024-06-14T23:20:15.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:14 smithi086 ceph-mon[102754]: pgmap v1573: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 5.8 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 50 MiB/s wr, 805 op/s; 8892/287193 objects degraded (3.096%); 449/287193 objects misplaced (0.156%); 5.4 MiB/s, 86 objects/s recovering 2024-06-14T23:20:15.356 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:15.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:15.365 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 79 16 100020 100004 79.109 60.1875 0.0130241 0.0126354 2024-06-14T23:20:15.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:15.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:15.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:15.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:15.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:15.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:15.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:15.763 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:15.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:15.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:15.898 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:15.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:16.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:16.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:16.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:16.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:16.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:16.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:16.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:16.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:16.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:16.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:2024-06-14T23:20:16.363777+0000 min lat: 0.00111701 max lat: 4.60763 avg lat: 0.0126603 2024-06-14T23:20:16.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-14T23:20:16.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 80 16 101090 101074 78.9558 66.875 0.0079386 0.0126603 2024-06-14T23:20:16.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:16.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:16.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:16.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:16.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:16.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:16.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:16.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-14T23:20:16.704 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:16.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:16.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:16.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:16.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:16.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:16.924 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:16.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:16.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:16.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:17.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:17.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:17.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:17.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:16 smithi137 ceph-mon[90634]: pgmap v1574: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 6.0 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 61 MiB/s wr, 969 op/s; 8892/296733 objects degraded (2.997%); 449/296733 objects misplaced (0.151%); 4.4 MiB/s, 69 objects/s recovering 2024-06-14T23:20:17.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:16 smithi086 ceph-mon[102754]: pgmap v1574: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 6.0 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 61 MiB/s wr, 969 op/s; 8892/296733 objects degraded (2.997%); 449/296733 objects misplaced (0.151%); 4.4 MiB/s, 69 objects/s recovering 2024-06-14T23:20:17.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:16 smithi086 ceph-mon[97913]: pgmap v1574: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 6.0 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 61 MiB/s wr, 969 op/s; 8892/296733 objects degraded (2.997%); 449/296733 objects misplaced (0.151%); 4.4 MiB/s, 69 objects/s recovering 2024-06-14T23:20:17.253 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:17.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:17.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:17.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 81 16 102134 102118 78.7865 65.25 0.00728602 0.0126872 2024-06-14T23:20:17.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:17.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:17.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:17.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:17.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:17.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:17.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:17.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:17.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:17.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:17.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:17.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:17.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:17.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:18.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:18.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:18.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:18.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:18.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:18.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:18.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:18.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 82 16 103176 103160 78.6199 65.125 0.0144469 0.0127142 2024-06-14T23:20:18.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:18.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:18.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:18.486 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.3 to 0.321606 (5254) 2024-06-14T23:20:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:18.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:18.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:18.646 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:18.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:18.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:18.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:18.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:18.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:18.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:18.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:19.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:19.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:19.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:19.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:19.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:19.166 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[97913]: pgmap v1575: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 6.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 61 MiB/s wr, 974 op/s; 8892/300291 objects degraded (2.961%); 91/300291 objects misplaced (0.030%); 4.1 MiB/s, 65 objects/s recovering 2024-06-14T23:20:19.166 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/617834543' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.32160640678487873}]: dispatch 2024-06-14T23:20:19.166 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 8892/300291 objects degraded (2.961%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:19.166 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/617834543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.32160640678487873}]': finished 2024-06-14T23:20:19.166 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[97913]: osdmap e432: 8 total, 8 up, 6 in 2024-06-14T23:20:19.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[102754]: pgmap v1575: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 6.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 61 MiB/s wr, 974 op/s; 8892/300291 objects degraded (2.961%); 91/300291 objects misplaced (0.030%); 4.1 MiB/s, 65 objects/s recovering 2024-06-14T23:20:19.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/617834543' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.32160640678487873}]: dispatch 2024-06-14T23:20:19.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 8892/300291 objects degraded (2.961%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:19.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/617834543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.32160640678487873}]': finished 2024-06-14T23:20:19.167 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:18 smithi086 ceph-mon[102754]: osdmap e432: 8 total, 8 up, 6 in 2024-06-14T23:20:19.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:18 smithi137 ceph-mon[90634]: pgmap v1575: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 113 active+clean; 6.1 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 61 MiB/s wr, 974 op/s; 8892/300291 objects degraded (2.961%); 91/300291 objects misplaced (0.030%); 4.1 MiB/s, 65 objects/s recovering 2024-06-14T23:20:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/617834543' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.32160640678487873}]: dispatch 2024-06-14T23:20:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:18 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 8892/300291 objects degraded (2.961%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/617834543' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.32160640678487873}]': finished 2024-06-14T23:20:19.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:18 smithi137 ceph-mon[90634]: osdmap e432: 8 total, 8 up, 6 in 2024-06-14T23:20:19.262 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:19.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:19.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:19.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 83 16 103329 103313 77.7879 9.5625 0.00336445 0.0127196 2024-06-14T23:20:19.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:19.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:19.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:19.516 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:19.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:19.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:19.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:19.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:19.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:19.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:19.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:19.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:19.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:19.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:19.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:20.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:20.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:20.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:20.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:20.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:20.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:20.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:20.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:20.367 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 84 16 103338 103322 76.8686 0.5625 0.28642 0.0127616 2024-06-14T23:20:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:20.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:20.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:20.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:20.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:20.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:20 smithi086 ceph-mon[97913]: osdmap e433: 8 total, 8 up, 6 in 2024-06-14T23:20:20.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:20 smithi086 ceph-mon[97913]: pgmap v1578: 119 pgs: 13 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 6.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 6.9 KiB/s rd, 58 MiB/s wr, 937 op/s; 8891/303831 objects degraded (2.926%); 91/303831 objects misplaced (0.030%); 5.2 MiB/s, 82 objects/s recovering 2024-06-14T23:20:20.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:20 smithi086 ceph-mon[102754]: osdmap e433: 8 total, 8 up, 6 in 2024-06-14T23:20:20.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:20 smithi086 ceph-mon[102754]: pgmap v1578: 119 pgs: 13 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 6.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 6.9 KiB/s rd, 58 MiB/s wr, 937 op/s; 8891/303831 objects degraded (2.926%); 91/303831 objects misplaced (0.030%); 5.2 MiB/s, 82 objects/s recovering 2024-06-14T23:20:20.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:20.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:20.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:20 smithi137 ceph-mon[90634]: osdmap e433: 8 total, 8 up, 6 in 2024-06-14T23:20:20.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:20 smithi137 ceph-mon[90634]: pgmap v1578: 119 pgs: 13 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 6.2 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 6.9 KiB/s rd, 58 MiB/s wr, 937 op/s; 8891/303831 objects degraded (2.926%); 91/303831 objects misplaced (0.030%); 5.2 MiB/s, 82 objects/s recovering 2024-06-14T23:20:20.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:20.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:20.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:20.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:20.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:20.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:21.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:21.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:21.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:21.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:21.203 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:21.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:21.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:21.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:21.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:21.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 85 16 104082 104066 76.5112 46.5 0.011557 0.0130646 2024-06-14T23:20:21.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:21.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:21.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:21.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:21.503 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-14T23:20:21.503 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[97913]: 14.2 starting backfill to osd.0 from (0'0,0'0] MAX to 431'6440 2024-06-14T23:20:21.503 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[97913]: 14.1 starting backfill to osd.7 from (0'0,0'0] MAX to 431'6432 2024-06-14T23:20:21.503 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[97913]: 14.a starting backfill to osd.6 from (0'0,0'0] MAX to 431'6461 2024-06-14T23:20:21.503 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[97913]: 14.3 starting backfill to osd.5 from (0'0,0'0] MAX to 431'6389 2024-06-14T23:20:21.503 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[97913]: osdmap e434: 8 total, 8 up, 6 in 2024-06-14T23:20:21.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:21.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:21.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:21.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:21.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:21.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:21.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:21.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-14T23:20:21.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[102754]: 14.2 starting backfill to osd.0 from (0'0,0'0] MAX to 431'6440 2024-06-14T23:20:21.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[102754]: 14.1 starting backfill to osd.7 from (0'0,0'0] MAX to 431'6432 2024-06-14T23:20:21.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[102754]: 14.a starting backfill to osd.6 from (0'0,0'0] MAX to 431'6461 2024-06-14T23:20:21.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[102754]: 14.3 starting backfill to osd.5 from (0'0,0'0] MAX to 431'6389 2024-06-14T23:20:21.762 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:21 smithi086 ceph-mon[102754]: osdmap e434: 8 total, 8 up, 6 in 2024-06-14T23:20:21.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:21.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:21.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:21.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:21.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:21 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-14T23:20:21.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:21 smithi137 ceph-mon[90634]: 14.2 starting backfill to osd.0 from (0'0,0'0] MAX to 431'6440 2024-06-14T23:20:21.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:21 smithi137 ceph-mon[90634]: 14.1 starting backfill to osd.7 from (0'0,0'0] MAX to 431'6432 2024-06-14T23:20:21.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:21 smithi137 ceph-mon[90634]: 14.a starting backfill to osd.6 from (0'0,0'0] MAX to 431'6461 2024-06-14T23:20:21.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:21 smithi137 ceph-mon[90634]: 14.3 starting backfill to osd.5 from (0'0,0'0] MAX to 431'6389 2024-06-14T23:20:21.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:21 smithi137 ceph-mon[90634]: osdmap e434: 8 total, 8 up, 6 in 2024-06-14T23:20:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:22.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:22.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:22.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:22.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:22.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:22.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:22.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:22.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 86 16 105044 105028 76.3206 60.125 0.0102034 0.0130971 2024-06-14T23:20:22.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:22.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:22.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:22.446 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:20:22.446 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:20:22.446 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:20:22.447 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:20:22.448 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:20:22.448 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:20:22.448 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:20:22.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:22.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:22.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:22.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:22.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:22.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:22.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:22.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:22.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[102754]: 14.f starting backfill to osd.6 from (0'0,0'0] MAX to 431'6429 2024-06-14T23:20:22.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[102754]: osdmap e435: 8 total, 8 up, 6 in 2024-06-14T23:20:22.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[102754]: pgmap v1581: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 1 active+remapped, 13 peering, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 97 active+clean; 6.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 14 KiB/s rd, 43 MiB/s wr, 691 op/s; 6531/308463 objects degraded (2.117%); 2204/308463 objects misplaced (0.715%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T23:20:22.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2430535076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:22.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:22.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:22.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[102754]: osdmap e436: 8 total, 8 up, 6 in 2024-06-14T23:20:22.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[97913]: 14.f starting backfill to osd.6 from (0'0,0'0] MAX to 431'6429 2024-06-14T23:20:22.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[97913]: osdmap e435: 8 total, 8 up, 6 in 2024-06-14T23:20:22.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[97913]: pgmap v1581: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 1 active+remapped, 13 peering, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 97 active+clean; 6.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 14 KiB/s rd, 43 MiB/s wr, 691 op/s; 6531/308463 objects degraded (2.117%); 2204/308463 objects misplaced (0.715%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T23:20:22.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2430535076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:22.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:22.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:22.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:22 smithi086 ceph-mon[97913]: osdmap e436: 8 total, 8 up, 6 in 2024-06-14T23:20:22.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:22.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:22.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:22.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:22 smithi137 ceph-mon[90634]: 14.f starting backfill to osd.6 from (0'0,0'0] MAX to 431'6429 2024-06-14T23:20:22.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:22 smithi137 ceph-mon[90634]: osdmap e435: 8 total, 8 up, 6 in 2024-06-14T23:20:22.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:22 smithi137 ceph-mon[90634]: pgmap v1581: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 1 active+remapped, 13 peering, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+degraded, 97 active+clean; 6.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 14 KiB/s rd, 43 MiB/s wr, 691 op/s; 6531/308463 objects degraded (2.117%); 2204/308463 objects misplaced (0.715%); 3.9 MiB/s, 61 objects/s recovering 2024-06-14T23:20:22.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:22 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2430535076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:22.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:22 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:22.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:22 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:22.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:22 smithi137 ceph-mon[90634]: osdmap e436: 8 total, 8 up, 6 in 2024-06-14T23:20:23.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:23.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:23.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:23.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:23.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:23.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:23.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:23.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:23.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 87 16 105929 105913 76.0791 55.3125 0.0149188 0.013139 2024-06-14T23:20:23.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:23.408 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:23.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:23.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:23.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:23.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:23.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:23.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:23.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:23.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:23.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:23.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:23.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:23.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:20:23.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:20:23.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:23.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:23 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6531/308463 objects degraded (2.117%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:23.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:20:23.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:20:23.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:23.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:23 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6531/308463 objects degraded (2.117%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:23.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:23.948 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:23.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:23.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:23.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:20:23.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:20:23.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:20:23.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:23 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6531/308463 objects degraded (2.117%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:24.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:24.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:24.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:24.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:24.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:24.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:24.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:24.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:24.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 88 16 106924 106908 75.9212 62.1875 0.0141716 0.0131666 2024-06-14T23:20:24.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:24.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:24.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:24.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:24.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:24.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:24.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:24.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:24 smithi137 ceph-mon[90634]: pgmap v1583: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 12 peering, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 97 active+clean; 6.4 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 4.8 KiB/s rd, 42 MiB/s wr, 675 op/s; 6531/312633 objects degraded (2.089%); 21619/312633 objects misplaced (6.915%); 7.4 MiB/s, 119 objects/s recovering 2024-06-14T23:20:24.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:24.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:24.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:24 smithi086 ceph-mon[97913]: pgmap v1583: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 12 peering, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 97 active+clean; 6.4 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 4.8 KiB/s rd, 42 MiB/s wr, 675 op/s; 6531/312633 objects degraded (2.089%); 21619/312633 objects misplaced (6.915%); 7.4 MiB/s, 119 objects/s recovering 2024-06-14T23:20:24.797 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:24 smithi086 ceph-mon[102754]: pgmap v1583: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 12 peering, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 97 active+clean; 6.4 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 4.8 KiB/s rd, 42 MiB/s wr, 675 op/s; 6531/312633 objects degraded (2.089%); 21619/312633 objects misplaced (6.915%); 7.4 MiB/s, 119 objects/s recovering 2024-06-14T23:20:24.818 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:24.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:24.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:24.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:25.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:25.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:25.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:25.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:20:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:20:24] "GET /metrics HTTP/1.1" 200 36709 "" "Prometheus/2.43.0" 2024-06-14T23:20:25.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:25.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:25.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:25.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:25.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:25.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:25.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 89 16 107810 107794 75.6903 55.375 0.0150123 0.0132066 2024-06-14T23:20:25.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:25.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:25.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:25.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:25.538 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:20:25.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:25.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:25.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:25.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:25.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:25.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:25.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/731383521' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:20:25.747 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0] out_osds: [1, 2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:20:25.747 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:20:25.748 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-06-14T23:20:25.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd in 2 2024-06-14T23:20:25.775 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/731383521' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:20:25.776 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/731383521' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:20:25.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:25.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:25.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:25.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:26.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:26.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:26.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:26.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:26.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:26.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:26.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:20:26.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:26.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:26.366 INFO:tasks.radosbench.radosbench.0.smithi086.stdout: 90 8 108787 108779 75.5333 61.5625 0.00915519 0.0132344 2024-06-14T23:20:26.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:26.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:26.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:26.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:26.557 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:26.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:26 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:20:26.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/731383521' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:20:26.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:26 smithi086 ceph-mon[102754]: osdmap e437: 8 total, 8 up, 6 in 2024-06-14T23:20:26.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:26 smithi086 ceph-mon[102754]: pgmap v1585: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 7 peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering, 103 active+clean; 6.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 3.9 KiB/s rd, 44 MiB/s wr, 712 op/s; 6319/319314 objects degraded (1.979%); 34373/319314 objects misplaced (10.765%); 12 MiB/s, 188 objects/s recovering 2024-06-14T23:20:26.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:26 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:20:26.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/731383521' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:20:26.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:26 smithi086 ceph-mon[97913]: osdmap e437: 8 total, 8 up, 6 in 2024-06-14T23:20:26.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:26 smithi086 ceph-mon[97913]: pgmap v1585: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 7 peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering, 103 active+clean; 6.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 3.9 KiB/s rd, 44 MiB/s wr, 712 op/s; 6319/319314 objects degraded (1.979%); 34373/319314 objects misplaced (10.765%); 12 MiB/s, 188 objects/s recovering 2024-06-14T23:20:26.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:26.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:26.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:26.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:26.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:26.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:26.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:26.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:26.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:26.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:26 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:20:26.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/731383521' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:20:26.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:26 smithi137 ceph-mon[90634]: osdmap e437: 8 total, 8 up, 6 in 2024-06-14T23:20:26.997 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:26 smithi137 ceph-mon[90634]: pgmap v1585: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 7 peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering, 103 active+clean; 6.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 3.9 KiB/s rd, 44 MiB/s wr, 712 op/s; 6319/319314 objects degraded (1.979%); 34373/319314 objects misplaced (10.765%); 12 MiB/s, 188 objects/s recovering 2024-06-14T23:20:27.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:27.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:27.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:27.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:27.177 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:27.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:27.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:27.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:27.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:27.367 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total time run: 90.0135 2024-06-14T23:20:27.367 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Total writes made: 108787 2024-06-14T23:20:27.367 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Write size: 65536 2024-06-14T23:20:27.367 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Object size: 65536 2024-06-14T23:20:27.367 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Bandwidth (MB/sec): 75.5352 2024-06-14T23:20:27.367 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Bandwidth: 45.9232 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max bandwidth (MB/sec): 243.5 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min bandwidth (MB/sec): 0 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average IOPS: 1208 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev IOPS: 734.775 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max IOPS: 3896 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min IOPS: 0 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Average Latency(s): 0.0132344 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Stddev Latency(s): 0.055459 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Max latency(s): 4.60763 2024-06-14T23:20:27.368 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Min latency(s): 0.00111701 2024-06-14T23:20:27.371 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Cleaning up (deleting benchmark objects) 2024-06-14T23:20:27.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:27.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:27.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:27.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:27.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:27.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:27.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:27.720 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:27.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:27.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:27.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:27 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-06-14T23:20:27.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:27.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:27 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-06-14T23:20:27.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:27.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:27.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:27.984 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:27.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:27.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:27 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-06-14T23:20:27.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:28.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:28.128 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:28.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:28.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:28.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:28.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:28.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:28.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:28.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:28.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:20:28.434 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:20:28.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:20:28.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:28.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:28.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:28.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:28.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:28.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:28.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:28.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:28 smithi086 ceph-mon[97913]: pgmap v1586: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering, 110 active+clean; 6.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 3.3 KiB/s rd, 48 MiB/s wr, 768 op/s; 6319/323082 objects degraded (1.956%); 33915/323082 objects misplaced (10.497%); 19 MiB/s, 306 objects/s recovering 2024-06-14T23:20:28.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6319/319314 objects degraded (1.979%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:28.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:28 smithi086 ceph-mon[102754]: pgmap v1586: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering, 110 active+clean; 6.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 3.3 KiB/s rd, 48 MiB/s wr, 768 op/s; 6319/323082 objects degraded (1.956%); 33915/323082 objects misplaced (10.497%); 19 MiB/s, 306 objects/s recovering 2024-06-14T23:20:28.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6319/319314 objects degraded (1.979%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:28.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:28.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:28.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:28.981 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:28.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:28.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:28 smithi137 ceph-mon[90634]: pgmap v1586: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering, 110 active+clean; 6.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 3.3 KiB/s rd, 48 MiB/s wr, 768 op/s; 6319/323082 objects degraded (1.956%); 33915/323082 objects misplaced (10.497%); 19 MiB/s, 306 objects/s recovering 2024-06-14T23:20:28.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6319/319314 objects degraded (1.979%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:29.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:29.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:29.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:29.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:29.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:29.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:29.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:29.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:29.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:29.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:29.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:29.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:29.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:29.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:29.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:29.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:29.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:29.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:29.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:29.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:29.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:30.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:30.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:30.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:30.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:30.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:30.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:30.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:30.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:30.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:30.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:30.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:30.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:30.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-14T23:20:30.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:30.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:30.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:30.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:30.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:30.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:30.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:30.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:30.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:31.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:31.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:31.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:31.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:31.165 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:30 smithi086 ceph-mon[97913]: pgmap v1587: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering, 110 active+clean; 6.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 2.6 KiB/s rd, 43 MiB/s wr, 734 op/s; 6319/323703 objects degraded (1.952%); 32944/323703 objects misplaced (10.177%); 24 MiB/s, 379 objects/s recovering 2024-06-14T23:20:31.165 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:30 smithi086 ceph-mon[102754]: pgmap v1587: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering, 110 active+clean; 6.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 2.6 KiB/s rd, 43 MiB/s wr, 734 op/s; 6319/323703 objects degraded (1.952%); 32944/323703 objects misplaced (10.177%); 24 MiB/s, 379 objects/s recovering 2024-06-14T23:20:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:31.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:31.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:30 smithi137 ceph-mon[90634]: pgmap v1587: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering, 110 active+clean; 6.6 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 2.6 KiB/s rd, 43 MiB/s wr, 734 op/s; 6319/323703 objects degraded (1.952%); 32944/323703 objects misplaced (10.177%); 24 MiB/s, 379 objects/s recovering 2024-06-14T23:20:31.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:31.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:31.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:31.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:31.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:31.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:31.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:31.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:31.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:31.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:31.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:31.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:31.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:31.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:32.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:32.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:32.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:32.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:32.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:32.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:32.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:32.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:32.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:32.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:32.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:32.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:32.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:32.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:32.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:32.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:32 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:20:32.314+0000 7fe0161db640 -1 mon.a@0(leader).osd e437 definitely_dead 0 2024-06-14T23:20:32.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:32.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:32.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:32.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:32.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:32.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:32.926 INFO:teuthology.orchestra.run.smithi086.stderr:marked in osd.2. 2024-06-14T23:20:32.958 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:32.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:33.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:33.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:33.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:33.162 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:32 smithi086 ceph-mon[97913]: pgmap v1588: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 1 active+recovering, 110 active+clean; 6.5 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 33 MiB/s wr, 700 op/s; 6153/321744 objects degraded (1.912%); 32542/321744 objects misplaced (10.114%); 22 MiB/s, 347 objects/s recovering 2024-06-14T23:20:33.162 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/554015172' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-14T23:20:33.162 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:32 smithi086 ceph-mon[102754]: pgmap v1588: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 1 active+recovering, 110 active+clean; 6.5 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 33 MiB/s wr, 700 op/s; 6153/321744 objects degraded (1.912%); 32542/321744 objects misplaced (10.114%); 22 MiB/s, 347 objects/s recovering 2024-06-14T23:20:33.162 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/554015172' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-14T23:20:33.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:33.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:33.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:33.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:32 smithi137 ceph-mon[90634]: pgmap v1588: 119 pgs: 2 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 1 active+recovering, 110 active+clean; 6.5 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 33 MiB/s wr, 700 op/s; 6153/321744 objects degraded (1.912%); 32542/321744 objects misplaced (10.114%); 22 MiB/s, 347 objects/s recovering 2024-06-14T23:20:33.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/554015172' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-14T23:20:33.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:33.337 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:33.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:33.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:33.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:33.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:33.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:33.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:33.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:33.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:33.709 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:33.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:33.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:33.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:33.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:33.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:33.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:33.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:33.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:34.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:34.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:34.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:34.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:34.213 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:34.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:34.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/554015172' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-06-14T23:20:34.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:33 smithi137 ceph-mon[90634]: osdmap e438: 8 total, 8 up, 7 in 2024-06-14T23:20:34.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:33 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/763966706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:34.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6153/321744 objects degraded (1.912%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:34.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:33 smithi137 ceph-mon[90634]: 14.1 starting backfill to osd.2 from (0'0,0'0] MAX to 437'7114 2024-06-14T23:20:34.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:33 smithi137 ceph-mon[90634]: osdmap e439: 8 total, 8 up, 7 in 2024-06-14T23:20:34.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:33 smithi137 ceph-mon[90634]: 14.f starting backfill to osd.2 from (0'0,0'0] MAX to 437'7144 2024-06-14T23:20:34.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:34.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/554015172' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-06-14T23:20:34.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[102754]: osdmap e438: 8 total, 8 up, 7 in 2024-06-14T23:20:34.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/763966706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:34.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6153/321744 objects degraded (1.912%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:34.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[102754]: 14.1 starting backfill to osd.2 from (0'0,0'0] MAX to 437'7114 2024-06-14T23:20:34.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[102754]: osdmap e439: 8 total, 8 up, 7 in 2024-06-14T23:20:34.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[102754]: 14.f starting backfill to osd.2 from (0'0,0'0] MAX to 437'7144 2024-06-14T23:20:34.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/554015172' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-06-14T23:20:34.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[97913]: osdmap e438: 8 total, 8 up, 7 in 2024-06-14T23:20:34.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/763966706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:34.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6153/321744 objects degraded (1.912%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-14T23:20:34.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[97913]: 14.1 starting backfill to osd.2 from (0'0,0'0] MAX to 437'7114 2024-06-14T23:20:34.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[97913]: osdmap e439: 8 total, 8 up, 7 in 2024-06-14T23:20:34.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:33 smithi086 ceph-mon[97913]: 14.f starting backfill to osd.2 from (0'0,0'0] MAX to 437'7144 2024-06-14T23:20:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:34.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:34.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:34.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:34.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:34.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:34.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:34.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:34.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:34.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:34.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:34.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:34.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:34.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:34.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:34.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:34.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:34.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:34.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:35.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:35.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:35.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:35.138 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:20:35.138 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:20:35.138 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:20:35.138 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:20:35.138 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:20:35.138 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:20:35.138 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:20:35.138 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:20:35.139 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:20:35.139 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:20:35.139 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:20:35.139 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:20:35.139 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:20:35.139 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:20:35.139 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:20:35.139 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:20:35.139 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:34 smithi086 ceph-mon[97913]: pgmap v1591: 119 pgs: 6 remapped+peering, 3 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 1 active+recovering, 104 active+clean; 6.4 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 19 MiB/s wr, 744 op/s; 6153/315648 objects degraded (1.949%); 30519/315648 objects misplaced (9.669%); 29 MiB/s, 470 objects/s recovering 2024-06-14T23:20:35.139 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:34 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-14T23:20:35.140 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:34 smithi086 ceph-mon[97913]: osdmap e440: 8 total, 8 up, 7 in 2024-06-14T23:20:35.140 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:20:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:20:34] "GET /metrics HTTP/1.1" 200 36731 "" "Prometheus/2.43.0" 2024-06-14T23:20:35.140 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:34 smithi086 ceph-mon[102754]: pgmap v1591: 119 pgs: 6 remapped+peering, 3 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 1 active+recovering, 104 active+clean; 6.4 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 19 MiB/s wr, 744 op/s; 6153/315648 objects degraded (1.949%); 30519/315648 objects misplaced (9.669%); 29 MiB/s, 470 objects/s recovering 2024-06-14T23:20:35.140 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:34 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-14T23:20:35.140 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:34 smithi086 ceph-mon[102754]: osdmap e440: 8 total, 8 up, 7 in 2024-06-14T23:20:35.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:35.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:34 smithi137 ceph-mon[90634]: pgmap v1591: 119 pgs: 6 remapped+peering, 3 active+remapped+backfilling, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 1 active+recovering, 104 active+clean; 6.4 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 19 MiB/s wr, 744 op/s; 6153/315648 objects degraded (1.949%); 30519/315648 objects misplaced (9.669%); 29 MiB/s, 470 objects/s recovering 2024-06-14T23:20:35.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:34 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-14T23:20:35.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:34 smithi137 ceph-mon[90634]: osdmap e440: 8 total, 8 up, 7 in 2024-06-14T23:20:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:35.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:35.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:35.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:35.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:35.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:35.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:35.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:35.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:35.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:35.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:35.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:35.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:35.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:35.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:35.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:35.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:35.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:36.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:36.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:36.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:36.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:36.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:36.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:36.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:35 smithi137 ceph-mon[90634]: 14.0 starting backfill to osd.2 from (0'0,415'2249] MAX to 437'7076 2024-06-14T23:20:36.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:35 smithi137 ceph-mon[90634]: 14.b starting backfill to osd.2 from (0'0,0'0] MAX to 437'7181 2024-06-14T23:20:36.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:35 smithi137 ceph-mon[90634]: 14.8 starting backfill to osd.2 from (0'0,415'2215] MAX to 437'7047 2024-06-14T23:20:36.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:35 smithi137 ceph-mon[90634]: 14.c starting backfill to osd.2 from (0'0,415'2258] MAX to 437'7187 2024-06-14T23:20:36.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:35 smithi137 ceph-mon[90634]: 14.6 starting backfill to osd.2 from (0'0,415'2170] MAX to 437'7060 2024-06-14T23:20:36.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:35 smithi137 ceph-mon[90634]: 14.e starting backfill to osd.2 from (0'0,0'0] MAX to 437'7290 2024-06-14T23:20:36.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:35 smithi137 ceph-mon[90634]: osdmap e441: 8 total, 8 up, 7 in 2024-06-14T23:20:36.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[102754]: 14.0 starting backfill to osd.2 from (0'0,415'2249] MAX to 437'7076 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[102754]: 14.b starting backfill to osd.2 from (0'0,0'0] MAX to 437'7181 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[102754]: 14.8 starting backfill to osd.2 from (0'0,415'2215] MAX to 437'7047 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[102754]: 14.c starting backfill to osd.2 from (0'0,415'2258] MAX to 437'7187 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[102754]: 14.6 starting backfill to osd.2 from (0'0,415'2170] MAX to 437'7060 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[102754]: 14.e starting backfill to osd.2 from (0'0,0'0] MAX to 437'7290 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[102754]: osdmap e441: 8 total, 8 up, 7 in 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[97913]: 14.0 starting backfill to osd.2 from (0'0,415'2249] MAX to 437'7076 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[97913]: 14.b starting backfill to osd.2 from (0'0,0'0] MAX to 437'7181 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[97913]: 14.8 starting backfill to osd.2 from (0'0,415'2215] MAX to 437'7047 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[97913]: 14.c starting backfill to osd.2 from (0'0,415'2258] MAX to 437'7187 2024-06-14T23:20:36.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[97913]: 14.6 starting backfill to osd.2 from (0'0,415'2170] MAX to 437'7060 2024-06-14T23:20:36.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[97913]: 14.e starting backfill to osd.2 from (0'0,0'0] MAX to 437'7290 2024-06-14T23:20:36.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:35 smithi086 ceph-mon[97913]: osdmap e441: 8 total, 8 up, 7 in 2024-06-14T23:20:36.352 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:36.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:36.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:36.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:36.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:36.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:36.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:36.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:36.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:36.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:36.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:36.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:36.884 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:36.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:36.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:37.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:37.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:37.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:37.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:37.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:37.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:37.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:36 smithi137 ceph-mon[90634]: pgmap v1594: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 6 remapped+peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_wait, 98 active+clean; 6.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 7299/309678 objects degraded (2.357%); 55424/309678 objects misplaced (17.897%); 49 MiB/s, 2 keys/s, 793 objects/s recovering 2024-06-14T23:20:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:36 smithi137 ceph-mon[90634]: osdmap e442: 8 total, 8 up, 7 in 2024-06-14T23:20:37.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:37.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:37.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:37.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:36 smithi086 ceph-mon[102754]: pgmap v1594: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 6 remapped+peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_wait, 98 active+clean; 6.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 7299/309678 objects degraded (2.357%); 55424/309678 objects misplaced (17.897%); 49 MiB/s, 2 keys/s, 793 objects/s recovering 2024-06-14T23:20:37.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:36 smithi086 ceph-mon[102754]: osdmap e442: 8 total, 8 up, 7 in 2024-06-14T23:20:37.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:36 smithi086 ceph-mon[97913]: pgmap v1594: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 6 remapped+peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_wait, 98 active+clean; 6.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 7299/309678 objects degraded (2.357%); 55424/309678 objects misplaced (17.897%); 49 MiB/s, 2 keys/s, 793 objects/s recovering 2024-06-14T23:20:37.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:36 smithi086 ceph-mon[97913]: osdmap e442: 8 total, 8 up, 7 in 2024-06-14T23:20:37.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:37.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:37.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:37.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:37.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:37.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:37.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:37.739 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:37.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:37.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:37.871 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:37.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:37.972 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:38.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:38.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:38.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:38.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:37 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-06-14T23:20:38.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:37 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-06-14T23:20:38.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:38.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:38.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:38.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:38.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:37 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-06-14T23:20:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:38.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:38.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:38.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:38.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:38.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:38.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:38.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:38.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:38.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:38.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:38.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:38.828 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:38.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:38.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:38.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:38.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:39.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:39.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:39.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:39.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:39.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:39.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:38 smithi137 ceph-mon[90634]: pgmap v1596: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 107 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 751 op/s; 7299/307281 objects degraded (2.375%); 54691/307281 objects misplaced (17.798%); 39 MiB/s, 2 keys/s, 631 objects/s recovering 2024-06-14T23:20:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:38 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 7299/307281 objects degraded (2.375%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:39.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:39.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:38 smithi086 ceph-mon[102754]: pgmap v1596: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 107 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 751 op/s; 7299/307281 objects degraded (2.375%); 54691/307281 objects misplaced (17.798%); 39 MiB/s, 2 keys/s, 631 objects/s recovering 2024-06-14T23:20:39.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:38 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 7299/307281 objects degraded (2.375%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:39.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:38 smithi086 ceph-mon[97913]: pgmap v1596: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 107 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 751 op/s; 7299/307281 objects degraded (2.375%); 54691/307281 objects misplaced (17.798%); 39 MiB/s, 2 keys/s, 631 objects/s recovering 2024-06-14T23:20:39.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:38 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 7299/307281 objects degraded (2.375%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:39.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:39.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:39.500 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:39.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:39.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:39.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:39.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:39.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:39.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:39.779 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:39.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:39.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:39.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:39.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:40.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:40.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:40.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:40.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:40.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:40.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:40.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:40.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:40.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:40.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:40.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:40.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:40.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:40.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:40.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:40.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:40.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:40.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:40.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:40.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:40.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:40.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:40.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:40.992 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:41.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:41.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:40 smithi086 ceph-mon[97913]: pgmap v1597: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 6.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 7299/304149 objects degraded (2.400%); 53513/304149 objects misplaced (17.594%); 32 MiB/s, 1 keys/s, 507 objects/s recovering 2024-06-14T23:20:41.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:40 smithi086 ceph-mon[102754]: pgmap v1597: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 6.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 7299/304149 objects degraded (2.400%); 53513/304149 objects misplaced (17.594%); 32 MiB/s, 1 keys/s, 507 objects/s recovering 2024-06-14T23:20:41.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:41.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:41.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:41.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:40 smithi137 ceph-mon[90634]: pgmap v1597: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 107 active+clean; 6.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 7299/304149 objects degraded (2.400%); 53513/304149 objects misplaced (17.594%); 32 MiB/s, 1 keys/s, 507 objects/s recovering 2024-06-14T23:20:41.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:41.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:41.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:41.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:41.426 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:41.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:41.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:41.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:41.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:41.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:41.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:41.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:41.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:41.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:41.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:41.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:41.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:42.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:42.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:42.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:42.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:42.237 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:42.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:42.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:42.375 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:42.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:42.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:42.508 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:42.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:42.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:42.649 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:42.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:42.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:42.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:42.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:42.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:42.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:42.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:42.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:43.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:43.050 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:43.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:43.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:43.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:43.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:43.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:42 smithi137 ceph-mon[90634]: pgmap v1598: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 6.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 6393/297096 objects degraded (2.152%); 51600/297096 objects misplaced (17.368%); 34 MiB/s, 547 objects/s recovering 2024-06-14T23:20:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:43.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:43.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:43.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:42 smithi086 ceph-mon[97913]: pgmap v1598: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 6.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 6393/297096 objects degraded (2.152%); 51600/297096 objects misplaced (17.368%); 34 MiB/s, 547 objects/s recovering 2024-06-14T23:20:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:43.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:42 smithi086 ceph-mon[102754]: pgmap v1598: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 6.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 6393/297096 objects degraded (2.152%); 51600/297096 objects misplaced (17.368%); 34 MiB/s, 547 objects/s recovering 2024-06-14T23:20:43.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:43.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:43.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:43.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:43.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:43.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:43.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:43.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:43.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:43.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:43.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:43.872 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:43.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:43.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:43.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:44.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:44.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:44.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:44.130 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:43 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 6393/297096 objects degraded (2.152%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:44.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:44.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:44.208 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:43 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 6393/297096 objects degraded (2.152%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:44.208 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:43 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 6393/297096 objects degraded (2.152%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:44.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:44.247 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-06-14T23:20:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:44.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:44.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:44.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:44.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:44.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:44.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:44.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:44.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:44.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:44.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:44.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:44.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:44.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:44.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:44.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:44.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:44.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:44.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:45.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:45.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:45.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:45.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:20:45.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:44 smithi086 ceph-mon[97913]: pgmap v1599: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 6.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 662 op/s; 6393/293574 objects degraded (2.178%); 51113/293574 objects misplaced (17.411%); 21 MiB/s, 337 objects/s recovering 2024-06-14T23:20:45.251 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:20:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:20:44] "GET /metrics HTTP/1.1" 200 36728 "" "Prometheus/2.43.0" 2024-06-14T23:20:45.252 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:44 smithi086 ceph-mon[102754]: pgmap v1599: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 6.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 662 op/s; 6393/293574 objects degraded (2.178%); 51113/293574 objects misplaced (17.411%); 21 MiB/s, 337 objects/s recovering 2024-06-14T23:20:45.253 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:45.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:45.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:45.388 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:44 smithi137 ceph-mon[90634]: pgmap v1599: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 6.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 662 op/s; 6393/293574 objects degraded (2.178%); 51113/293574 objects misplaced (17.411%); 21 MiB/s, 337 objects/s recovering 2024-06-14T23:20:45.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:45.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:45.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:45.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:45.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:45.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:45.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:45.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:45.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:45.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:45.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:45.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:45.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:45.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:46.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:46.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:46.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:46.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:46.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:46.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:46.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:46.363 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:46.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:46.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:46.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:46.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:46.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:46.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:46.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:46.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:46.774 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:46.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:46.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:46.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:47.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:47.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:47.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:47.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:47.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:47.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:47.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:47.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:47.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:47.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:47.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:47 smithi086 ceph-mon[102754]: pgmap v1600: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 866 op/s; 5304/283668 objects degraded (1.870%); 49106/283668 objects misplaced (17.311%); 29 MiB/s, 468 objects/s recovering 2024-06-14T23:20:47.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:46 smithi086 ceph-mon[97913]: pgmap v1600: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 866 op/s; 5304/283668 objects degraded (1.870%); 49106/283668 objects misplaced (17.311%); 29 MiB/s, 468 objects/s recovering 2024-06-14T23:20:47.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:47.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:47.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:47.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:46 smithi137 ceph-mon[90634]: pgmap v1600: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 866 op/s; 5304/283668 objects degraded (1.870%); 49106/283668 objects misplaced (17.311%); 29 MiB/s, 468 objects/s recovering 2024-06-14T23:20:47.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:47.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:47.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:47.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:47.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:47.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:47.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:47.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:47.869 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:47.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:47.879 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:47.891 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:20:47.891 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-06-14T23:20:47.892 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-06-14T23:20:47.892 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:20:47.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:48.021 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:48.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:48.033 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:20:48.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:48.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:48.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:48.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:48.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:48.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:48.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/887811049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:20:48.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/887811049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:20:48.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:48.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:48.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:48.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/887811049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:20:48.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:48.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:48.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:48.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:48.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:48.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:48.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:48.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:48.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:48.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:49.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:49.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:49.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:49.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:49.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:49.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:49.248 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0, 2] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:20:49.248 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:20:49.248 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_9 2024-06-14T23:20:49.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:20:49.283 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[102754]: pgmap v1601: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 5304/280452 objects degraded (1.891%); 48684/280452 objects misplaced (17.359%); 26 MiB/s, 412 objects/s recovering 2024-06-14T23:20:49.283 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:20:49.283 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/887811049' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:20:49.283 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[102754]: osdmap e443: 8 total, 8 up, 7 in 2024-06-14T23:20:49.283 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5304/280452 objects degraded (1.891%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:49.283 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[97913]: pgmap v1601: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 5304/280452 objects degraded (1.891%); 48684/280452 objects misplaced (17.359%); 26 MiB/s, 412 objects/s recovering 2024-06-14T23:20:49.283 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:20:49.284 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/887811049' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:20:49.284 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[97913]: osdmap e443: 8 total, 8 up, 7 in 2024-06-14T23:20:49.284 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:49 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5304/280452 objects degraded (1.891%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:49.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:49.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:49.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:49.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:49.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:49.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:49.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:49 smithi137 ceph-mon[90634]: pgmap v1601: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 5304/280452 objects degraded (1.891%); 48684/280452 objects misplaced (17.359%); 26 MiB/s, 412 objects/s recovering 2024-06-14T23:20:49.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:49 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:20:49.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:49 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/887811049' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:20:49.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:49 smithi137 ceph-mon[90634]: osdmap e443: 8 total, 8 up, 7 in 2024-06-14T23:20:49.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:49 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5304/280452 objects degraded (1.891%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:49.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:49.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:49.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:49.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:49.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:49.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:49.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:49.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:49.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:49.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:49.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:50.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:50.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:50.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:50.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:50.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:50.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:50.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:50.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:50.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:50.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:50.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:50.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:50.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:50.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:50.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:50.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:50.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:50.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:50.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:50.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:50.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:50.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:51.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:51.080 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:51.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:51.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:51.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:51.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:51.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:51 smithi086 ceph-mon[97913]: pgmap v1603: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.6 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 5304/277191 objects degraded (1.913%); 48217/277191 objects misplaced (17.395%); 22 MiB/s, 356 objects/s recovering 2024-06-14T23:20:51.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:51 smithi086 ceph-mon[102754]: pgmap v1603: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.6 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 5304/277191 objects degraded (1.913%); 48217/277191 objects misplaced (17.395%); 22 MiB/s, 356 objects/s recovering 2024-06-14T23:20:51.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:51.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:51.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:51.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:51 smithi137 ceph-mon[90634]: pgmap v1603: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.6 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 5304/277191 objects degraded (1.913%); 48217/277191 objects misplaced (17.395%); 22 MiB/s, 356 objects/s recovering 2024-06-14T23:20:51.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:51.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:51.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:51.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:51.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:51.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:51.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:51.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:52.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:52.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:52.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:52.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:52.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:52.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:52.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:52.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:52.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:52.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:52.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:52.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:52.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:52.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:52.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:52.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:52.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:52.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:52.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:52.872 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:52.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:52.946 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:52.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:53.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:53.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:53.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:53.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:53.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:53.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:53.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:53.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:53.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:53 smithi086 ceph-mon[97913]: pgmap v1604: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 881 op/s; 4249/270633 objects degraded (1.570%); 46822/270633 objects misplaced (17.301%); 22 MiB/s, 357 objects/s recovering 2024-06-14T23:20:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:53 smithi086 ceph-mon[102754]: pgmap v1604: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 881 op/s; 4249/270633 objects degraded (1.570%); 46822/270633 objects misplaced (17.301%); 22 MiB/s, 357 objects/s recovering 2024-06-14T23:20:53.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:53.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:53.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:53 smithi137 ceph-mon[90634]: pgmap v1604: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 881 op/s; 4249/270633 objects degraded (1.570%); 46822/270633 objects misplaced (17.301%); 22 MiB/s, 357 objects/s recovering 2024-06-14T23:20:53.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:53.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:53.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-14T23:20:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:53.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:53.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:53.684 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:53.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:53.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:53.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:53.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:53.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:53.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:53.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:20:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:53.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:54.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:54 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 4249/270633 objects degraded (1.570%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:54.060 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:54 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 4249/270633 objects degraded (1.570%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:54.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:54.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:54.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:54.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:54.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:54.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:54.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:54.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:54.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:54.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:54 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 4249/270633 objects degraded (1.570%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:54.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:54.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:54.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:54.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:54.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:54.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:54.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:54.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:54.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:54.885 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:54.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:54.966 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:54.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:55.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:55.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:55.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:55.127 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:55 smithi086 ceph-mon[97913]: pgmap v1605: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 867 op/s; 4249/267537 objects degraded (1.588%); 46377/267537 objects misplaced (17.335%); 22 MiB/s, 357 objects/s recovering 2024-06-14T23:20:55.127 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:55 smithi086 ceph-mon[102754]: pgmap v1605: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 867 op/s; 4249/267537 objects degraded (1.588%); 46377/267537 objects misplaced (17.335%); 22 MiB/s, 357 objects/s recovering 2024-06-14T23:20:55.128 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:20:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:20:54] "GET /metrics HTTP/1.1" 200 36728 "" "Prometheus/2.43.0" 2024-06-14T23:20:55.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:55.227 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:55.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:55.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:55.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:55.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:55.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:55 smithi137 ceph-mon[90634]: pgmap v1605: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 867 op/s; 4249/267537 objects degraded (1.588%); 46377/267537 objects misplaced (17.335%); 22 MiB/s, 357 objects/s recovering 2024-06-14T23:20:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:55.505 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:55.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:55.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:55.616 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:20:55.616 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:20:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:55.649 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:55.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:55.657 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:20:55.658 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:20:55.694 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:20:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:20:55.723 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:20:55.723 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:20:55.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:55.759 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:20:55.759 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:20:55.785 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:55.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:55.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:55.799 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:20:55.799 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:20:55.843 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:20:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:20:55.872 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:20:55.872 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:20:55.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:55.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:56.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:56.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:56.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:56.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:56.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:56.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:56.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:56.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:56.334 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:56.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:56.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2144825080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:56.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2144825080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:56.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:56.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:56.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:56.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2144825080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:20:56.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:56.598 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:56.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:56.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:56.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:56.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:56.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:56.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:56.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:57.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:57.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:57.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:57.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:57.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:57.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:57.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:57.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:57.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:57.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:57 smithi086 ceph-mon[102754]: pgmap v1606: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 3099/257028 objects degraded (1.206%); 44395/257028 objects misplaced (17.272%); 22 MiB/s, 358 objects/s recovering 2024-06-14T23:20:57.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:57 smithi086 ceph-mon[97913]: pgmap v1606: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 3099/257028 objects degraded (1.206%); 44395/257028 objects misplaced (17.272%); 22 MiB/s, 358 objects/s recovering 2024-06-14T23:20:57.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:57.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:57.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:57 smithi137 ceph-mon[90634]: pgmap v1606: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 3099/257028 objects degraded (1.206%); 44395/257028 objects misplaced (17.272%); 22 MiB/s, 358 objects/s recovering 2024-06-14T23:20:57.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:57.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:57.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:57.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:57.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:57.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:57.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:20:57.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:57.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:57.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:57.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:57.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:57.956 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:57.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:58.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:58.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:58.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:58.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:58.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:58.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:58.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:58.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:58.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:58.363 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:58.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:58.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:20:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:20:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:58.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:58.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:20:58.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:58.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:58.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:58.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:20:58.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:58.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:20:58.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:58.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:58.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:58.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:20:59.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:59.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:59.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:59.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:20:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:59.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:20:59.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:20:59.308 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:59.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:20:59.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:59 smithi086 ceph-mon[102754]: pgmap v1607: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 3099/253575 objects degraded (1.222%); 43917/253575 objects misplaced (17.319%); 22 MiB/s, 358 objects/s recovering 2024-06-14T23:20:59.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:20:59 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 3099/253575 objects degraded (1.222%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:59.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:59 smithi086 ceph-mon[97913]: pgmap v1607: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 3099/253575 objects degraded (1.222%); 43917/253575 objects misplaced (17.319%); 22 MiB/s, 358 objects/s recovering 2024-06-14T23:20:59.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:20:59 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 3099/253575 objects degraded (1.222%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:59.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:59.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:59.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:59.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:59 smithi137 ceph-mon[90634]: pgmap v1607: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 3099/253575 objects degraded (1.222%); 43917/253575 objects misplaced (17.319%); 22 MiB/s, 358 objects/s recovering 2024-06-14T23:20:59.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:20:59 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 3099/253575 objects degraded (1.222%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:20:59.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:20:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:59.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:59.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:20:59.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:20:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:59.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:59.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:20:59.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:20:59.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:59.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:20:59.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:20:59.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:20:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:20:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:20:59.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:00.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:00.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:00.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:00.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:00.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:00.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:00.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:00.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:00.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:00.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:00.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:00.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:00.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:00.673 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:00.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:00.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:00.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:00.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:00.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:00.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:00.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:01.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:01.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:01.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:01.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:01.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:01.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:01 smithi086 ceph-mon[97913]: pgmap v1608: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 3099/250140 objects degraded (1.239%); 43391/250140 objects misplaced (17.347%); 19 MiB/s, 303 objects/s recovering 2024-06-14T23:21:01.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:01 smithi086 ceph-mon[102754]: pgmap v1608: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 3099/250140 objects degraded (1.239%); 43391/250140 objects misplaced (17.347%); 19 MiB/s, 303 objects/s recovering 2024-06-14T23:21:01.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:01.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:01.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:01.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:01.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:01.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:01 smithi137 ceph-mon[90634]: pgmap v1608: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 5.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 853 op/s; 3099/250140 objects degraded (1.239%); 43391/250140 objects misplaced (17.347%); 19 MiB/s, 303 objects/s recovering 2024-06-14T23:21:01.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:01.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:01.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:01.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:01.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:01.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:01.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:01.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:01.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:01.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:02.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:02.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:02.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:02.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:02.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:02.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:02.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:02.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:02.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:02.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:02.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:02.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:02.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:02.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:02.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:02.673 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:02.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:02.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:02.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:02.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:02.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:21:02.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:02.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:02.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:03.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:03.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:03.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:03.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:03.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:03.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:03.299 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:03 smithi086 ceph-mon[97913]: pgmap v1609: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 944 op/s; 1940/243150 objects degraded (0.798%); 41996/243150 objects misplaced (17.272%); 28 MiB/s, 451 objects/s recovering 2024-06-14T23:21:03.299 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:03 smithi086 ceph-mon[102754]: pgmap v1609: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 944 op/s; 1940/243150 objects degraded (0.798%); 41996/243150 objects misplaced (17.272%); 28 MiB/s, 451 objects/s recovering 2024-06-14T23:21:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:21:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:21:03.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:03.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:03.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:03.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:03 smithi137 ceph-mon[90634]: pgmap v1609: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 944 op/s; 1940/243150 objects degraded (0.798%); 41996/243150 objects misplaced (17.272%); 28 MiB/s, 451 objects/s recovering 2024-06-14T23:21:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:03.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:03.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:03.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:03.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:03.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:03.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:03.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:03.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:03.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:03.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:03.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:04.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:04.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:04.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:04.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:04.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:04.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:04.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-14T23:21:04.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:04.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:04.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:04.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:04 smithi086 ceph-mon[97913]: from='client.47333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:21:04.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:04 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 1940/243150 objects degraded (0.798%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:21:04.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:04 smithi086 ceph-mon[102754]: from='client.47333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:21:04.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:04 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 1940/243150 objects degraded (0.798%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:21:04.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:04.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:04.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:04.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:04 smithi137 ceph-mon[90634]: from='client.47333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:21:04.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:04 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 1940/243150 objects degraded (0.798%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:21:04.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:04.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:04.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:04.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:04.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:04.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:04.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:04.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:04.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:04.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:04.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:05.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:05.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:05.021 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:21:05.023 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:21:05.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:05.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:05.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:05.143 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:05 smithi086 ceph-mon[102754]: pgmap v1610: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 862 op/s; 1940/239556 objects degraded (0.810%); 41509/239556 objects misplaced (17.327%); 19 MiB/s, 308 objects/s recovering 2024-06-14T23:21:05.143 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:05 smithi086 ceph-mon[97913]: pgmap v1610: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 862 op/s; 1940/239556 objects degraded (0.810%); 41509/239556 objects misplaced (17.327%); 19 MiB/s, 308 objects/s recovering 2024-06-14T23:21:05.144 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:21:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:21:04] "GET /metrics HTTP/1.1" 200 36725 "" "Prometheus/2.43.0" 2024-06-14T23:21:05.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:05.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:05.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:05.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:05.403 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:05.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:05 smithi137 ceph-mon[90634]: pgmap v1610: 119 pgs: 1 active+remapped, 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.9 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 862 op/s; 1940/239556 objects degraded (0.810%); 41509/239556 objects misplaced (17.327%); 19 MiB/s, 308 objects/s recovering 2024-06-14T23:21:05.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:05.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:05.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:05.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:05.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:05.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:05.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:05.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:05.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:05.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:06.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:06.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:06.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:06.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:06.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:06.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:06.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:06.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:06.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:06.481 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:06.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:06.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:06.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:06.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:06.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:06.747 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:06.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:06.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:06.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:06.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:06.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:07.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:07.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:07.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:07.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:07.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:07.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:07.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:07.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:07.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:07 smithi086 ceph-mon[97913]: pgmap v1611: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 716/228756 objects degraded (0.313%); 39658/228756 objects misplaced (17.336%); 29 MiB/s, 464 objects/s recovering 2024-06-14T23:21:07.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:07 smithi086 ceph-mon[102754]: pgmap v1611: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 716/228756 objects degraded (0.313%); 39658/228756 objects misplaced (17.336%); 29 MiB/s, 464 objects/s recovering 2024-06-14T23:21:07.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:07.403 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:07.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:07.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:07 smithi137 ceph-mon[90634]: pgmap v1611: 119 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 1 active+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 716/228756 objects degraded (0.313%); 39658/228756 objects misplaced (17.336%); 29 MiB/s, 464 objects/s recovering 2024-06-14T23:21:07.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:07.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:07.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:07.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:07.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:07.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:07.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:07.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:07.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:07.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:07.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:07.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:08.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:08.084 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:08.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:08.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:08.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:08.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:08.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:08.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:08.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:08.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:08.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:08.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:08.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:08.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:08.610 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:08.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:08.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:08.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:08.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:08.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:08.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:08.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:08.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:08.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:09.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:09.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:09.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:09.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:09.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:09.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:09.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:09.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:09.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:09.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:09 smithi086 ceph-mon[102754]: 1.13 scrub starts 2024-06-14T23:21:09.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:09 smithi086 ceph-mon[102754]: pgmap v1612: 119 pgs: 1 active+recovering+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 879 op/s; 716/225336 objects degraded (0.318%); 38756/225336 objects misplaced (17.199%); 22 MiB/s, 345 objects/s recovering 2024-06-14T23:21:09.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:09 smithi086 ceph-mon[102754]: osdmap e444: 8 total, 8 up, 7 in 2024-06-14T23:21:09.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:09 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 716/225336 objects degraded (0.318%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:21:09.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:09 smithi086 ceph-mon[97913]: 1.13 scrub starts 2024-06-14T23:21:09.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:09 smithi086 ceph-mon[97913]: pgmap v1612: 119 pgs: 1 active+recovering+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 879 op/s; 716/225336 objects degraded (0.318%); 38756/225336 objects misplaced (17.199%); 22 MiB/s, 345 objects/s recovering 2024-06-14T23:21:09.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:09 smithi086 ceph-mon[97913]: osdmap e444: 8 total, 8 up, 7 in 2024-06-14T23:21:09.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:09 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 716/225336 objects degraded (0.318%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:21:09.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:09.389 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2024-06-14T23:21:01.867719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5310831212,"num_objects":81050,"num_object_clones":0,"num_object_copies":243150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":41996,"num_objects_unfound":0,"num_objects_dirty":81050,"num_whiteouts":0,"num_read":140,"num_read_kb":262,"num_write":136678,"num_write_kb":6963091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23965,"num_bytes_recovered":1568853790,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87713,"ondisk_log_size":87713,"up":357,"acting":357,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":371,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":24866884,"kb_used_data":16659212,"kb_used_omap":33,"kb_used_meta":8207390,"kb_avail":631349180,"statfs":{"total":671965249536,"available":646501560320,"internally_reserved":0,"allocated":17059033088,"data_stored":17043076635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34272,"internal_metadata":8404367904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-743636992,"num_objects":-11347,"num_object_clones":0,"num_object_copies":-34041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3364,"num_objects_misplaced":-6221,"num_objects_unfound":0,"num_objects_dirty":-11347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11347,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5423,"num_bytes_recovered":355401728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007437"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":689,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507339+0000","last_change":"2024-06-14T23:20:36.533068+0000","last_active":"2024-06-14T23:20:37.507339+0000","last_peered":"2024-06-14T23:20:37.507339+0000","last_clean":"2024-06-14T23:20:37.507339+0000","last_became_active":"2024-06-14T23:20:36.532738+0000","last_became_peered":"2024-06-14T23:20:36.532738+0000","last_unstale":"2024-06-14T23:20:37.507339+0000","last_undegraded":"2024-06-14T23:20:37.507339+0000","last_fullsized":"2024-06-14T23:20:37.507339+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:59.302146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:59.302146+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025287560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":21,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507453+0000","last_change":"2024-06-14T23:20:33.597941+0000","last_active":"2024-06-14T23:20:37.507453+0000","last_peered":"2024-06-14T23:20:37.507453+0000","last_clean":"2024-06-14T23:20:37.507453+0000","last_became_active":"2024-06-14T23:20:33.515706+0000","last_became_peered":"2024-06-14T23:20:33.515706+0000","last_unstale":"2024-06-14T23:20:37.507453+0000","last_undegraded":"2024-06-14T23:20:37.507453+0000","last_fullsized":"2024-06-14T23:20:37.507453+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:06:05.294468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.506967+0000","last_change":"2024-06-14T23:20:33.524528+0000","last_active":"2024-06-14T23:20:37.506967+0000","last_peered":"2024-06-14T23:20:37.506967+0000","last_clean":"2024-06-14T23:20:37.506967+0000","last_became_active":"2024-06-14T23:20:33.524283+0000","last_became_peered":"2024-06-14T23:20:33.524283+0000","last_unstale":"2024-06-14T23:20:37.506967+0000","last_undegraded":"2024-06-14T23:20:37.506967+0000","last_fullsized":"2024-06-14T23:20:37.506967+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:19:13.967392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507398+0000","last_change":"2024-06-14T23:20:33.524484+0000","last_active":"2024-06-14T23:20:37.507398+0000","last_peered":"2024-06-14T23:20:37.507398+0000","last_clean":"2024-06-14T23:20:37.507398+0000","last_became_active":"2024-06-14T23:20:33.524150+0000","last_became_peered":"2024-06-14T23:20:33.524150+0000","last_unstale":"2024-06-14T23:20:37.507398+0000","last_undegraded":"2024-06-14T23:20:37.507398+0000","last_fullsized":"2024-06-14T23:20:37.507398+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:58.933924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507067+0000","last_change":"2024-06-14T23:20:33.515893+0000","last_active":"2024-06-14T23:20:37.507067+0000","last_peered":"2024-06-14T23:20:37.507067+0000","last_clean":"2024-06-14T23:20:37.507067+0000","last_became_active":"2024-06-14T23:20:33.515584+0000","last_became_peered":"2024-06-14T23:20:33.515584+0000","last_unstale":"2024-06-14T23:20:37.507067+0000","last_undegraded":"2024-06-14T23:20:37.507067+0000","last_fullsized":"2024-06-14T23:20:37.507067+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:06:56.884875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":676,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.506277+0000","last_change":"2024-06-14T23:19:23.472062+0000","last_active":"2024-06-14T23:20:36.506277+0000","last_peered":"2024-06-14T23:20:36.506277+0000","last_clean":"2024-06-14T23:20:36.506277+0000","last_became_active":"2024-06-14T23:19:23.471190+0000","last_became_peered":"2024-06-14T23:19:23.471190+0000","last_unstale":"2024-06-14T23:20:36.506277+0000","last_undegraded":"2024-06-14T23:20:36.506277+0000","last_fullsized":"2024-06-14T23:20:36.506277+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:14:16.297860+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:14:16.297860+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:50:51.694519+0000","scrub_duration":0.0039154719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":74,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502036+0000","last_change":"2024-06-14T23:17:36.973694+0000","last_active":"2024-06-14T23:20:36.502036+0000","last_peered":"2024-06-14T23:20:36.502036+0000","last_clean":"2024-06-14T23:20:36.502036+0000","last_became_active":"2024-06-14T23:17:36.973328+0000","last_became_peered":"2024-06-14T23:17:36.973328+0000","last_unstale":"2024-06-14T23:20:36.502036+0000","last_undegraded":"2024-06-14T23:20:36.502036+0000","last_fullsized":"2024-06-14T23:20:36.502036+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:42:16.457436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":431,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501163+0000","last_change":"2024-06-14T23:17:22.897703+0000","last_active":"2024-06-14T23:20:36.501163+0000","last_peered":"2024-06-14T23:20:36.501163+0000","last_clean":"2024-06-14T23:20:36.501163+0000","last_became_active":"2024-06-14T23:17:22.896776+0000","last_became_peered":"2024-06-14T23:17:22.896776+0000","last_unstale":"2024-06-14T23:20:36.501163+0000","last_undegraded":"2024-06-14T23:20:36.501163+0000","last_fullsized":"2024-06-14T23:20:36.501163+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:30:54.965205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":444,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.503126+0000","last_change":"2024-06-14T23:20:22.523472+0000","last_active":"2024-06-14T23:20:36.503126+0000","last_peered":"2024-06-14T23:20:36.503126+0000","last_clean":"2024-06-14T23:20:36.503126+0000","last_became_active":"2024-06-14T23:20:22.523120+0000","last_became_peered":"2024-06-14T23:20:22.523120+0000","last_unstale":"2024-06-14T23:20:36.503126+0000","last_undegraded":"2024-06-14T23:20:36.503126+0000","last_fullsized":"2024-06-14T23:20:36.503126+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:51.419531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:51.419531+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":40253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":420,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.503369+0000","last_change":"2024-06-14T23:10:26.056119+0000","last_active":"2024-06-14T23:20:36.503369+0000","last_peered":"2024-06-14T23:20:36.503369+0000","last_clean":"2024-06-14T23:20:36.503369+0000","last_became_active":"2024-06-14T23:10:26.046724+0000","last_became_peered":"2024-06-14T23:10:26.046724+0000","last_unstale":"2024-06-14T23:20:36.503369+0000","last_undegraded":"2024-06-14T23:20:36.503369+0000","last_fullsized":"2024-06-14T23:20:36.503369+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:38:20.114394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":451,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504517+0000","last_change":"2024-06-14T23:20:33.492885+0000","last_active":"2024-06-14T23:20:36.504517+0000","last_peered":"2024-06-14T23:20:36.504517+0000","last_clean":"2024-06-14T23:20:36.504517+0000","last_became_active":"2024-06-14T23:20:33.492383+0000","last_became_peered":"2024-06-14T23:20:33.492383+0000","last_unstale":"2024-06-14T23:20:36.504517+0000","last_undegraded":"2024-06-14T23:20:36.504517+0000","last_fullsized":"2024-06-14T23:20:36.504517+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:24:32.029275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":69,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504724+0000","last_change":"2024-06-14T23:17:22.888832+0000","last_active":"2024-06-14T23:20:36.504724+0000","last_peered":"2024-06-14T23:20:36.504724+0000","last_clean":"2024-06-14T23:20:36.504724+0000","last_became_active":"2024-06-14T23:17:22.888525+0000","last_became_peered":"2024-06-14T23:17:22.888525+0000","last_unstale":"2024-06-14T23:20:36.504724+0000","last_undegraded":"2024-06-14T23:20:36.504724+0000","last_fullsized":"2024-06-14T23:20:36.504724+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:36.414213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507238+0000","last_change":"2024-06-14T23:20:33.513367+0000","last_active":"2024-06-14T23:20:37.507238+0000","last_peered":"2024-06-14T23:20:37.507238+0000","last_clean":"2024-06-14T23:20:37.507238+0000","last_became_active":"2024-06-14T23:20:33.512816+0000","last_became_peered":"2024-06-14T23:20:33.512816+0000","last_unstale":"2024-06-14T23:20:37.507238+0000","last_undegraded":"2024-06-14T23:20:37.507238+0000","last_fullsized":"2024-06-14T23:20:37.507238+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:31:50.997555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":786,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499731+0000","last_change":"2024-06-14T23:20:22.519959+0000","last_active":"2024-06-14T23:20:36.499731+0000","last_peered":"2024-06-14T23:20:36.499731+0000","last_clean":"2024-06-14T23:20:36.499731+0000","last_became_active":"2024-06-14T23:20:22.519634+0000","last_became_peered":"2024-06-14T23:20:22.519634+0000","last_unstale":"2024-06-14T23:20:36.499731+0000","last_undegraded":"2024-06-14T23:20:36.499731+0000","last_fullsized":"2024-06-14T23:20:36.499731+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:35:36.889974+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507146+0000","last_change":"2024-06-14T23:20:33.492415+0000","last_active":"2024-06-14T23:20:37.507146+0000","last_peered":"2024-06-14T23:20:37.507146+0000","last_clean":"2024-06-14T23:20:37.507146+0000","last_became_active":"2024-06-14T23:20:33.492089+0000","last_became_peered":"2024-06-14T23:20:33.492089+0000","last_unstale":"2024-06-14T23:20:37.507146+0000","last_undegraded":"2024-06-14T23:20:37.507146+0000","last_fullsized":"2024-06-14T23:20:37.507146+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:40:01.500895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":192,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501879+0000","last_change":"2024-06-14T23:20:19.503838+0000","last_active":"2024-06-14T23:20:36.501879+0000","last_peered":"2024-06-14T23:20:36.501879+0000","last_clean":"2024-06-14T23:20:36.501879+0000","last_became_active":"2024-06-14T23:20:19.503397+0000","last_became_peered":"2024-06-14T23:20:19.503397+0000","last_unstale":"2024-06-14T23:20:36.501879+0000","last_undegraded":"2024-06-14T23:20:36.501879+0000","last_fullsized":"2024-06-14T23:20:36.501879+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:12:44.519521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":841,"reported_epoch":442,"state":"active+recovery_wait+remapped","last_fresh":"2024-06-14T23:20:36.503125+0000","last_change":"2024-06-14T23:20:34.653016+0000","last_active":"2024-06-14T23:20:36.503125+0000","last_peered":"2024-06-14T23:20:36.503125+0000","last_clean":"2024-06-14T23:18:58.404718+0000","last_became_active":"2024-06-14T23:20:34.644716+0000","last_became_peered":"2024-06-14T23:20:34.644716+0000","last_unstale":"2024-06-14T23:20:36.503125+0000","last_undegraded":"2024-06-14T23:20:36.503125+0000","last_fullsized":"2024-06-14T23:20:36.503125+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:35:02.426590+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":243,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504743+0000","last_change":"2024-06-14T23:10:26.048569+0000","last_active":"2024-06-14T23:20:36.504743+0000","last_peered":"2024-06-14T23:20:36.504743+0000","last_clean":"2024-06-14T23:20:36.504743+0000","last_became_active":"2024-06-14T23:10:26.046823+0000","last_became_peered":"2024-06-14T23:10:26.046823+0000","last_unstale":"2024-06-14T23:20:36.504743+0000","last_undegraded":"2024-06-14T23:20:36.504743+0000","last_fullsized":"2024-06-14T23:20:36.504743+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:54:03.048039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":269,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.505541+0000","last_change":"2024-06-14T23:20:19.497063+0000","last_active":"2024-06-14T23:20:36.505541+0000","last_peered":"2024-06-14T23:20:36.505541+0000","last_clean":"2024-06-14T23:20:36.505541+0000","last_became_active":"2024-06-14T23:20:19.496694+0000","last_became_peered":"2024-06-14T23:20:19.496694+0000","last_unstale":"2024-06-14T23:20:36.505541+0000","last_undegraded":"2024-06-14T23:20:36.505541+0000","last_fullsized":"2024-06-14T23:20:36.505541+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:20:11.696156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":817,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.503291+0000","last_change":"2024-06-14T23:17:25.906387+0000","last_active":"2024-06-14T23:20:36.503291+0000","last_peered":"2024-06-14T23:20:36.503291+0000","last_clean":"2024-06-14T23:20:36.503291+0000","last_became_active":"2024-06-14T23:17:25.906076+0000","last_became_peered":"2024-06-14T23:17:25.906076+0000","last_unstale":"2024-06-14T23:20:36.503291+0000","last_undegraded":"2024-06-14T23:20:36.503291+0000","last_fullsized":"2024-06-14T23:20:36.503291+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:02.302579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:02.302579+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:11:42.673437+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507003+0000","last_change":"2024-06-14T23:20:33.519035+0000","last_active":"2024-06-14T23:20:37.507003+0000","last_peered":"2024-06-14T23:20:37.507003+0000","last_clean":"2024-06-14T23:20:37.507003+0000","last_became_active":"2024-06-14T23:20:33.518741+0000","last_became_peered":"2024-06-14T23:20:33.518741+0000","last_unstale":"2024-06-14T23:20:37.507003+0000","last_undegraded":"2024-06-14T23:20:37.507003+0000","last_fullsized":"2024-06-14T23:20:37.507003+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:57:31.831100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":404,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501757+0000","last_change":"2024-06-14T23:17:22.900785+0000","last_active":"2024-06-14T23:20:36.501757+0000","last_peered":"2024-06-14T23:20:36.501757+0000","last_clean":"2024-06-14T23:20:36.501757+0000","last_became_active":"2024-06-14T23:17:22.900478+0000","last_became_peered":"2024-06-14T23:17:22.900478+0000","last_unstale":"2024-06-14T23:20:36.501757+0000","last_undegraded":"2024-06-14T23:20:36.501757+0000","last_fullsized":"2024-06-14T23:20:36.501757+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:36:48.160082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":726,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.500893+0000","last_change":"2024-06-14T23:17:36.971150+0000","last_active":"2024-06-14T23:20:36.500893+0000","last_peered":"2024-06-14T23:20:36.500893+0000","last_clean":"2024-06-14T23:20:36.500893+0000","last_became_active":"2024-06-14T23:17:36.970756+0000","last_became_peered":"2024-06-14T23:17:36.970756+0000","last_unstale":"2024-06-14T23:20:36.500893+0000","last_undegraded":"2024-06-14T23:20:36.500893+0000","last_fullsized":"2024-06-14T23:20:36.500893+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:32:06.129704+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":306,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501979+0000","last_change":"2024-06-14T23:20:19.501101+0000","last_active":"2024-06-14T23:20:36.501979+0000","last_peered":"2024-06-14T23:20:36.501979+0000","last_clean":"2024-06-14T23:20:36.501979+0000","last_became_active":"2024-06-14T23:20:19.500789+0000","last_became_peered":"2024-06-14T23:20:19.500789+0000","last_unstale":"2024-06-14T23:20:36.501979+0000","last_undegraded":"2024-06-14T23:20:36.501979+0000","last_fullsized":"2024-06-14T23:20:36.501979+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:48:45.904500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499646+0000","last_change":"2024-06-14T23:20:19.518229+0000","last_active":"2024-06-14T23:20:36.499646+0000","last_peered":"2024-06-14T23:20:36.499646+0000","last_clean":"2024-06-14T23:20:36.499646+0000","last_became_active":"2024-06-14T23:20:19.516920+0000","last_became_peered":"2024-06-14T23:20:19.516920+0000","last_unstale":"2024-06-14T23:20:36.499646+0000","last_undegraded":"2024-06-14T23:20:36.499646+0000","last_fullsized":"2024-06-14T23:20:36.499646+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:10:04.124831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":749,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.507184+0000","last_change":"2024-06-14T23:10:29.068876+0000","last_active":"2024-06-14T23:20:36.507184+0000","last_peered":"2024-06-14T23:20:36.507184+0000","last_clean":"2024-06-14T23:20:36.507184+0000","last_became_active":"2024-06-14T23:10:29.068524+0000","last_became_peered":"2024-06-14T23:10:29.068524+0000","last_unstale":"2024-06-14T23:20:36.507184+0000","last_undegraded":"2024-06-14T23:20:36.507184+0000","last_fullsized":"2024-06-14T23:20:36.507184+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:58:05.964299+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":312,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504591+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:20:36.504591+0000","last_peered":"2024-06-14T23:20:36.504591+0000","last_clean":"2024-06-14T23:20:36.504591+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:20:36.504591+0000","last_undegraded":"2024-06-14T23:20:36.504591+0000","last_fullsized":"2024-06-14T23:20:36.504591+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507413+0000","last_change":"2024-06-14T23:20:33.519920+0000","last_active":"2024-06-14T23:20:37.507413+0000","last_peered":"2024-06-14T23:20:37.507413+0000","last_clean":"2024-06-14T23:20:37.507413+0000","last_became_active":"2024-06-14T23:20:33.517941+0000","last_became_peered":"2024-06-14T23:20:33.517941+0000","last_unstale":"2024-06-14T23:20:37.507413+0000","last_undegraded":"2024-06-14T23:20:37.507413+0000","last_fullsized":"2024-06-14T23:20:37.507413+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:12:45.394589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":588,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507223+0000","last_change":"2024-06-14T23:20:36.538318+0000","last_active":"2024-06-14T23:20:37.507223+0000","last_peered":"2024-06-14T23:20:37.507223+0000","last_clean":"2024-06-14T23:20:37.507223+0000","last_became_active":"2024-06-14T23:20:36.537993+0000","last_became_peered":"2024-06-14T23:20:36.537993+0000","last_unstale":"2024-06-14T23:20:37.507223+0000","last_undegraded":"2024-06-14T23:20:37.507223+0000","last_fullsized":"2024-06-14T23:20:37.507223+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:17.053258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:17.053258+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001812674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":330,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504729+0000","last_change":"2024-06-14T22:55:53.664567+0000","last_active":"2024-06-14T23:20:36.504729+0000","last_peered":"2024-06-14T23:20:36.504729+0000","last_clean":"2024-06-14T23:20:36.504729+0000","last_became_active":"2024-06-14T22:55:53.664002+0000","last_became_peered":"2024-06-14T22:55:53.664002+0000","last_unstale":"2024-06-14T23:20:36.504729+0000","last_undegraded":"2024-06-14T23:20:36.504729+0000","last_fullsized":"2024-06-14T23:20:36.504729+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:44.104092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":486,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502447+0000","last_change":"2024-06-14T23:20:33.516758+0000","last_active":"2024-06-14T23:20:36.502447+0000","last_peered":"2024-06-14T23:20:36.502447+0000","last_clean":"2024-06-14T23:20:36.502447+0000","last_became_active":"2024-06-14T23:20:33.516122+0000","last_became_peered":"2024-06-14T23:20:33.516122+0000","last_unstale":"2024-06-14T23:20:36.502447+0000","last_undegraded":"2024-06-14T23:20:36.502447+0000","last_fullsized":"2024-06-14T23:20:36.502447+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:51:14.428188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":586,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499731+0000","last_change":"2024-06-14T23:19:23.473395+0000","last_active":"2024-06-14T23:20:36.499731+0000","last_peered":"2024-06-14T23:20:36.499731+0000","last_clean":"2024-06-14T23:20:36.499731+0000","last_became_active":"2024-06-14T23:19:23.472999+0000","last_became_peered":"2024-06-14T23:19:23.472999+0000","last_unstale":"2024-06-14T23:20:36.499731+0000","last_undegraded":"2024-06-14T23:20:36.499731+0000","last_fullsized":"2024-06-14T23:20:36.499731+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:03.262561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:03.262561+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":6,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":142,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.506389+0000","last_change":"2024-06-14T23:15:09.030189+0000","last_active":"2024-06-14T23:20:36.506389+0000","last_peered":"2024-06-14T23:20:36.506389+0000","last_clean":"2024-06-14T23:20:36.506389+0000","last_became_active":"2024-06-14T23:15:09.029757+0000","last_became_peered":"2024-06-14T23:15:09.029757+0000","last_unstale":"2024-06-14T23:20:36.506389+0000","last_undegraded":"2024-06-14T23:20:36.506389+0000","last_fullsized":"2024-06-14T23:20:36.506389+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:58:48.305434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":384,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501610+0000","last_change":"2024-06-14T23:15:09.044391+0000","last_active":"2024-06-14T23:20:36.501610+0000","last_peered":"2024-06-14T23:20:36.501610+0000","last_clean":"2024-06-14T23:20:36.501610+0000","last_became_active":"2024-06-14T23:15:09.043763+0000","last_became_peered":"2024-06-14T23:15:09.043763+0000","last_unstale":"2024-06-14T23:20:36.501610+0000","last_undegraded":"2024-06-14T23:20:36.501610+0000","last_fullsized":"2024-06-14T23:20:36.501610+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:46:25.597564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":505,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.505687+0000","last_change":"2024-06-14T23:20:22.507214+0000","last_active":"2024-06-14T23:20:36.505687+0000","last_peered":"2024-06-14T23:20:36.505687+0000","last_clean":"2024-06-14T23:20:36.505687+0000","last_became_active":"2024-06-14T23:20:22.506459+0000","last_became_peered":"2024-06-14T23:20:22.506459+0000","last_unstale":"2024-06-14T23:20:36.505687+0000","last_undegraded":"2024-06-14T23:20:36.505687+0000","last_fullsized":"2024-06-14T23:20:36.505687+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.634078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.634078+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:07:16.747252+0000","scrub_duration":0.00160432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.b","version":"443'8679","reported_seq":13428,"reported_epoch":443,"state":"active+remapped+backfill_wait","last_fresh":"2024-06-14T23:20:59.926753+0000","last_change":"2024-06-14T23:20:34.648334+0000","last_active":"2024-06-14T23:20:59.926753+0000","last_peered":"2024-06-14T23:20:59.926753+0000","last_clean":"2024-06-14T23:20:32.889385+0000","last_became_active":"2024-06-14T23:20:34.637776+0000","last_became_peered":"2024-06-14T23:20:34.637776+0000","last_unstale":"2024-06-14T23:20:59.926753+0000","last_undegraded":"2024-06-14T23:20:59.926753+0000","last_fullsized":"2024-06-14T23:20:59.926753+0000","mapping_epoch":439,"log_start":"425'3534","ondisk_log_start":"425'3534","created":411,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:03:40.250435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323813376,"num_objects":4941,"num_object_clones":0,"num_object_copies":14823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4941,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8679,"num_write_kb":435840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2342,"num_bytes_recovered":153485312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[0,6,7],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,6,7","objects":4941}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":260,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499983+0000","last_change":"2024-06-14T23:19:23.895377+0000","last_active":"2024-06-14T23:20:36.499983+0000","last_peered":"2024-06-14T23:20:36.499983+0000","last_clean":"2024-06-14T23:20:36.499983+0000","last_became_active":"2024-06-14T23:19:23.895084+0000","last_became_peered":"2024-06-14T23:19:23.895084+0000","last_unstale":"2024-06-14T23:20:36.499983+0000","last_undegraded":"2024-06-14T23:20:36.499983+0000","last_fullsized":"2024-06-14T23:20:36.499983+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:15:21.807921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":435,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499800+0000","last_change":"2024-06-14T23:20:33.516210+0000","last_active":"2024-06-14T23:20:36.499800+0000","last_peered":"2024-06-14T23:20:36.499800+0000","last_clean":"2024-06-14T23:20:36.499800+0000","last_became_active":"2024-06-14T23:20:33.515892+0000","last_became_peered":"2024-06-14T23:20:33.515892+0000","last_unstale":"2024-06-14T23:20:36.499800+0000","last_undegraded":"2024-06-14T23:20:36.499800+0000","last_fullsized":"2024-06-14T23:20:36.499800+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:52.415552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":141,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.505299+0000","last_change":"2024-06-14T23:17:22.914912+0000","last_active":"2024-06-14T23:20:36.505299+0000","last_peered":"2024-06-14T23:20:36.505299+0000","last_clean":"2024-06-14T23:20:36.505299+0000","last_became_active":"2024-06-14T23:17:22.896173+0000","last_became_peered":"2024-06-14T23:17:22.896173+0000","last_unstale":"2024-06-14T23:20:36.505299+0000","last_undegraded":"2024-06-14T23:20:36.505299+0000","last_fullsized":"2024-06-14T23:20:36.505299+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:58:10.727766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":491,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.503280+0000","last_change":"2024-06-14T23:20:22.511982+0000","last_active":"2024-06-14T23:20:36.503280+0000","last_peered":"2024-06-14T23:20:36.503280+0000","last_clean":"2024-06-14T23:20:36.503280+0000","last_became_active":"2024-06-14T23:20:22.511630+0000","last_became_peered":"2024-06-14T23:20:22.511630+0000","last_unstale":"2024-06-14T23:20:36.503280+0000","last_undegraded":"2024-06-14T23:20:36.503280+0000","last_fullsized":"2024-06-14T23:20:36.503280+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:10.229178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:10.229178+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":27,"num_write":7,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.a","version":"443'8511","reported_seq":10430,"reported_epoch":443,"state":"active+remapped+backfill_wait","last_fresh":"2024-06-14T23:20:58.875154+0000","last_change":"2024-06-14T23:20:34.660810+0000","last_active":"2024-06-14T23:20:58.875154+0000","last_peered":"2024-06-14T23:20:58.875154+0000","last_clean":"2024-06-14T23:20:18.473852+0000","last_became_active":"2024-06-14T23:20:20.608464+0000","last_became_peered":"2024-06-14T23:20:20.608464+0000","last_unstale":"2024-06-14T23:20:58.875154+0000","last_undegraded":"2024-06-14T23:20:58.875154+0000","last_fullsized":"2024-06-14T23:20:58.875154+0000","mapping_epoch":433,"log_start":"425'3495","ondisk_log_start":"425'3495","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:49:46.201757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332857344,"num_objects":5079,"num_object_clones":0,"num_object_copies":15237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4358,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":60948480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,3],"avail_no_missing":["5","3","4"],"object_location_counts":[{"shards":"3,4,5","objects":5079}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":324,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504263+0000","last_change":"2024-06-14T23:20:19.499145+0000","last_active":"2024-06-14T23:20:36.504263+0000","last_peered":"2024-06-14T23:20:36.504263+0000","last_clean":"2024-06-14T23:20:36.504263+0000","last_became_active":"2024-06-14T23:20:19.498774+0000","last_became_peered":"2024-06-14T23:20:19.498774+0000","last_unstale":"2024-06-14T23:20:36.504263+0000","last_undegraded":"2024-06-14T23:20:36.504263+0000","last_fullsized":"2024-06-14T23:20:36.504263+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:44:13.452541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":361,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504596+0000","last_change":"2024-06-14T23:15:09.029369+0000","last_active":"2024-06-14T23:20:36.504596+0000","last_peered":"2024-06-14T23:20:36.504596+0000","last_clean":"2024-06-14T23:20:36.504596+0000","last_became_active":"2024-06-14T23:15:09.029112+0000","last_became_peered":"2024-06-14T23:15:09.029112+0000","last_unstale":"2024-06-14T23:20:36.504596+0000","last_undegraded":"2024-06-14T23:20:36.504596+0000","last_fullsized":"2024-06-14T23:20:36.504596+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:39:40.009379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":263,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.505077+0000","last_change":"2024-06-14T23:17:22.899160+0000","last_active":"2024-06-14T23:20:36.505077+0000","last_peered":"2024-06-14T23:20:36.505077+0000","last_clean":"2024-06-14T23:20:36.505077+0000","last_became_active":"2024-06-14T23:17:22.892666+0000","last_became_peered":"2024-06-14T23:17:22.892666+0000","last_unstale":"2024-06-14T23:20:36.505077+0000","last_undegraded":"2024-06-14T23:20:36.505077+0000","last_fullsized":"2024-06-14T23:20:36.505077+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:21:27.960103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":799,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507295+0000","last_change":"2024-06-14T23:20:36.534472+0000","last_active":"2024-06-14T23:20:37.507295+0000","last_peered":"2024-06-14T23:20:37.507295+0000","last_clean":"2024-06-14T23:20:37.507295+0000","last_became_active":"2024-06-14T23:20:36.534150+0000","last_became_peered":"2024-06-14T23:20:36.534150+0000","last_unstale":"2024-06-14T23:20:37.507295+0000","last_undegraded":"2024-06-14T23:20:37.507295+0000","last_fullsized":"2024-06-14T23:20:37.507295+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:51.748589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:51.748589+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:09:53.180585+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"443'8599","reported_seq":8641,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-14T23:21:00.830091+0000","last_change":"2024-06-14T23:18:48.032161+0000","last_active":"2024-06-14T23:21:00.830091+0000","last_peered":"2024-06-14T23:21:00.830091+0000","last_clean":"2024-06-14T23:21:00.830091+0000","last_became_active":"2024-06-14T23:18:48.031847+0000","last_became_peered":"2024-06-14T23:18:48.031847+0000","last_unstale":"2024-06-14T23:21:00.830091+0000","last_undegraded":"2024-06-14T23:21:00.830091+0000","last_fullsized":"2024-06-14T23:21:00.830091+0000","mapping_epoch":411,"log_start":"415'1600","ondisk_log_start":"415'1600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":6999,"log_dups_size":0,"ondisk_log_size":6999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:18:59.248622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326041600,"num_objects":4975,"num_object_clones":0,"num_object_copies":14925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":434368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":239,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504524+0000","last_change":"2024-06-14T23:19:23.471579+0000","last_active":"2024-06-14T23:20:36.504524+0000","last_peered":"2024-06-14T23:20:36.504524+0000","last_clean":"2024-06-14T23:20:36.504524+0000","last_became_active":"2024-06-14T23:19:23.471172+0000","last_became_peered":"2024-06-14T23:19:23.471172+0000","last_unstale":"2024-06-14T23:20:36.504524+0000","last_undegraded":"2024-06-14T23:20:36.504524+0000","last_fullsized":"2024-06-14T23:20:36.504524+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:46:26.499184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":438,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.507174+0000","last_change":"2024-06-14T23:17:22.890776+0000","last_active":"2024-06-14T23:20:36.507174+0000","last_peered":"2024-06-14T23:20:36.507174+0000","last_clean":"2024-06-14T23:20:36.507174+0000","last_became_active":"2024-06-14T23:17:22.890463+0000","last_became_peered":"2024-06-14T23:17:22.890463+0000","last_unstale":"2024-06-14T23:20:36.507174+0000","last_undegraded":"2024-06-14T23:20:36.507174+0000","last_fullsized":"2024-06-14T23:20:36.507174+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:45:42.223422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507193+0000","last_change":"2024-06-14T23:20:33.515949+0000","last_active":"2024-06-14T23:20:37.507193+0000","last_peered":"2024-06-14T23:20:37.507193+0000","last_clean":"2024-06-14T23:20:37.507193+0000","last_became_active":"2024-06-14T23:20:33.515653+0000","last_became_peered":"2024-06-14T23:20:33.515653+0000","last_unstale":"2024-06-14T23:20:37.507193+0000","last_undegraded":"2024-06-14T23:20:37.507193+0000","last_fullsized":"2024-06-14T23:20:37.507193+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:25:36.918704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":782,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.503482+0000","last_change":"2024-06-14T23:20:22.518019+0000","last_active":"2024-06-14T23:20:36.503482+0000","last_peered":"2024-06-14T23:20:36.503482+0000","last_clean":"2024-06-14T23:20:36.503482+0000","last_became_active":"2024-06-14T23:20:22.517708+0000","last_became_peered":"2024-06-14T23:20:22.517708+0000","last_unstale":"2024-06-14T23:20:36.503482+0000","last_undegraded":"2024-06-14T23:20:36.503482+0000","last_fullsized":"2024-06-14T23:20:36.503482+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T11:02:40.596440+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"443'8445","reported_seq":12015,"reported_epoch":443,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-06-14T23:21:00.059692+0000","last_change":"2024-06-14T23:20:34.656991+0000","last_active":"2024-06-14T23:21:00.059692+0000","last_peered":"2024-06-14T23:21:00.059692+0000","last_clean":"2024-06-14T23:19:09.846924+0000","last_became_active":"2024-06-14T23:20:34.644488+0000","last_became_peered":"2024-06-14T23:20:34.644488+0000","last_unstale":"2024-06-14T23:21:00.059692+0000","last_undegraded":"2024-06-14T23:20:34.618629+0000","last_fullsized":"2024-06-14T23:21:00.059692+0000","mapping_epoch":439,"log_start":"425'3284","ondisk_log_start":"425'3284","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:56.021530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323682304,"num_objects":4939,"num_object_clones":0,"num_object_copies":14817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":461,"num_objects_misplaced":4478,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8445,"num_write_kb":428288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1754,"num_bytes_recovered":114950144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3,5],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":461},{"shards":"0,3,5","objects":4478}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":239,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504015+0000","last_change":"2024-06-14T23:17:22.905260+0000","last_active":"2024-06-14T23:20:36.504015+0000","last_peered":"2024-06-14T23:20:36.504015+0000","last_clean":"2024-06-14T23:20:36.504015+0000","last_became_active":"2024-06-14T23:17:22.903093+0000","last_became_peered":"2024-06-14T23:17:22.903093+0000","last_unstale":"2024-06-14T23:20:36.504015+0000","last_undegraded":"2024-06-14T23:20:36.504015+0000","last_fullsized":"2024-06-14T23:20:36.504015+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:14:57.000855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507252+0000","last_change":"2024-06-14T23:20:33.518719+0000","last_active":"2024-06-14T23:20:37.507252+0000","last_peered":"2024-06-14T23:20:37.507252+0000","last_clean":"2024-06-14T23:20:37.507252+0000","last_became_active":"2024-06-14T23:20:33.518229+0000","last_became_peered":"2024-06-14T23:20:33.518229+0000","last_unstale":"2024-06-14T23:20:37.507252+0000","last_undegraded":"2024-06-14T23:20:37.507252+0000","last_fullsized":"2024-06-14T23:20:37.507252+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:12:40.540689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507280+0000","last_change":"2024-06-14T23:20:33.515314+0000","last_active":"2024-06-14T23:20:37.507280+0000","last_peered":"2024-06-14T23:20:37.507280+0000","last_clean":"2024-06-14T23:20:37.507280+0000","last_became_active":"2024-06-14T23:20:33.515011+0000","last_became_peered":"2024-06-14T23:20:33.515011+0000","last_unstale":"2024-06-14T23:20:37.507280+0000","last_undegraded":"2024-06-14T23:20:37.507280+0000","last_fullsized":"2024-06-14T23:20:37.507280+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:47:33.574708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":742,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499937+0000","last_change":"2024-06-14T23:20:22.523039+0000","last_active":"2024-06-14T23:20:36.499937+0000","last_peered":"2024-06-14T23:20:36.499937+0000","last_clean":"2024-06-14T23:20:36.499937+0000","last_became_active":"2024-06-14T23:20:22.522726+0000","last_became_peered":"2024-06-14T23:20:22.522726+0000","last_unstale":"2024-06-14T23:20:36.499937+0000","last_undegraded":"2024-06-14T23:20:36.499937+0000","last_fullsized":"2024-06-14T23:20:36.499937+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:17.272250+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"443'8812","reported_seq":13652,"reported_epoch":443,"state":"active+remapped+backfill_wait","last_fresh":"2024-06-14T23:20:59.940332+0000","last_change":"2024-06-14T23:20:34.647202+0000","last_active":"2024-06-14T23:20:59.940332+0000","last_peered":"2024-06-14T23:20:59.940332+0000","last_clean":"2024-06-14T23:20:32.902182+0000","last_became_active":"2024-06-14T23:20:34.635938+0000","last_became_peered":"2024-06-14T23:20:34.635938+0000","last_unstale":"2024-06-14T23:20:59.940332+0000","last_undegraded":"2024-06-14T23:20:59.940332+0000","last_fullsized":"2024-06-14T23:20:59.940332+0000","mapping_epoch":439,"log_start":"425'3681","ondisk_log_start":"425'3681","created":411,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:03:21.748339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330694656,"num_objects":5046,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5046,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":443456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2387,"num_bytes_recovered":156434432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[0,4,6],"avail_no_missing":["0","4","6"],"object_location_counts":[{"shards":"0,4,6","objects":5046}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507208+0000","last_change":"2024-06-14T23:20:33.518934+0000","last_active":"2024-06-14T23:20:37.507208+0000","last_peered":"2024-06-14T23:20:37.507208+0000","last_clean":"2024-06-14T23:20:37.507208+0000","last_became_active":"2024-06-14T23:20:33.518659+0000","last_became_peered":"2024-06-14T23:20:33.518659+0000","last_unstale":"2024-06-14T23:20:37.507208+0000","last_undegraded":"2024-06-14T23:20:37.507208+0000","last_fullsized":"2024-06-14T23:20:37.507208+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:16:57.240614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":405,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499908+0000","last_change":"2024-06-14T23:17:22.896004+0000","last_active":"2024-06-14T23:20:36.499908+0000","last_peered":"2024-06-14T23:20:36.499908+0000","last_clean":"2024-06-14T23:20:36.499908+0000","last_became_active":"2024-06-14T23:17:22.895677+0000","last_became_peered":"2024-06-14T23:17:22.895677+0000","last_unstale":"2024-06-14T23:20:36.499908+0000","last_undegraded":"2024-06-14T23:20:36.499908+0000","last_fullsized":"2024-06-14T23:20:36.499908+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:07:44.738249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":442,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502285+0000","last_change":"2024-06-14T23:17:36.972069+0000","last_active":"2024-06-14T23:20:36.502285+0000","last_peered":"2024-06-14T23:20:36.502285+0000","last_clean":"2024-06-14T23:20:36.502285+0000","last_became_active":"2024-06-14T23:17:36.971783+0000","last_became_peered":"2024-06-14T23:17:36.971783+0000","last_unstale":"2024-06-14T23:20:36.502285+0000","last_undegraded":"2024-06-14T23:20:36.502285+0000","last_fullsized":"2024-06-14T23:20:36.502285+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:04:41.244348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":799,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507033+0000","last_change":"2024-06-14T23:20:36.538197+0000","last_active":"2024-06-14T23:20:37.507033+0000","last_peered":"2024-06-14T23:20:37.507033+0000","last_clean":"2024-06-14T23:20:37.507033+0000","last_became_active":"2024-06-14T23:20:36.534739+0000","last_became_peered":"2024-06-14T23:20:36.534739+0000","last_unstale":"2024-06-14T23:20:37.507033+0000","last_undegraded":"2024-06-14T23:20:37.507033+0000","last_fullsized":"2024-06-14T23:20:37.507033+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.522124+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:44:24.221647+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.d","version":"443'8432","reported_seq":8473,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-14T23:20:57.214673+0000","last_change":"2024-06-14T23:18:48.032086+0000","last_active":"2024-06-14T23:20:57.214673+0000","last_peered":"2024-06-14T23:20:57.214673+0000","last_clean":"2024-06-14T23:20:57.214673+0000","last_became_active":"2024-06-14T23:18:48.031779+0000","last_became_peered":"2024-06-14T23:18:48.031779+0000","last_unstale":"2024-06-14T23:20:57.214673+0000","last_undegraded":"2024-06-14T23:20:57.214673+0000","last_fullsized":"2024-06-14T23:20:57.214673+0000","mapping_epoch":411,"log_start":"425'3663","ondisk_log_start":"425'3663","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4769,"log_dups_size":0,"ondisk_log_size":4769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:07:21.593686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341377048,"num_objects":5210,"num_object_clones":0,"num_object_copies":15630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8432,"num_write_kb":436481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":413,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504150+0000","last_change":"2024-06-14T23:17:22.902652+0000","last_active":"2024-06-14T23:20:36.504150+0000","last_peered":"2024-06-14T23:20:36.504150+0000","last_clean":"2024-06-14T23:20:36.504150+0000","last_became_active":"2024-06-14T23:17:22.902142+0000","last_became_peered":"2024-06-14T23:17:22.902142+0000","last_unstale":"2024-06-14T23:20:36.504150+0000","last_undegraded":"2024-06-14T23:20:36.504150+0000","last_fullsized":"2024-06-14T23:20:36.504150+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:08:45.185030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.506947+0000","last_change":"2024-06-14T23:20:33.524579+0000","last_active":"2024-06-14T23:20:37.506947+0000","last_peered":"2024-06-14T23:20:37.506947+0000","last_clean":"2024-06-14T23:20:37.506947+0000","last_became_active":"2024-06-14T23:20:33.523777+0000","last_became_peered":"2024-06-14T23:20:33.523777+0000","last_unstale":"2024-06-14T23:20:37.506947+0000","last_undegraded":"2024-06-14T23:20:37.506947+0000","last_fullsized":"2024-06-14T23:20:37.506947+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:06:26.206711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507099+0000","last_change":"2024-06-14T23:20:33.523363+0000","last_active":"2024-06-14T23:20:37.507099+0000","last_peered":"2024-06-14T23:20:37.507099+0000","last_clean":"2024-06-14T23:20:37.507099+0000","last_became_active":"2024-06-14T23:20:33.523094+0000","last_became_peered":"2024-06-14T23:20:33.523094+0000","last_unstale":"2024-06-14T23:20:37.507099+0000","last_undegraded":"2024-06-14T23:20:37.507099+0000","last_fullsized":"2024-06-14T23:20:37.507099+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:35:35.422168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":580,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504801+0000","last_change":"2024-06-14T23:17:52.317490+0000","last_active":"2024-06-14T23:20:36.504801+0000","last_peered":"2024-06-14T23:20:36.504801+0000","last_clean":"2024-06-14T23:20:36.504801+0000","last_became_active":"2024-06-14T23:17:25.912414+0000","last_became_peered":"2024-06-14T23:17:25.912414+0000","last_unstale":"2024-06-14T23:20:36.504801+0000","last_undegraded":"2024-06-14T23:20:36.504801+0000","last_fullsized":"2024-06-14T23:20:36.504801+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:52.317329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:52.317329+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:25:14.327597+0000","scrub_duration":0.0087017850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"443'8601","reported_seq":12026,"reported_epoch":443,"state":"active+recovering+degraded+remapped","last_fresh":"2024-06-14T23:21:00.055970+0000","last_change":"2024-06-14T23:20:34.662612+0000","last_active":"2024-06-14T23:21:00.055970+0000","last_peered":"2024-06-14T23:21:00.055970+0000","last_clean":"2024-06-14T23:19:09.848418+0000","last_became_active":"2024-06-14T23:20:34.641651+0000","last_became_peered":"2024-06-14T23:20:34.641651+0000","last_unstale":"2024-06-14T23:21:00.055970+0000","last_undegraded":"2024-06-14T23:20:34.628193+0000","last_fullsized":"2024-06-14T23:21:00.055970+0000","mapping_epoch":439,"log_start":"425'3481","ondisk_log_start":"425'3481","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5120,"log_dups_size":0,"ondisk_log_size":5120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:35:01.053961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329056256,"num_objects":5021,"num_object_clones":0,"num_object_copies":15063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8601,"num_write_kb":435904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":110166016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[0,4,6],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":391},{"shards":"0,4,6","objects":4630}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507383+0000","last_change":"2024-06-14T23:20:33.519905+0000","last_active":"2024-06-14T23:20:37.507383+0000","last_peered":"2024-06-14T23:20:37.507383+0000","last_clean":"2024-06-14T23:20:37.507383+0000","last_became_active":"2024-06-14T23:20:33.519000+0000","last_became_peered":"2024-06-14T23:20:33.519000+0000","last_unstale":"2024-06-14T23:20:37.507383+0000","last_undegraded":"2024-06-14T23:20:37.507383+0000","last_fullsized":"2024-06-14T23:20:37.507383+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:47:17.908824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":200,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.503635+0000","last_change":"2024-06-14T23:20:19.493516+0000","last_active":"2024-06-14T23:20:36.503635+0000","last_peered":"2024-06-14T23:20:36.503635+0000","last_clean":"2024-06-14T23:20:36.503635+0000","last_became_active":"2024-06-14T23:20:19.493209+0000","last_became_peered":"2024-06-14T23:20:19.493209+0000","last_unstale":"2024-06-14T23:20:36.503635+0000","last_undegraded":"2024-06-14T23:20:36.503635+0000","last_fullsized":"2024-06-14T23:20:36.503635+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:17:00.579072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":421,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504432+0000","last_change":"2024-06-14T23:17:22.888817+0000","last_active":"2024-06-14T23:20:36.504432+0000","last_peered":"2024-06-14T23:20:36.504432+0000","last_clean":"2024-06-14T23:20:36.504432+0000","last_became_active":"2024-06-14T23:17:22.888355+0000","last_became_peered":"2024-06-14T23:17:22.888355+0000","last_unstale":"2024-06-14T23:20:36.504432+0000","last_undegraded":"2024-06-14T23:20:36.504432+0000","last_fullsized":"2024-06-14T23:20:36.504432+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:10:02.306355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":472,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499839+0000","last_change":"2024-06-14T23:17:53.735170+0000","last_active":"2024-06-14T23:20:36.499839+0000","last_peered":"2024-06-14T23:20:36.499839+0000","last_clean":"2024-06-14T23:20:36.499839+0000","last_became_active":"2024-06-14T23:17:36.972022+0000","last_became_peered":"2024-06-14T23:17:36.972022+0000","last_unstale":"2024-06-14T23:20:36.499839+0000","last_undegraded":"2024-06-14T23:20:36.499839+0000","last_fullsized":"2024-06-14T23:20:36.499839+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.735114+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:17:53.735114+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.735114+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:44:04.045839+0000","scrub_duration":0.0011940379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.f","version":"443'8544","reported_seq":13549,"reported_epoch":443,"state":"active+remapped+backfill_wait","last_fresh":"2024-06-14T23:20:59.054431+0000","last_change":"2024-06-14T23:20:34.653698+0000","last_active":"2024-06-14T23:20:59.054431+0000","last_peered":"2024-06-14T23:20:59.054431+0000","last_clean":"2024-06-14T23:20:18.476676+0000","last_became_active":"2024-06-14T23:20:33.597041+0000","last_became_peered":"2024-06-14T23:20:33.597041+0000","last_unstale":"2024-06-14T23:20:59.054431+0000","last_undegraded":"2024-06-14T23:20:59.054431+0000","last_fullsized":"2024-06-14T23:20:59.054431+0000","mapping_epoch":438,"log_start":"425'3311","ondisk_log_start":"425'3311","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:58:09.876018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328466432,"num_objects":5012,"num_object_clones":0,"num_object_copies":15036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4662,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8544,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2468,"num_bytes_recovered":161742848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,3],"avail_no_missing":["7","3","5"],"object_location_counts":[{"shards":"3,5,7","objects":5012}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":358,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502292+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:20:36.502292+0000","last_peered":"2024-06-14T23:20:36.502292+0000","last_clean":"2024-06-14T23:20:36.502292+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:20:36.502292+0000","last_undegraded":"2024-06-14T23:20:36.502292+0000","last_fullsized":"2024-06-14T23:20:36.502292+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":227,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502431+0000","last_change":"2024-06-14T23:20:19.502117+0000","last_active":"2024-06-14T23:20:36.502431+0000","last_peered":"2024-06-14T23:20:36.502431+0000","last_clean":"2024-06-14T23:20:36.502431+0000","last_became_active":"2024-06-14T23:20:19.501780+0000","last_became_peered":"2024-06-14T23:20:19.501780+0000","last_unstale":"2024-06-14T23:20:36.502431+0000","last_undegraded":"2024-06-14T23:20:36.502431+0000","last_fullsized":"2024-06-14T23:20:36.502431+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:30:15.566019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507162+0000","last_change":"2024-06-14T23:20:33.518180+0000","last_active":"2024-06-14T23:20:37.507162+0000","last_peered":"2024-06-14T23:20:37.507162+0000","last_clean":"2024-06-14T23:20:37.507162+0000","last_became_active":"2024-06-14T23:20:33.517893+0000","last_became_peered":"2024-06-14T23:20:33.517893+0000","last_unstale":"2024-06-14T23:20:37.507162+0000","last_undegraded":"2024-06-14T23:20:37.507162+0000","last_fullsized":"2024-06-14T23:20:37.507162+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:56:03.041933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":687,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504626+0000","last_change":"2024-06-14T23:17:53.277122+0000","last_active":"2024-06-14T23:20:36.504626+0000","last_peered":"2024-06-14T23:20:36.504626+0000","last_clean":"2024-06-14T23:20:36.504626+0000","last_became_active":"2024-06-14T23:17:25.909540+0000","last_became_peered":"2024-06-14T23:17:25.909540+0000","last_unstale":"2024-06-14T23:20:36.504626+0000","last_undegraded":"2024-06-14T23:20:36.504626+0000","last_fullsized":"2024-06-14T23:20:36.504626+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.277028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.277028+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:37:17.362500+0000","scrub_duration":0.002049375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"443'8616","reported_seq":8656,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-14T23:20:59.070515+0000","last_change":"2024-06-14T23:18:48.033959+0000","last_active":"2024-06-14T23:20:59.070515+0000","last_peered":"2024-06-14T23:20:59.070515+0000","last_clean":"2024-06-14T23:20:59.070515+0000","last_became_active":"2024-06-14T23:18:48.033347+0000","last_became_peered":"2024-06-14T23:18:48.033347+0000","last_unstale":"2024-06-14T23:20:59.070515+0000","last_undegraded":"2024-06-14T23:20:59.070515+0000","last_fullsized":"2024-06-14T23:20:59.070515+0000","mapping_epoch":411,"log_start":"425'3374","ondisk_log_start":"425'3374","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:44:39.633537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333053952,"num_objects":5082,"num_object_clones":0,"num_object_copies":15246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8616,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":244,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504998+0000","last_change":"2024-06-14T23:10:26.043810+0000","last_active":"2024-06-14T23:20:36.504998+0000","last_peered":"2024-06-14T23:20:36.504998+0000","last_clean":"2024-06-14T23:20:36.504998+0000","last_became_active":"2024-06-14T23:10:26.043455+0000","last_became_peered":"2024-06-14T23:10:26.043455+0000","last_unstale":"2024-06-14T23:20:36.504998+0000","last_undegraded":"2024-06-14T23:20:36.504998+0000","last_fullsized":"2024-06-14T23:20:36.504998+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:30:56.548754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":292,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504352+0000","last_change":"2024-06-14T23:20:19.493842+0000","last_active":"2024-06-14T23:20:36.504352+0000","last_peered":"2024-06-14T23:20:36.504352+0000","last_clean":"2024-06-14T23:20:36.504352+0000","last_became_active":"2024-06-14T23:20:19.493526+0000","last_became_peered":"2024-06-14T23:20:19.493526+0000","last_unstale":"2024-06-14T23:20:36.504352+0000","last_undegraded":"2024-06-14T23:20:36.504352+0000","last_fullsized":"2024-06-14T23:20:36.504352+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:53:09.616716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":424,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507115+0000","last_change":"2024-06-14T23:20:36.534675+0000","last_active":"2024-06-14T23:20:37.507115+0000","last_peered":"2024-06-14T23:20:37.507115+0000","last_clean":"2024-06-14T23:20:37.507115+0000","last_became_active":"2024-06-14T23:20:36.533578+0000","last_became_peered":"2024-06-14T23:20:36.533578+0000","last_unstale":"2024-06-14T23:20:37.507115+0000","last_undegraded":"2024-06-14T23:20:37.507115+0000","last_fullsized":"2024-06-14T23:20:37.507115+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:43.479469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:43.479469+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"443'8519","reported_seq":8559,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-14T23:20:57.179890+0000","last_change":"2024-06-14T23:18:47.964784+0000","last_active":"2024-06-14T23:20:57.179890+0000","last_peered":"2024-06-14T23:20:57.179890+0000","last_clean":"2024-06-14T23:20:57.179890+0000","last_became_active":"2024-06-14T23:18:47.964479+0000","last_became_peered":"2024-06-14T23:18:47.964479+0000","last_unstale":"2024-06-14T23:20:57.179890+0000","last_undegraded":"2024-06-14T23:20:57.179890+0000","last_fullsized":"2024-06-14T23:20:57.179890+0000","mapping_epoch":411,"log_start":"425'3724","ondisk_log_start":"425'3724","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:57.833878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345178112,"num_objects":5267,"num_object_clones":0,"num_object_copies":15801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":441152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":440,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501397+0000","last_change":"2024-06-14T23:20:19.500890+0000","last_active":"2024-06-14T23:20:36.501397+0000","last_peered":"2024-06-14T23:20:36.501397+0000","last_clean":"2024-06-14T23:20:36.501397+0000","last_became_active":"2024-06-14T23:20:19.500495+0000","last_became_peered":"2024-06-14T23:20:19.500495+0000","last_unstale":"2024-06-14T23:20:36.501397+0000","last_undegraded":"2024-06-14T23:20:36.501397+0000","last_fullsized":"2024-06-14T23:20:36.501397+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:28:29.510210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507017+0000","last_change":"2024-06-14T23:20:33.518396+0000","last_active":"2024-06-14T23:20:37.507017+0000","last_peered":"2024-06-14T23:20:37.507017+0000","last_clean":"2024-06-14T23:20:37.507017+0000","last_became_active":"2024-06-14T23:20:33.518120+0000","last_became_peered":"2024-06-14T23:20:33.518120+0000","last_unstale":"2024-06-14T23:20:37.507017+0000","last_undegraded":"2024-06-14T23:20:37.507017+0000","last_fullsized":"2024-06-14T23:20:37.507017+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:14:01.291968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":765,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507177+0000","last_change":"2024-06-14T23:20:36.541486+0000","last_active":"2024-06-14T23:20:37.507177+0000","last_peered":"2024-06-14T23:20:37.507177+0000","last_clean":"2024-06-14T23:20:37.507177+0000","last_became_active":"2024-06-14T23:20:36.541051+0000","last_became_peered":"2024-06-14T23:20:36.541051+0000","last_unstale":"2024-06-14T23:20:37.507177+0000","last_undegraded":"2024-06-14T23:20:37.507177+0000","last_fullsized":"2024-06-14T23:20:37.507177+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:47.375725+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.7","version":"443'8582","reported_seq":8622,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-06-14T23:20:58.875848+0000","last_change":"2024-06-14T23:18:48.032106+0000","last_active":"2024-06-14T23:20:58.875848+0000","last_peered":"2024-06-14T23:20:58.875848+0000","last_clean":"2024-06-14T23:20:58.875848+0000","last_became_active":"2024-06-14T23:18:48.031823+0000","last_became_peered":"2024-06-14T23:18:48.031823+0000","last_unstale":"2024-06-14T23:20:58.875848+0000","last_undegraded":"2024-06-14T23:20:58.875848+0000","last_fullsized":"2024-06-14T23:20:58.875848+0000","mapping_epoch":411,"log_start":"425'3566","ondisk_log_start":"425'3566","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:11:19.113099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341311488,"num_objects":5208,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":441280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507354+0000","last_change":"2024-06-14T23:20:33.516727+0000","last_active":"2024-06-14T23:20:37.507354+0000","last_peered":"2024-06-14T23:20:37.507354+0000","last_clean":"2024-06-14T23:20:37.507354+0000","last_became_active":"2024-06-14T23:20:33.516412+0000","last_became_peered":"2024-06-14T23:20:33.516412+0000","last_unstale":"2024-06-14T23:20:37.507354+0000","last_undegraded":"2024-06-14T23:20:37.507354+0000","last_fullsized":"2024-06-14T23:20:37.507354+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:21:08.956274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":295,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502510+0000","last_change":"2024-06-14T23:15:09.037746+0000","last_active":"2024-06-14T23:20:36.502510+0000","last_peered":"2024-06-14T23:20:36.502510+0000","last_clean":"2024-06-14T23:20:36.502510+0000","last_became_active":"2024-06-14T23:15:09.037427+0000","last_became_peered":"2024-06-14T23:15:09.037427+0000","last_unstale":"2024-06-14T23:20:36.502510+0000","last_undegraded":"2024-06-14T23:20:36.502510+0000","last_fullsized":"2024-06-14T23:20:36.502510+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:40:34.511973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":488,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.506987+0000","last_change":"2024-06-14T23:20:36.541861+0000","last_active":"2024-06-14T23:20:37.506987+0000","last_peered":"2024-06-14T23:20:37.506987+0000","last_clean":"2024-06-14T23:20:37.506987+0000","last_became_active":"2024-06-14T23:20:36.540619+0000","last_became_peered":"2024-06-14T23:20:36.540619+0000","last_unstale":"2024-06-14T23:20:37.506987+0000","last_undegraded":"2024-06-14T23:20:37.506987+0000","last_fullsized":"2024-06-14T23:20:37.506987+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:44.494532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:44.494532+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.6","version":"443'8429","reported_seq":11560,"reported_epoch":443,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-06-14T23:21:00.056515+0000","last_change":"2024-06-14T23:20:34.658902+0000","last_active":"2024-06-14T23:21:00.056515+0000","last_peered":"2024-06-14T23:21:00.056515+0000","last_clean":"2024-06-14T23:19:09.855587+0000","last_became_active":"2024-06-14T23:20:34.648564+0000","last_became_peered":"2024-06-14T23:20:34.648564+0000","last_unstale":"2024-06-14T23:21:00.056515+0000","last_undegraded":"2024-06-14T23:20:34.629858+0000","last_fullsized":"2024-06-14T23:21:00.056515+0000","mapping_epoch":439,"log_start":"425'3295","ondisk_log_start":"425'3295","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5134,"log_dups_size":0,"ondisk_log_size":5134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:07:31.602084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330498048,"num_objects":5043,"num_object_clones":0,"num_object_copies":15129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":633,"num_objects_misplaced":4410,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8429,"num_write_kb":431104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1537,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3,6],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":477},{"shards":"0,3,6","objects":4566}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":333,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.503235+0000","last_change":"2024-06-14T23:19:23.895155+0000","last_active":"2024-06-14T23:20:36.503235+0000","last_peered":"2024-06-14T23:20:36.503235+0000","last_clean":"2024-06-14T23:20:36.503235+0000","last_became_active":"2024-06-14T23:19:23.894798+0000","last_became_peered":"2024-06-14T23:19:23.894798+0000","last_unstale":"2024-06-14T23:20:36.503235+0000","last_undegraded":"2024-06-14T23:20:36.503235+0000","last_fullsized":"2024-06-14T23:20:36.503235+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:42:35.298855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":24,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.506006+0000","last_change":"2024-06-14T23:20:19.499960+0000","last_active":"2024-06-14T23:20:36.506006+0000","last_peered":"2024-06-14T23:20:36.506006+0000","last_clean":"2024-06-14T23:20:36.506006+0000","last_became_active":"2024-06-14T23:20:19.494657+0000","last_became_peered":"2024-06-14T23:20:19.494657+0000","last_unstale":"2024-06-14T23:20:36.506006+0000","last_undegraded":"2024-06-14T23:20:36.506006+0000","last_fullsized":"2024-06-14T23:20:36.506006+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:00:57.983055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":482,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507050+0000","last_change":"2024-06-14T23:20:36.538824+0000","last_active":"2024-06-14T23:20:37.507050+0000","last_peered":"2024-06-14T23:20:37.507050+0000","last_clean":"2024-06-14T23:20:37.507050+0000","last_became_active":"2024-06-14T23:20:36.535896+0000","last_became_peered":"2024-06-14T23:20:36.535896+0000","last_unstale":"2024-06-14T23:20:37.507050+0000","last_undegraded":"2024-06-14T23:20:37.507050+0000","last_fullsized":"2024-06-14T23:20:37.507050+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:14:14.052631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:14:14.052631+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":10,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"443'8372","reported_seq":12936,"reported_epoch":443,"state":"active+remapped+backfill_wait","last_fresh":"2024-06-14T23:20:57.211753+0000","last_change":"2024-06-14T23:20:33.624019+0000","last_active":"2024-06-14T23:20:57.211753+0000","last_peered":"2024-06-14T23:20:57.211753+0000","last_clean":"2024-06-14T23:20:18.465446+0000","last_became_active":"2024-06-14T23:20:33.589761+0000","last_became_peered":"2024-06-14T23:20:33.589761+0000","last_unstale":"2024-06-14T23:20:57.211753+0000","last_undegraded":"2024-06-14T23:20:57.211753+0000","last_fullsized":"2024-06-14T23:20:57.211753+0000","mapping_epoch":438,"log_start":"425'3523","ondisk_log_start":"425'3523","created":411,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:35:35.594124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337379328,"num_objects":5148,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5148,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8372,"num_write_kb":432640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":146997248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[6,4,3],"avail_no_missing":["6","3","4"],"object_location_counts":[{"shards":"3,4,6","objects":5148}],"blocked_by":[],"up_primary":2,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":240,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.505181+0000","last_change":"2024-06-14T23:20:19.497061+0000","last_active":"2024-06-14T23:20:36.505181+0000","last_peered":"2024-06-14T23:20:36.505181+0000","last_clean":"2024-06-14T23:20:36.505181+0000","last_became_active":"2024-06-14T23:20:19.496795+0000","last_became_peered":"2024-06-14T23:20:19.496795+0000","last_unstale":"2024-06-14T23:20:36.505181+0000","last_undegraded":"2024-06-14T23:20:36.505181+0000","last_fullsized":"2024-06-14T23:20:36.505181+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:37:41.524322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":267,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.503076+0000","last_change":"2024-06-14T23:20:19.518619+0000","last_active":"2024-06-14T23:20:36.503076+0000","last_peered":"2024-06-14T23:20:36.503076+0000","last_clean":"2024-06-14T23:20:36.503076+0000","last_became_active":"2024-06-14T23:20:19.517810+0000","last_became_peered":"2024-06-14T23:20:19.517810+0000","last_unstale":"2024-06-14T23:20:36.503076+0000","last_undegraded":"2024-06-14T23:20:36.503076+0000","last_fullsized":"2024-06-14T23:20:36.503076+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:37:50.948702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":564,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.505193+0000","last_change":"2024-06-14T23:15:12.943908+0000","last_active":"2024-06-14T23:20:36.505193+0000","last_peered":"2024-06-14T23:20:36.505193+0000","last_clean":"2024-06-14T23:20:36.505193+0000","last_became_active":"2024-06-14T23:15:09.031629+0000","last_became_peered":"2024-06-14T23:15:09.031629+0000","last_unstale":"2024-06-14T23:20:36.505193+0000","last_undegraded":"2024-06-14T23:20:36.505193+0000","last_fullsized":"2024-06-14T23:20:36.505193+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:12.943827+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:15:12.943827+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:48:02.155745+0000","scrub_duration":0.0029576450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.0","version":"443'8523","reported_seq":12169,"reported_epoch":443,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-06-14T23:21:00.053150+0000","last_change":"2024-06-14T23:20:34.657021+0000","last_active":"2024-06-14T23:21:00.053150+0000","last_peered":"2024-06-14T23:21:00.053150+0000","last_clean":"2024-06-14T23:19:09.847157+0000","last_became_active":"2024-06-14T23:20:34.646901+0000","last_became_peered":"2024-06-14T23:20:34.646901+0000","last_unstale":"2024-06-14T23:21:00.053150+0000","last_undegraded":"2024-06-14T23:20:34.636390+0000","last_fullsized":"2024-06-14T23:21:00.053150+0000","mapping_epoch":439,"log_start":"425'3420","ondisk_log_start":"425'3420","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5103,"log_dups_size":0,"ondisk_log_size":5103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:35:11.146578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323682304,"num_objects":4939,"num_object_clones":0,"num_object_copies":14817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":455,"num_objects_misplaced":4484,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8523,"num_write_kb":430784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1794,"num_bytes_recovered":117571584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[0,5,6],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":455},{"shards":"0,5,6","objects":4484}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":394,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502359+0000","last_change":"2024-06-14T23:10:26.040257+0000","last_active":"2024-06-14T23:20:36.502359+0000","last_peered":"2024-06-14T23:20:36.502359+0000","last_clean":"2024-06-14T23:20:36.502359+0000","last_became_active":"2024-06-14T23:10:26.039591+0000","last_became_peered":"2024-06-14T23:10:26.039591+0000","last_unstale":"2024-06-14T23:20:36.502359+0000","last_undegraded":"2024-06-14T23:20:36.502359+0000","last_fullsized":"2024-06-14T23:20:36.502359+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:07:21.630340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":43,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502129+0000","last_change":"2024-06-14T23:19:23.476782+0000","last_active":"2024-06-14T23:20:36.502129+0000","last_peered":"2024-06-14T23:20:36.502129+0000","last_clean":"2024-06-14T23:20:36.502129+0000","last_became_active":"2024-06-14T23:19:23.476292+0000","last_became_peered":"2024-06-14T23:19:23.476292+0000","last_unstale":"2024-06-14T23:20:36.502129+0000","last_undegraded":"2024-06-14T23:20:36.502129+0000","last_fullsized":"2024-06-14T23:20:36.502129+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:42:11.010204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":722,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501676+0000","last_change":"2024-06-14T23:09:52.265802+0000","last_active":"2024-06-14T23:20:36.501676+0000","last_peered":"2024-06-14T23:20:36.501676+0000","last_clean":"2024-06-14T23:20:36.501676+0000","last_became_active":"2024-06-14T23:07:59.003283+0000","last_became_peered":"2024-06-14T23:07:59.003283+0000","last_unstale":"2024-06-14T23:20:36.501676+0000","last_undegraded":"2024-06-14T23:20:36.501676+0000","last_fullsized":"2024-06-14T23:20:36.501676+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:53:15.920555+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"443'8497","reported_seq":17494,"reported_epoch":443,"state":"active+remapped+backfilling","last_fresh":"2024-06-14T23:21:00.060197+0000","last_change":"2024-06-14T23:20:33.598210+0000","last_active":"2024-06-14T23:21:00.060197+0000","last_peered":"2024-06-14T23:21:00.060197+0000","last_clean":"2024-06-14T23:20:18.458235+0000","last_became_active":"2024-06-14T23:20:20.598977+0000","last_became_peered":"2024-06-14T23:20:20.598977+0000","last_unstale":"2024-06-14T23:21:00.060197+0000","last_undegraded":"2024-06-14T23:21:00.060197+0000","last_fullsized":"2024-06-14T23:21:00.060197+0000","mapping_epoch":433,"log_start":"425'3402","ondisk_log_start":"425'3402","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:33:04.833983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323289088,"num_objects":4933,"num_object_clones":0,"num_object_copies":14799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1113,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8497,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4468,"num_bytes_recovered":292814848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":4933}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":313,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499655+0000","last_change":"2024-06-14T23:20:19.517928+0000","last_active":"2024-06-14T23:20:36.499655+0000","last_peered":"2024-06-14T23:20:36.499655+0000","last_clean":"2024-06-14T23:20:36.499655+0000","last_became_active":"2024-06-14T23:20:19.517637+0000","last_became_peered":"2024-06-14T23:20:19.517637+0000","last_unstale":"2024-06-14T23:20:36.499655+0000","last_undegraded":"2024-06-14T23:20:36.499655+0000","last_fullsized":"2024-06-14T23:20:36.499655+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:58:46.848855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":422,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.505607+0000","last_change":"2024-06-14T23:17:22.916573+0000","last_active":"2024-06-14T23:20:36.505607+0000","last_peered":"2024-06-14T23:20:36.505607+0000","last_clean":"2024-06-14T23:20:36.505607+0000","last_became_active":"2024-06-14T23:17:22.897253+0000","last_became_peered":"2024-06-14T23:17:22.897253+0000","last_unstale":"2024-06-14T23:20:36.505607+0000","last_undegraded":"2024-06-14T23:20:36.505607+0000","last_fullsized":"2024-06-14T23:20:36.505607+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:33:59.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":660,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502926+0000","last_change":"2024-06-14T23:20:22.522823+0000","last_active":"2024-06-14T23:20:36.502926+0000","last_peered":"2024-06-14T23:20:36.502926+0000","last_clean":"2024-06-14T23:20:36.502926+0000","last_became_active":"2024-06-14T23:20:22.522002+0000","last_became_peered":"2024-06-14T23:20:22.522002+0000","last_unstale":"2024-06-14T23:20:36.502926+0000","last_undegraded":"2024-06-14T23:20:36.502926+0000","last_fullsized":"2024-06-14T23:20:36.502926+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:10:06.551486+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:10:06.551486+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019682448000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.2","version":"443'8388","reported_seq":13096,"reported_epoch":443,"state":"active+remapped","last_fresh":"2024-06-14T23:20:57.207396+0000","last_change":"2024-06-14T23:20:33.550727+0000","last_active":"2024-06-14T23:20:57.207396+0000","last_peered":"2024-06-14T23:20:57.207396+0000","last_clean":"2024-06-14T23:20:18.468084+0000","last_became_active":"2024-06-14T23:20:20.562565+0000","last_became_peered":"2024-06-14T23:20:20.562565+0000","last_unstale":"2024-06-14T23:20:57.207396+0000","last_undegraded":"2024-06-14T23:20:57.207396+0000","last_fullsized":"2024-06-14T23:20:57.207396+0000","mapping_epoch":433,"log_start":"425'3531","ondisk_log_start":"425'3531","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T23:18:46.942049+0000","last_clean_scrub_stamp":"2024-06-14T23:18:46.942049+0000","objects_scrubbed":0,"log_size":4857,"log_dups_size":0,"ondisk_log_size":4857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:12:14.198184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":5184,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3355,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8388,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2323,"num_bytes_recovered":152240128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,3],"avail_no_missing":["6","3","5"],"object_location_counts":[{"shards":"3,5,6","objects":5184}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":256,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.502032+0000","last_change":"2024-06-14T23:19:23.479101+0000","last_active":"2024-06-14T23:20:36.502032+0000","last_peered":"2024-06-14T23:20:36.502032+0000","last_clean":"2024-06-14T23:20:36.502032+0000","last_became_active":"2024-06-14T23:19:23.478364+0000","last_became_peered":"2024-06-14T23:19:23.478364+0000","last_unstale":"2024-06-14T23:20:36.502032+0000","last_undegraded":"2024-06-14T23:20:36.502032+0000","last_fullsized":"2024-06-14T23:20:36.502032+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:57:30.628738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":435,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.505933+0000","last_change":"2024-06-14T23:17:22.904229+0000","last_active":"2024-06-14T23:20:36.505933+0000","last_peered":"2024-06-14T23:20:36.505933+0000","last_clean":"2024-06-14T23:20:36.505933+0000","last_became_active":"2024-06-14T23:17:22.896077+0000","last_became_peered":"2024-06-14T23:17:22.896077+0000","last_unstale":"2024-06-14T23:20:36.505933+0000","last_undegraded":"2024-06-14T23:20:36.505933+0000","last_fullsized":"2024-06-14T23:20:36.505933+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:05.630800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":808,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507323+0000","last_change":"2024-06-14T23:20:36.537083+0000","last_active":"2024-06-14T23:20:37.507323+0000","last_peered":"2024-06-14T23:20:37.507323+0000","last_clean":"2024-06-14T23:20:37.507323+0000","last_became_active":"2024-06-14T23:20:36.536771+0000","last_became_peered":"2024-06-14T23:20:36.536771+0000","last_unstale":"2024-06-14T23:20:37.507323+0000","last_undegraded":"2024-06-14T23:20:37.507323+0000","last_fullsized":"2024-06-14T23:20:37.507323+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.146838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.146838+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:59:24.797795+0000","scrub_duration":0.0040813639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":449,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501516+0000","last_change":"2024-06-14T23:17:22.910264+0000","last_active":"2024-06-14T23:20:36.501516+0000","last_peered":"2024-06-14T23:20:36.501516+0000","last_clean":"2024-06-14T23:20:36.501516+0000","last_became_active":"2024-06-14T23:17:22.900551+0000","last_became_peered":"2024-06-14T23:17:22.900551+0000","last_unstale":"2024-06-14T23:20:36.501516+0000","last_undegraded":"2024-06-14T23:20:36.501516+0000","last_fullsized":"2024-06-14T23:20:36.501516+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:39:01.672829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507427+0000","last_change":"2024-06-14T23:20:33.527187+0000","last_active":"2024-06-14T23:20:37.507427+0000","last_peered":"2024-06-14T23:20:37.507427+0000","last_clean":"2024-06-14T23:20:37.507427+0000","last_became_active":"2024-06-14T23:20:33.526950+0000","last_became_peered":"2024-06-14T23:20:33.526950+0000","last_unstale":"2024-06-14T23:20:37.507427+0000","last_undegraded":"2024-06-14T23:20:37.507427+0000","last_fullsized":"2024-06-14T23:20:37.507427+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:26:03.584991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":593,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499953+0000","last_change":"2024-06-14T23:19:23.473442+0000","last_active":"2024-06-14T23:20:36.499953+0000","last_peered":"2024-06-14T23:20:36.499953+0000","last_clean":"2024-06-14T23:20:36.499953+0000","last_became_active":"2024-06-14T23:19:23.472785+0000","last_became_peered":"2024-06-14T23:19:23.472785+0000","last_unstale":"2024-06-14T23:20:36.499953+0000","last_undegraded":"2024-06-14T23:20:36.499953+0000","last_fullsized":"2024-06-14T23:20:36.499953+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:07:16.093566+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001647253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":244,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.499853+0000","last_change":"2024-06-14T23:15:09.039681+0000","last_active":"2024-06-14T23:20:36.499853+0000","last_peered":"2024-06-14T23:20:36.499853+0000","last_clean":"2024-06-14T23:20:36.499853+0000","last_became_active":"2024-06-14T23:15:09.039180+0000","last_became_peered":"2024-06-14T23:15:09.039180+0000","last_unstale":"2024-06-14T23:20:36.499853+0000","last_undegraded":"2024-06-14T23:20:36.499853+0000","last_fullsized":"2024-06-14T23:20:36.499853+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:54.312788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507368+0000","last_change":"2024-06-14T23:20:33.520097+0000","last_active":"2024-06-14T23:20:37.507368+0000","last_peered":"2024-06-14T23:20:37.507368+0000","last_clean":"2024-06-14T23:20:37.507368+0000","last_became_active":"2024-06-14T23:20:33.519495+0000","last_became_peered":"2024-06-14T23:20:33.519495+0000","last_unstale":"2024-06-14T23:20:37.507368+0000","last_undegraded":"2024-06-14T23:20:37.507368+0000","last_fullsized":"2024-06-14T23:20:37.507368+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:21:40.919827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":491,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501026+0000","last_change":"2024-06-14T23:20:22.507021+0000","last_active":"2024-06-14T23:20:36.501026+0000","last_peered":"2024-06-14T23:20:36.501026+0000","last_clean":"2024-06-14T23:20:36.501026+0000","last_became_active":"2024-06-14T23:20:22.506745+0000","last_became_peered":"2024-06-14T23:20:22.506745+0000","last_unstale":"2024-06-14T23:20:36.501026+0000","last_undegraded":"2024-06-14T23:20:36.501026+0000","last_fullsized":"2024-06-14T23:20:36.501026+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":28,"num_write":7,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507131+0000","last_change":"2024-06-14T23:20:33.517267+0000","last_active":"2024-06-14T23:20:37.507131+0000","last_peered":"2024-06-14T23:20:37.507131+0000","last_clean":"2024-06-14T23:20:37.507131+0000","last_became_active":"2024-06-14T23:20:33.516735+0000","last_became_peered":"2024-06-14T23:20:33.516735+0000","last_unstale":"2024-06-14T23:20:37.507131+0000","last_undegraded":"2024-06-14T23:20:37.507131+0000","last_fullsized":"2024-06-14T23:20:37.507131+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:31:34.979464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":121,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.504641+0000","last_change":"2024-06-14T23:19:23.471655+0000","last_active":"2024-06-14T23:20:36.504641+0000","last_peered":"2024-06-14T23:20:36.504641+0000","last_clean":"2024-06-14T23:20:36.504641+0000","last_became_active":"2024-06-14T23:19:23.470993+0000","last_became_peered":"2024-06-14T23:19:23.470993+0000","last_unstale":"2024-06-14T23:20:36.504641+0000","last_undegraded":"2024-06-14T23:20:36.504641+0000","last_fullsized":"2024-06-14T23:20:36.504641+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:57:59.933536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":431,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.501604+0000","last_change":"2024-06-14T23:19:23.479406+0000","last_active":"2024-06-14T23:20:36.501604+0000","last_peered":"2024-06-14T23:20:36.501604+0000","last_clean":"2024-06-14T23:20:36.501604+0000","last_became_active":"2024-06-14T23:19:23.478774+0000","last_became_peered":"2024-06-14T23:19:23.478774+0000","last_unstale":"2024-06-14T23:20:36.501604+0000","last_undegraded":"2024-06-14T23:20:36.501604+0000","last_fullsized":"2024-06-14T23:20:36.501604+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:14.072476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:14.072476+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":41,"num_write":11,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":487,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:36.506157+0000","last_change":"2024-06-14T23:20:33.536774+0000","last_active":"2024-06-14T23:20:36.506157+0000","last_peered":"2024-06-14T23:20:36.506157+0000","last_clean":"2024-06-14T23:20:36.506157+0000","last_became_active":"2024-06-14T23:20:33.515459+0000","last_became_peered":"2024-06-14T23:20:33.515459+0000","last_unstale":"2024-06-14T23:20:36.506157+0000","last_undegraded":"2024-06-14T23:20:36.506157+0000","last_fullsized":"2024-06-14T23:20:36.506157+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:39.163824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507266+0000","last_change":"2024-06-14T23:20:33.518884+0000","last_active":"2024-06-14T23:20:37.507266+0000","last_peered":"2024-06-14T23:20:37.507266+0000","last_clean":"2024-06-14T23:20:37.507266+0000","last_became_active":"2024-06-14T23:20:33.518444+0000","last_became_peered":"2024-06-14T23:20:33.518444+0000","last_unstale":"2024-06-14T23:20:37.507266+0000","last_undegraded":"2024-06-14T23:20:37.507266+0000","last_fullsized":"2024-06-14T23:20:37.507266+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:59.847306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":881,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-06-14T23:20:37.507083+0000","last_change":"2024-06-14T23:20:36.538374+0000","last_active":"2024-06-14T23:20:37.507083+0000","last_peered":"2024-06-14T23:20:37.507083+0000","last_clean":"2024-06-14T23:20:37.507083+0000","last_became_active":"2024-06-14T23:20:36.535319+0000","last_became_peered":"2024-06-14T23:20:36.535319+0000","last_unstale":"2024-06-14T23:20:37.507083+0000","last_undegraded":"2024-06-14T23:20:37.507083+0000","last_fullsized":"2024-06-14T23:20:37.507083+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:28:20.993479+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1912,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22040,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":14,"num_pg":16,"stat_sum":{"num_bytes":5310119960,"num_objects":81027,"num_object_clones":0,"num_object_copies":243081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":41995,"num_objects_unfound":0,"num_objects_dirty":81027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136549,"num_write_kb":6962369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23932,"num_bytes_recovered":1568014336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16904892416,"data_stored":16904880200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82665,"ondisk_log_size":82665,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":705710,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":261,"num_write":68,"num_write_kb":693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":837542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234563,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2808668,"kb_used_data":1309972,"kb_used_omap":1,"kb_used_meta":1498686,"kb_avail":90936484,"statfs":{"total":95995035648,"available":93118959616,"internally_reserved":0,"allocated":1341411328,"data_stored":1339116551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1534654808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.0409999999999999,"15min":0.96399999999999997},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.214},"max":{"1min":2.629,"5min":12.263999999999999,"15min":12.263999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.349,"5min":1.2230000000000001,"15min":1.1080000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.224},"max":{"1min":3.1459999999999999,"5min":5.6719999999999997,"15min":6.5149999999999997},"last":1.101}]},{"osd":1,"last update":"Fri Jun 14 23:20:27 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.2170000000000001,"15min":1.24},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.22600000000000001},"max":{"1min":3.98,"5min":5.7309999999999999,"15min":12.010999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.125,"15min":1.294},"min":{"1min":0.33000000000000002,"5min":0.27700000000000002,"15min":0.27500000000000002},"max":{"1min":4.8710000000000004,"5min":12.295,"15min":13.531000000000001},"last":0.36599999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.0069999999999999,"15min":1.2250000000000001},"min":{"1min":0.30399999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.6079999999999997,"5min":6.0819999999999999,"15min":10.539},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.099,"15min":1.2250000000000001},"min":{"1min":0.36399999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.6980000000000004,"5min":6.1529999999999996,"15min":10.56},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.99399999999999999,"15min":1.294},"min":{"1min":0.29999999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.5499999999999998,"5min":6.1130000000000004,"15min":20.645},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.022,"5min":1.1459999999999999,"15min":1.4450000000000001},"min":{"1min":0.313,"5min":0.29399999999999998,"15min":0.28299999999999997},"max":{"1min":2.7749999999999999,"5min":5.7080000000000002,"15min":22.042999999999999},"last":1.077}]},{"osd":4,"last update":"Fri Jun 14 23:20:38 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.655,"15min":1.224},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23100000000000001},"max":{"1min":4.2610000000000001,"5min":6.1539999999999999,"15min":6.1539999999999999},"last":1.165},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3799999999999999,"15min":1.1799999999999999},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":4.6699999999999999,"5min":6.2199999999999998,"15min":10.499000000000001},"last":0.83899999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:20:38 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.631,"15min":1.2170000000000001},"min":{"1min":0.42299999999999999,"5min":0.28799999999999998,"15min":0.23699999999999999},"max":{"1min":5.234,"5min":12.236000000000001,"15min":12.236000000000001},"last":1.028},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.9339999999999999,"15min":1.3919999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.24399999999999999},"max":{"1min":6.5090000000000003,"5min":12.842000000000001,"15min":12.842000000000001},"last":1.0489999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:20:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0819999999999999,"5min":1.8200000000000001,"15min":1.651},"min":{"1min":0.39400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.2779999999999996,"5min":12.317,"15min":12.317},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.655,"5min":1.7430000000000001,"15min":1.5960000000000001},"min":{"1min":0.35099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.9509999999999996,"5min":10.050000000000001,"15min":10.087999999999999},"last":1.1459999999999999}]}]},{"osd":6,"up_from":272,"seq":1168231104669,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4023564,"kb_used_data":3054468,"kb_used_omap":7,"kb_used_meta":969080,"kb_avail":89721588,"statfs":{"total":95995035648,"available":91874906112,"internally_reserved":0,"allocated":3127775232,"data_stored":3125495850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":992338362},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:20:20 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.337,"15min":1.298},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.8700000000000001,"5min":6.109,"15min":6.6440000000000001},"last":0.624},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.508,"15min":1.397},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.28899999999999998},"max":{"1min":6.0620000000000003,"5min":14.148999999999999,"15min":14.148999999999999},"last":0.64600000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:20:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.194,"15min":1.4419999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":3.9089999999999998,"5min":5.6710000000000003,"15min":8.173},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.425,"5min":1.478,"15min":1.5029999999999999},"min":{"1min":0.28100000000000003,"5min":0.248,"15min":0.248},"max":{"1min":5.9980000000000002,"5min":14.086,"15min":14.086},"last":0.33000000000000002}]},{"osd":2,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.2969999999999999,"15min":1.27},"min":{"1min":0.32700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":7.8019999999999996,"5min":7.8019999999999996,"15min":29.940999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.847,"5min":1.3939999999999999,"15min":1.359},"min":{"1min":0.38600000000000001,"5min":0.29099999999999998,"15min":0.28999999999999998},"max":{"1min":7.8609999999999998,"5min":8.0679999999999996,"15min":30.135999999999999},"last":0.442}]},{"osd":3,"last update":"Fri Jun 14 23:20:20 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.4399999999999999,"15min":1.421},"min":{"1min":0.33900000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.0339999999999998,"5min":6.0839999999999996,"15min":6.7089999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.3740000000000001,"15min":1.585},"min":{"1min":0.34699999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":7.8319999999999999,"5min":7.8319999999999999,"15min":8.4740000000000002},"last":0.68899999999999995}]},{"osd":4,"last update":"Fri Jun 14 23:20:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.5189999999999999,"15min":1.417},"min":{"1min":0.24399999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.9430000000000001,"5min":6.6299999999999999,"15min":11.010999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.974,"5min":1.9690000000000001,"15min":1.45},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":8.7550000000000008,"5min":14.199999999999999,"15min":14.199999999999999},"last":0.29899999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:20:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":2.5270000000000001,"15min":1.629},"min":{"1min":0.317,"5min":0.308,"15min":0.28199999999999997},"max":{"1min":6.0949999999999998,"5min":26.172000000000001,"15min":26.172000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":2.2149999999999999,"5min":2.2349999999999999,"15min":1.7110000000000001},"min":{"1min":0.29199999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":7.7249999999999996,"5min":25.946999999999999,"15min":25.946999999999999},"last":0.38300000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:20:20 2024","interfaces":[{"interface":"back","average":{"1min":2.0880000000000001,"5min":1.889,"15min":1.734},"min":{"1min":0.35499999999999998,"5min":0.29999999999999999,"15min":0.29399999999999998},"max":{"1min":7.7720000000000002,"5min":7.7720000000000002,"15min":10.016999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.986,"15min":1.6659999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":3.8639999999999999,"5min":10.906000000000001,"15min":10.906000000000001},"last":0.31}]}]},{"osd":5,"up_from":38,"seq":163208757838,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4309896,"kb_used_data":2826048,"kb_used_omap":1,"kb_used_meta":1483774,"kb_avail":89435256,"statfs":{"total":95995035648,"available":91581702144,"internally_reserved":0,"allocated":2893873152,"data_stored":2891594224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1519384920},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:20:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.5389999999999999,"15min":1.087},"min":{"1min":0.22,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":4.3090000000000002,"5min":8.3770000000000007,"15min":8.3770000000000007},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.859,"5min":1.5109999999999999,"15min":1.1319999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":6.0019999999999998,"5min":13.398999999999999,"15min":13.398999999999999},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:20:31 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.3480000000000001,"15min":1.0700000000000001},"min":{"1min":0.318,"5min":0.26500000000000001,"15min":0.245},"max":{"1min":4.7270000000000003,"5min":7.6219999999999999,"15min":7.6219999999999999},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.5760000000000001,"15min":1.165},"min":{"1min":0.28499999999999998,"5min":0.27100000000000002,"15min":0.22900000000000001},"max":{"1min":4.3799999999999999,"5min":8.4480000000000004,"15min":8.4480000000000004},"last":1.891}]},{"osd":2,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.5349999999999999,"15min":1.048},"min":{"1min":0.26800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":4.3550000000000004,"5min":8.3989999999999991,"15min":8.3989999999999991},"last":1.911},{"interface":"front","average":{"1min":1.008,"5min":1.5620000000000001,"15min":1.0649999999999999},"min":{"1min":0.439,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.4050000000000002,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":1.9299999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:20:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5800000000000001,"15min":1.331},"min":{"1min":0.26600000000000001,"5min":0.25,"15min":0.25},"max":{"1min":6.0270000000000001,"5min":13.352,"15min":21.815000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.7789999999999999,"15min":1.399},"min":{"1min":0.31,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":3.9100000000000001,"5min":8.4269999999999996,"15min":21.797000000000001},"last":1.8700000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.964,"5min":1.944,"15min":1.2390000000000001},"min":{"1min":0.27700000000000002,"5min":0.248,"15min":0.23599999999999999},"max":{"1min":8.3160000000000007,"5min":8.3160000000000007,"15min":8.3160000000000007},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.538,"5min":2.0670000000000002,"15min":1.357},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.253},"max":{"1min":6.2140000000000004,"5min":13.443,"15min":13.443},"last":0.91200000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:20:31 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.2549999999999999,"15min":1.6100000000000001},"min":{"1min":0.65300000000000002,"5min":0.33900000000000002,"15min":0.26100000000000001},"max":{"1min":5.5880000000000001,"5min":10.893000000000001,"15min":30.934000000000001},"last":3.4990000000000001},{"interface":"front","average":{"1min":2.7589999999999999,"5min":2.5529999999999999,"15min":1.8819999999999999},"min":{"1min":0.39900000000000002,"5min":0.28499999999999998,"15min":0.26100000000000001},"max":{"1min":6.1769999999999996,"5min":13.465999999999999,"15min":31.059999999999999},"last":0.81899999999999995}]},{"osd":7,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":2.181,"5min":2.202,"15min":1.6160000000000001},"min":{"1min":0.34899999999999998,"5min":0.32400000000000001,"15min":0.25},"max":{"1min":21.844000000000001,"5min":21.844000000000001,"15min":26.981999999999999},"last":2.2639999999999998},{"interface":"front","average":{"1min":2.6899999999999999,"5min":2.1379999999999999,"15min":1.6299999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.26400000000000001},"max":{"1min":20.067,"5min":20.067,"15min":23.326000000000001},"last":3.5310000000000001}]}]},{"osd":4,"up_from":30,"seq":128849019474,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4461012,"kb_used_data":2972876,"kb_used_omap":1,"kb_used_meta":1488126,"kb_avail":89284140,"statfs":{"total":95995035648,"available":91426959360,"internally_reserved":0,"allocated":3044225024,"data_stored":3041937584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1523841368},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:20:11 2024","interfaces":[{"interface":"back","average":{"1min":2.2000000000000002,"5min":1.3240000000000001,"15min":0.97899999999999998},"min":{"1min":0.23400000000000001,"5min":0.221,"15min":0.217},"max":{"1min":21.262,"5min":21.262,"15min":21.262},"last":3.0659999999999998},{"interface":"front","average":{"1min":2.2120000000000002,"5min":1.405,"15min":1.03},"min":{"1min":0.27200000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":21.491,"5min":21.491,"15min":21.491},"last":3.4500000000000002}]},{"osd":1,"last update":"Fri Jun 14 23:20:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.1950000000000001,"15min":1.141},"min":{"1min":0.313,"5min":0.29899999999999999,"15min":0.26400000000000001},"max":{"1min":6.0179999999999998,"5min":6.6790000000000003,"15min":42.656999999999996},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.1359999999999999,"15min":1.141},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.249},"max":{"1min":6.2169999999999996,"5min":6.2169999999999996,"15min":42.192999999999998},"last":0.69599999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.3160000000000001,"15min":0.99399999999999999},"min":{"1min":0.33500000000000002,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":6.0469999999999997,"5min":6.7110000000000003,"15min":6.7110000000000003},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.609,"5min":1.3560000000000001,"15min":1.0469999999999999},"min":{"1min":0.34799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.0750000000000002,"5min":6.6550000000000002,"15min":6.6550000000000002},"last":1.3899999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:20:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3400000000000001,"15min":1.2949999999999999},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.25},"max":{"1min":4.9690000000000003,"5min":5.5700000000000003,"15min":44.389000000000003},"last":3.367},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.2609999999999999,"15min":1.3440000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":6.1950000000000003,"5min":6.1950000000000003,"15min":46.101999999999997},"last":0.66800000000000004}]},{"osd":5,"last update":"Fri Jun 14 23:20:54 2024","interfaces":[{"interface":"back","average":{"1min":2.2970000000000002,"5min":2.1539999999999999,"15min":1.296},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.20300000000000001},"max":{"1min":6.3019999999999996,"5min":8.173,"15min":8.173},"last":3.427},{"interface":"front","average":{"1min":1.327,"5min":1.8440000000000001,"15min":1.2290000000000001},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.23300000000000001},"max":{"1min":3.77,"5min":7.7789999999999999,"15min":7.7789999999999999},"last":2.3140000000000001}]},{"osd":6,"last update":"Fri Jun 14 23:20:21 2024","interfaces":[{"interface":"back","average":{"1min":2.7490000000000001,"5min":1.9790000000000001,"15min":1.829},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25800000000000001},"max":{"1min":8.1669999999999998,"5min":8.1669999999999998,"15min":122.88500000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":2.8460000000000001,"5min":2.0990000000000002,"15min":1.9810000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":9.1929999999999996,"5min":9.1929999999999996,"15min":122.21299999999999},"last":3.5150000000000001}]},{"osd":7,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":2.2759999999999998,"15min":1.4590000000000001},"min":{"1min":0.378,"5min":0.255,"15min":0.255},"max":{"1min":6.0739999999999998,"5min":11.430999999999999,"15min":11.430999999999999},"last":1.369},{"interface":"front","average":{"1min":1.835,"5min":2.0150000000000001,"15min":1.401},"min":{"1min":0.38100000000000001,"5min":0.36299999999999999,"15min":0.27300000000000002},"max":{"1min":6.2850000000000001,"5min":7.2930000000000001,"15min":8.5340000000000007},"last":3.403}]}]},{"osd":3,"up_from":138,"seq":592705487173,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4097472,"kb_used_data":2890424,"kb_used_omap":4,"kb_used_meta":1207035,"kb_avail":89647680,"statfs":{"total":95995035648,"available":91799224320,"internally_reserved":0,"allocated":2959794176,"data_stored":2957522645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":1236004318},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.378,"15min":0.79800000000000004},"min":{"1min":0.188,"5min":0.186,"15min":0.16800000000000001},"max":{"1min":1.1419999999999999,"5min":1.361,"15min":25.190000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.50700000000000001,"15min":0.92300000000000004},"min":{"1min":0.215,"5min":0.214,"15min":0.17799999999999999},"max":{"1min":2.2349999999999999,"5min":2.399,"15min":25.652000000000001},"last":0.30199999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46400000000000002,"15min":0.95599999999999996},"min":{"1min":0.23799999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8540000000000001,"5min":2.375,"15min":25.754000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57299999999999995,"15min":1.131},"min":{"1min":0.28399999999999997,"5min":0.25900000000000001,"15min":0.249},"max":{"1min":2.181,"5min":3.956,"15min":27.992999999999999},"last":2.181}]},{"osd":2,"last update":"Fri Jun 14 23:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.53600000000000003,"15min":0.72499999999999998},"min":{"1min":0.24199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":1.6819999999999999,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":0.246},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.503,"15min":0.63700000000000001},"min":{"1min":0.28199999999999997,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":1.0009999999999999,"5min":2.2909999999999999,"15min":3.5550000000000002},"last":0.28299999999999997}]},{"osd":4,"last update":"Fri Jun 14 23:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.1499999999999999,"15min":1.645},"min":{"1min":0.28999999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.5359999999999996,"5min":6.5110000000000001,"15min":73.263000000000005},"last":5.5359999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":1.135,"15min":1.5860000000000001},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.27300000000000002},"max":{"1min":2.1579999999999999,"5min":5.7130000000000001,"15min":73.5},"last":0.32000000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.365,"15min":1.708},"min":{"1min":0.32500000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.6459999999999999,"5min":12.718999999999999,"15min":81.438999999999993},"last":1.8120000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.48,"15min":1.7470000000000001},"min":{"1min":0.34599999999999997,"5min":0.318,"15min":0.29299999999999998},"max":{"1min":4.8310000000000004,"5min":12.706,"15min":81.626999999999995},"last":0.53500000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":1.675,"15min":1.635},"min":{"1min":0.379,"5min":0.35899999999999999,"15min":0.34599999999999997},"max":{"1min":5.907,"5min":15.458,"15min":25.869},"last":0.55300000000000005},{"interface":"front","average":{"1min":2.2519999999999998,"5min":1.5609999999999999,"15min":1.5149999999999999},"min":{"1min":0.32000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.9309999999999992,"5min":15.699,"15min":25.143000000000001},"last":0.34300000000000003}]},{"osd":7,"last update":"Fri Jun 14 23:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.107,"15min":1.401},"min":{"1min":0.41299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.4769999999999999,"5min":6.085,"15min":44.887999999999998},"last":2.016},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1479999999999999,"15min":1.462},"min":{"1min":0.41499999999999998,"5min":0.38600000000000001,"15min":0.33800000000000002},"max":{"1min":2.657,"5min":4.1920000000000002,"15min":45.835000000000001},"last":2.1680000000000001}]}]},{"osd":2,"up_from":322,"seq":1382979469428,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1051520,"kb_used_data":597688,"kb_used_omap":10,"kb_used_meta":453813,"kb_avail":92693632,"statfs":{"total":95995035648,"available":94918279168,"internally_reserved":0,"allocated":612032512,"data_stored":609761195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":464705062},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.504,"15min":0.42399999999999999},"min":{"1min":0.192,"5min":0.191,"15min":0.191},"max":{"1min":5.423,"5min":5.423,"15min":6.1319999999999997},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.54700000000000004,"15min":0.495},"min":{"1min":0.24199999999999999,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":5.3869999999999996,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.44700000000000001,"15min":0.435},"min":{"1min":0.21299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.8839999999999999,"5min":4.04,"15min":4.04},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52900000000000003,"15min":0.504},"min":{"1min":0.23499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":1.478,"5min":6.9889999999999999,"15min":6.9889999999999999},"last":0.47099999999999997}]},{"osd":3,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.52700000000000002,"15min":0.58899999999999997},"min":{"1min":0.26200000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.319,"5min":3.319,"15min":3.7519999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.53600000000000003,"15min":0.56200000000000006},"min":{"1min":0.26300000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.917,"5min":4.1379999999999999,"15min":5.5890000000000004},"last":0.45600000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.1870000000000001,"15min":1.0580000000000001},"min":{"1min":0.32300000000000001,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":4.1820000000000004,"5min":6.9820000000000002,"15min":6.9820000000000002},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":1.113,"15min":0.88900000000000001},"min":{"1min":0.30599999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.1299999999999999,"5min":5.0759999999999996,"15min":5.0759999999999996},"last":0.69399999999999995}]},{"osd":5,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.708,"15min":1.1160000000000001},"min":{"1min":0.318,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.0819999999999999,"5min":54.969999999999999,"15min":54.969999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.855,"15min":1.2190000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.2519999999999998,"5min":57.284999999999997,"15min":57.284999999999997},"last":0.49199999999999999}]},{"osd":6,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1990000000000001,"15min":1.0109999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.468,"5min":5.3380000000000001,"15min":6.6989999999999998},"last":0.753},{"interface":"front","average":{"1min":1.006,"5min":1.1859999999999999,"15min":1.0549999999999999},"min":{"1min":0.41399999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.3540000000000001,"5min":4.3970000000000002,"15min":4.6950000000000003},"last":1.5669999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.86499999999999999,"15min":0.95199999999999996},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.071,"5min":5,"15min":5},"last":1.405},{"interface":"front","average":{"1min":0.628,"5min":1.675,"15min":1.343},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1919999999999999,"5min":49.552999999999997,"15min":49.552999999999997},"last":0.52700000000000002}]}]},{"osd":1,"up_from":0,"seq":498216206717,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":91,"seq":390842024372,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4114752,"kb_used_data":3007736,"kb_used_omap":5,"kb_used_meta":1106874,"kb_avail":89630400,"statfs":{"total":95995035648,"available":91781529600,"internally_reserved":0,"allocated":3079921664,"data_stored":3077648586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":1133439066},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35699999999999998,"15min":0.84599999999999997},"min":{"1min":0.191,"5min":0.191,"15min":0.17899999999999999},"max":{"1min":1.4990000000000001,"5min":2.5099999999999998,"15min":34.448},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.41599999999999998,"15min":0.86899999999999999},"min":{"1min":0.25900000000000001,"5min":0.19500000000000001,"15min":0.16900000000000001},"max":{"1min":4.3769999999999998,"5min":4.3769999999999998,"15min":34.354999999999997},"last":0.50900000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:20:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.40699999999999997,"15min":0.54100000000000004},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.70099999999999996,"5min":2.6549999999999998,"15min":4.0659999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.46999999999999997,"15min":0.50700000000000001},"min":{"1min":0.26700000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.544,"5min":2.657,"15min":3.2509999999999999},"last":0.23100000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.48699999999999999,"15min":0.99299999999999999},"min":{"1min":0.253,"5min":0.22,"15min":0.21199999999999999},"max":{"1min":4.4199999999999999,"5min":4.4199999999999999,"15min":35.094999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.497,"15min":1.093},"min":{"1min":0.26000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":1.9370000000000001,"5min":2.6059999999999999,"15min":38.444000000000003},"last":0.26100000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:20:05 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.29,"15min":1.147},"min":{"1min":0.41199999999999998,"5min":0.32200000000000001,"15min":0.27000000000000002},"max":{"1min":3.4550000000000001,"5min":6.556,"15min":24.170999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.248,"5min":1.2769999999999999,"15min":1.125},"min":{"1min":0.314,"5min":0.29299999999999998,"15min":0.27600000000000002},"max":{"1min":4.5540000000000003,"5min":5.0439999999999996,"15min":24.204000000000001},"last":1.321}]},{"osd":5,"last update":"Fri Jun 14 23:20:05 2024","interfaces":[{"interface":"back","average":{"1min":2.7970000000000002,"5min":1.9930000000000001,"15min":1.268},"min":{"1min":0.35799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":36.189,"5min":45.274999999999999,"15min":45.274999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":2.6509999999999998,"5min":1.9790000000000001,"15min":1.2310000000000001},"min":{"1min":0.33200000000000002,"5min":0.315,"15min":0.315},"max":{"1min":36.319000000000003,"5min":45.241999999999997,"15min":45.241999999999997},"last":0.81299999999999994}]},{"osd":6,"last update":"Fri Jun 14 23:20:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5270000000000001,"5min":1.673,"15min":1.454},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.33200000000000002},"max":{"1min":36.478000000000002,"5min":36.478000000000002,"15min":36.478000000000002},"last":1.7529999999999999},{"interface":"front","average":{"1min":3.7440000000000002,"5min":1.766,"15min":1.5940000000000001},"min":{"1min":0.39900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":36.494,"5min":36.494,"15min":36.494},"last":1.744}]},{"osd":7,"last update":"Fri Jun 14 23:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.1140000000000001,"15min":1.1719999999999999},"min":{"1min":0.372,"5min":0.35999999999999999,"15min":0.35499999999999998},"max":{"1min":2.4369999999999998,"5min":5.8879999999999999,"15min":24.131},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.161,"15min":1.254},"min":{"1min":0.44400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.786,"5min":5.8289999999999997,"15min":24.257000000000001},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3066888192,"data_stored":3066888192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":606142464,"data_stored":606142464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2921136128,"data_stored":2921136128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3006730240,"data_stored":3006726168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2867597312,"data_stored":2867593240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3111129088,"data_stored":3111125016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1325268992,"data_stored":1325268992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:21:09.395 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-14T23:21:09.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set unique_pool_9 pg_num 26 2024-06-14T23:21:09.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:09.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:09 smithi137 ceph-mon[90634]: 1.13 scrub starts 2024-06-14T23:21:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:09 smithi137 ceph-mon[90634]: pgmap v1612: 119 pgs: 1 active+recovering+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 879 op/s; 716/225336 objects degraded (0.318%); 38756/225336 objects misplaced (17.199%); 22 MiB/s, 345 objects/s recovering 2024-06-14T23:21:09.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:09 smithi137 ceph-mon[90634]: osdmap e444: 8 total, 8 up, 7 in 2024-06-14T23:21:09.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:09 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 716/225336 objects degraded (0.318%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:21:09.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:09.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:09.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:09.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:09.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:09.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:09.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:09.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:09.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:09.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:10.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:10.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:10.031 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:10.040 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:10.070 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:10.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:10.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:10.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:10.207 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:10 smithi086 ceph-mon[97913]: 1.13 scrub starts 2024-06-14T23:21:10.207 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:10 smithi086 ceph-mon[97913]: osdmap e445: 8 total, 8 up, 7 in 2024-06-14T23:21:10.208 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:10 smithi086 ceph-mon[102754]: 1.13 scrub starts 2024-06-14T23:21:10.208 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:10 smithi086 ceph-mon[102754]: osdmap e445: 8 total, 8 up, 7 in 2024-06-14T23:21:10.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:10.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:10.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:10.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:10.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:10.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:10.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:10.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:10 smithi137 ceph-mon[90634]: 1.13 scrub starts 2024-06-14T23:21:10.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:10 smithi137 ceph-mon[90634]: osdmap e445: 8 total, 8 up, 7 in 2024-06-14T23:21:10.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:10.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:10.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:10.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:10.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:10.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:10.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:10.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:10.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:10.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:10.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:10.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:10.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:11.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:11.114 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:11.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:11.144 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:21:11.162 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:11 smithi086 ceph-mon[97913]: pgmap v1615: 119 pgs: 1 active+recovering+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 880 op/s; 716/222006 objects degraded (0.323%); 38143/222006 objects misplaced (17.181%); 19 MiB/s, 304 objects/s recovering 2024-06-14T23:21:11.162 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:11 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2602098486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:11.162 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:11 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:11.162 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:11 smithi086 ceph-mon[102754]: pgmap v1615: 119 pgs: 1 active+recovering+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 880 op/s; 716/222006 objects degraded (0.323%); 38143/222006 objects misplaced (17.181%); 19 MiB/s, 304 objects/s recovering 2024-06-14T23:21:11.162 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:11 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2602098486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:11.163 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:11 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:11.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:11.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:11.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:11.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:11.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:11.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:11.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:11.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:11.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:11 smithi137 ceph-mon[90634]: pgmap v1615: 119 pgs: 1 active+recovering+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovery_wait+remapped, 3 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 107 active+clean; 4.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 880 op/s; 716/222006 objects degraded (0.323%); 38143/222006 objects misplaced (17.181%); 19 MiB/s, 304 objects/s recovering 2024-06-14T23:21:11.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:11 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2602098486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:11.498 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:11 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:11.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:11.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:11.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:11.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:11.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:11.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:11.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:11.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:11.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:11.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:11.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:12.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:12.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:12.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:12.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:12.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:12.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:12.245 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:12 smithi086 ceph-mon[97913]: 1.13 scrub starts 2024-06-14T23:21:12.245 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:12 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:21:12.245 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:12 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:21:12.245 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:12 smithi086 ceph-mon[97913]: osdmap e446: 8 total, 8 up, 7 in 2024-06-14T23:21:12.245 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:12 smithi086 ceph-mon[102754]: 1.13 scrub starts 2024-06-14T23:21:12.245 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:12 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:21:12.245 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:12 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:21:12.245 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:12 smithi086 ceph-mon[102754]: osdmap e446: 8 total, 8 up, 7 in 2024-06-14T23:21:12.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:12.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:12.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:12.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:12.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:12.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:12.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:12 smithi137 ceph-mon[90634]: 1.13 scrub starts 2024-06-14T23:21:12.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:12 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:21:12.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:12 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:21:12.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:12 smithi137 ceph-mon[90634]: osdmap e446: 8 total, 8 up, 7 in 2024-06-14T23:21:12.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:12.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:12.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:12.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:12.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:12.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:12.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:12.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:12.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:13.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:13.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:13.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:13.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:13.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:13.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:13.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:13.330 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:13.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:13.397 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:13 smithi137 ceph-mon[90634]: pgmap v1617: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 10 active+remapped+backfill_wait, 107 active+clean; 4.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 747 op/s; 41571/215289 objects misplaced (19.309%); 15 MiB/s, 302 objects/s recovering 2024-06-14T23:21:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:13 smithi137 ceph-mon[90634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 716/222006 objects degraded (0.323%), 4 pgs degraded) 2024-06-14T23:21:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:13 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:21:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:13.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:13.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:13.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:13.496 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[102754]: pgmap v1617: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 10 active+remapped+backfill_wait, 107 active+clean; 4.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 747 op/s; 41571/215289 objects misplaced (19.309%); 15 MiB/s, 302 objects/s recovering 2024-06-14T23:21:13.496 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[102754]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 716/222006 objects degraded (0.323%), 4 pgs degraded) 2024-06-14T23:21:13.496 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:21:13.496 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:13.496 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:13.496 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[97913]: pgmap v1617: 119 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+remapped, 10 active+remapped+backfill_wait, 107 active+clean; 4.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 747 op/s; 41571/215289 objects misplaced (19.309%); 15 MiB/s, 302 objects/s recovering 2024-06-14T23:21:13.496 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[97913]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 716/222006 objects degraded (0.323%), 4 pgs degraded) 2024-06-14T23:21:13.496 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:21:13.497 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:13.497 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:13.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:13.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:13.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:13.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:13.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:13.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:13.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:13.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:13.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:13.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:13.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:14.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:14.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:14.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:14.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:14.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:14.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:14.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:14.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:14.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:14.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:14.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:14.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:14.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:14.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:14.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:14.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:14.809 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:14.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:14.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:14.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:14.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:14.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:15.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:15.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:15.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:15.137 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:21:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:21:14] "GET /metrics HTTP/1.1" 200 36724 "" "Prometheus/2.43.0" 2024-06-14T23:21:15.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:15.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:15.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:15.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:15.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:21:15.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:15.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:15.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:15.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:15.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:15.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:15.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:15 smithi137 ceph-mon[90634]: pgmap v1618: 119 pgs: 1 active+remapped+backfilling, 10 active+remapped+backfill_wait, 108 active+clean; 4.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 40511/211698 objects misplaced (19.136%); 17 MiB/s, 331 objects/s recovering 2024-06-14T23:21:15.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:15 smithi086 ceph-mon[97913]: pgmap v1618: 119 pgs: 1 active+remapped+backfilling, 10 active+remapped+backfill_wait, 108 active+clean; 4.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 40511/211698 objects misplaced (19.136%); 17 MiB/s, 331 objects/s recovering 2024-06-14T23:21:15.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:15 smithi086 ceph-mon[102754]: pgmap v1618: 119 pgs: 1 active+remapped+backfilling, 10 active+remapped+backfill_wait, 108 active+clean; 4.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 40511/211698 objects misplaced (19.136%); 17 MiB/s, 331 objects/s recovering 2024-06-14T23:21:15.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:15.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:15.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:15.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:15.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:15.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:15.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:15.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:15.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:15.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:15.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:16.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:16.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:16.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:16.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:16.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:16.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:16.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:16.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:16.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:16.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:16 smithi086 ceph-mon[97913]: 1.13 scrub starts 2024-06-14T23:21:16.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:16 smithi086 ceph-mon[102754]: 1.13 scrub starts 2024-06-14T23:21:16.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:16.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:16.457 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:16.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:16.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:16 smithi137 ceph-mon[90634]: 1.13 scrub starts 2024-06-14T23:21:16.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:16.594 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:16.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:16.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:16.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:16.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:16.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:16.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:16.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:17.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:17.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:17.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:17.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:17.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:17.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:17.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:17.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:17.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:17.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:17 smithi086 ceph-mon[102754]: pgmap v1619: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 4.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 37080/199845 objects misplaced (18.554%); 21 MiB/s, 379 objects/s recovering 2024-06-14T23:21:17.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:17 smithi086 ceph-mon[97913]: pgmap v1619: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 4.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 37080/199845 objects misplaced (18.554%); 21 MiB/s, 379 objects/s recovering 2024-06-14T23:21:17.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:17.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:17.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:17.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:17 smithi137 ceph-mon[90634]: pgmap v1619: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 4.1 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 37080/199845 objects misplaced (18.554%); 21 MiB/s, 379 objects/s recovering 2024-06-14T23:21:17.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:17.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:17.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:17.631 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:21:17.631 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:21:17.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:17.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:17.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:17.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:17.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:17.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:17.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:17.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:17.965 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:21:17.965 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:21:17.979 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:17.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:18.003 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:21:18.003 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:21:18.037 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:21:18.037 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:21:18.074 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:21:18.074 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:21:18.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:18.109 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:21:18.109 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:21:18.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:18.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:18.151 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:21:18.151 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:21:18.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:18 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4291826821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:21:18.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:18 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4291826821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:21:18.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:18.224 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:21:18.224 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:21:18.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:21:18.256 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:21:18.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:18.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:18.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:18.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:18.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:18.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:18.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:18.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:18 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4291826821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:21:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:18.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:18.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:18.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:18.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:18.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:18.802 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:18.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:18.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:18.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:18.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:18.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:19.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:19.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:19.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:19.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:19.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:19.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:19.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:19.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-14T23:21:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:19.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:19.413 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:19 smithi086 ceph-mon[97913]: pgmap v1620: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 4.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 36074/195708 objects misplaced (18.433%); 22 MiB/s, 384 objects/s recovering 2024-06-14T23:21:19.413 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:19 smithi086 ceph-mon[102754]: pgmap v1620: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 4.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 36074/195708 objects misplaced (18.433%); 22 MiB/s, 384 objects/s recovering 2024-06-14T23:21:19.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:19.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:19 smithi137 ceph-mon[90634]: pgmap v1620: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 4.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 36074/195708 objects misplaced (18.433%); 22 MiB/s, 384 objects/s recovering 2024-06-14T23:21:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:19.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:19.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:19.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:19.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:19.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:19.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:19.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:19.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:19.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:19.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:19.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:20.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:20.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:20.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:20.181 INFO:teuthology.orchestra.run.smithi086.stderr:set pool 14 pg_num to 26 2024-06-14T23:21:20.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:20.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:20.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:20.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:20.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:20.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:20.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:20.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:20.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:20 smithi137 ceph-mon[90634]: pgmap v1621: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 3.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 34329/191892 objects misplaced (17.890%); 27 MiB/s, 471 objects/s recovering 2024-06-14T23:21:20.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3369986989' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]: dispatch 2024-06-14T23:21:20.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:20 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]: dispatch 2024-06-14T23:21:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:20.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:20.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:20.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:20 smithi086 ceph-mon[97913]: pgmap v1621: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 3.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 34329/191892 objects misplaced (17.890%); 27 MiB/s, 471 objects/s recovering 2024-06-14T23:21:20.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3369986989' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]: dispatch 2024-06-14T23:21:20.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:20 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]: dispatch 2024-06-14T23:21:20.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:20 smithi086 ceph-mon[102754]: pgmap v1621: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 3.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 34329/191892 objects misplaced (17.890%); 27 MiB/s, 471 objects/s recovering 2024-06-14T23:21:20.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3369986989' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]: dispatch 2024-06-14T23:21:20.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:20 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]: dispatch 2024-06-14T23:21:20.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:20.652 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:20.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:20.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:20.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:20.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:20.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:20.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:20.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:21.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:21.071 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:21.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:21.075 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:21.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:21.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:21.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:21.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:21.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:21.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:21.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:21.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:21 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]': finished 2024-06-14T23:21:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:21 smithi137 ceph-mon[90634]: osdmap e447: 8 total, 8 up, 7 in 2024-06-14T23:21:21.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:21.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:21 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]': finished 2024-06-14T23:21:21.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:21 smithi086 ceph-mon[97913]: osdmap e447: 8 total, 8 up, 7 in 2024-06-14T23:21:21.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:21 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]': finished 2024-06-14T23:21:21.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:21 smithi086 ceph-mon[102754]: osdmap e447: 8 total, 8 up, 7 in 2024-06-14T23:21:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:21.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:21.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:21.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:21.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:21.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:21.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:21.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:21.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:22.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:22.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:22.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:22.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:22.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:22.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:22.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:22.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:22.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:22.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:22.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:22 smithi137 ceph-mon[90634]: pgmap v1623: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 3.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 32422/184380 objects misplaced (17.584%); 22 MiB/s, 345 objects/s recovering 2024-06-14T23:21:22.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:22 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-14T23:21:22.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:22.584 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:22.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:22.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:22 smithi086 ceph-mon[102754]: pgmap v1623: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 3.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 32422/184380 objects misplaced (17.584%); 22 MiB/s, 345 objects/s recovering 2024-06-14T23:21:22.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:22 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-14T23:21:22.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:22 smithi086 ceph-mon[97913]: pgmap v1623: 119 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 108 active+clean; 3.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 32422/184380 objects misplaced (17.584%); 22 MiB/s, 345 objects/s recovering 2024-06-14T23:21:22.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:22 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-14T23:21:22.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:22.709 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:22.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:22.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:21:22.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:22.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:22.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:22.848 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:22.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:22.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:22.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:23.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:23.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:23.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:23.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:23.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:23.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:23.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:23.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:23.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:23.439 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-14T23:21:23.439 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:23 smithi086 ceph-mon[97913]: osdmap e448: 8 total, 8 up, 7 in 2024-06-14T23:21:23.439 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:23 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:21:23.440 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-14T23:21:23.440 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:23 smithi086 ceph-mon[102754]: osdmap e448: 8 total, 8 up, 7 in 2024-06-14T23:21:23.440 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:23 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:21:23.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:23.488 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:23.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:23.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-14T23:21:23.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:23 smithi137 ceph-mon[90634]: osdmap e448: 8 total, 8 up, 7 in 2024-06-14T23:21:23.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:23 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:21:23.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:23.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:23.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:23.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:23.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:23.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:23.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:23.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:23.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:23.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:23.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:23.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:24.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:24.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:24.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:24.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:24.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:24.234 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:24.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:24.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:24.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:24.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:24.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:24.482 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:24.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:24.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: 1.13 scrub starts 2024-06-14T23:21:24.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: 14.1 continuing backfill to osd.2 from (428'5123,447'9922] MIN to 447'9922 2024-06-14T23:21:24.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: 14.2 continuing backfill to osd.0 from (428'5131,447'9926] MIN to 447'9926 2024-06-14T23:21:24.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: 14.8 continuing backfill to osd.2 from (428'4584,447'9773] MIN to 447'9773 2024-06-14T23:21:24.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: 14.0 continuing backfill to osd.2 from (428'4620,447'9834] MIN to 447'9834 2024-06-14T23:21:24.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: 14.6 continuing backfill to osd.2 from (428'4595,447'9813] MIN to 447'9813 2024-06-14T23:21:24.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: osdmap e449: 8 total, 8 up, 7 in 2024-06-14T23:21:24.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: 14.3 continuing backfill to osd.5 from (428'4702,447'9899] MIN to 447'9899 2024-06-14T23:21:24.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: osdmap e450: 8 total, 8 up, 7 in 2024-06-14T23:21:24.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:24 smithi137 ceph-mon[90634]: pgmap v1627: 129 pgs: 1 peering, 2 remapped+peering, 10 unknown, 2 active+remapped+backfilling, 7 active+remapped+backfill_wait, 107 active+clean; 3.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 27373/164055 objects misplaced (16.685%) 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: 1.13 scrub starts 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: 14.1 continuing backfill to osd.2 from (428'5123,447'9922] MIN to 447'9922 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: 14.2 continuing backfill to osd.0 from (428'5131,447'9926] MIN to 447'9926 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: 14.8 continuing backfill to osd.2 from (428'4584,447'9773] MIN to 447'9773 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: 14.0 continuing backfill to osd.2 from (428'4620,447'9834] MIN to 447'9834 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: 14.6 continuing backfill to osd.2 from (428'4595,447'9813] MIN to 447'9813 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: osdmap e449: 8 total, 8 up, 7 in 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: 14.3 continuing backfill to osd.5 from (428'4702,447'9899] MIN to 447'9899 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: osdmap e450: 8 total, 8 up, 7 in 2024-06-14T23:21:24.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[97913]: pgmap v1627: 129 pgs: 1 peering, 2 remapped+peering, 10 unknown, 2 active+remapped+backfilling, 7 active+remapped+backfill_wait, 107 active+clean; 3.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 27373/164055 objects misplaced (16.685%) 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: 1.13 scrub starts 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: 14.1 continuing backfill to osd.2 from (428'5123,447'9922] MIN to 447'9922 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: 14.2 continuing backfill to osd.0 from (428'5131,447'9926] MIN to 447'9926 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: 14.8 continuing backfill to osd.2 from (428'4584,447'9773] MIN to 447'9773 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: 14.0 continuing backfill to osd.2 from (428'4620,447'9834] MIN to 447'9834 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: 14.6 continuing backfill to osd.2 from (428'4595,447'9813] MIN to 447'9813 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: osdmap e449: 8 total, 8 up, 7 in 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: 14.3 continuing backfill to osd.5 from (428'4702,447'9899] MIN to 447'9899 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: osdmap e450: 8 total, 8 up, 7 in 2024-06-14T23:21:24.577 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-mon[102754]: pgmap v1627: 129 pgs: 1 peering, 2 remapped+peering, 10 unknown, 2 active+remapped+backfilling, 7 active+remapped+backfill_wait, 107 active+clean; 3.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 27373/164055 objects misplaced (16.685%) 2024-06-14T23:21:24.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:24.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:24.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:24.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:24.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:24.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:24.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:24.868 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:24.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:24.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:24.998 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:24.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:24.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:25.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:25.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:25.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:25.134 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:21:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:21:24] "GET /metrics HTTP/1.1" 200 36724 "" "Prometheus/2.43.0" 2024-06-14T23:21:25.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:25.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:25.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:25.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:25.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:25.469 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:25 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-14T23:21:25.469 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:25 smithi137 ceph-mon[90634]: osdmap e451: 8 total, 8 up, 7 in 2024-06-14T23:21:25.469 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:25 smithi137 ceph-mon[90634]: 14.10 continuing backfill to osd.2 from (428'4620,447'9834] MIN to 447'9834 2024-06-14T23:21:25.469 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:25 smithi137 ceph-mon[90634]: 14.18 continuing backfill to osd.2 from (428'4584,447'9773] MIN to 447'9773 2024-06-14T23:21:25.469 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:25 smithi137 ceph-mon[90634]: 14.16 continuing backfill to osd.2 from (428'4595,447'9813] MIN to 447'9813 2024-06-14T23:21:25.469 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:25 smithi137 ceph-mon[90634]: 14.12 continuing backfill to osd.0 from (428'5131,447'9926] MIN to 447'9926 2024-06-14T23:21:25.469 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:25 smithi137 ceph-mon[90634]: 14.13 continuing backfill to osd.5 from (428'4702,447'9899] MIN to 447'9899 2024-06-14T23:21:25.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:25.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:25.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:25.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-14T23:21:25.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[102754]: osdmap e451: 8 total, 8 up, 7 in 2024-06-14T23:21:25.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[102754]: 14.10 continuing backfill to osd.2 from (428'4620,447'9834] MIN to 447'9834 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[102754]: 14.18 continuing backfill to osd.2 from (428'4584,447'9773] MIN to 447'9773 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[102754]: 14.16 continuing backfill to osd.2 from (428'4595,447'9813] MIN to 447'9813 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[102754]: 14.12 continuing backfill to osd.0 from (428'5131,447'9926] MIN to 447'9926 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[102754]: 14.13 continuing backfill to osd.5 from (428'4702,447'9899] MIN to 447'9899 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[97913]: osdmap e451: 8 total, 8 up, 7 in 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[97913]: 14.10 continuing backfill to osd.2 from (428'4620,447'9834] MIN to 447'9834 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[97913]: 14.18 continuing backfill to osd.2 from (428'4584,447'9773] MIN to 447'9773 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[97913]: 14.16 continuing backfill to osd.2 from (428'4595,447'9813] MIN to 447'9813 2024-06-14T23:21:25.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[97913]: 14.12 continuing backfill to osd.0 from (428'5131,447'9926] MIN to 447'9926 2024-06-14T23:21:25.590 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:25 smithi086 ceph-mon[97913]: 14.13 continuing backfill to osd.5 from (428'4702,447'9899] MIN to 447'9899 2024-06-14T23:21:25.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:25.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:25.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:25.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:25.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:25.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:25.815 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:25.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:25.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:25.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:26.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:26.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:26.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:26.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:26.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:26.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:26.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:26.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:26.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:26.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:26.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:26.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:26.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:26 smithi137 ceph-mon[90634]: 14.11 continuing backfill to osd.2 from (428'5123,447'9922] MIN to 447'9922 2024-06-14T23:21:26.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:26 smithi137 ceph-mon[90634]: osdmap e452: 8 total, 8 up, 7 in 2024-06-14T23:21:26.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:26 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3902139646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:21:26.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:26 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:21:26.497 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:26 smithi137 ceph-mon[90634]: pgmap v1630: 129 pgs: 1 peering, 2 remapped+peering, 2 unknown, 3 active+remapped+backfilling, 11 active+remapped+backfill_wait, 110 active+clean; 3.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 25528/163140 objects misplaced (15.648%); 12 MiB/s, 198 objects/s recovering 2024-06-14T23:21:26.519 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:21:26.520 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[97913]: 14.11 continuing backfill to osd.2 from (428'5123,447'9922] MIN to 447'9922 2024-06-14T23:21:26.520 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[97913]: osdmap e452: 8 total, 8 up, 7 in 2024-06-14T23:21:26.520 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3902139646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:21:26.520 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:21:26.520 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[97913]: pgmap v1630: 129 pgs: 1 peering, 2 remapped+peering, 2 unknown, 3 active+remapped+backfilling, 11 active+remapped+backfill_wait, 110 active+clean; 3.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 25528/163140 objects misplaced (15.648%); 12 MiB/s, 198 objects/s recovering 2024-06-14T23:21:26.520 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[102754]: 14.11 continuing backfill to osd.2 from (428'5123,447'9922] MIN to 447'9922 2024-06-14T23:21:26.520 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[102754]: osdmap e452: 8 total, 8 up, 7 in 2024-06-14T23:21:26.520 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3902139646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:21:26.521 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:21:26.521 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:26 smithi086 ceph-mon[102754]: pgmap v1630: 129 pgs: 1 peering, 2 remapped+peering, 2 unknown, 3 active+remapped+backfilling, 11 active+remapped+backfill_wait, 110 active+clean; 3.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 25528/163140 objects misplaced (15.648%); 12 MiB/s, 198 objects/s recovering 2024-06-14T23:21:26.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:26.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:26.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:26.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:26.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:26.858 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:26.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:26.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:26.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:27.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:27.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:27.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:27.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:27.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:27.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:27.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:27.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:27.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:27.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:27.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:27 smithi137 ceph-mon[90634]: 14.9 scrub starts 2024-06-14T23:21:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:27 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-06-14T23:21:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:27 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:21:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:27 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:21:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:27 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:21:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:27 smithi137 ceph-mon[90634]: osdmap e453: 8 total, 8 up, 7 in 2024-06-14T23:21:27.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[97913]: 14.9 scrub starts 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[97913]: osdmap e453: 8 total, 8 up, 7 in 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[102754]: 14.9 scrub starts 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:21:27.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:21:27.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:27 smithi086 ceph-mon[102754]: osdmap e453: 8 total, 8 up, 7 in 2024-06-14T23:21:27.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:27.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:27.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:27.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:27.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:27.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:27.834 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0, 2] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:21:27.834 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:21:27.834 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-06-14T23:21:27.835 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:21:27.835 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:21:27.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-06-14T23:21:27.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:27.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:27.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:27.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:28.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:28.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:28.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:28.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:28.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:28.217 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:28 smithi137 ceph-mon[90634]: 1.13 scrub starts 2024-06-14T23:21:28.217 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:28.217 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:28.217 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:28 smithi137 ceph-mon[90634]: pgmap v1632: 129 pgs: 4 active+remapped+backfilling, 12 active+remapped+backfill_wait, 113 active+clean; 3.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 145 MiB/s wr, 4.04k op/s; 27218/172107 objects misplaced (15.815%); 88 MiB/s, 1.41k objects/s recovering 2024-06-14T23:21:28.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:28.308 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:21:28.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:28.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:28.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:28.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:28.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:28.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:28.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:28.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:28.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:28 smithi086 ceph-mon[97913]: 1.13 scrub starts 2024-06-14T23:21:28.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:28.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:28.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:28 smithi086 ceph-mon[97913]: pgmap v1632: 129 pgs: 4 active+remapped+backfilling, 12 active+remapped+backfill_wait, 113 active+clean; 3.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 145 MiB/s wr, 4.04k op/s; 27218/172107 objects misplaced (15.815%); 88 MiB/s, 1.41k objects/s recovering 2024-06-14T23:21:28.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:28 smithi086 ceph-mon[102754]: 1.13 scrub starts 2024-06-14T23:21:28.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:28.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:28.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:28 smithi086 ceph-mon[102754]: pgmap v1632: 129 pgs: 4 active+remapped+backfilling, 12 active+remapped+backfill_wait, 113 active+clean; 3.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 145 MiB/s wr, 4.04k op/s; 27218/172107 objects misplaced (15.815%); 88 MiB/s, 1.41k objects/s recovering 2024-06-14T23:21:28.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:28.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:28.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:28.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:28.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:28.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:28.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:28.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:29.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:29.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:29.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:29.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:29.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:29.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:29.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:29.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:29.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:29.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:29.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:29.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:29.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:29.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:29.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:29.739 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:29.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:29.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:29 smithi137 ceph-mon[90634]: osdmap e454: 8 total, 8 up, 7 in 2024-06-14T23:21:29.751 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:29 smithi086 ceph-mon[97913]: osdmap e454: 8 total, 8 up, 7 in 2024-06-14T23:21:29.751 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:29 smithi086 ceph-mon[102754]: osdmap e454: 8 total, 8 up, 7 in 2024-06-14T23:21:29.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:29.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:29.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:29.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:30.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:30.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:30.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:30.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:30.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:30.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:30.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:30.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:30.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:30.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:30.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:30.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:30.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:30.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:30.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:30 smithi086 ceph-mon[102754]: osdmap e455: 8 total, 8 up, 7 in 2024-06-14T23:21:30.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:30 smithi086 ceph-mon[102754]: pgmap v1635: 129 pgs: 3 active+remapped+backfilling, 13 active+remapped+backfill_wait, 113 active+clean; 3.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 146 MiB/s wr, 3.88k op/s; 25915/168606 objects misplaced (15.370%); 71 MiB/s, 1.14k objects/s recovering 2024-06-14T23:21:30.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:30 smithi086 ceph-mon[97913]: osdmap e455: 8 total, 8 up, 7 in 2024-06-14T23:21:30.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:30 smithi086 ceph-mon[97913]: pgmap v1635: 129 pgs: 3 active+remapped+backfilling, 13 active+remapped+backfill_wait, 113 active+clean; 3.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 146 MiB/s wr, 3.88k op/s; 25915/168606 objects misplaced (15.370%); 71 MiB/s, 1.14k objects/s recovering 2024-06-14T23:21:30.628 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:30 smithi137 ceph-mon[90634]: osdmap e455: 8 total, 8 up, 7 in 2024-06-14T23:21:30.628 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:30 smithi137 ceph-mon[90634]: pgmap v1635: 129 pgs: 3 active+remapped+backfilling, 13 active+remapped+backfill_wait, 113 active+clean; 3.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 146 MiB/s wr, 3.88k op/s; 25915/168606 objects misplaced (15.370%); 71 MiB/s, 1.14k objects/s recovering 2024-06-14T23:21:30.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:30.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:30.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:30.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:30.847 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:30.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:30.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:30.984 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:30.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:30.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:31.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:31.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:31.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:31.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:31.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:31.233 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:21:31.233 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-06-14T23:21:31.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.233 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.37287.0:165998 14.9 14:91a61fba:::benchmark_data_smithi086_333440_object57208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-06-14T23:21:31.233 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:21:30.982000+0000", 2024-06-14T23:21:31.233 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0064952589999999998, 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0065237339999999998, 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.37287", 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 165998 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982000+0000", 2024-06-14T23:21:31.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982000+0000", 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.981998+0000", 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999976 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:21:31.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982011+0000", 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2499e-05 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982035+0000", 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4477999999999998e-05 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982046+0000", 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1092e-05 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982104+0000", 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7426999999999998e-05 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982248+0000", 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014427900000000001 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,5", 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982371+0000", 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012289000000000001 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.985358+0000", 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002986991 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.37287.0:166001 14.11 14:89778811:::benchmark_data_smithi086_333440_object57211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:21:30.982513+0000", 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0059825340000000003, 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060902179999999997, 2024-06-14T23:21:31.238 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.37287", 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 166001 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982513+0000", 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:21:31.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982513+0000", 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982512+0000", 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982517+0000", 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8199999999999996e-06 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:21:31.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982532+0000", 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4793e-05 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982539+0000", 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5240000000000003e-06 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982606+0000", 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6933000000000004e-05 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:21:31.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982746+0000", 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013956299999999999 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4,6", 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982860+0000", 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114512 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.985235+0000", 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00237419 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.985304+0000", 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9845000000000006e-05 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.986649+0000", 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013447789999999999 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.37287.0:166002 14.9 14:951fc280:::benchmark_data_smithi086_333440_object57212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-06-14T23:21:31.243 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:21:30.982652+0000", 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0058439199999999998, 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006032801, 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.37287", 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 166002 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982652+0000", 2024-06-14T23:21:31.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982652+0000", 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982651+0000", 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:21:31.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982655+0000", 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1219999999999997e-06 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982664+0000", 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2390000000000008e-06 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.982672+0000", 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.221e-06 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.985384+0000", 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027128740000000001 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.985502+0000", 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011791 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,5", 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.985611+0000", 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109094 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.247 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop_reply(client.37287.0:165998 14.9 e455/448)", 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:21:30.986758+0000", 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.001737819, 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019882530000000002, 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "queued for pg", 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.986758+0000", 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.986758+0000", 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.986757+0000", 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.986762+0000", 2024-06-14T23:21:31.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0719999999999999e-06 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.986775+0000", 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2539e-05 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.986782+0000", 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9779999999999999e-06 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.37287.0:166004 14.a e455/433 14:587c1185:::benchmark_data_smithi086_333440_object57214:head v 455'10332, mlcod=455'10330)", 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:21:30.987223+0000", 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.00127268, 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001566055, 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987223+0000", 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987223+0000", 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987222+0000", 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999995 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987228+0000", 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9309999999999996e-06 2024-06-14T23:21:31.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987241+0000", 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2697000000000001e-05 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987246+0000", 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1819999999999997e-06 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987300+0000", 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4194000000000003e-05 2024-06-14T23:21:31.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987324+0000", 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3703999999999999e-05 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.37287.0:166005 14.6 e455/448 14:617d4d71:::benchmark_data_smithi086_333440_object57215:head v 455'10012, mlcod=455'10010)", 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:21:30.987489+0000", 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0010063019999999999, 2024-06-14T23:21:31.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013507949999999999, 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987489+0000", 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987489+0000", 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987489+0000", 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.999999 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987495+0000", 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8709999999999999e-06 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987506+0000", 2024-06-14T23:21:31.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0034000000000001e-05 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987511+0000", 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8050000000000003e-06 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987571+0000", 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9778999999999998e-05 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987594+0000", 2024-06-14T23:21:31.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2501e-05 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.37287.0:166012 14.11 14.51a23991 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-06-14T23:21:30.987648+0000", 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.00084736700000000002, 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012427370000000001, 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "queued for pg", 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-06-14T23:21:31.258 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.37287", 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 166012 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987648+0000", 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987648+0000", 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987647+0000", 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4294967295.9999986 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987653+0000", 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6420000000000004e-06 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987665+0000", 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2315e-05 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-06-14T23:21:31.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-06-14T23:21:30.987672+0000", 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1350000000000003e-06 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 7 2024-06-14T23:21:31.261 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:21:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:31.262 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:31.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:31.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:31.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:31.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:31.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:31.536 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:31.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:31.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:31.665 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:31.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:31.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:31.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:31.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:31.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:31.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:31.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:31.987 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:21:31.987 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-14T23:21:31.988 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:21:32.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:32.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:32.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:32.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:32.160 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:32.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:32.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-14T23:21:32.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:32.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:32.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:32.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-14T23:21:32.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:32.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:32.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:32.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:32.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:32.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:32.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:32.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:32 smithi086 ceph-mon[97913]: osdmap e456: 8 total, 8 up, 7 in 2024-06-14T23:21:32.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:32 smithi086 ceph-mon[97913]: pgmap v1637: 129 pgs: 3 active+remapped+backfilling, 13 active+remapped+backfill_wait, 113 active+clean; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 80 MiB/s wr, 2.73k op/s; 22453/160128 objects misplaced (14.022%); 61 MiB/s, 979 objects/s recovering 2024-06-14T23:21:32.676 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:32 smithi086 ceph-mon[102754]: osdmap e456: 8 total, 8 up, 7 in 2024-06-14T23:21:32.677 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:32 smithi086 ceph-mon[102754]: pgmap v1637: 129 pgs: 3 active+remapped+backfilling, 13 active+remapped+backfill_wait, 113 active+clean; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 80 MiB/s wr, 2.73k op/s; 22453/160128 objects misplaced (14.022%); 61 MiB/s, 979 objects/s recovering 2024-06-14T23:21:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:32.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:32.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:32.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:32 smithi137 ceph-mon[90634]: osdmap e456: 8 total, 8 up, 7 in 2024-06-14T23:21:32.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:32 smithi137 ceph-mon[90634]: pgmap v1637: 129 pgs: 3 active+remapped+backfilling, 13 active+remapped+backfill_wait, 113 active+clean; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 80 MiB/s wr, 2.73k op/s; 22453/160128 objects misplaced (14.022%); 61 MiB/s, 979 objects/s recovering 2024-06-14T23:21:32.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:32.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:32.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:32.990 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:32.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:33.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:33.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:33.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:33.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:33.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:33.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:33.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:33.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:33.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:33.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:33.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:33.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:33.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:33 smithi086 ceph-mon[97913]: osdmap e457: 8 total, 8 up, 7 in 2024-06-14T23:21:33.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:33 smithi086 ceph-mon[102754]: osdmap e457: 8 total, 8 up, 7 in 2024-06-14T23:21:33.633 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:33 smithi137 ceph-mon[90634]: osdmap e457: 8 total, 8 up, 7 in 2024-06-14T23:21:33.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:33.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:33.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:33.781 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:21:33.782 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:21:33.818 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:21:33.818 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:21:33.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:33.860 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:21:33.860 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:21:33.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:33.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:33.895 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:21:33.895 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:21:33.909 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:33.919 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:33.924 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:21:33.924 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:21:33.962 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:21:33.962 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:21:33.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:34.002 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:21:34.002 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:21:34.024 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:34.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:34.033 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:21:34.033 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:21:34.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:34.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:34.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:34.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:34.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:34.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:34.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:34.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:34.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:34.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:34.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:34.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1384934151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:21:34.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:34 smithi086 ceph-mon[97913]: pgmap v1639: 129 pgs: 2 active+remapped+backfilling, 13 active+remapped+backfill_wait, 114 active+clean; 3.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 22334/157251 objects misplaced (14.203%); 30 MiB/s, 483 objects/s recovering 2024-06-14T23:21:34.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1384934151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:21:34.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:34 smithi086 ceph-mon[102754]: pgmap v1639: 129 pgs: 2 active+remapped+backfilling, 13 active+remapped+backfill_wait, 114 active+clean; 3.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 22334/157251 objects misplaced (14.203%); 30 MiB/s, 483 objects/s recovering 2024-06-14T23:21:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:34.612 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:34.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:34.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:34.742 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:34.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:34.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1384934151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:21:34.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:34 smithi137 ceph-mon[90634]: pgmap v1639: 129 pgs: 2 active+remapped+backfilling, 13 active+remapped+backfill_wait, 114 active+clean; 3.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 22334/157251 objects misplaced (14.203%); 30 MiB/s, 483 objects/s recovering 2024-06-14T23:21:34.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:34.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:34.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:35.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:35.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:35.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:35.161 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:35.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:35.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:35.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:35.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:35.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:35.317 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:21:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:21:34] "GET /metrics HTTP/1.1" 200 36728 "" "Prometheus/2.43.0" 2024-06-14T23:21:35.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:35.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:35.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:35.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:35.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:35.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:35.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:35.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:35.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:35.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:35.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:35.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:35.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:35.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:35.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:36.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:36.106 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:36.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:36.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:36.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:36.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:36.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:36.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:36.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:36.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:36.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:36.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:36.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:36.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:36.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:36.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:36.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:36.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:36.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:36.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:37.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:37.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:37.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:37.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:37.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:37.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:36 smithi137 ceph-mon[90634]: pgmap v1640: 129 pgs: 1 active+remapped+backfilling, 13 active+remapped+backfill_wait, 115 active+clean; 2.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.23k op/s; 18409/144816 objects misplaced (12.712%); 32 MiB/s, 515 objects/s recovering 2024-06-14T23:21:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:36 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:36 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:37.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:37.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:37.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:37.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:36 smithi086 ceph-mon[97913]: pgmap v1640: 129 pgs: 1 active+remapped+backfilling, 13 active+remapped+backfill_wait, 115 active+clean; 2.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.23k op/s; 18409/144816 objects misplaced (12.712%); 32 MiB/s, 515 objects/s recovering 2024-06-14T23:21:37.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:36 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:37.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:36 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:37.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:36 smithi086 ceph-mon[102754]: pgmap v1640: 129 pgs: 1 active+remapped+backfilling, 13 active+remapped+backfill_wait, 115 active+clean; 2.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.23k op/s; 18409/144816 objects misplaced (12.712%); 32 MiB/s, 515 objects/s recovering 2024-06-14T23:21:37.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:36 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:37.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:36 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:37.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:37.450 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:37.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:37.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:37.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:37.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:37.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:37.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:37.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:37.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:37.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:37.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:37.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:37.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:38.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:38.079 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:38.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:38.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:38.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:38.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:38.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:37 smithi137 ceph-mon[90634]: osdmap e458: 8 total, 8 up, 7 in 2024-06-14T23:21:38.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:38.331 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0, 2] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:21:38.331 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:21:38.331 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-06-14T23:21:38.332 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:21:38.332 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:21:38.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-06-14T23:21:38.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:37 smithi086 ceph-mon[102754]: osdmap e458: 8 total, 8 up, 7 in 2024-06-14T23:21:38.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:37 smithi086 ceph-mon[97913]: osdmap e458: 8 total, 8 up, 7 in 2024-06-14T23:21:38.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:38.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:38.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:38.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:38.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:38.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:38.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:38.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:38.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:38.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:38.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:38.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:38.879 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:38.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:38.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:38.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:39.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:39.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:39.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:39.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:39.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:38 smithi137 ceph-mon[90634]: pgmap v1642: 129 pgs: 1 peering, 1 active+remapped+backfilling, 12 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 17591/139983 objects misplaced (12.567%); 25 MiB/s, 392 objects/s recovering 2024-06-14T23:21:39.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:38 smithi137 ceph-mon[90634]: osdmap e459: 8 total, 8 up, 7 in 2024-06-14T23:21:39.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:39.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:39.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:39.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:38 smithi086 ceph-mon[97913]: pgmap v1642: 129 pgs: 1 peering, 1 active+remapped+backfilling, 12 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 17591/139983 objects misplaced (12.567%); 25 MiB/s, 392 objects/s recovering 2024-06-14T23:21:39.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:38 smithi086 ceph-mon[97913]: osdmap e459: 8 total, 8 up, 7 in 2024-06-14T23:21:39.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:38 smithi086 ceph-mon[102754]: pgmap v1642: 129 pgs: 1 peering, 1 active+remapped+backfilling, 12 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 17591/139983 objects misplaced (12.567%); 25 MiB/s, 392 objects/s recovering 2024-06-14T23:21:39.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:38 smithi086 ceph-mon[102754]: osdmap e459: 8 total, 8 up, 7 in 2024-06-14T23:21:39.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:39.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:39.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:39.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:39.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:39.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:39.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:39.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:39.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:39.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:39.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:39.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:39.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:39.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:39.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:39.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:40.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:40.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:40.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:40.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:40.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:40.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:40.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:40.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:40.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:40.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:40.490 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:40.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:40.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:40.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:40.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:40.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:40.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:40.766 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:40.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:40.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:40.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:40.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:41.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:41.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:41.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:41.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:41.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:41.209 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:40 smithi086 ceph-mon[97913]: pgmap v1644: 129 pgs: 1 peering, 1 active+remapped+backfilling, 12 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 17543/135855 objects misplaced (12.913%); 21 MiB/s, 340 objects/s recovering 2024-06-14T23:21:41.209 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:40 smithi086 ceph-mon[102754]: pgmap v1644: 129 pgs: 1 peering, 1 active+remapped+backfilling, 12 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 17543/135855 objects misplaced (12.913%); 21 MiB/s, 340 objects/s recovering 2024-06-14T23:21:41.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:40 smithi137 ceph-mon[90634]: pgmap v1644: 129 pgs: 1 peering, 1 active+remapped+backfilling, 12 active+remapped+backfill_wait, 115 active+clean; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 17543/135855 objects misplaced (12.913%); 21 MiB/s, 340 objects/s recovering 2024-06-14T23:21:41.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:41.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:41.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:41.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:41.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:41.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:41.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:41.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:41.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:41.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:41.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:41.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:41.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:41.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:41.901 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:41.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:42.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:42.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:42.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:42.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:42.186 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:42.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:42.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:42.342 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:42.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:42.403 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:21:42.403 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:21:42.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:42.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:42.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:21:42.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:42.633 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:42.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:42.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:42.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:42.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:42.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:42.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:42.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:43.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:43.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:43.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:43.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:43.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:43.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:42 smithi137 ceph-mon[90634]: pgmap v1645: 129 pgs: 1 peering, 1 active+remapped+backfilling, 12 active+remapped+backfill_wait, 115 active+clean; 2.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 15729/128409 objects misplaced (12.249%); 19 MiB/s, 309 objects/s recovering 2024-06-14T23:21:43.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:43.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:43.292 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:43.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:42 smithi086 ceph-mon[97913]: pgmap v1645: 129 pgs: 1 peering, 1 active+remapped+backfilling, 12 active+remapped+backfill_wait, 115 active+clean; 2.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 15729/128409 objects misplaced (12.249%); 19 MiB/s, 309 objects/s recovering 2024-06-14T23:21:43.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:43.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:42 smithi086 ceph-mon[102754]: pgmap v1645: 129 pgs: 1 peering, 1 active+remapped+backfilling, 12 active+remapped+backfill_wait, 115 active+clean; 2.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 15729/128409 objects misplaced (12.249%); 19 MiB/s, 309 objects/s recovering 2024-06-14T23:21:43.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:43.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:43.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:43.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:43.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:43.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:43.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:43.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:43.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-14T23:21:43.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:43.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:43.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:43.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:43.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:43.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:43.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:43.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:43.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:43.975 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:43 smithi086 ceph-mon[97913]: osdmap e460: 8 total, 8 up, 7 in 2024-06-14T23:21:44.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:44.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:44.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:44.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:44.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:44.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:44.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:43 smithi137 ceph-mon[90634]: osdmap e460: 8 total, 8 up, 7 in 2024-06-14T23:21:44.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:44.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:44.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:44.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:43 smithi086 ceph-mon[102754]: osdmap e460: 8 total, 8 up, 7 in 2024-06-14T23:21:44.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:44.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:44.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:44.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:44.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:44.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:44.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:44.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:44.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:44.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:44.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:44.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:44.873 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:44.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:44.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:45.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:45.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:45.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:45.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:45.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:45.199 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:21:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:21:44] "GET /metrics HTTP/1.1" 200 36727 "" "Prometheus/2.43.0" 2024-06-14T23:21:45.200 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:44 smithi086 ceph-mon[102754]: pgmap v1647: 129 pgs: 1 active+clean+remapped, 12 active+remapped+backfill_wait, 116 active+clean; 2.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 14768/122931 objects misplaced (12.013%); 14 MiB/s, 228 objects/s recovering 2024-06-14T23:21:45.200 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:44 smithi086 ceph-mon[102754]: osdmap e461: 8 total, 8 up, 7 in 2024-06-14T23:21:45.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:44 smithi086 ceph-mon[97913]: pgmap v1647: 129 pgs: 1 active+clean+remapped, 12 active+remapped+backfill_wait, 116 active+clean; 2.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 14768/122931 objects misplaced (12.013%); 14 MiB/s, 228 objects/s recovering 2024-06-14T23:21:45.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:44 smithi086 ceph-mon[97913]: osdmap e461: 8 total, 8 up, 7 in 2024-06-14T23:21:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:44 smithi137 ceph-mon[90634]: pgmap v1647: 129 pgs: 1 active+clean+remapped, 12 active+remapped+backfill_wait, 116 active+clean; 2.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 14768/122931 objects misplaced (12.013%); 14 MiB/s, 228 objects/s recovering 2024-06-14T23:21:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:44 smithi137 ceph-mon[90634]: osdmap e461: 8 total, 8 up, 7 in 2024-06-14T23:21:45.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:45.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:45.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:45.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:45.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:45.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:45.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:45.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:45.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:45.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:45.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:45.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:45.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:45.856 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:45.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:45.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:45.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:45.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:45.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:46.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:46.131 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:46.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:46.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:46.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:46.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:46.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:46.414 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:46.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:46.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:46.551 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:46.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:46.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:46.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:46.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:46.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:46.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:46.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:46.966 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:46.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:47.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:47.103 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:47.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:47.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:47.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:47.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:47.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:46 smithi137 ceph-mon[90634]: pgmap v1649: 129 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 11 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.26k op/s; 11980/109899 objects misplaced (10.901%); 13 MiB/s, 210 objects/s recovering 2024-06-14T23:21:47.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:46 smithi086 ceph-mon[97913]: pgmap v1649: 129 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 11 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.26k op/s; 11980/109899 objects misplaced (10.901%); 13 MiB/s, 210 objects/s recovering 2024-06-14T23:21:47.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:46 smithi086 ceph-mon[102754]: pgmap v1649: 129 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 11 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.26k op/s; 11980/109899 objects misplaced (10.901%); 13 MiB/s, 210 objects/s recovering 2024-06-14T23:21:47.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:47.372 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:47.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:47.456 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:47.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:47.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:47.606 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-06-14T23:21:47.606 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "" 2024-06-14T23:21:47.606 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-06-14T23:21:47.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:47.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:47.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:47.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:47.808 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:47.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:47.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:47.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:47.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:48.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:48.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:48.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:48.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:48.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:48.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:48.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:48.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:48.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:48.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:48.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:48.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:48.531 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:48.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:48.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:48.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:48.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:48.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:48.807 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:48.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:48.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:48.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:48.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-06-14T23:21:49.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:49.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:49.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:49.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:49.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:49.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:48 smithi086 ceph-mon[97913]: pgmap v1650: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 2.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.30k op/s; 11980/104727 objects misplaced (11.439%); 13 MiB/s, 208 objects/s recovering 2024-06-14T23:21:49.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/151153893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:49.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:48 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:49.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:48 smithi086 ceph-mon[102754]: pgmap v1650: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 2.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.30k op/s; 11980/104727 objects misplaced (11.439%); 13 MiB/s, 208 objects/s recovering 2024-06-14T23:21:49.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/151153893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:49.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:48 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:49.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:49.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:49.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:49.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:49.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:49.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:49.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:48 smithi137 ceph-mon[90634]: pgmap v1650: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 2.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.30k op/s; 11980/104727 objects misplaced (11.439%); 13 MiB/s, 208 objects/s recovering 2024-06-14T23:21:49.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/151153893' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:49.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:48 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:21:49.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:49.628 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:49.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:49.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:49.773 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:49.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:49.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:49.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:49.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:49.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:50.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:50.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:50.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:50.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:50.175 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:50.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:50.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:50.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:50.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:50.314 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:50 smithi086 ceph-mon[102754]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:21:50.314 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:50 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:21:50.314 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:50 smithi086 ceph-mon[102754]: osdmap e462: 8 total, 8 up, 7 in 2024-06-14T23:21:50.314 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:50 smithi086 ceph-mon[97913]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:21:50.314 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:50 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:21:50.314 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:50 smithi086 ceph-mon[97913]: osdmap e462: 8 total, 8 up, 7 in 2024-06-14T23:21:50.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:50.447 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:50.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:50.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:50 smithi137 ceph-mon[90634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:21:50.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:50 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-14T23:21:50.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:50 smithi137 ceph-mon[90634]: osdmap e462: 8 total, 8 up, 7 in 2024-06-14T23:21:50.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:50.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:50.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:50.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:50.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:50.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:50.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:50.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:50.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:50.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:50.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:50.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:51.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:51.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:51.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:51.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:51.270 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:51 smithi086 ceph-mon[97913]: pgmap v1652: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 2.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 11921/99312 objects misplaced (12.004%); 6.2 MiB/s, 99 objects/s recovering 2024-06-14T23:21:51.270 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:51.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:51 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:51.271 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:51 smithi086 ceph-mon[102754]: pgmap v1652: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 2.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 11921/99312 objects misplaced (12.004%); 6.2 MiB/s, 99 objects/s recovering 2024-06-14T23:21:51.271 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:51.271 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:51 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:51.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:51.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:51.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:51.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:51 smithi137 ceph-mon[90634]: pgmap v1652: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 2.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 11921/99312 objects misplaced (12.004%); 6.2 MiB/s, 99 objects/s recovering 2024-06-14T23:21:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:51.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:51.527 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:51.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:51.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:51.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:51.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:51.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:51.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:51.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:51.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:51.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:51.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:51.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:51.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:52.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:52 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:21:52.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:52 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:21:52.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:52 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:52.089 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:52 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:21:52.089 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:52 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:21:52.089 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:52 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:52.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:52.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:21:52.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:52.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:52.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:52.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:52.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:52.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:52.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:52.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:52.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:52 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:21:52.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:52 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:21:52.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:52 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:21:52.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:52.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:52.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:52.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:52.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:52.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:52.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:52.816 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:52.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:52.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:52.952 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:52.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:52.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:53.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:53.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:53.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:53.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:53.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:53.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:53.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:53.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:53 smithi086 ceph-mon[97913]: pgmap v1653: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 1.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.39k op/s; 8909/89466 objects misplaced (9.958%); 11 MiB/s, 180 objects/s recovering 2024-06-14T23:21:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:53 smithi086 ceph-mon[102754]: pgmap v1653: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 1.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.39k op/s; 8909/89466 objects misplaced (9.958%); 11 MiB/s, 180 objects/s recovering 2024-06-14T23:21:53.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:53.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:53.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:53.493 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:53.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:53.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:53 smithi137 ceph-mon[90634]: pgmap v1653: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 1.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.39k op/s; 8909/89466 objects misplaced (9.958%); 11 MiB/s, 180 objects/s recovering 2024-06-14T23:21:53.516 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:21:53.516 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:21:53.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:53.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:53.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:53.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:53.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:53.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:53.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:53.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:54.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:54.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:54.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:54.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:54.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:54.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:54.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:54.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:54.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:54.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:54.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:54.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:54.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:54.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:54.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:54.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:54.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:54.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:54.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-14T23:21:54.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:54.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:54.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:54.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:54.918 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:54.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:54.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:54.972 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:21:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:21:54] "GET /metrics HTTP/1.1" 200 36727 "" "Prometheus/2.43.0" 2024-06-14T23:21:54.985 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:21:55.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:55.050 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:55.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:55.056 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:21:55.056 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:21:55.056 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:21:55.056 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:21:55.057 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:21:55.057 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:21:55.057 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:21:55.057 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:21:55.062 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:21:55.062 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:21:55.085 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5: {} 2024-06-14T23:21:55.085 INFO:teuthology.orchestra.run.smithi086.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-14T23:21:55.113 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:55.116 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:21:55.116 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:21:55.148 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:21:55.149 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:21:55.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:55.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:55.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:55.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:55 smithi086 ceph-mon[102754]: pgmap v1654: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 1.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.25k op/s; 8909/83523 objects misplaced (10.667%); 9.1 MiB/s, 145 objects/s recovering 2024-06-14T23:21:55.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:55 smithi086 ceph-mon[102754]: osdmap e463: 8 total, 8 up, 7 in 2024-06-14T23:21:55.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:55 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3969604588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:21:55.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:55 smithi086 ceph-mon[97913]: pgmap v1654: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 1.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.25k op/s; 8909/83523 objects misplaced (10.667%); 9.1 MiB/s, 145 objects/s recovering 2024-06-14T23:21:55.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:55 smithi086 ceph-mon[97913]: osdmap e463: 8 total, 8 up, 7 in 2024-06-14T23:21:55.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:55 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3969604588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:21:55.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:55.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:55.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:55.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:55.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:55.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:55 smithi137 ceph-mon[90634]: pgmap v1654: 129 pgs: 1 active+remapped+backfilling, 11 active+remapped+backfill_wait, 117 active+clean; 1.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.25k op/s; 8909/83523 objects misplaced (10.667%); 9.1 MiB/s, 145 objects/s recovering 2024-06-14T23:21:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:55 smithi137 ceph-mon[90634]: osdmap e463: 8 total, 8 up, 7 in 2024-06-14T23:21:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3969604588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:21:55.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:55.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:55.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:55.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:55.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:55.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:55.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:55.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:55.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:55.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:55.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:55.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:56.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:56.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:56.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:56.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:56.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:56.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:56.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:56 smithi086 ceph-mon[102754]: osdmap e464: 8 total, 8 up, 7 in 2024-06-14T23:21:56.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:56 smithi086 ceph-mon[97913]: osdmap e464: 8 total, 8 up, 7 in 2024-06-14T23:21:56.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5 2024-06-14T23:21:56.386 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:56 smithi137 ceph-mon[90634]: osdmap e464: 8 total, 8 up, 7 in 2024-06-14T23:21:56.392 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:56.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.5' 2024-06-14T23:21:56.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:56.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:56.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:56.571 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0, 2] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:21:56.571 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:21:56.571 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 3, 4, 5, 7, 6, 2] 2024-06-14T23:21:56.571 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-14T23:21:56.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.5 2024-06-14T23:21:56.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:56.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:56.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:56.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:21:56 smithi137 systemd[1]: Stopping Ceph osd.5 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:21:56.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:56.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:56.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:56.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:56.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:56.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:56.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:57.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:57.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:57.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:57.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:57.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:57.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:57.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:57 smithi137 ceph-mon[90634]: pgmap v1657: 129 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 10 active+remapped+backfill_wait, 117 active+clean; 1.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.47k op/s; 6495/71415 objects misplaced (9.095%); 13 MiB/s, 209 objects/s recovering 2024-06-14T23:21:57.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:57 smithi137 ceph-mon[90634]: osdmap e465: 8 total, 8 up, 7 in 2024-06-14T23:21:57.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:57.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:57.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:57.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:57 smithi086 ceph-mon[97913]: pgmap v1657: 129 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 10 active+remapped+backfill_wait, 117 active+clean; 1.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.47k op/s; 6495/71415 objects misplaced (9.095%); 13 MiB/s, 209 objects/s recovering 2024-06-14T23:21:57.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:57 smithi086 ceph-mon[97913]: osdmap e465: 8 total, 8 up, 7 in 2024-06-14T23:21:57.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:57 smithi086 ceph-mon[102754]: pgmap v1657: 129 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 10 active+remapped+backfill_wait, 117 active+clean; 1.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.47k op/s; 6495/71415 objects misplaced (9.095%); 13 MiB/s, 209 objects/s recovering 2024-06-14T23:21:57.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:57 smithi086 ceph-mon[102754]: osdmap e465: 8 total, 8 up, 7 in 2024-06-14T23:21:57.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:57.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:57.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:57.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:57.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:57.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:57.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:57.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:57.710 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:21:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[44686]: 2024-06-14T23:21:57.586+0000 7f5c44610640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:21:57.710 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:21:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[44686]: 2024-06-14T23:21:57.586+0000 7f5c44610640 -1 osd.5 466 *** Got signal Terminated *** 2024-06-14T23:21:57.710 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:21:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[44686]: 2024-06-14T23:21:57.586+0000 7f5c44610640 -1 osd.5 466 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:21:57.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:57.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:57.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:57.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:57.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:57.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:58.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:58.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:58.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:58.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:58.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:58.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:58.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[97913]: osdmap e466: 8 total, 8 up, 7 in 2024-06-14T23:21:58.226 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:58.226 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[97913]: osd.5 marked itself down and dead 2024-06-14T23:21:58.226 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[97913]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T23:21:58.226 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[97913]: osdmap e467: 8 total, 7 up, 7 in 2024-06-14T23:21:58.226 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[102754]: osdmap e466: 8 total, 8 up, 7 in 2024-06-14T23:21:58.226 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:58.226 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[102754]: osd.5 marked itself down and dead 2024-06-14T23:21:58.226 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[102754]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T23:21:58.226 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:58 smithi086 ceph-mon[102754]: osdmap e467: 8 total, 7 up, 7 in 2024-06-14T23:21:58.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:58.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:58.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:58.320 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:21:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:21:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-06-14T23:21:58.321 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:21:58.321 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:58 smithi137 ceph-mon[90634]: osdmap e466: 8 total, 8 up, 7 in 2024-06-14T23:21:58.321 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:21:58.321 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:58 smithi137 ceph-mon[90634]: osd.5 marked itself down and dead 2024-06-14T23:21:58.321 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:58 smithi137 ceph-mon[90634]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-14T23:21:58.321 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:58 smithi137 ceph-mon[90634]: osdmap e467: 8 total, 7 up, 7 in 2024-06-14T23:21:58.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:58.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:58.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:58.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:21:58.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:58.548 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:58.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:58.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:58.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:58.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:58.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:21:58.806 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:58.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:21:58.828 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:21:58.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:58.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:58.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:58.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-14T23:21:59.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:21:59.054 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:59.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:21:59.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:21:59.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:59.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:59.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:21:59.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:59 smithi086 ceph-mon[97913]: pgmap v1660: 129 pgs: 10 active+remapped+backfill_wait, 119 active+clean; 1.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.44k op/s; 5928/66267 objects misplaced (8.946%); 19 MiB/s, 297 objects/s recovering 2024-06-14T23:21:59.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:59 smithi086 ceph-mon[97913]: osdmap e468: 8 total, 7 up, 7 in 2024-06-14T23:21:59.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:59 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1067456064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:21:59.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:21:59 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:21:59.212 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:59 smithi086 ceph-mon[102754]: pgmap v1660: 129 pgs: 10 active+remapped+backfill_wait, 119 active+clean; 1.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.44k op/s; 5928/66267 objects misplaced (8.946%); 19 MiB/s, 297 objects/s recovering 2024-06-14T23:21:59.212 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:59 smithi086 ceph-mon[102754]: osdmap e468: 8 total, 7 up, 7 in 2024-06-14T23:21:59.212 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:59 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1067456064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:21:59.212 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:21:59 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:21:59.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:59 smithi137 ceph-mon[90634]: pgmap v1660: 129 pgs: 10 active+remapped+backfill_wait, 119 active+clean; 1.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.44k op/s; 5928/66267 objects misplaced (8.946%); 19 MiB/s, 297 objects/s recovering 2024-06-14T23:21:59.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:59 smithi137 ceph-mon[90634]: osdmap e468: 8 total, 7 up, 7 in 2024-06-14T23:21:59.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:59 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1067456064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:21:59.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:21:59 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:21:59.247 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:21:58 smithi137 podman[234927]: 2024-06-14 23:21:58.978343127 +0000 UTC m=+1.510729065 container died e01c2182297f8bec57e9d0235fa4d3085a1a2b308f9c7bbf89228f85ce798228 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b) 2024-06-14T23:21:59.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:21:59.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:59.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:21:59.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:21:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:21:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:59.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:21:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:21:59.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:59.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:59.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:21:59.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:59.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:59.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:59.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:21:59 smithi137 podman[234927]: 2024-06-14 23:21:59.416736022 +0000 UTC m=+1.949121963 container cleanup e01c2182297f8bec57e9d0235fa4d3085a1a2b308f9c7bbf89228f85ce798228 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T23:21:59.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:21:59 smithi137 bash[234927]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5 2024-06-14T23:21:59.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:21:59.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:59.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:21:59.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:21:59.960 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:21:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:21:59.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:21:59.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:00.069 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:21:59 smithi137 podman[235059]: 2024-06-14 23:21:59.768771013 +0000 UTC m=+1.008293497 container remove e01c2182297f8bec57e9d0235fa4d3085a1a2b308f9c7bbf89228f85ce798228 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612) 2024-06-14T23:22:00.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:00.116 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:00.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:00.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:00.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:00.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[97913]: 14.10 continuing backfill to osd.2 from (428'4620,466'10979] MIN to 466'10979 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[97913]: 14.8 continuing backfill to osd.2 from (428'4584,466'10930] MIN to 466'10930 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[97913]: 14.0 continuing backfill to osd.2 from (428'4620,466'10997] MIN to 466'10997 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[97913]: 14.18 continuing backfill to osd.2 from (428'4584,466'10902] MIN to 466'10902 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[97913]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[97913]: osdmap e469: 8 total, 7 up, 7 in 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[102754]: 14.10 continuing backfill to osd.2 from (428'4620,466'10979] MIN to 466'10979 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[102754]: 14.8 continuing backfill to osd.2 from (428'4584,466'10930] MIN to 466'10930 2024-06-14T23:22:00.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[102754]: 14.0 continuing backfill to osd.2 from (428'4620,466'10997] MIN to 466'10997 2024-06-14T23:22:00.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[102754]: 14.18 continuing backfill to osd.2 from (428'4584,466'10902] MIN to 466'10902 2024-06-14T23:22:00.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[102754]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:22:00.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:22:00.339 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:00 smithi086 ceph-mon[102754]: osdmap e469: 8 total, 7 up, 7 in 2024-06-14T23:22:00.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:00.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:00.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:00.459 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:00 smithi137 ceph-mon[90634]: 14.10 continuing backfill to osd.2 from (428'4620,466'10979] MIN to 466'10979 2024-06-14T23:22:00.459 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:00 smithi137 ceph-mon[90634]: 14.8 continuing backfill to osd.2 from (428'4584,466'10930] MIN to 466'10930 2024-06-14T23:22:00.459 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:00 smithi137 ceph-mon[90634]: 14.0 continuing backfill to osd.2 from (428'4620,466'10997] MIN to 466'10997 2024-06-14T23:22:00.459 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:00 smithi137 ceph-mon[90634]: 14.18 continuing backfill to osd.2 from (428'4584,466'10902] MIN to 466'10902 2024-06-14T23:22:00.459 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:00 smithi137 ceph-mon[90634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:22:00.459 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:00 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:22:00.460 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:00 smithi137 ceph-mon[90634]: osdmap e469: 8 total, 7 up, 7 in 2024-06-14T23:22:00.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:00.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:00.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:00.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:00.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:00.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:00.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:00.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:00.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:00.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:00.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:00.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:01.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:01.073 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:01.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:01.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:01.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:01.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:01.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:01 smithi137 ceph-mon[90634]: pgmap v1664: 129 pgs: 1 remapped+peering, 1 stale+active+clean, 11 active+undersized, 9 active+remapped+backfill_wait, 4 active+undersized+degraded, 103 active+clean; 1.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 265 B/s rd, 0 B/s wr, 576 op/s; 2648/64758 objects degraded (4.089%); 5730/64758 objects misplaced (8.848%); 11 MiB/s, 178 objects/s recovering 2024-06-14T23:22:01.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:01 smithi137 ceph-mon[90634]: Health check failed: Degraded data redundancy: 2648/64758 objects degraded (4.089%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:01.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:01.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:01 smithi086 ceph-mon[97913]: pgmap v1664: 129 pgs: 1 remapped+peering, 1 stale+active+clean, 11 active+undersized, 9 active+remapped+backfill_wait, 4 active+undersized+degraded, 103 active+clean; 1.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 265 B/s rd, 0 B/s wr, 576 op/s; 2648/64758 objects degraded (4.089%); 5730/64758 objects misplaced (8.848%); 11 MiB/s, 178 objects/s recovering 2024-06-14T23:22:01.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:01 smithi086 ceph-mon[97913]: Health check failed: Degraded data redundancy: 2648/64758 objects degraded (4.089%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:01.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:01 smithi086 ceph-mon[102754]: pgmap v1664: 129 pgs: 1 remapped+peering, 1 stale+active+clean, 11 active+undersized, 9 active+remapped+backfill_wait, 4 active+undersized+degraded, 103 active+clean; 1.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 265 B/s rd, 0 B/s wr, 576 op/s; 2648/64758 objects degraded (4.089%); 5730/64758 objects misplaced (8.848%); 11 MiB/s, 178 objects/s recovering 2024-06-14T23:22:01.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:01 smithi086 ceph-mon[102754]: Health check failed: Degraded data redundancy: 2648/64758 objects degraded (4.089%), 4 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:01.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:01.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:01.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:01.481 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:01.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:01.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:01.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:01.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:01.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:01.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:01.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:01.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:01.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:01.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:02.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:02.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:02.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:02.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:02.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:02.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:02.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:02.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:02.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:02 smithi086 ceph-mon[97913]: 14.5 scrub starts 2024-06-14T23:22:02.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:02 smithi086 ceph-mon[102754]: 14.5 scrub starts 2024-06-14T23:22:02.353 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:02 smithi137 ceph-mon[90634]: 14.5 scrub starts 2024-06-14T23:22:02.353 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:02 smithi137 podman[235477]: 2024-06-14 23:22:02.077607327 +0000 UTC m=+0.033863247 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:22:02.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:02.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:02.434 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:02.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:02.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:02.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:02.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:02.588 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:22:02.588 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:22:02.588 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-06-14T23:22:02.589 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-06-14T23:22:02.589 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:22:02.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:02.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:02.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:02.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:02 smithi137 podman[235477]: 2024-06-14 23:22:02.349845776 +0000 UTC m=+0.306101679 container create ea6acaee2c863b21fe9ac97e82db4b7743021c953538c9bb1a4c1f44d045b161 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-14T23:22:02.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:02.849 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:02.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:02.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:02.973 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:02.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:03.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:03.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:03.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:03.236 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:03 smithi137 ceph-mon[90634]: pgmap v1665: 129 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_wait, 1 remapped+peering, 18 active+undersized, 3 active+remapped+backfill_wait, 6 active+undersized+degraded, 95 active+clean; 1.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 211 B/s rd, 0 B/s wr, 676 op/s; 5949/61569 objects degraded (9.662%); 3846/61569 objects misplaced (6.247%); 30 MiB/s, 484 objects/s recovering 2024-06-14T23:22:03.236 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:03 smithi137 ceph-mon[90634]: 1.c scrub starts 2024-06-14T23:22:03.237 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:02 smithi137 podman[235477]: 2024-06-14 23:22:02.963689319 +0000 UTC m=+0.919945224 container init ea6acaee2c863b21fe9ac97e82db4b7743021c953538c9bb1a4c1f44d045b161 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-06-14T23:22:03.237 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:02 smithi137 podman[235477]: 2024-06-14 23:22:02.970681412 +0000 UTC m=+0.926937319 container start ea6acaee2c863b21fe9ac97e82db4b7743021c953538c9bb1a4c1f44d045b161 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-06-14T23:22:03.237 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:03 smithi137 podman[235477]: 2024-06-14 23:22:03.105645987 +0000 UTC m=+1.061901906 container attach ea6acaee2c863b21fe9ac97e82db4b7743021c953538c9bb1a4c1f44d045b161 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T23:22:03.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:03.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:03.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:03 smithi086 ceph-mon[97913]: pgmap v1665: 129 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_wait, 1 remapped+peering, 18 active+undersized, 3 active+remapped+backfill_wait, 6 active+undersized+degraded, 95 active+clean; 1.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 211 B/s rd, 0 B/s wr, 676 op/s; 5949/61569 objects degraded (9.662%); 3846/61569 objects misplaced (6.247%); 30 MiB/s, 484 objects/s recovering 2024-06-14T23:22:03.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:03 smithi086 ceph-mon[97913]: 1.c scrub starts 2024-06-14T23:22:03.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:03 smithi086 ceph-mon[102754]: pgmap v1665: 129 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_wait, 1 remapped+peering, 18 active+undersized, 3 active+remapped+backfill_wait, 6 active+undersized+degraded, 95 active+clean; 1.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 211 B/s rd, 0 B/s wr, 676 op/s; 5949/61569 objects degraded (9.662%); 3846/61569 objects misplaced (6.247%); 30 MiB/s, 484 objects/s recovering 2024-06-14T23:22:03.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:03 smithi086 ceph-mon[102754]: 1.c scrub starts 2024-06-14T23:22:03.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:03.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:03.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:03.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:03.520 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:03.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:03.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:03.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:03.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:03.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:03.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:03.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-14T23:22:03.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:03.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:03.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:03.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:03.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:03.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:03.996 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:03 smithi137 podman[235477]: 2024-06-14 23:22:03.6272472 +0000 UTC m=+1.583503103 container died ea6acaee2c863b21fe9ac97e82db4b7743021c953538c9bb1a4c1f44d045b161 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240612) 2024-06-14T23:22:04.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:04.024 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:04.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:04.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:04.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:04.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:04.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:04.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:04.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:04.317 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:04 smithi137 ceph-mon[90634]: osdmap e470: 8 total, 7 up, 7 in 2024-06-14T23:22:04.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.5.service' 2024-06-14T23:22:04.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:04.406 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:22:04.406 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-14T23:22:04.411 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:04.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:04.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:04.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:04.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:04.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:04.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:04 smithi086 ceph-mon[97913]: osdmap e470: 8 total, 7 up, 7 in 2024-06-14T23:22:04.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:04 smithi086 ceph-mon[102754]: osdmap e470: 8 total, 7 up, 7 in 2024-06-14T23:22:04.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:04.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:04.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:04.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:04.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:04.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:04.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:04.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:04.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:05.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:05.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:05.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:05.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:05.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:05 smithi137 ceph-mon[90634]: pgmap v1667: 129 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_wait, 1 remapped+peering, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 59 active+clean; 1.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 175 B/s rd, 0 B/s wr, 678 op/s; 7207/54360 objects degraded (13.258%); 3846/54360 objects misplaced (7.075%); 20 MiB/s, 316 objects/s recovering 2024-06-14T23:22:05.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:05 smithi137 ceph-mon[90634]: osdmap e471: 8 total, 7 up, 7 in 2024-06-14T23:22:05.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:05 smithi137 ceph-mon[90634]: 14.15 scrub starts 2024-06-14T23:22:05.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:05.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:05 smithi086 ceph-mon[97913]: pgmap v1667: 129 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_wait, 1 remapped+peering, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 59 active+clean; 1.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 175 B/s rd, 0 B/s wr, 678 op/s; 7207/54360 objects degraded (13.258%); 3846/54360 objects misplaced (7.075%); 20 MiB/s, 316 objects/s recovering 2024-06-14T23:22:05.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:05 smithi086 ceph-mon[97913]: osdmap e471: 8 total, 7 up, 7 in 2024-06-14T23:22:05.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:05 smithi086 ceph-mon[97913]: 14.15 scrub starts 2024-06-14T23:22:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:05 smithi086 ceph-mon[102754]: pgmap v1667: 129 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 3 active+undersized+degraded+remapped+backfill_wait, 1 remapped+peering, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 59 active+clean; 1.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 175 B/s rd, 0 B/s wr, 678 op/s; 7207/54360 objects degraded (13.258%); 3846/54360 objects misplaced (7.075%); 20 MiB/s, 316 objects/s recovering 2024-06-14T23:22:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:05 smithi086 ceph-mon[102754]: osdmap e471: 8 total, 7 up, 7 in 2024-06-14T23:22:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:05 smithi086 ceph-mon[102754]: 14.15 scrub starts 2024-06-14T23:22:05.338 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:22:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:22:04] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.43.0" 2024-06-14T23:22:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:05.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:05.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:05.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:05.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:05.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:05.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:05.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:05.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:05.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:05.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:05.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:05.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:05.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:06.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:06.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:06.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:06.115 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:06.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:06.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:06.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:06.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:06.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:06.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:06.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:06.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:06.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:06 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 5820/45300 objects degraded (12.848%), 18 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:06.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:06 smithi137 ceph-mon[90634]: osdmap e472: 8 total, 7 up, 7 in 2024-06-14T23:22:06.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:06.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:06.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:06.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:06 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 5820/45300 objects degraded (12.848%), 18 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:06.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:06 smithi086 ceph-mon[97913]: osdmap e472: 8 total, 7 up, 7 in 2024-06-14T23:22:06.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:06 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 5820/45300 objects degraded (12.848%), 18 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:06.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:06 smithi086 ceph-mon[102754]: osdmap e472: 8 total, 7 up, 7 in 2024-06-14T23:22:06.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:06.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:06.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:06.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:06.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:06.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:06.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:06.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:06.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:22:07.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:07.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:07.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:07.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:07.212 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:07 smithi137 ceph-mon[90634]: pgmap v1669: 129 pgs: 2 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 59 active+clean; 943 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 5820/45300 objects degraded (12.848%); 3543/45300 objects misplaced (7.821%); 20 MiB/s, 326 objects/s recovering 2024-06-14T23:22:07.212 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:07 smithi137 ceph-mon[90634]: osdmap e473: 8 total, 7 up, 7 in 2024-06-14T23:22:07.245 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:185279 12.2 12:45f8f1ef:::benchmark_data_smithi086_261492_object68774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:04.083409+0000", 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 543.15662668200002, 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.105249811, 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.246 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 185279 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.083409+0000", 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.083409+0000", 2024-06-14T23:22:07.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.083408+0000", 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.083415+0000", 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2570000000000001e-06 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.083431+0000", 2024-06-14T23:22:07.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6274e-05 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.083437+0000", 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2360000000000001e-06 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.083572+0000", 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000134735 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.083856+0000", 2024-06-14T23:22:07.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028388399999999998 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.084261+0000", 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040506400000000001 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.087589+0000", 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033284 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.088286+0000", 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069620999999999995 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.187275+0000", 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.098989141000000003 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.188534+0000", 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001259359 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.188659+0000", 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012475499999999999 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46469.0:185288 12.e 12:74ea73c2:::benchmark_data_smithi086_261492_object68783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:13:04.096389+0000", 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 543.14364684600002, 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.091968396999999993, 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.252 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46469", 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/3068301209", 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 185288 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.096389+0000", 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.096389+0000", 2024-06-14T23:22:07.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.096388+0000", 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.096396+0000", 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2660000000000001e-06 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.096409+0000", 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3027000000000001e-05 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.096414+0000", 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.972e-06 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.096455+0000", 2024-06-14T23:22:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1273999999999998e-05 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.096536+0000", 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0999000000000002e-05 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.096606+0000", 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0263000000000001e-05 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.098829+0000", 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022228090000000001 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.099161+0000", 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033130200000000001 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.188213+0000", 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.089052082000000005 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.188265+0000", 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2512999999999997e-05 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:13:04.188357+0000", 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2112000000000002e-05 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.258 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:156545 13.a 13:5a2f2ee9:::benchmark_data_smithi086_299538_object35707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:16:49.304963+0000", 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 317.93507260199999, 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.043857119999999999, 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 156545 2024-06-14T23:22:07.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.260 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.304963+0000", 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.304963+0000", 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.304962+0000", 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.304968+0000", 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3380000000000001e-06 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.304980+0000", 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152e-05 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.304986+0000", 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3749999999999999e-06 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.315364+0000", 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010377478000000001 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.318702+0000", 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033385260000000001 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3,7", 2024-06-14T23:22:07.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.318832+0000", 2024-06-14T23:22:07.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013039099999999999 2024-06-14T23:22:07.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.325752+0000", 2024-06-14T23:22:07.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0069192580000000002 2024-06-14T23:22:07.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.325780+0000", 2024-06-14T23:22:07.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8223999999999999e-05 2024-06-14T23:22:07.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.325813+0000", 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3349999999999997e-05 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.348613+0000", 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022799532000000001 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.348677+0000", 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4455999999999994e-05 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:49.348820+0000", 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014296 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.267 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:165433 13.a 13:5517798b:::benchmark_data_smithi086_299538_object44595:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:16:57.546486+0000", 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 309.69354965399998, 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.069017152999999998, 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 165433 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546486+0000", 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546486+0000", 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546484+0000", 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546497+0000", 2024-06-14T23:22:07.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2853e-05 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546510+0000", 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3135999999999999e-05 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546517+0000", 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4649999999999999e-06 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546589+0000", 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2185000000000001e-05 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546715+0000", 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012567700000000001 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3,7", 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546834+0000", 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011968099999999999 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.550277+0000", 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034427889999999999 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.550285+0000", 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1669999999999999e-06 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.555179+0000", 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048937590000000001 2024-06-14T23:22:07.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.615354+0000", 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.060174932 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.615400+0000", 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6232000000000001e-05 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.615503+0000", 2024-06-14T23:22:07.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103058 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:165434 13.a 13:5c67633d:::benchmark_data_smithi086_299538_object44596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:16:57.546530+0000", 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 309.69350606299997, 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.071240200000000004, 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:22:07.274 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 165434 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546530+0000", 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546530+0000", 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546529+0000", 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546532+0000", 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1769999999999998e-06 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546540+0000", 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2790000000000001e-06 2024-06-14T23:22:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.546544+0000", 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4830000000000001e-06 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.550319+0000", 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037747190000000002 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.550479+0000", 2024-06-14T23:22:07.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016071099999999999 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3,7", 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.550638+0000", 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015846299999999999 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.553322+0000", 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002684503 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.555219+0000", 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018967249999999999 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.555248+0000", 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8954000000000001e-05 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.617614+0000", 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.062365444999999999 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.617670+0000", 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6700000000000003e-05 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:57.617770+0000", 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9617999999999994e-05 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:167197 13.a 13:5c556976:::benchmark_data_smithi086_299538_object46359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-06-14T23:22:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:16:59.209244+0000", 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 308.0307914, 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.068815229000000006, 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 167197 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.209244+0000", 2024-06-14T23:22:07.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.209244+0000", 2024-06-14T23:22:07.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.209243+0000", 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.209250+0000", 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6720000000000004e-06 2024-06-14T23:22:07.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.209263+0000", 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2639e-05 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.209269+0000", 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5799999999999999e-06 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.211092+0000", 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001823781 2024-06-14T23:22:07.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.211375+0000", 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028246400000000001 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3,7", 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.211772+0000", 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039669 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.218249+0000", 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006477394 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.218282+0000", 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3290999999999997e-05 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.222476+0000", 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041939389999999998 2024-06-14T23:22:07.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.277883+0000", 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.055407310000000001 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.277947+0000", 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3979000000000005e-05 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.278060+0000", 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011212100000000001 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:167203 13.a 13:58b054d6:::benchmark_data_smithi086_299538_object46365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:16:59.213234+0000", 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 308.02680142299999, 2024-06-14T23:22:07.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.069234902000000001, 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 167203 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.213234+0000", 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.213234+0000", 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.213233+0000", 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.213242+0000", 2024-06-14T23:22:07.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7649999999999999e-06 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.213258+0000", 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6436999999999999e-05 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.213264+0000", 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7409999999999998e-06 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.218311+0000", 2024-06-14T23:22:07.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050468889999999997 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.218396+0000", 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4958999999999998e-05 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3,7", 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.218532+0000", 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000136799 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.221352+0000", 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028200489999999998 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.221386+0000", 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3905999999999997e-05 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.222504+0000", 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011181959999999999 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.282330+0000", 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059825362999999999 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.282359+0000", 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8700999999999998e-05 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.282469+0000", 2024-06-14T23:22:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110646 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.46844.0:167214 13.a 13:59986563:::benchmark_data_smithi086_299538_object46376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:16:59.226185+0000", 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 308.01385120999998, 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.056588762000000001, 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.46844", 2024-06-14T23:22:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/355209090", 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 167214 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.226185+0000", 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.226185+0000", 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.226183+0000", 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.226190+0000", 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9779999999999999e-06 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.226203+0000", 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2901000000000001e-05 2024-06-14T23:22:07.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.226214+0000", 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1358e-05 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.226364+0000", 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015042300000000001 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.226446+0000", 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2095000000000006e-05 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,3,7", 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.226529+0000", 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2560999999999998e-05 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.228669+0000", 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021395429999999998 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.229309+0000", 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064076200000000002 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.229332+0000", 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2483000000000001e-05 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.282508+0000", 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.053176270999999997 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.282557+0000", 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8931999999999998e-05 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:16:59.282773+0000", 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000216295 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.37287.0:75641 14.7 e428/411 14:ecf9834a:::benchmark_data_smithi086_333440_object75640:head v 428'4759, mlcod=428'4757)", 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:50.231815+0000", 2024-06-14T23:22:07.301 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 137.008220408, 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047033705000000002, 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.231815+0000", 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.231815+0000", 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.231815+0000", 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.231836+0000", 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1058999999999999e-05 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.231851+0000", 2024-06-14T23:22:07.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5067e-05 2024-06-14T23:22:07.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.231856+0000", 2024-06-14T23:22:07.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:07.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:07 smithi086 ceph-mon[97913]: pgmap v1669: 129 pgs: 2 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 59 active+clean; 943 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 5820/45300 objects degraded (12.848%); 3543/45300 objects misplaced (7.821%); 20 MiB/s, 326 objects/s recovering 2024-06-14T23:22:07.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:07 smithi086 ceph-mon[97913]: osdmap e473: 8 total, 7 up, 7 in 2024-06-14T23:22:07.305 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:07 smithi086 ceph-mon[102754]: pgmap v1669: 129 pgs: 2 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 59 active+clean; 943 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 5820/45300 objects degraded (12.848%); 3543/45300 objects misplaced (7.821%); 20 MiB/s, 326 objects/s recovering 2024-06-14T23:22:07.305 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:07 smithi086 ceph-mon[102754]: osdmap e473: 8 total, 7 up, 7 in 2024-06-14T23:22:07.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5979999999999996e-06 2024-06-14T23:22:07.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.234383+0000", 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025264699999999998 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.234404+0000", 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0894e-05 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.278804+0000", 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.044400556000000001 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:50.278849+0000", 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4774000000000002e-05 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.37287.0:159228 14.5 e447/411 14:a9cd1d7b:::benchmark_data_smithi086_333440_object50438:head v 447'10084, mlcod=447'10082)", 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:22.199369+0000", 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 45.040666856000001, 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.242772988, 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:22:07.308 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.199369+0000", 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.199369+0000", 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.199368+0000", 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.199373+0000", 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5129999999999998e-06 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.199380+0000", 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2690000000000002e-06 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.199385+0000", 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8890000000000001e-06 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202125+0000", 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002739918 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202196+0000", 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1023000000000001e-05 2024-06-14T23:22:07.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.442142+0000", 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23994584199999999 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.37287.0:159223 14.d e447/411 14:babb6c7a:::benchmark_data_smithi086_333440_object50433:head v 447'9993, mlcod=447'9990)", 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:22.202499+0000", 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 45.037537135999997, 2024-06-14T23:22:07.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.24229335199999999, 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202499+0000", 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202499+0000", 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202497+0000", 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202505+0000", 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7030000000000006e-06 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202520+0000", 2024-06-14T23:22:07.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5318000000000001e-05 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202532+0000", 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1263999999999999e-05 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.206205+0000", 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003673277 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.206249+0000", 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4409999999999997e-05 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.444686+0000", 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.238436964 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.444792+0000", 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105719 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.316 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.317 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.37287.0:159230 14.d e447/411 14:b6cafc14:::benchmark_data_smithi086_333440_object50440:head v 447'9994, mlcod=447'9990)", 2024-06-14T23:22:07.317 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:22.202551+0000", 2024-06-14T23:22:07.317 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 45.037484741, 2024-06-14T23:22:07.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.24232626199999999, 2024-06-14T23:22:07.317 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202551+0000", 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202551+0000", 2024-06-14T23:22:07.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202551+0000", 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967296 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202554+0000", 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.704e-06 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202562+0000", 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6190000000000002e-06 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.202566+0000", 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8530000000000002e-06 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.222351+0000", 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019784759999999998 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.222406+0000", 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5123000000000001e-05 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.444796+0000", 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.222390375 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.444877+0000", 2024-06-14T23:22:07.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1007000000000006e-05 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.37287.0:159235 14.7 e447/411 14:e635b3ee:::benchmark_data_smithi086_333440_object50445:head v 447'10065, mlcod=447'10062)", 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:22.205655+0000", 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 45.034380882999997, 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.25975379799999998, 2024-06-14T23:22:07.322 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.205655+0000", 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.205655+0000", 2024-06-14T23:22:07.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.205653+0000", 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.205665+0000", 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1603999999999999e-05 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.205676+0000", 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0485999999999999e-05 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.205683+0000", 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0140000000000001e-06 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.205749+0000", 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6307000000000001e-05 2024-06-14T23:22:07.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.205771+0000", 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2276000000000002e-05 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.465409+0000", 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.25963749400000002 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.37287.0:159236 14.7 e447/411)", 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:22.224963+0000", 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 45.015072291000003, 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.24020540600000001, 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.224963+0000", 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.224963+0000", 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.224962+0000", 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.224974+0000", 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2529e-05 2024-06-14T23:22:07.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.224997+0000", 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2493e-05 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.225004+0000", 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.79e-06 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.465149+0000", 2024-06-14T23:22:07.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.24014494 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:22.465169+0000", 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9395e-05 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:198565 14.7 14:e0fc211b:::benchmark_data_smithi086_333440_object89775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:22:00.786840+0000", 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 6.4531955290000003, 2024-06-14T23:22:07.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.044775115999999997, 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 198565 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.786840+0000", 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.786840+0000", 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.786839+0000", 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.786844+0000", 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0300000000000001e-06 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.786854+0000", 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0115000000000001e-05 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.786859+0000", 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5469999999999998e-06 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.786902+0000", 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3486999999999999e-05 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.786992+0000", 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9876000000000005e-05 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0", 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.787307+0000", 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031519600000000001 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.790096+0000", 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002788507 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.831507+0000", 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041411627999999999 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.831538+0000", 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0397000000000001e-05 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.831615+0000", 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7478000000000006e-05 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:198578 14.17 14:ef9c89a6:::benchmark_data_smithi086_333440_object89788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:22:00.791811+0000", 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 6.4482242510000001, 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.044517656000000003, 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 198578 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.336 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.791811+0000", 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.791811+0000", 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.791810+0000", 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.791816+0000", 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0419999999999999e-06 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.791830+0000", 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.358e-05 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.791835+0000", 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.665e-06 2024-06-14T23:22:07.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.791886+0000", 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0853999999999998e-05 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.792000+0000", 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114177 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0", 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.792322+0000", 2024-06-14T23:22:07.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032204400000000001 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.794191+0000", 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018689799999999999 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.836192+0000", 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.042000682999999997 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.836241+0000", 2024-06-14T23:22:07.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8542999999999999e-05 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:00.836329+0000", 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8566999999999994e-05 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:201629 14.17 14:e940f13b:::benchmark_data_smithi086_333440_object92839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:22:02.914779+0000", 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4.3252566110000004, 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.053441713000000002, 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.341 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 201629 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.914779+0000", 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.914779+0000", 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.914778+0000", 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.914784+0000", 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.426e-06 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.914796+0000", 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2208000000000001e-05 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.914802+0000", 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6799999999999998e-06 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.914853+0000", 2024-06-14T23:22:07.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1082000000000003e-05 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.914999+0000", 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014591199999999999 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0", 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.915331+0000", 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033215399999999998 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.918222+0000", 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028909209999999999 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.967759+0000", 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.049536919999999998 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.968124+0000", 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036508299999999999 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.968221+0000", 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6617e-05 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:201644 14.17 14:ea3b0696:::benchmark_data_smithi086_333440_object92854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:22:02.926537+0000", 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4.3134983890000003, 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046055916000000002, 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.347 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 201644 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.926537+0000", 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.926537+0000", 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.926537+0000", 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.926540+0000", 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5329999999999999e-06 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.926550+0000", 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.029e-05 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.926554+0000", 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1189999999999999e-06 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.926593+0000", 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8581000000000002e-05 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.926673+0000", 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0291999999999996e-05 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0", 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.926940+0000", 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000266712 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.928090+0000", 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001149813 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.972425+0000", 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.044335499 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.972509+0000", 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3605999999999999e-05 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:02.972593+0000", 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4250000000000001e-05 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:203181 14.7 14:e0a388e7:::benchmark_data_smithi086_333440_object94391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-06-14T23:22:07.352 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:22:04.565117+0000", 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.6749189480000002, 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.077788311999999998, 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 203181 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565117+0000", 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565117+0000", 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565116+0000", 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565121+0000", 2024-06-14T23:22:07.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2440000000000001e-06 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565133+0000", 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2254e-05 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565140+0000", 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6510000000000004e-06 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565190+0000", 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0253000000000001e-05 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565316+0000", 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012595 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0", 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565777+0000", 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046134700000000003 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.568137+0000", 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023595790000000001 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.642726+0000", 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074589235000000004 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.642776+0000", 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0118999999999999e-05 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.642905+0000", 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000128837 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:203188 14.17 14:e89daeef:::benchmark_data_smithi086_333440_object94398:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:22:04.565990+0000", 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.6740453, 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.079992898000000007, 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:07.358 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 203188 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565990+0000", 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565990+0000", 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:07.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565989+0000", 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.565995+0000", 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3199999999999999e-06 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.566006+0000", 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1674e-05 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.566012+0000", 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8830000000000003e-06 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.566154+0000", 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014156699999999999 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.566259+0000", 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010507 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0", 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.566581+0000", 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032155600000000002 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.568270+0000", 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001689664 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.645850+0000", 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.077579314999999996 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.645901+0000", 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1728999999999999e-05 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:22:04.645983+0000", 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2070000000000005e-05 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:07.363 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:22:07.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:07.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:07.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:07.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:07.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:07.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:07.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:07.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:07.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:07.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:07.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:07.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:07.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:07.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:07.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:08.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:08.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:08.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:08.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:08.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:08.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:08.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:08.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:08.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-14T23:22:08.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:08.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:08.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:08.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:08.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:08.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:08.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:08.657 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:08.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:08.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:08.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:08.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:08.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:08.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:08.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:08.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:09.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:09.064 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:09.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:09.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:09.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:09.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:09.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:09.329 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:22:09.329 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:22:09.329 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-06-14T23:22:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:22:09.331 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:22:09.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:09.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:09.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:09.407 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0, 2] out_osds: [1] dead_osds: [5] live_osds: [0, 1, 3, 4, 7, 6, 2] 2024-06-14T23:22:09.407 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:22:09.407 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-06-14T23:22:09.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd in 1 2024-06-14T23:22:09.432 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:09 smithi086 ceph-mon[97913]: pgmap v1672: 129 pgs: 4 active+clean+remapped, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 60 active+clean; 818 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 4622/39318 objects degraded (11.755%); 3060/39318 objects misplaced (7.783%); 14 MiB/s, 216 objects/s recovering 2024-06-14T23:22:09.432 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:09 smithi086 ceph-mon[97913]: osdmap e474: 8 total, 7 up, 7 in 2024-06-14T23:22:09.432 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1101360889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:22:09.433 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:09 smithi086 ceph-mon[97913]: 14.5 scrub starts 2024-06-14T23:22:09.433 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:09 smithi086 ceph-mon[102754]: pgmap v1672: 129 pgs: 4 active+clean+remapped, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 60 active+clean; 818 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 4622/39318 objects degraded (11.755%); 3060/39318 objects misplaced (7.783%); 14 MiB/s, 216 objects/s recovering 2024-06-14T23:22:09.433 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:09 smithi086 ceph-mon[102754]: osdmap e474: 8 total, 7 up, 7 in 2024-06-14T23:22:09.433 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1101360889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:22:09.433 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:09 smithi086 ceph-mon[102754]: 14.5 scrub starts 2024-06-14T23:22:09.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:09.470 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:09.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:09 smithi137 ceph-mon[90634]: pgmap v1672: 129 pgs: 4 active+clean+remapped, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 60 active+clean; 818 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 4622/39318 objects degraded (11.755%); 3060/39318 objects misplaced (7.783%); 14 MiB/s, 216 objects/s recovering 2024-06-14T23:22:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:09 smithi137 ceph-mon[90634]: osdmap e474: 8 total, 7 up, 7 in 2024-06-14T23:22:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1101360889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:22:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:09 smithi137 ceph-mon[90634]: 14.5 scrub starts 2024-06-14T23:22:09.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:09.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:09.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:09.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:09.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:09.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:09.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:09.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:09.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:09.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:09.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:10.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:10.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:10.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:10.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:10.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:10.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:10 smithi137 ceph-mon[90634]: 1.1e scrub starts 2024-06-14T23:22:10.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:10 smithi137 ceph-mon[90634]: 1.1e scrub ok 2024-06-14T23:22:10.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:10 smithi137 ceph-mon[90634]: osdmap e475: 8 total, 7 up, 7 in 2024-06-14T23:22:10.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:10.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:10.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:10.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:10.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:10.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:10.573 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:22:10.573 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:22:10.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:10 smithi086 ceph-mon[97913]: 1.1e scrub starts 2024-06-14T23:22:10.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:10 smithi086 ceph-mon[97913]: 1.1e scrub ok 2024-06-14T23:22:10.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:10 smithi086 ceph-mon[97913]: osdmap e475: 8 total, 7 up, 7 in 2024-06-14T23:22:10.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:10 smithi086 ceph-mon[102754]: 1.1e scrub starts 2024-06-14T23:22:10.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:10 smithi086 ceph-mon[102754]: 1.1e scrub ok 2024-06-14T23:22:10.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:10 smithi086 ceph-mon[102754]: osdmap e475: 8 total, 7 up, 7 in 2024-06-14T23:22:10.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:10.636 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:10.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:10.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:10.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:10.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:10.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:10.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:10.914 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:10.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:11.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:11.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:11.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:11.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:11.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:11.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:11.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:11.301 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:11.337 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:11.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:11.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:11.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:11.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:11.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:11 smithi137 ceph-mon[90634]: pgmap v1675: 129 pgs: 4 active+clean+remapped, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 60 active+clean; 752 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 762 op/s; 4148/36150 objects degraded (11.474%); 2774/36150 objects misplaced (7.674%); 9.6 MiB/s, 153 objects/s recovering 2024-06-14T23:22:11.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:11 smithi137 ceph-mon[90634]: osdmap e476: 8 total, 7 up, 7 in 2024-06-14T23:22:11.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:11.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:11 smithi086 ceph-mon[97913]: pgmap v1675: 129 pgs: 4 active+clean+remapped, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 60 active+clean; 752 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 762 op/s; 4148/36150 objects degraded (11.474%); 2774/36150 objects misplaced (7.674%); 9.6 MiB/s, 153 objects/s recovering 2024-06-14T23:22:11.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:11 smithi086 ceph-mon[97913]: osdmap e476: 8 total, 7 up, 7 in 2024-06-14T23:22:11.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:11 smithi086 ceph-mon[102754]: pgmap v1675: 129 pgs: 4 active+clean+remapped, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 3 active+remapped+backfill_wait, 14 active+undersized+degraded, 60 active+clean; 752 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 762 op/s; 4148/36150 objects degraded (11.474%); 2774/36150 objects misplaced (7.674%); 9.6 MiB/s, 153 objects/s recovering 2024-06-14T23:22:11.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:11 smithi086 ceph-mon[102754]: osdmap e476: 8 total, 7 up, 7 in 2024-06-14T23:22:11.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:11.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:11.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-14T23:22:11.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:11.726 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:11.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:11.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:11.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:11.851 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:11.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:11.899 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:11.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:11.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:11.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:12.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:12.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:12.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:12.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:12.225 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:12.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:12.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:12.339 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:12 smithi137 ceph-mon[90634]: osdmap e477: 8 total, 7 up, 7 in 2024-06-14T23:22:12.339 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:12 smithi137 ceph-mon[90634]: pgmap v1678: 129 pgs: 1 active+remapped+backfilling, 4 active+clean+remapped, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 2 active+remapped+backfill_wait, 14 active+undersized+degraded, 60 active+clean; 694 MiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 495 op/s; 3896/33372 objects degraded (11.674%); 2059/33372 objects misplaced (6.170%); 3.8 MiB/s, 60 objects/s recovering 2024-06-14T23:22:12.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:12.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:12.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:12.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:12.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:12.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:12 smithi086 ceph-mon[102754]: osdmap e477: 8 total, 7 up, 7 in 2024-06-14T23:22:12.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:12 smithi086 ceph-mon[102754]: pgmap v1678: 129 pgs: 1 active+remapped+backfilling, 4 active+clean+remapped, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 2 active+remapped+backfill_wait, 14 active+undersized+degraded, 60 active+clean; 694 MiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 495 op/s; 3896/33372 objects degraded (11.674%); 2059/33372 objects misplaced (6.170%); 3.8 MiB/s, 60 objects/s recovering 2024-06-14T23:22:12.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:12 smithi086 ceph-mon[97913]: osdmap e477: 8 total, 7 up, 7 in 2024-06-14T23:22:12.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:12 smithi086 ceph-mon[97913]: pgmap v1678: 129 pgs: 1 active+remapped+backfilling, 4 active+clean+remapped, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 2 active+remapped+backfill_wait, 14 active+undersized+degraded, 60 active+clean; 694 MiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 495 op/s; 3896/33372 objects degraded (11.674%); 2059/33372 objects misplaced (6.170%); 3.8 MiB/s, 60 objects/s recovering 2024-06-14T23:22:12.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:12.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:12.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:12.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:12.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:12.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:12.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:12.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:12.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:13.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:13.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:13.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:13.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:13.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:13.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:13.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:13.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:13.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:13.429 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:13 smithi137 ceph-mon[90634]: 14.4 scrub starts 2024-06-14T23:22:13.430 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:13 smithi137 ceph-mon[90634]: 14.4 scrub ok 2024-06-14T23:22:13.430 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:13 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 3896/33372 objects degraded (11.674%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:13.430 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:13 smithi137 ceph-mon[90634]: osdmap e478: 8 total, 7 up, 7 in 2024-06-14T23:22:13.430 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:13 smithi137 ceph-mon[90634]: 14.15 scrub starts 2024-06-14T23:22:13.430 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:13.430 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:13.450 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:13.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:13.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:13.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:13.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:13.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[97913]: 14.4 scrub starts 2024-06-14T23:22:13.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[97913]: 14.4 scrub ok 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 3896/33372 objects degraded (11.674%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[97913]: osdmap e478: 8 total, 7 up, 7 in 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[97913]: 14.15 scrub starts 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[102754]: 14.4 scrub starts 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[102754]: 14.4 scrub ok 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 3896/33372 objects degraded (11.674%), 15 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[102754]: osdmap e478: 8 total, 7 up, 7 in 2024-06-14T23:22:13.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[102754]: 14.15 scrub starts 2024-06-14T23:22:13.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:13.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:13.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:13.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:13.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:13.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:13.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:13.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:13.861 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:22:13.862 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:22:13.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:13.947 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:13.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:13.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:14.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:14.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:14.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:14.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:14.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:14.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:14.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:14.320 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:14.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:14.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:14.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:14.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:14.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:14.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:14.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:14.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:14.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:14.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:14 smithi137 ceph-mon[90634]: 14.5 scrub starts 2024-06-14T23:22:14.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:14 smithi137 ceph-mon[90634]: osdmap e479: 8 total, 7 up, 7 in 2024-06-14T23:22:14.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:14 smithi137 ceph-mon[90634]: 14.5 scrub ok 2024-06-14T23:22:14.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:14 smithi137 ceph-mon[90634]: 14.13 scrub starts 2024-06-14T23:22:14.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:14 smithi137 ceph-mon[90634]: 14.13 scrub ok 2024-06-14T23:22:14.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:14 smithi137 ceph-mon[90634]: pgmap v1681: 129 pgs: 5 active+clean+remapped, 46 active+undersized, 2 active+remapped+backfill_wait, 14 active+undersized+degraded, 62 active+clean; 511 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2290/24585 objects degraded (9.315%); 2012/24585 objects misplaced (8.184%); 11 MiB/s, 173 objects/s recovering 2024-06-14T23:22:14.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:14.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[97913]: 14.5 scrub starts 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[97913]: osdmap e479: 8 total, 7 up, 7 in 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[97913]: 14.5 scrub ok 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[97913]: 14.13 scrub starts 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[97913]: 14.13 scrub ok 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[97913]: pgmap v1681: 129 pgs: 5 active+clean+remapped, 46 active+undersized, 2 active+remapped+backfill_wait, 14 active+undersized+degraded, 62 active+clean; 511 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2290/24585 objects degraded (9.315%); 2012/24585 objects misplaced (8.184%); 11 MiB/s, 173 objects/s recovering 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[102754]: 14.5 scrub starts 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[102754]: osdmap e479: 8 total, 7 up, 7 in 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[102754]: 14.5 scrub ok 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[102754]: 14.13 scrub starts 2024-06-14T23:22:14.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[102754]: 14.13 scrub ok 2024-06-14T23:22:14.839 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-mon[102754]: pgmap v1681: 129 pgs: 5 active+clean+remapped, 46 active+undersized, 2 active+remapped+backfill_wait, 14 active+undersized+degraded, 62 active+clean; 511 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2290/24585 objects degraded (9.315%); 2012/24585 objects misplaced (8.184%); 11 MiB/s, 173 objects/s recovering 2024-06-14T23:22:14.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:14.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:14.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:14.979 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:14.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:15.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-14T23:22:15.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:15.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:15.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:15.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:15.238 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:22:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:22:14] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.43.0" 2024-06-14T23:22:15.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:15.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:22:15.274 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:22:15.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:15.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:15.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:15.412 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:15 smithi137 ceph-mon[90634]: 1.9 scrub starts 2024-06-14T23:22:15.452 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:15.468 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:15.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:15.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:15.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:15.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:15 smithi086 ceph-mon[102754]: 1.9 scrub starts 2024-06-14T23:22:15.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:15 smithi086 ceph-mon[102754]: 1.9 scrub ok 2024-06-14T23:22:15.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:15 smithi086 ceph-mon[102754]: osdmap e480: 8 total, 7 up, 7 in 2024-06-14T23:22:15.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:15 smithi086 ceph-mon[97913]: 1.9 scrub starts 2024-06-14T23:22:15.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:15 smithi086 ceph-mon[97913]: 1.9 scrub ok 2024-06-14T23:22:15.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:15 smithi086 ceph-mon[97913]: osdmap e480: 8 total, 7 up, 7 in 2024-06-14T23:22:15.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:15.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:15.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:15.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:15 smithi137 ceph-mon[90634]: 1.9 scrub ok 2024-06-14T23:22:15.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:15 smithi137 ceph-mon[90634]: osdmap e480: 8 total, 7 up, 7 in 2024-06-14T23:22:15.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:15.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:15.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:15.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:15.958 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:15.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:16.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:16.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:16.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:16.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:16.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:16.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:16.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:16.371 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:16.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:16.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:16.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:16.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:16.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:16.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:16.749 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:16 smithi137 ceph-mon[90634]: 1.14 scrub starts 2024-06-14T23:22:16.749 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:16 smithi137 ceph-mon[90634]: 1.14 scrub ok 2024-06-14T23:22:16.749 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:16 smithi137 ceph-mon[90634]: pgmap v1683: 129 pgs: 5 active+clean+remapped, 46 active+undersized, 2 active+remapped+backfill_wait, 14 active+undersized+degraded, 62 active+clean; 439 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2009/21102 objects degraded (9.520%); 1394/21102 objects misplaced (6.606%) 2024-06-14T23:22:16.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:16.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:16.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:16.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:16 smithi086 ceph-mon[97913]: 1.14 scrub starts 2024-06-14T23:22:16.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:16 smithi086 ceph-mon[97913]: 1.14 scrub ok 2024-06-14T23:22:16.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:16 smithi086 ceph-mon[97913]: pgmap v1683: 129 pgs: 5 active+clean+remapped, 46 active+undersized, 2 active+remapped+backfill_wait, 14 active+undersized+degraded, 62 active+clean; 439 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2009/21102 objects degraded (9.520%); 1394/21102 objects misplaced (6.606%) 2024-06-14T23:22:16.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:16 smithi086 ceph-mon[102754]: 1.14 scrub starts 2024-06-14T23:22:16.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:16 smithi086 ceph-mon[102754]: 1.14 scrub ok 2024-06-14T23:22:16.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:16 smithi086 ceph-mon[102754]: pgmap v1683: 129 pgs: 5 active+clean+remapped, 46 active+undersized, 2 active+remapped+backfill_wait, 14 active+undersized+degraded, 62 active+clean; 439 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2009/21102 objects degraded (9.520%); 1394/21102 objects misplaced (6.606%) 2024-06-14T23:22:16.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:16.924 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:16.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:17.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:17.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:17.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:17.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:17.194 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:17.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:17.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-14T23:22:17.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:17.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:17.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:17.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:17.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:17.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:17.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:17 smithi086 ceph-mon[97913]: 14.1 scrub starts 2024-06-14T23:22:17.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:17 smithi086 ceph-mon[97913]: 14.1 scrub ok 2024-06-14T23:22:17.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:17 smithi086 ceph-mon[97913]: 14.15 scrub starts 2024-06-14T23:22:17.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:17 smithi086 ceph-mon[97913]: 14.15 scrub ok 2024-06-14T23:22:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:17 smithi086 ceph-mon[102754]: 14.1 scrub starts 2024-06-14T23:22:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:17 smithi086 ceph-mon[102754]: 14.1 scrub ok 2024-06-14T23:22:17.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:17 smithi086 ceph-mon[102754]: 14.15 scrub starts 2024-06-14T23:22:17.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:17 smithi086 ceph-mon[102754]: 14.15 scrub ok 2024-06-14T23:22:17.618 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:17.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:17.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:17 smithi137 ceph-mon[90634]: 14.1 scrub starts 2024-06-14T23:22:17.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:17 smithi137 ceph-mon[90634]: 14.1 scrub ok 2024-06-14T23:22:17.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:17 smithi137 ceph-mon[90634]: 14.15 scrub starts 2024-06-14T23:22:17.639 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:17 smithi137 ceph-mon[90634]: 14.15 scrub ok 2024-06-14T23:22:17.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:17.755 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:17.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:17.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:17.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:17.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:17.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:18.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:18.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:18.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:18.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:18.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:18.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:18.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:18.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:18.371 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:22:18.371 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:22:18.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:18.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:18.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:18.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:18.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:18.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:18.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:18.717 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:18 smithi137 ceph-mon[90634]: 14.3 scrub starts 2024-06-14T23:22:18.717 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:18 smithi137 ceph-mon[90634]: 14.3 scrub ok 2024-06-14T23:22:18.717 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:18 smithi137 ceph-mon[90634]: pgmap v1684: 129 pgs: 5 active+clean+remapped, 46 active+undersized, 14 active+undersized+degraded, 64 active+clean; 206 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1292/9921 objects degraded (13.023%); 757/9921 objects misplaced (7.630%) 2024-06-14T23:22:18.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:18.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:18.750 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Removed 108787 objects 2024-06-14T23:22:18.750 INFO:tasks.radosbench.radosbench.0.smithi086.stdout:Clean up completed and total clean up time :111.379 2024-06-14T23:22:18.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:18.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:18 smithi086 ceph-mon[102754]: 14.3 scrub starts 2024-06-14T23:22:18.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:18 smithi086 ceph-mon[102754]: 14.3 scrub ok 2024-06-14T23:22:18.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:18 smithi086 ceph-mon[102754]: pgmap v1684: 129 pgs: 5 active+clean+remapped, 46 active+undersized, 14 active+undersized+degraded, 64 active+clean; 206 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1292/9921 objects degraded (13.023%); 757/9921 objects misplaced (7.630%) 2024-06-14T23:22:18.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:18 smithi086 ceph-mon[97913]: 14.3 scrub starts 2024-06-14T23:22:18.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:18 smithi086 ceph-mon[97913]: 14.3 scrub ok 2024-06-14T23:22:18.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:18 smithi086 ceph-mon[97913]: pgmap v1684: 129 pgs: 5 active+clean+remapped, 46 active+undersized, 14 active+undersized+degraded, 64 active+clean; 206 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1292/9921 objects degraded (13.023%); 757/9921 objects misplaced (7.630%) 2024-06-14T23:22:18.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:18.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:18.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:18.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:19.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:19.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:19.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-14T23:22:19.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:19.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:19.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:19.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:19.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:19.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:19.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:19.351 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:19.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:19.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:19.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:19.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:19.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:19.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:19.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:19.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:19.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:19.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:19.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:19 smithi137 ceph-mon[90634]: 14.14 scrub starts 2024-06-14T23:22:19.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:19 smithi137 ceph-mon[90634]: 14.14 scrub ok 2024-06-14T23:22:19.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:19 smithi137 ceph-mon[90634]: 14.11 scrub starts 2024-06-14T23:22:19.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:19 smithi137 ceph-mon[90634]: 14.11 scrub ok 2024-06-14T23:22:19.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:19 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 1292/9921 objects degraded (13.023%), 14 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:19.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:19.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[97913]: 14.14 scrub starts 2024-06-14T23:22:19.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[97913]: 14.14 scrub ok 2024-06-14T23:22:19.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[97913]: 14.11 scrub starts 2024-06-14T23:22:19.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[97913]: 14.11 scrub ok 2024-06-14T23:22:19.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 1292/9921 objects degraded (13.023%), 14 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:19.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[102754]: 14.14 scrub starts 2024-06-14T23:22:19.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[102754]: 14.14 scrub ok 2024-06-14T23:22:19.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[102754]: 14.11 scrub starts 2024-06-14T23:22:19.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[102754]: 14.11 scrub ok 2024-06-14T23:22:19.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:19 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 1292/9921 objects degraded (13.023%), 14 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:19.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:19.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:19.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:19.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:19.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:19.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:20.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:20.065 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:20.090 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:20.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:20.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:20.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:20.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:20.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:20.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:20.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:20.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:20.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:20.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:20.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:20.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:20.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:20.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:20 smithi086 ceph-mon[97913]: 14.8 scrub starts 2024-06-14T23:22:20.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:20 smithi086 ceph-mon[97913]: 14.8 scrub ok 2024-06-14T23:22:20.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:20 smithi086 ceph-mon[97913]: pgmap v1685: 129 pgs: 5 active+clean+remapped, 47 active+undersized, 13 active+undersized+degraded, 64 active+clean; 111 MiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 570/5382 objects degraded (10.591%); 368/5382 objects misplaced (6.838%) 2024-06-14T23:22:20.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:20 smithi086 ceph-mon[102754]: 14.8 scrub starts 2024-06-14T23:22:20.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:20 smithi086 ceph-mon[102754]: 14.8 scrub ok 2024-06-14T23:22:20.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:20 smithi086 ceph-mon[102754]: pgmap v1685: 129 pgs: 5 active+clean+remapped, 47 active+undersized, 13 active+undersized+degraded, 64 active+clean; 111 MiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 570/5382 objects degraded (10.591%); 368/5382 objects misplaced (6.838%) 2024-06-14T23:22:20.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:20.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:20.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:20.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:20 smithi137 ceph-mon[90634]: 14.8 scrub starts 2024-06-14T23:22:20.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:20 smithi137 ceph-mon[90634]: 14.8 scrub ok 2024-06-14T23:22:20.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:20 smithi137 ceph-mon[90634]: pgmap v1685: 129 pgs: 5 active+clean+remapped, 47 active+undersized, 13 active+undersized+degraded, 64 active+clean; 111 MiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 570/5382 objects degraded (10.591%); 368/5382 objects misplaced (6.838%) 2024-06-14T23:22:20.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:20.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:20.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:20.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:20.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:21.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:21.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:21.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:21.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:21.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:21.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:21.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:21.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:21.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:21.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:21.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:21.447 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:21.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:21.452 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:22:21.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:21.571 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:21.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:21.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:21.675 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:21 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1685053068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:22:21.675 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:21 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:22:21.676 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:22:21.676 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:22:21.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:21 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1685053068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:22:21.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:21 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:22:21.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:21 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1685053068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:22:21.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:21 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:22:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:21.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:21.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:21.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:21.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:21.940 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:21.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:22.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:22.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:22.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:22.164 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:22.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:22.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:22.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:22.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-14T23:22:22.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:22.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:22.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:22.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:22.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:22.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:22.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:22.546 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:22.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:22.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:22.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:22.668 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:22 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:22:22.669 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:22 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:22:22.669 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:22 smithi137 ceph-mon[90634]: osdmap e481: 8 total, 7 up, 7 in 2024-06-14T23:22:22.669 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:22 smithi137 ceph-mon[90634]: pgmap v1687: 129 pgs: 5 active+clean+remapped, 48 active+undersized, 12 active+undersized+degraded, 64 active+clean; 80 MiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 339/3909 objects degraded (8.672%); 108/3909 objects misplaced (2.763%) 2024-06-14T23:22:22.679 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:22.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:22.745 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:22 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:22:22.745 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:22 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:22:22.746 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:22 smithi086 ceph-mon[102754]: osdmap e481: 8 total, 7 up, 7 in 2024-06-14T23:22:22.746 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:22 smithi086 ceph-mon[102754]: pgmap v1687: 129 pgs: 5 active+clean+remapped, 48 active+undersized, 12 active+undersized+degraded, 64 active+clean; 80 MiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 339/3909 objects degraded (8.672%); 108/3909 objects misplaced (2.763%) 2024-06-14T23:22:22.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:22 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-14T23:22:22.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:22 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:22:22.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:22 smithi086 ceph-mon[97913]: osdmap e481: 8 total, 7 up, 7 in 2024-06-14T23:22:22.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:22 smithi086 ceph-mon[97913]: pgmap v1687: 129 pgs: 5 active+clean+remapped, 48 active+undersized, 12 active+undersized+degraded, 64 active+clean; 80 MiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 339/3909 objects degraded (8.672%); 108/3909 objects misplaced (2.763%) 2024-06-14T23:22:22.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:22.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:22.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:22.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:22.926 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:22.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:23.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:23.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:23.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:23.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:23.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:23.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:23.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:23.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:23.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:23.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:23.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:23.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:23.451 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:23 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:22:23.141+0000 7fe0161db640 -1 mon.a@0(leader).osd e481 definitely_dead 0 2024-06-14T23:22:23.469 INFO:teuthology.orchestra.run.smithi086.stderr:marked in osd.1. 2024-06-14T23:22:23.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:23.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:23.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:23.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:23.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:23.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:23.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3081468646' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-06-14T23:22:23.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:23 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-06-14T23:22:23.755 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_9 2024-06-14T23:22:23.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool rm unique_pool_9 unique_pool_9 --yes-i-really-really-mean-it 2024-06-14T23:22:23.776 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3081468646' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-06-14T23:22:23.776 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:23 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-06-14T23:22:23.776 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3081468646' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-06-14T23:22:23.776 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:23 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-06-14T23:22:23.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:23.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:23.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:23.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:23.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:23.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:24.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:24.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:24.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:24.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:24.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:24.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:24.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:24.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:24.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:24.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:24.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:24.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:24.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd set noscrub 2024-06-14T23:22:24.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:24.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:24.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:24.647 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-06-14T23:22:24.648 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:22:24.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:24.711 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: 14.10 scrub starts 2024-06-14T23:22:24.711 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: 14.10 scrub ok 2024-06-14T23:22:24.711 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-06-14T23:22:24.711 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: osdmap e482: 8 total, 7 up, 8 in 2024-06-14T23:22:24.711 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 339/3909 objects degraded (8.672%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:24.711 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: osdmap e483: 8 total, 7 up, 8 in 2024-06-14T23:22:24.711 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: 14.f starting backfill to osd.1 from (0'0,0'0] MAX to 480'13556 2024-06-14T23:22:24.711 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: 14.13 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11624 2024-06-14T23:22:24.712 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: 14.3 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11705 2024-06-14T23:22:24.712 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[102754]: pgmap v1690: 129 pgs: 8 peering, 9 unknown, 4 remapped+peering, 5 active+clean+remapped, 37 active+undersized, 8 active+undersized+degraded, 58 active+clean; 11 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 106/579 objects degraded (18.307%); 11/579 objects misplaced (1.900%) 2024-06-14T23:22:24.712 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: 14.10 scrub starts 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: 14.10 scrub ok 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: osdmap e482: 8 total, 7 up, 8 in 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 339/3909 objects degraded (8.672%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: osdmap e483: 8 total, 7 up, 8 in 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: 14.f starting backfill to osd.1 from (0'0,0'0] MAX to 480'13556 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: 14.13 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11624 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: 14.3 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11705 2024-06-14T23:22:24.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-mon[97913]: pgmap v1690: 129 pgs: 8 peering, 9 unknown, 4 remapped+peering, 5 active+clean+remapped, 37 active+undersized, 8 active+undersized+degraded, 58 active+clean; 11 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 106/579 objects degraded (18.307%); 11/579 objects misplaced (1.900%) 2024-06-14T23:22:24.730 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:24.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:24.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: 14.10 scrub starts 2024-06-14T23:22:24.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: 14.10 scrub ok 2024-06-14T23:22:24.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-06-14T23:22:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: osdmap e482: 8 total, 7 up, 8 in 2024-06-14T23:22:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 339/3909 objects degraded (8.672%), 12 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: osdmap e483: 8 total, 7 up, 8 in 2024-06-14T23:22:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: 14.f starting backfill to osd.1 from (0'0,0'0] MAX to 480'13556 2024-06-14T23:22:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: 14.13 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11624 2024-06-14T23:22:24.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: 14.3 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11705 2024-06-14T23:22:24.748 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:24 smithi137 ceph-mon[90634]: pgmap v1690: 129 pgs: 8 peering, 9 unknown, 4 remapped+peering, 5 active+clean+remapped, 37 active+undersized, 8 active+undersized+degraded, 58 active+clean; 11 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 106/579 objects degraded (18.307%); 11/579 objects misplaced (1.900%) 2024-06-14T23:22:24.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:24.857 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:24.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:24.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:24.966 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:22:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:22:24] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.43.0" 2024-06-14T23:22:24.981 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:24.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:25.065 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:22:25.066 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:22:25.081 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:22:25.082 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:22:25.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:25.101 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:22:25.101 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:22:25.104 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:25.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:25.120 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:22:25.120 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:22:25.123 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:25.141 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:22:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:22:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-06-14T23:22:25.161 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:22:25.162 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:22:25.178 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:22:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:22:25.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:25.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:25.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:25.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:25.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:25.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:25.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-14T23:22:25.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:25.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:25.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:25.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.0 scrub starts 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.0 scrub ok 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.11 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11660 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: osdmap e484: 8 total, 7 up, 8 in 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.b starting backfill to osd.1 from (0'0,0'0] MAX to 480'13620 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.16 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11705 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.12 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11741 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.2 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11757 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.6 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11580 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.1 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11782 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.a starting backfill to osd.1 from (0'0,0'0] MAX to 480'13590 2024-06-14T23:22:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.18 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11565 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: 14.8 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11592 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2309153012' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.0 scrub starts 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.0 scrub ok 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.11 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11660 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: osdmap e484: 8 total, 7 up, 8 in 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.b starting backfill to osd.1 from (0'0,0'0] MAX to 480'13620 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.16 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11705 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.12 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11741 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.2 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11757 2024-06-14T23:22:25.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.6 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11580 2024-06-14T23:22:25.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.1 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11782 2024-06-14T23:22:25.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.a starting backfill to osd.1 from (0'0,0'0] MAX to 480'13590 2024-06-14T23:22:25.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.18 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11565 2024-06-14T23:22:25.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: 14.8 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11592 2024-06-14T23:22:25.590 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:25 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2309153012' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:22:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:25.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:25.679 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:25.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:25.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:25.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:25.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.0 scrub starts 2024-06-14T23:22:25.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.0 scrub ok 2024-06-14T23:22:25.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-06-14T23:22:25.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.11 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11660 2024-06-14T23:22:25.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: osdmap e484: 8 total, 7 up, 8 in 2024-06-14T23:22:25.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.b starting backfill to osd.1 from (0'0,0'0] MAX to 480'13620 2024-06-14T23:22:25.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.16 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11705 2024-06-14T23:22:25.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.12 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11741 2024-06-14T23:22:25.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.2 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11757 2024-06-14T23:22:25.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.6 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11580 2024-06-14T23:22:25.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.1 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11782 2024-06-14T23:22:25.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.a starting backfill to osd.1 from (0'0,0'0] MAX to 480'13590 2024-06-14T23:22:25.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.18 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11565 2024-06-14T23:22:25.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: 14.8 starting backfill to osd.1 from (0'0,0'0] MAX to 480'11592 2024-06-14T23:22:25.747 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:25 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2309153012' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:22:25.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:25.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:25.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:25.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:25.970 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:25.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:25.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:26.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:26.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:26.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:26.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:26.217 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:26.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:22:26.230 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:22:26.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:26.343 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:26.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:26.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:26.467 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:26.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:26.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:26.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:26.647 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:26.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:26.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:26.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:26.770 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:26 smithi086 ceph-mon[97913]: 1.c scrub starts 2024-06-14T23:22:26.770 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:26 smithi086 ceph-mon[97913]: 1.c scrub ok 2024-06-14T23:22:26.770 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:26 smithi086 ceph-mon[97913]: osdmap e485: 8 total, 7 up, 8 in 2024-06-14T23:22:26.770 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:26 smithi086 ceph-mon[97913]: pgmap v1693: 129 pgs: 4 active+undersized+remapped, 10 peering, 4 unknown, 8 remapped+peering, 5 active+clean+remapped, 31 active+undersized, 8 active+undersized+degraded, 59 active+clean; 12 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 106/621 objects degraded (17.069%); 11/621 objects misplaced (1.771%) 2024-06-14T23:22:26.770 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:26 smithi086 ceph-mon[102754]: 1.c scrub starts 2024-06-14T23:22:26.770 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:26 smithi086 ceph-mon[102754]: 1.c scrub ok 2024-06-14T23:22:26.770 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:26 smithi086 ceph-mon[102754]: osdmap e485: 8 total, 7 up, 8 in 2024-06-14T23:22:26.770 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:26 smithi086 ceph-mon[102754]: pgmap v1693: 129 pgs: 4 active+undersized+remapped, 10 peering, 4 unknown, 8 remapped+peering, 5 active+clean+remapped, 31 active+undersized, 8 active+undersized+degraded, 59 active+clean; 12 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 106/621 objects degraded (17.069%); 11/621 objects misplaced (1.771%) 2024-06-14T23:22:26.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:26.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:26.908 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:26 smithi137 ceph-mon[90634]: 1.c scrub starts 2024-06-14T23:22:26.908 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:26 smithi137 ceph-mon[90634]: 1.c scrub ok 2024-06-14T23:22:26.908 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:26 smithi137 ceph-mon[90634]: osdmap e485: 8 total, 7 up, 8 in 2024-06-14T23:22:26.908 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:26 smithi137 ceph-mon[90634]: pgmap v1693: 129 pgs: 4 active+undersized+remapped, 10 peering, 4 unknown, 8 remapped+peering, 5 active+clean+remapped, 31 active+undersized, 8 active+undersized+degraded, 59 active+clean; 12 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 106/621 objects degraded (17.069%); 11/621 objects misplaced (1.771%) 2024-06-14T23:22:26.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:26.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:26.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:27.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:27.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:27.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:27.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:27.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:27.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:27.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:27.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:27.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:27.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:27.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:27.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:36381 14.e 14:7c932a93:::benchmark_data_smithi086_333440_object36380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:19:09.907483+0000", 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 197.57711992700001, 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0754551530000001, 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.490 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 36381 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:09.907483+0000", 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:09.907483+0000", 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:09.907482+0000", 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:09.907535+0000", 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3097999999999999e-05 2024-06-14T23:22:27.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:09.907542+0000", 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6220000000000003e-06 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:09.907545+0000", 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4020000000000002e-06 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:09.907577+0000", 2024-06-14T23:22:27.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1384999999999999e-05 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:09.907582+0000", 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4600000000000002e-06 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:10.871091+0000", 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96350893800000004 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:10.871111+0000", 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9894999999999998e-05 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:11.977867+0000", 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.106756383 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:11.978851+0000", 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00098342900000000007 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:11.978984+0000", 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013324699999999999 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:11.981534+0000", 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025500760000000001 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:11.981683+0000", 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014860900000000001 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:11.981827+0000", 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014408300000000001 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:11.981879+0000", 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2478999999999997e-05 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:19:11.982938+0000", 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010586510000000001 2024-06-14T23:22:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103291 14.e 14:79b57b88:::benchmark_data_smithi086_333440_object103290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:18.477518+0000", 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.00708504100001, 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0447275439999999, 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.498 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103291 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.477518+0000", 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.477518+0000", 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.477517+0000", 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.477547+0000", 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9689999999999999e-05 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.477597+0000", 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0346000000000002e-05 2024-06-14T23:22:27.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.477602+0000", 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2979999999999999e-06 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.477643+0000", 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0592e-05 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.477778+0000", 2024-06-14T23:22:27.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013458599999999999 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.478157+0000", 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037995200000000002 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.482153+0000", 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039955420000000004 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.482305+0000", 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015203600000000001 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.499191+0000", 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016886102 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.499200+0000", 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2280000000000004e-06 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.522186+0000", 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.022985273 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.522245+0000", 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9416000000000002e-05 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.504 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103299 14.e 14:7a922f95:::benchmark_data_smithi086_333440_object103298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:18.487044+0000", 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.99755862399999, 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.046499252, 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103299 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.505 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.487044+0000", 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.487044+0000", 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.487043+0000", 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.487081+0000", 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7134000000000001e-05 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.487088+0000", 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9470000000000003e-06 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.487093+0000", 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7539999999999997e-06 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.503488+0000", 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016395257999999999 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.503497+0000", 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6459999999999999e-06 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.522287+0000", 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.018789814 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.522421+0000", 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013425000000000001 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.522891+0000", 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046964799999999997 2024-06-14T23:22:27.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.532844+0000", 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0099532709999999996 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533101+0000", 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025679499999999998 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533471+0000", 2024-06-14T23:22:27.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000370324 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533507+0000", 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5639000000000002e-05 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533543+0000", 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6347999999999998e-05 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103300 14.2 14:4698aeb1:::benchmark_data_smithi086_333440_object103299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:18.509540+0000", 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.97506251199999, 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0918255929999998, 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103300 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509540+0000", 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509540+0000", 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509539+0000", 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509579+0000", 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0178000000000001e-05 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509591+0000", 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2704000000000001e-05 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509599+0000", 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.221e-06 2024-06-14T23:22:27.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509661+0000", 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2168999999999994e-05 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509670+0000", 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1800000000000002e-06 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.490534+0000", 2024-06-14T23:22:27.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98086404900000002 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.490545+0000", 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0513000000000001e-05 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.567113+0000", 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0765688419999999 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.567272+0000", 2024-06-14T23:22:27.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015862100000000001 2024-06-14T23:22:27.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.517 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.517 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,5", 2024-06-14T23:22:27.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.567757+0000", 2024-06-14T23:22:27.518 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048515199999999998 2024-06-14T23:22:27.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.518 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.518 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.518 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.584558+0000", 2024-06-14T23:22:27.518 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016800691 2024-06-14T23:22:27.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.518 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.518 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601204+0000", 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016645752999999999 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601215+0000", 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.119e-05 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601255+0000", 2024-06-14T23:22:27.519 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9920999999999999e-05 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601290+0000", 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5321000000000003e-05 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601366+0000", 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5619999999999998e-05 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103293 14.1 14:80c222ee:::benchmark_data_smithi086_333440_object103292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:18.509605+0000", 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.97499796299999, 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1020400170000002, 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103293 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509605+0000", 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509605+0000", 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509604+0000", 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:22:27.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509638+0000", 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3730999999999999e-05 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509645+0000", 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7190000000000004e-06 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509649+0000", 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3950000000000003e-06 2024-06-14T23:22:27.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509697+0000", 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7979000000000003e-05 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509703+0000", 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9000000000000003e-06 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.501322+0000", 2024-06-14T23:22:27.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99161896199999999 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.501332+0000", 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7950000000000002e-06 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.590026+0000", 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0886942399999999 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-06-14T23:22:27.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.590124+0000", 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8573000000000006e-05 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.611548+0000", 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021424021000000001 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.611645+0000", 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6163999999999996e-05 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.526 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103304 14.1 14:8bc50412:::benchmark_data_smithi086_333440_object103303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:18.509706+0000", 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.97489697200001, 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1037629020000002, 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103304 2024-06-14T23:22:27.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509706+0000", 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509706+0000", 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509705+0000", 2024-06-14T23:22:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509736+0000", 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0528999999999999e-05 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509741+0000", 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7230000000000001e-06 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.509745+0000", 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2780000000000002e-06 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.510310+0000", 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00056519399999999998 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.510318+0000", 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0099999999999995e-06 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.501476+0000", 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99115816599999995 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.501485+0000", 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8140000000000007e-06 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.590428+0000", 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0889430849999999 2024-06-14T23:22:27.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.590566+0000", 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013793099999999999 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.591183+0000", 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00061742000000000003 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.611744+0000", 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020560563 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.611777+0000", 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3479999999999998e-05 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.611807+0000", 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9442e-05 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.613309+0000", 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001502544 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.613357+0000", 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7902999999999999e-05 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.613468+0000", 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111203 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.534 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103314 14.e 14:79bd421c:::benchmark_data_smithi086_333440_object103313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:18.523648+0000", 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.96095462700001, 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0140330630000001, 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103314 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.535 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.523648+0000", 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.523648+0000", 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.523647+0000", 2024-06-14T23:22:27.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.523703+0000", 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6317999999999998e-05 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.523720+0000", 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7159e-05 2024-06-14T23:22:27.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.523725+0000", 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3630000000000004e-06 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.523787+0000", 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2384000000000004e-05 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.523794+0000", 2024-06-14T23:22:27.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6000000000000003e-06 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.527778+0000", 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.003983973 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.527883+0000", 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104937 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.528001+0000", 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011803699999999999 2024-06-14T23:22:27.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.532852+0000", 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048514070000000003 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533127+0000", 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027483599999999998 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.537624+0000", 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044975800000000002 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.537656+0000", 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1130000000000002e-05 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.537681+0000", 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5449999999999999e-05 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.541 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103320 14.e 14:7fdebe47:::benchmark_data_smithi086_333440_object103319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:18.529734+0000", 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.95486821, 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0080407549999999, 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103320 2024-06-14T23:22:27.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.529734+0000", 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.529734+0000", 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.529733+0000", 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.529799+0000", 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5636999999999995e-05 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.529810+0000", 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1012000000000001e-05 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.529815+0000", 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2680000000000002e-06 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.529856+0000", 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1090000000000001e-05 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.529862+0000", 2024-06-14T23:22:27.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7100000000000004e-06 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.530040+0000", 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.000178043 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.530185+0000", 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000145091 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,4", 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.530349+0000", 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016425700000000001 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533134+0000", 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002784431 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533244+0000", 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110572 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.537711+0000", 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044669219999999999 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.537742+0000", 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0752e-05 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.537775+0000", 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.311e-05 2024-06-14T23:22:27.548 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103322 14.2 14:483ea328:::benchmark_data_smithi086_333440_object103321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:18.533942+0000", 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.950660857, 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0677597809999999, 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103322 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.533942+0000", 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.533942+0000", 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.533941+0000", 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.533971+0000", 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0137e-05 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.533984+0000", 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3015e-05 2024-06-14T23:22:27.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.533989+0000", 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5679999999999999e-06 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.534066+0000", 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.661e-05 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:18.534075+0000", 2024-06-14T23:22:27.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4010000000000003e-06 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.490790+0000", 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95671434 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.490798+0000", 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.585e-06 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.573788+0000", 2024-06-14T23:22:27.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0829901 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.573962+0000", 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000173072 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,5", 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.574157+0000", 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019540200000000001 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.588679+0000", 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014522017 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601222+0000", 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012542739000000001 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601555+0000", 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033300400000000003 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.555 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601603+0000", 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7939000000000003e-05 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601642+0000", 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9632999999999998e-05 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601701+0000", 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9228999999999998e-05 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.556 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103332 14.1 14:8bc7522e:::benchmark_data_smithi086_333440_object103331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:19.533754+0000", 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 127.95084878199999, 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.079895619, 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103332 2024-06-14T23:22:27.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533754+0000", 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533754+0000", 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.558 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533753+0000", 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533812+0000", 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9647e-05 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533821+0000", 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3699999999999995e-06 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533824+0000", 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5389999999999999e-06 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533863+0000", 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9131999999999997e-05 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.533870+0000", 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7100000000000001e-06 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.606334+0000", 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0724642120000001 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.606491+0000", 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000156611 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.606671+0000", 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018052500000000001 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.611838+0000", 2024-06-14T23:22:27.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051666500000000001 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.612759+0000", 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000921265 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.612789+0000", 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9305000000000001e-05 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.613477+0000", 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00068831700000000003 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.613517+0000", 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9910999999999998e-05 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.613649+0000", 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013260299999999999 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.563 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:103334 14.2 14:4c07ba2a:::benchmark_data_smithi086_333440_object103333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:20:19.537975+0000", 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 127.94662732, 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0642876750000001, 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 103334 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.564 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.537975+0000", 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.537975+0000", 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.537974+0000", 2024-06-14T23:22:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.538088+0000", 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011398799999999999 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.538096+0000", 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.388e-06 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.538100+0000", 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3610000000000001e-06 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.538137+0000", 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7332000000000001e-05 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:19.538145+0000", 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.486e-06 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.576236+0000", 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.038091761 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.576348+0000", 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111316 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,5", 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.576481+0000", 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000132965 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.588711+0000", 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012230782000000001 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601228+0000", 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012516922 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601752+0000", 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052348300000000004 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601944+0000", 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000191881 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.601984+0000", 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0710000000000002e-05 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:20:20.602263+0000", 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027848800000000002 2024-06-14T23:22:27.570 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:159248 14.12 14:4de46917:::benchmark_data_smithi086_333440_object50458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:23.229348+0000", 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 64.255254809999997, 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.303401652, 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.571 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159248 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.229348+0000", 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.229348+0000", 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.229346+0000", 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.229352+0000", 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.049e-06 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.229362+0000", 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0162e-05 2024-06-14T23:22:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.229370+0000", 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7800000000000001e-06 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.229543+0000", 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017295799999999999 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.229550+0000", 2024-06-14T23:22:27.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.686e-06 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.516155+0000", 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.286604888 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:23.516163+0000", 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1119999999999996e-06 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.528190+0000", 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.012027287 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.528291+0000", 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100345 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,5", 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.528495+0000", 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020454800000000001 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.531060+0000", 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025653949999999998 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.531071+0000", 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0791000000000001e-05 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.531105+0000", 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3416e-05 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.532566+0000", 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001461459 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.532621+0000", 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5274e-05 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.532749+0000", 2024-06-14T23:22:27.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012797199999999999 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:159339 14.2 14:466382c3:::benchmark_data_smithi086_333440_object50549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e451)", 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:24.516239+0000", 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 62.968363816, 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0086425560000001, 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.579 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159339 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.516239+0000", 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.516239+0000", 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.516237+0000", 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.516244+0000", 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3400000000000003e-06 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.516257+0000", 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.343e-05 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.516263+0000", 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3690000000000003e-06 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.516301+0000", 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7088999999999997e-05 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:24.516306+0000", 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7420000000000003e-06 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:25.521347+0000", 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.005040253 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:25.521470+0000", 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012354400000000001 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:25.521689+0000", 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021868300000000001 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:25.523784+0000", 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002094767 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:25.524721+0000", 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00093723400000000001 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:25.524823+0000", 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102501 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:25.524861+0000", 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8047999999999999e-05 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:25.524881+0000", 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0004e-05 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:163549 14.12 14:4c5ed4ae:::benchmark_data_smithi086_333440_object54759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:28.400253+0000", 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.084349942000003, 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.070891861, 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 163549 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.587 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.400253+0000", 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.400253+0000", 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.400252+0000", 2024-06-14T23:22:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999995 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.400255+0000", 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3189999999999999e-06 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.400263+0000", 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.221e-06 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.400267+0000", 2024-06-14T23:22:27.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4360000000000004e-06 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.408526+0000", 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082592640000000005 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.408687+0000", 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160249 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,5", 2024-06-14T23:22:27.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.408936+0000", 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024959899999999998 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.412007+0000", 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003071112 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.412580+0000", 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00057231399999999998 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.414945+0000", 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023658429999999999 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.429325+0000", 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014379708 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.429332+0000", 2024-06-14T23:22:27.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.99e-06 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.471095+0000", 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0417631650000001 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.471144+0000", 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9159999999999997e-05 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.593 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:163568 14.12 14:4862e4fc:::benchmark_data_smithi086_333440_object54778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:28.418456+0000", 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.066146562, 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0573474009999999, 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 163568 2024-06-14T23:22:27.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.418456+0000", 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.418456+0000", 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.418455+0000", 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.418459+0000", 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6870000000000002e-06 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.418469+0000", 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2499999999999995e-06 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.418473+0000", 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2320000000000003e-06 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.418569+0000", 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6335999999999994e-05 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.418691+0000", 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121533 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,5", 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.418963+0000", 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027197899999999999 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.422426+0000", 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003462924 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.429356+0000", 2024-06-14T23:22:27.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0069300580000000002 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.429362+0000", 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9710000000000001e-06 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.471343+0000", 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.041981466 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.471421+0000", 2024-06-14T23:22:27.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7503999999999995e-05 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.475748+0000", 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043275040000000002 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.475803+0000", 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5194e-05 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:163571 14.12 14:4ec5e02f:::benchmark_data_smithi086_333440_object54781:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e453)", 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:28.422998+0000", 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.061604180000003, 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0530998469999999, 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 163571 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.601 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.422998+0000", 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.422998+0000", 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.422996+0000", 2024-06-14T23:22:27.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.423003+0000", 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0679999999999999e-06 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.423016+0000", 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2262000000000001e-05 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.423022+0000", 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4400000000000002e-06 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.423093+0000", 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0658e-05 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.423220+0000", 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012717100000000001 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,5", 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.423625+0000", 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000404554 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.429383+0000", 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057589590000000001 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.429389+0000", 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8810000000000001e-06 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.471483+0000", 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0420933859999999 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.471611+0000", 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012840900000000001 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476054+0000", 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044433190000000003 2024-06-14T23:22:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476098+0000", 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3741e-05 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:163617 14.12 14:48813db6:::benchmark_data_smithi086_333440_object54827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:28.456626+0000", 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.027976496999997, 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.019597758, 2024-06-14T23:22:27.607 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 163617 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.456626+0000", 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.456626+0000", 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.456625+0000", 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.456629+0000", 2024-06-14T23:22:27.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5669999999999996e-06 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.456637+0000", 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9820000000000008e-06 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.456642+0000", 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7929999999999997e-06 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.456701+0000", 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8848000000000003e-05 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.456710+0000", 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.85e-06 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.471654+0000", 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0149446609999999 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.471762+0000", 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010802999999999999 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.472019+0000", 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025697600000000001 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.475280+0000", 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032609599999999998 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476128+0000", 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00084748500000000004 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476153+0000", 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4814999999999999e-05 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476178+0000", 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5375999999999999e-05 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476224+0000", 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5763000000000002e-05 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:163625 14.12 14:4d9f0099:::benchmark_data_smithi086_333440_object54835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:28.459252+0000", 2024-06-14T23:22:27.614 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.025350236000001, 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.017096014, 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 163625 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459252+0000", 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459252+0000", 2024-06-14T23:22:27.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459251+0000", 2024-06-14T23:22:27.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999986 2024-06-14T23:22:27.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459257+0000", 2024-06-14T23:22:27.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7769999999999999e-06 2024-06-14T23:22:27.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459274+0000", 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7302000000000001e-05 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459280+0000", 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4280000000000001e-06 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459337+0000", 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6100999999999997e-05 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459346+0000", 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2650000000000002e-06 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.473216+0000", 2024-06-14T23:22:27.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0138705180000001 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.473334+0000", 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117868 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.473530+0000", 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019569699999999999 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.475473+0000", 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00194342 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476254+0000", 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00078116299999999997 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476279+0000", 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5040000000000001e-05 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476303+0000", 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3107000000000001e-05 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476348+0000", 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5754999999999998e-05 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.622 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:163627 14.12 14:4f61a27a:::benchmark_data_smithi086_333440_object54837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:28.459581+0000", 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.025021531, 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.016935667, 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 163627 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459581+0000", 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459581+0000", 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459580+0000", 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.999999 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459585+0000", 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7870000000000001e-06 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459596+0000", 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0815999999999999e-05 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459601+0000", 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0669999999999999e-06 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459639+0000", 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8642e-05 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.459644+0000", 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7530000000000001e-06 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.474344+0000", 2024-06-14T23:22:27.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0147001980000001 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.474453+0000", 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108397 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.474616+0000", 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016371800000000001 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.475802+0000", 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011854299999999999 2024-06-14T23:22:27.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476375+0000", 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00057315100000000004 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476401+0000", 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6225999999999999e-05 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476425+0000", 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4207e-05 2024-06-14T23:22:27.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476517+0000", 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1205000000000005e-05 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.37287.0:163692 14.12 14:4dbeebc3:::benchmark_data_smithi086_333440_object54902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-06-14T23:21:28.503384+0000", 2024-06-14T23:22:27.629 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 58.981218734999999, 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.980973541, 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.37287", 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "172.21.15.86:0/520292708", 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 163692 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.503384+0000", 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.503384+0000", 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.503382+0000", 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4294967295.9999981 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.503388+0000", 2024-06-14T23:22:27.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.682e-06 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.503398+0000", 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0438000000000001e-05 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.503403+0000", 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0900000000000004e-06 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.503457+0000", 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3797e-05 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:28.503466+0000", 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0890000000000006e-06 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476550+0000", 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97308375199999997 2024-06-14T23:22:27.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476618+0000", 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7900999999999999e-05 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.476730+0000", 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011183699999999999 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.479798+0000", 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030683170000000001 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.484170+0000", 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043719329999999997 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.484224+0000", 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3566000000000002e-05 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.484275+0000", 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1953999999999998e-05 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-06-14T23:21:29.484357+0000", 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1903000000000005e-05 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-06-14T23:22:27.637 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-06-14T23:22:27.637 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:22:27.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:27.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:27.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:27 smithi137 ceph-mon[90634]: osdmap e486: 8 total, 7 up, 8 in 2024-06-14T23:22:27.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:27.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:27.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:27.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:27.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:27.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:27 smithi086 ceph-mon[97913]: osdmap e486: 8 total, 7 up, 8 in 2024-06-14T23:22:27.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:27.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:27.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:27 smithi086 ceph-mon[102754]: osdmap e486: 8 total, 7 up, 8 in 2024-06-14T23:22:27.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:27.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:27.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:27.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:27.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:28.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:28.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:28.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:28.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:28.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:28.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:28.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-14T23:22:28.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:28.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:28.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:28.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:28.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:28.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:28.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:28.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:28.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:28.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:28.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:28.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:28.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:28.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:28.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:28.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:28.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[97913]: 1.1d scrub starts 2024-06-14T23:22:28.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[97913]: 1.1d scrub ok 2024-06-14T23:22:28.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[97913]: 1.d scrub starts 2024-06-14T23:22:28.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[97913]: 1.d scrub ok 2024-06-14T23:22:28.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[97913]: pgmap v1695: 129 pgs: 4 active+undersized+remapped, 10 peering, 4 unknown, 8 remapped+peering, 10 active+clean+remapped, 23 active+undersized, 7 active+undersized+degraded, 63 active+clean; 11 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 98/591 objects degraded (16.582%); 8/591 objects misplaced (1.354%) 2024-06-14T23:22:28.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 106/621 objects degraded (17.069%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:28.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[102754]: 1.1d scrub starts 2024-06-14T23:22:28.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[102754]: 1.1d scrub ok 2024-06-14T23:22:28.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[102754]: 1.d scrub starts 2024-06-14T23:22:28.839 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[102754]: 1.d scrub ok 2024-06-14T23:22:28.839 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[102754]: pgmap v1695: 129 pgs: 4 active+undersized+remapped, 10 peering, 4 unknown, 8 remapped+peering, 10 active+clean+remapped, 23 active+undersized, 7 active+undersized+degraded, 63 active+clean; 11 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 98/591 objects degraded (16.582%); 8/591 objects misplaced (1.354%) 2024-06-14T23:22:28.839 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:28 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 106/621 objects degraded (17.069%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:28.894 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:28 smithi137 ceph-mon[90634]: 1.1d scrub starts 2024-06-14T23:22:28.895 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:28 smithi137 ceph-mon[90634]: 1.1d scrub ok 2024-06-14T23:22:28.895 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:28 smithi137 ceph-mon[90634]: 1.d scrub starts 2024-06-14T23:22:28.895 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:28 smithi137 ceph-mon[90634]: 1.d scrub ok 2024-06-14T23:22:28.895 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:28 smithi137 ceph-mon[90634]: pgmap v1695: 129 pgs: 4 active+undersized+remapped, 10 peering, 4 unknown, 8 remapped+peering, 10 active+clean+remapped, 23 active+undersized, 7 active+undersized+degraded, 63 active+clean; 11 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 98/591 objects degraded (16.582%); 8/591 objects misplaced (1.354%) 2024-06-14T23:22:28.895 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:28 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 106/621 objects degraded (17.069%), 8 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:28.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3 2024-06-14T23:22:28.946 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:28.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.3' 2024-06-14T23:22:29.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:29.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:29.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:29.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:29.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:29.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:29.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:29.315 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:29.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:29.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:29.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-14T23:22:29.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:29.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:29.526 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_9' removed 2024-06-14T23:22:29.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:29.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:29.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:29.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2 2024-06-14T23:22:29.694 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:29.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.2' 2024-06-14T23:22:29.713 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 6, 7, 0, 2, 1] out_osds: [] dead_osds: [5] live_osds: [0, 1, 3, 4, 7, 6, 2] 2024-06-14T23:22:29.713 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-14T23:22:29.713 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-06-14T23:22:29.713 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-14T23:22:29.713 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-14T23:22:29.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-06-14T23:22:29.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:29 smithi137 ceph-mon[90634]: 14.2 deep-scrub starts 2024-06-14T23:22:29.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:29 smithi137 ceph-mon[90634]: 14.2 deep-scrub ok 2024-06-14T23:22:29.739 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:29 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4171801054' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:22:29.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:29 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:22:29.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:29 smithi137 ceph-mon[90634]: 14.18 scrub starts 2024-06-14T23:22:29.740 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:29 smithi137 ceph-mon[90634]: 14.18 scrub ok 2024-06-14T23:22:29.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:29.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:29.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:29.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[97913]: 14.2 deep-scrub starts 2024-06-14T23:22:29.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[97913]: 14.2 deep-scrub ok 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4171801054' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[97913]: 14.18 scrub starts 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[97913]: 14.18 scrub ok 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[102754]: 14.2 deep-scrub starts 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[102754]: 14.2 deep-scrub ok 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4171801054' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[102754]: 14.18 scrub starts 2024-06-14T23:22:29.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:29 smithi086 ceph-mon[102754]: 14.18 scrub ok 2024-06-14T23:22:29.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:29.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:29.990 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:30.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:30.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:30.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:30.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:30.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:30.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:30.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6 2024-06-14T23:22:30.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:30.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.6' 2024-06-14T23:22:30.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0 2024-06-14T23:22:30.437 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:30.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.0' 2024-06-14T23:22:30.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:30.560 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:30.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:30.648 INFO:teuthology.task.sequential:In sequential, running task print... 2024-06-14T23:22:30.648 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-06-14T23:22:30.648 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-06-14T23:22:30.653 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-06-14T23:22:30.653 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-14T23:22:30.653 DEBUG:teuthology.orchestra.run.smithi086:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-14T23:22:30.656 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:30.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:30.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:30.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-06-14T23:22:30.691 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-06-14T23:22:30.781 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-06-14T23:22:30.781 DEBUG:teuthology.orchestra.run.smithi086:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-06-14T23:22:30.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[97913]: 14.16 scrub starts 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[97913]: 14.16 scrub ok 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[97913]: osdmap e487: 8 total, 7 up, 8 in 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[97913]: pgmap v1697: 103 pgs: 1 active+undersized+remapped, 11 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[102754]: 14.16 scrub starts 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[102754]: 14.16 scrub ok 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[102754]: osdmap e487: 8 total, 7 up, 8 in 2024-06-14T23:22:30.788 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:30 smithi086 ceph-mon[102754]: pgmap v1697: 103 pgs: 1 active+undersized+remapped, 11 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:30.797 INFO:tasks.workunit:timeout=3h 2024-06-14T23:22:30.797 INFO:tasks.workunit:cleanup=True 2024-06-14T23:22:30.798 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://github.com/cbodley/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-06-14T23:22:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:30.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-06-14T23:22:30.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.1' 2024-06-14T23:22:30.829 INFO:tasks.workunit.client.0.smithi086.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-14T23:22:30.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:30.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:30.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:30.964 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:30 smithi137 ceph-mon[90634]: 14.16 scrub starts 2024-06-14T23:22:30.964 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:30 smithi137 ceph-mon[90634]: 14.16 scrub ok 2024-06-14T23:22:30.964 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:30 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-06-14T23:22:30.964 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:30 smithi137 ceph-mon[90634]: osdmap e487: 8 total, 7 up, 8 in 2024-06-14T23:22:30.964 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:30 smithi137 ceph-mon[90634]: pgmap v1697: 103 pgs: 1 active+undersized+remapped, 11 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:31.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4 2024-06-14T23:22:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:31.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.4' 2024-06-14T23:22:31.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7 2024-06-14T23:22:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:22:31.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-06-14T23:22:31.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd.7' 2024-06-14T23:22:31.191 INFO:tasks.workunit.client.0.smithi086.stderr:warning: Could not find remote branch quincy to clone. 2024-06-14T23:22:31.191 INFO:tasks.workunit.client.0.smithi086.stderr:fatal: Remote branch quincy not found in upstream origin 2024-06-14T23:22:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 128 2024-06-14T23:22:31.203 INFO:tasks.thrashosds:joining thrashosds 2024-06-14T23:22:31.272 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:31.544 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-06-14T23:22:31.578 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:31 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-06-14T23:22:31.578 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:31 smithi137 ceph-mon[90634]: 1.12 scrub starts 2024-06-14T23:22:31.579 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:31 smithi137 ceph-mon[90634]: 1.12 scrub ok 2024-06-14T23:22:31.579 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/865194050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:22:31.827 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:22:31.827 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "" 2024-06-14T23:22:31.827 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:22:31.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:31 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-06-14T23:22:31.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:31 smithi086 ceph-mon[97913]: 1.12 scrub starts 2024-06-14T23:22:31.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:31 smithi086 ceph-mon[97913]: 1.12 scrub ok 2024-06-14T23:22:31.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/865194050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:22:31.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:31 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-06-14T23:22:31.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:31 smithi086 ceph-mon[102754]: 1.12 scrub starts 2024-06-14T23:22:31.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:31 smithi086 ceph-mon[102754]: 1.12 scrub ok 2024-06-14T23:22:31.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/865194050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-14T23:22:32.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:32 smithi137 ceph-mon[90634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:22:32.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/865194050' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:22:32.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:32 smithi137 ceph-mon[90634]: osdmap e488: 8 total, 7 up, 8 in 2024-06-14T23:22:32.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:32 smithi137 ceph-mon[90634]: pgmap v1699: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:32.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:32 smithi086 ceph-mon[102754]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:22:32.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/865194050' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:22:32.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:32 smithi086 ceph-mon[102754]: osdmap e488: 8 total, 7 up, 8 in 2024-06-14T23:22:32.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:32 smithi086 ceph-mon[102754]: pgmap v1699: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:32.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:32 smithi086 ceph-mon[97913]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-14T23:22:32.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/865194050' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-14T23:22:32.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:32 smithi086 ceph-mon[97913]: osdmap e488: 8 total, 7 up, 8 in 2024-06-14T23:22:32.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:32 smithi086 ceph-mon[97913]: pgmap v1699: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:33.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:33 smithi086 ceph-mon[97913]: Health check update: Degraded data redundancy: 9/72 objects degraded (12.500%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:33.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:33 smithi086 ceph-mon[102754]: Health check update: Degraded data redundancy: 9/72 objects degraded (12.500%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:33.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:33 smithi137 ceph-mon[90634]: Health check update: Degraded data redundancy: 9/72 objects degraded (12.500%), 5 pgs degraded (PG_DEGRADED) 2024-06-14T23:22:34.429 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-06-14T23:22:34.429 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-14T23:22:34.441 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1: {} 2024-06-14T23:22:34.442 INFO:teuthology.orchestra.run.smithi086.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-14T23:22:34.459 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-06-14T23:22:34.459 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-14T23:22:34.476 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-06-14T23:22:34.477 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-14T23:22:34.493 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4: {} 2024-06-14T23:22:34.493 INFO:teuthology.orchestra.run.smithi086.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-14T23:22:34.493 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-06-14T23:22:34.509 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-06-14T23:22:34.510 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-14T23:22:34.526 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7: {} 2024-06-14T23:22:34.526 INFO:teuthology.orchestra.run.smithi086.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-14T23:22:34.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:34 smithi086 ceph-mon[97913]: pgmap v1700: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:34.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2113626294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:22:34.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:34 smithi086 ceph-mon[102754]: pgmap v1700: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:34.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2113626294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:22:34.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:34 smithi137 ceph-mon[90634]: pgmap v1700: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:34.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2113626294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-14T23:22:35.145 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:22:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:22:34] "GET /metrics HTTP/1.1" 200 36724 "" "Prometheus/2.43.0" 2024-06-14T23:22:35.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset noscrub 2024-06-14T23:22:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-14T23:22:36.057 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-14T23:22:36.256 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:22:36.256 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:22:37.210 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:36 smithi086 ceph-mon[97913]: pgmap v1701: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:37.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:36 smithi086 ceph-mon[102754]: pgmap v1701: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:37.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-14T23:22:37.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:36 smithi137 ceph-mon[90634]: pgmap v1701: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:37.494 INFO:tasks.thrashosds.thrasher:reviving osd 2024-06-14T23:22:37.495 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-06-14T23:22:37.495 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-06-14T23:22:37.495 DEBUG:teuthology.orchestra.run.smithi137:osd.5> sudo journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.5.service 2024-06-14T23:22:37.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl start ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.5 2024-06-14T23:22:37.805 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:37 smithi137 systemd[1]: Starting Ceph osd.5 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:22:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:38.086 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:37 smithi137 podman[238722]: 2024-06-14 23:22:37.705151068 +0000 UTC m=+0.017212060 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:22:38.086 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:37 smithi137 podman[238722]: 2024-06-14 23:22:37.828211896 +0000 UTC m=+0.140272887 container create f64c21635a0e20b96e619a04c3c77d4c33777c4c377e44e85f7e22a083d6b25f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS) 2024-06-14T23:22:38.086 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 podman[238722]: 2024-06-14 23:22:38.083255509 +0000 UTC m=+0.395316503 container init f64c21635a0e20b96e619a04c3c77d4c33777c4c377e44e85f7e22a083d6b25f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-06-14T23:22:38.442 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 podman[238722]: 2024-06-14 23:22:38.089400926 +0000 UTC m=+0.401461918 container start f64c21635a0e20b96e619a04c3c77d4c33777c4c377e44e85f7e22a083d6b25f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-06-14T23:22:38.443 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 podman[238722]: 2024-06-14 23:22:38.182603438 +0000 UTC m=+0.494664430 container attach f64c21635a0e20b96e619a04c3c77d4c33777c4c377e44e85f7e22a083d6b25f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-14T23:22:38.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate[238737]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 bash[238722]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate[238737]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 bash[238722]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate[238737]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 bash[238722]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate[238737]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 bash[238722]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate[238737]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 bash[238722]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate[238737]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 bash[238722]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate[238737]: --> ceph-volume raw activate successful for osd ID: 5 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 bash[238722]: --> ceph-volume raw activate successful for osd ID: 5 2024-06-14T23:22:38.747 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:38 smithi137 podman[238722]: 2024-06-14 23:22:38.488496482 +0000 UTC m=+0.800557476 container died f64c21635a0e20b96e619a04c3c77d4c33777c4c377e44e85f7e22a083d6b25f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-14T23:22:39.058 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:38 smithi137 ceph-mon[90634]: pgmap v1702: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:39.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:38 smithi086 ceph-mon[97913]: pgmap v1702: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:39.255 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:38 smithi086 ceph-mon[102754]: pgmap v1702: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:39.309 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:39 smithi137 podman[238885]: 2024-06-14 23:22:39.055475375 +0000 UTC m=+0.561123092 container remove f64c21635a0e20b96e619a04c3c77d4c33777c4c377e44e85f7e22a083d6b25f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-activate, GIT_CLEAN=True, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-06-14T23:22:39.309 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:39 smithi137 podman[238952]: 2024-06-14 23:22:39.174285346 +0000 UTC m=+0.017078644 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:22:39.309 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:39 smithi137 podman[238952]: 2024-06-14 23:22:39.306351721 +0000 UTC m=+0.149145019 container create c89b91667200ed2f7ec6750ce19a7e8561708e2c83d2ca45967f093bf0e5563d (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-14T23:22:39.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T23:22:39.957 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:39.960 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:39 smithi137 podman[238952]: 2024-06-14 23:22:39.576139138 +0000 UTC m=+0.418932448 container init c89b91667200ed2f7ec6750ce19a7e8561708e2c83d2ca45967f093bf0e5563d (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:22:39.960 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:39 smithi137 podman[238952]: 2024-06-14 23:22:39.581488516 +0000 UTC m=+0.424281807 container start c89b91667200ed2f7ec6750ce19a7e8561708e2c83d2ca45967f093bf0e5563d (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-14T23:22:39.961 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:39 smithi137 ceph-osd[238990]: -- 172.21.15.137:0/545641344 <== mon.1 v2:172.21.15.137:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5574315d21a0 con 0x5574315a5400 2024-06-14T23:22:39.961 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:39 smithi137 bash[238952]: c89b91667200ed2f7ec6750ce19a7e8561708e2c83d2ca45967f093bf0e5563d 2024-06-14T23:22:39.961 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:39 smithi137 systemd[1]: Started Ceph osd.5 for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T23:22:39.968 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-06-14T23:22:40.226 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:22:40.147+0000 7fb682ae32c0 -1 Falling back to public interface 2024-06-14T23:22:40.227 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:39 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4228750587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:22:40.227 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:39 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:22:40.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:39 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4228750587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:22:40.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:39 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:22:40.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:39 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4228750587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:22:40.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:39 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-14T23:22:41.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:40 smithi086 ceph-mon[97913]: pgmap v1703: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:41.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:40 smithi086 ceph-mon[97913]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-06-14T23:22:41.016 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:40 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:22:41.016 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:40 smithi086 ceph-mon[97913]: osdmap e489: 8 total, 7 up, 8 in 2024-06-14T23:22:41.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:40 smithi086 ceph-mon[102754]: pgmap v1703: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:41.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:40 smithi086 ceph-mon[102754]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-06-14T23:22:41.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:40 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:22:41.016 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:40 smithi086 ceph-mon[102754]: osdmap e489: 8 total, 7 up, 8 in 2024-06-14T23:22:41.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-14T23:22:41.088 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:40 smithi137 ceph-mon[90634]: pgmap v1703: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:41.088 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:40 smithi137 ceph-mon[90634]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-06-14T23:22:41.088 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:40 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-14T23:22:41.088 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:40 smithi137 ceph-mon[90634]: osdmap e489: 8 total, 7 up, 8 in 2024-06-14T23:22:41.618 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:22:41.618 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:22:41.641 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:config diff 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:config diff get 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:config get 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:config help [] 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:config set ... 2024-06-14T23:22:41.642 INFO:teuthology.orchestra.run.smithi137.stderr:admin_socket: invalid command 2024-06-14T23:22:41.856 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-14T23:22:42.286 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-06-14T23:22:42.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-14T23:22:43.176 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:42 smithi086 ceph-mon[97913]: pgmap v1705: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:43.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:43.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:42 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:43.177 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:42 smithi086 ceph-mon[102754]: pgmap v1705: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:43.177 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:43.177 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:42 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:42 smithi137 ceph-mon[90634]: pgmap v1705: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:43.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:42 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:43.929 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:44.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:44 smithi086 ceph-mon[97913]: pgmap v1706: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:44.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:44 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3996095785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:22:44.990 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:22:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:22:44] "GET /metrics HTTP/1.1" 200 34284 "" "Prometheus/2.43.0" 2024-06-14T23:22:44.990 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:44 smithi086 ceph-mon[102754]: pgmap v1706: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:44.990 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:44 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3996095785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:22:45.079 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-06-14T23:22:45.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:44 smithi137 ceph-mon[90634]: pgmap v1706: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:45.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:44 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3996095785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-14T23:22:46.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3996095785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:22:46.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:45 smithi086 ceph-mon[97913]: osdmap e490: 8 total, 7 up, 8 in 2024-06-14T23:22:46.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3996095785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:22:46.072 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:45 smithi086 ceph-mon[102754]: osdmap e490: 8 total, 7 up, 8 in 2024-06-14T23:22:46.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3996095785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-14T23:22:46.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:45 smithi137 ceph-mon[90634]: osdmap e490: 8 total, 7 up, 8 in 2024-06-14T23:22:46.350 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-06-14T23:22:46.350 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-14T23:22:47.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:46 smithi137 ceph-mon[90634]: pgmap v1708: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:47.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-14T23:22:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:46 smithi086 ceph-mon[97913]: pgmap v1708: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:47.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:46 smithi086 ceph-mon[102754]: pgmap v1708: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:47.567 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.b/config 2024-06-14T23:22:49.106 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:48 smithi137 ceph-mon[90634]: pgmap v1709: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:49.106 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:48 smithi137 ceph-mon[90634]: from='osd.5 [v2:172.21.15.137:6808/4252012775,v1:172.21.15.137:6809/4252012775]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T23:22:49.106 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:48 smithi137 ceph-mon[90634]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T23:22:49.106 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:22:48.965+0000 7fb682ae32c0 -1 osd.5 466 log_to_monitors true 2024-06-14T23:22:49.107 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:22:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:22:49.006+0000 7fb677088640 -1 osd.5 466 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-14T23:22:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:48 smithi086 ceph-mon[97913]: pgmap v1709: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:48 smithi086 ceph-mon[97913]: from='osd.5 [v2:172.21.15.137:6808/4252012775,v1:172.21.15.137:6809/4252012775]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T23:22:49.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:48 smithi086 ceph-mon[97913]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T23:22:49.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:48 smithi086 ceph-mon[102754]: pgmap v1709: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:49.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:48 smithi086 ceph-mon[102754]: from='osd.5 [v2:172.21.15.137:6808/4252012775,v1:172.21.15.137:6809/4252012775]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T23:22:49.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:48 smithi086 ceph-mon[102754]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-14T23:22:49.651 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-06-14T23:22:49.651 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-06-14T23:22:49.651 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-06-14T23:22:49.652 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-06-14T23:22:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:50 smithi137 ceph-mon[90634]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-14T23:22:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:50 smithi137 ceph-mon[90634]: osdmap e491: 8 total, 7 up, 8 in 2024-06-14T23:22:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:50 smithi137 ceph-mon[90634]: from='osd.5 [v2:172.21.15.137:6808/4252012775,v1:172.21.15.137:6809/4252012775]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:22:50.240 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:50 smithi137 ceph-mon[90634]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:22:50.264 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-06-14T23:22:50.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd pool set .mgr pgp_num 51 2024-06-14T23:22:50.293 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:49 smithi086 ceph-mon[102754]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-14T23:22:50.293 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:49 smithi086 ceph-mon[102754]: osdmap e491: 8 total, 7 up, 8 in 2024-06-14T23:22:50.293 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:49 smithi086 ceph-mon[102754]: from='osd.5 [v2:172.21.15.137:6808/4252012775,v1:172.21.15.137:6809/4252012775]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:22:50.293 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:49 smithi086 ceph-mon[102754]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:22:50.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:49 smithi086 ceph-mon[97913]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-14T23:22:50.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:49 smithi086 ceph-mon[97913]: osdmap e491: 8 total, 7 up, 8 in 2024-06-14T23:22:50.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:49 smithi086 ceph-mon[97913]: from='osd.5 [v2:172.21.15.137:6808/4252012775,v1:172.21.15.137:6809/4252012775]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:22:50.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:49 smithi086 ceph-mon[97913]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-06-14T23:22:50.531 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:51.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[102754]: pgmap v1711: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[102754]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[102754]: osd.5 [v2:172.21.15.137:6808/4252012775,v1:172.21.15.137:6809/4252012775] boot 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[102754]: osdmap e492: 8 total, 8 up, 8 in 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[102754]: osdmap e493: 8 total, 8 up, 8 in 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[97913]: pgmap v1711: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[97913]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[97913]: osd.5 [v2:172.21.15.137:6808/4252012775,v1:172.21.15.137:6809/4252012775] boot 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[97913]: osdmap e492: 8 total, 8 up, 8 in 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T23:22:51.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:50 smithi086 ceph-mon[97913]: osdmap e493: 8 total, 8 up, 8 in 2024-06-14T23:22:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:50 smithi137 ceph-mon[90634]: pgmap v1711: 103 pgs: 12 active+clean+remapped, 20 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:51 smithi137 ceph-mon[90634]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-14T23:22:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:51 smithi137 ceph-mon[90634]: osd.5 [v2:172.21.15.137:6808/4252012775,v1:172.21.15.137:6809/4252012775] boot 2024-06-14T23:22:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:51 smithi137 ceph-mon[90634]: osdmap e492: 8 total, 8 up, 8 in 2024-06-14T23:22:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:51 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-14T23:22:51.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:51 smithi137 ceph-mon[90634]: osdmap e493: 8 total, 8 up, 8 in 2024-06-14T23:22:52.173 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[102754]: 1.a scrub starts 2024-06-14T23:22:52.173 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[102754]: 1.a scrub ok 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[102754]: osdmap e494: 8 total, 8 up, 8 in 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[97913]: 1.a scrub starts 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[97913]: 1.a scrub ok 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:52.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:52 smithi086 ceph-mon[97913]: osdmap e494: 8 total, 8 up, 8 in 2024-06-14T23:22:52.259 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:52 smithi137 ceph-mon[90634]: 1.a scrub starts 2024-06-14T23:22:52.259 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:52 smithi137 ceph-mon[90634]: 1.a scrub ok 2024-06-14T23:22:52.259 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:52 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:22:52.259 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:52 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:52.259 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:52 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:52.259 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:52 smithi137 ceph-mon[90634]: osdmap e494: 8 total, 8 up, 8 in 2024-06-14T23:22:52.659 INFO:teuthology.orchestra.run.smithi086.stderr:Error EINVAL: specified pgp_num 51 > pg_num 31 2024-06-14T23:22:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:53 smithi137 ceph-mon[90634]: pgmap v1714: 103 pgs: 3 peering, 1 remapped+peering, 11 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:53 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4164216634' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-06-14T23:22:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:53 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-06-14T23:22:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:53 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:53 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:53.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[102754]: pgmap v1714: 103 pgs: 3 peering, 1 remapped+peering, 11 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:53.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4164216634' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-06-14T23:22:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-06-14T23:22:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:53.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:53.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[97913]: pgmap v1714: 103 pgs: 3 peering, 1 remapped+peering, 11 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 66 active+clean; 691 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 9/72 objects degraded (12.500%); 1/72 objects misplaced (1.389%) 2024-06-14T23:22:53.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4164216634' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-06-14T23:22:53.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "51"}]: dispatch 2024-06-14T23:22:53.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:53.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:53 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-14T23:22:53.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-06-14T23:22:54.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:54 smithi086 ceph-mon[97913]: 1.4 scrub starts 2024-06-14T23:22:54.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:54 smithi086 ceph-mon[97913]: 1.4 scrub ok 2024-06-14T23:22:54.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:54 smithi086 ceph-mon[97913]: 1.8 scrub starts 2024-06-14T23:22:54.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:54 smithi086 ceph-mon[97913]: 1.8 scrub ok 2024-06-14T23:22:54.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:54 smithi086 ceph-mon[102754]: 1.4 scrub starts 2024-06-14T23:22:54.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:54 smithi086 ceph-mon[102754]: 1.4 scrub ok 2024-06-14T23:22:54.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:54 smithi086 ceph-mon[102754]: 1.8 scrub starts 2024-06-14T23:22:54.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:54 smithi086 ceph-mon[102754]: 1.8 scrub ok 2024-06-14T23:22:54.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:54 smithi137 ceph-mon[90634]: 1.4 scrub starts 2024-06-14T23:22:54.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:54 smithi137 ceph-mon[90634]: 1.4 scrub ok 2024-06-14T23:22:54.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:54 smithi137 ceph-mon[90634]: 1.8 scrub starts 2024-06-14T23:22:54.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:54 smithi137 ceph-mon[90634]: 1.8 scrub ok 2024-06-14T23:22:54.266 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:55.009 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:22:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:22:54] "GET /metrics HTTP/1.1" 200 34284 "" "Prometheus/2.43.0" 2024-06-14T23:22:55.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:55 smithi137 ceph-mon[90634]: pgmap v1716: 103 pgs: 3 peering, 1 remapped+peering, 3 active+clean+remapped, 2 active+undersized, 94 active+clean; 665 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1/72 objects misplaced (1.389%) 2024-06-14T23:22:55.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:55 smithi137 ceph-mon[90634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/72 objects degraded (12.500%), 5 pgs degraded) 2024-06-14T23:22:55.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:55 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:22:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:55 smithi086 ceph-mon[102754]: pgmap v1716: 103 pgs: 3 peering, 1 remapped+peering, 3 active+clean+remapped, 2 active+undersized, 94 active+clean; 665 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1/72 objects misplaced (1.389%) 2024-06-14T23:22:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:55 smithi086 ceph-mon[102754]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/72 objects degraded (12.500%), 5 pgs degraded) 2024-06-14T23:22:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:55 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:22:55.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:55 smithi086 ceph-mon[97913]: pgmap v1716: 103 pgs: 3 peering, 1 remapped+peering, 3 active+clean+remapped, 2 active+undersized, 94 active+clean; 665 KiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1/72 objects misplaced (1.389%) 2024-06-14T23:22:55.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:55 smithi086 ceph-mon[97913]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/72 objects degraded (12.500%), 5 pgs degraded) 2024-06-14T23:22:55.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:55 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:22:56.162 INFO:teuthology.orchestra.run.smithi086.stdout:mon.a: {} 2024-06-14T23:22:56.162 INFO:teuthology.orchestra.run.smithi086.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-14T23:22:56.173 INFO:teuthology.orchestra.run.smithi086.stdout:mon.b: {} 2024-06-14T23:22:56.173 INFO:teuthology.orchestra.run.smithi086.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-14T23:22:56.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:56 smithi086 ceph-mon[97913]: 1.10 scrub starts 2024-06-14T23:22:56.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:56 smithi086 ceph-mon[97913]: 1.10 scrub ok 2024-06-14T23:22:56.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:56.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:56 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:56.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:56 smithi086 ceph-mon[102754]: 1.10 scrub starts 2024-06-14T23:22:56.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:56 smithi086 ceph-mon[102754]: 1.10 scrub ok 2024-06-14T23:22:56.174 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:56.175 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:56 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:56.185 INFO:teuthology.orchestra.run.smithi086.stdout:mon.c: {} 2024-06-14T23:22:56.185 INFO:teuthology.orchestra.run.smithi086.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-14T23:22:56.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:56 smithi137 ceph-mon[90634]: 1.10 scrub starts 2024-06-14T23:22:56.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:56 smithi137 ceph-mon[90634]: 1.10 scrub ok 2024-06-14T23:22:56.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:56.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:56 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:56.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 0 1 2024-06-14T23:22:57.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:57 smithi086 ceph-mon[97913]: pgmap v1717: 103 pgs: 103 active+clean; 647 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:22:57.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:57 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2486076035' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T23:22:57.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:57 smithi086 ceph-mon[102754]: pgmap v1717: 103 pgs: 103 active+clean; 647 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:22:57.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:57 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2486076035' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T23:22:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:57 smithi137 ceph-mon[90634]: pgmap v1717: 103 pgs: 103 active+clean; 647 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:22:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:57 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2486076035' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-14T23:22:57.734 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:22:58.429 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:58.430 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:58.430 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:58 smithi086 ceph-mon[97913]: pgmap v1718: 103 pgs: 103 active+clean; 595 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:22:58.430 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:58.430 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:58.430 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:58 smithi086 ceph-mon[102754]: pgmap v1718: 103 pgs: 103 active+clean; 595 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:22:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:22:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:22:58.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:58 smithi137 ceph-mon[90634]: pgmap v1718: 103 pgs: 103 active+clean; 595 KiB data, 8.7 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:00.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:59 smithi137 ceph-mon[90634]: 1.17 scrub starts 2024-06-14T23:23:00.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:22:59 smithi137 ceph-mon[90634]: 1.17 scrub ok 2024-06-14T23:23:00.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:59 smithi086 ceph-mon[97913]: 1.17 scrub starts 2024-06-14T23:23:00.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:22:59 smithi086 ceph-mon[97913]: 1.17 scrub ok 2024-06-14T23:23:00.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:59 smithi086 ceph-mon[102754]: 1.17 scrub starts 2024-06-14T23:23:00.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:22:59 smithi086 ceph-mon[102754]: 1.17 scrub ok 2024-06-14T23:23:00.897 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.0 to 1 (10000) 2024-06-14T23:23:01.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:00 smithi137 ceph-mon[90634]: pgmap v1719: 103 pgs: 103 active+clean; 595 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:01.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:00 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3316738115' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:23:01.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:00 smithi086 ceph-mon[97913]: pgmap v1719: 103 pgs: 103 active+clean; 595 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:01.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:00 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3316738115' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:23:01.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:00 smithi086 ceph-mon[102754]: pgmap v1719: 103 pgs: 103 active+clean; 595 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:01.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:00 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3316738115' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:23:01.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 0 1 2024-06-14T23:23:02.012 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:02.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3316738115' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-14T23:23:02.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[97913]: osdmap e495: 8 total, 8 up, 8 in 2024-06-14T23:23:02.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:02.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:02.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:23:02.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:23:02.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:02.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3316738115' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-14T23:23:02.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[102754]: osdmap e495: 8 total, 8 up, 8 in 2024-06-14T23:23:02.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:02.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:02.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:23:02.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:23:02.088 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:01 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:02.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3316738115' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-14T23:23:02.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:01 smithi137 ceph-mon[90634]: osdmap e495: 8 total, 8 up, 8 in 2024-06-14T23:23:02.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:02.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:02.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:23:02.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:23:02.247 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:01 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:03.184 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:02 smithi086 ceph-mon[97913]: pgmap v1721: 103 pgs: 103 active+clean; 595 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:03.185 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:02 smithi086 ceph-mon[102754]: pgmap v1721: 103 pgs: 103 active+clean; 595 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:03.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:02 smithi137 ceph-mon[90634]: pgmap v1721: 103 pgs: 103 active+clean; 595 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:04.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:04 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:04.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:04 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:04.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:04 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-06-14T23:23:05.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:23:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:23:04] "GET /metrics HTTP/1.1" 200 34286 "" "Prometheus/2.43.0" 2024-06-14T23:23:05.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:05 smithi086 ceph-mon[97913]: pgmap v1722: 103 pgs: 103 active+clean; 582 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:05.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2253770321' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:23:05.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:05 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:23:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:05 smithi086 ceph-mon[102754]: pgmap v1722: 103 pgs: 103 active+clean; 582 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2253770321' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:23:05.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:05 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:23:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:05 smithi137 ceph-mon[90634]: pgmap v1722: 103 pgs: 103 active+clean; 582 KiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2253770321' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:23:05.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:05 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-14T23:23:05.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 1 1 2024-06-14T23:23:06.244 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:06.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:06 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-06-14T23:23:06.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:06 smithi086 ceph-mon[97913]: osdmap e496: 8 total, 8 up, 8 in 2024-06-14T23:23:06.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:06 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-06-14T23:23:06.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:06 smithi086 ceph-mon[102754]: osdmap e496: 8 total, 8 up, 8 in 2024-06-14T23:23:06.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:06 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-06-14T23:23:06.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:06 smithi137 ceph-mon[90634]: osdmap e496: 8 total, 8 up, 8 in 2024-06-14T23:23:07.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:07 smithi137 ceph-mon[90634]: pgmap v1724: 103 pgs: 17 peering, 86 active+clean; 608 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.6 KiB/s rd, 6.9 KiB/s wr, 2 op/s; 4.8 KiB/s, 0 objects/s recovering 2024-06-14T23:23:07.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:07 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 12 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-06-14T23:23:07.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:07 smithi137 ceph-mon[90634]: osdmap e497: 8 total, 8 up, 8 in 2024-06-14T23:23:07.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:07 smithi086 ceph-mon[97913]: pgmap v1724: 103 pgs: 17 peering, 86 active+clean; 608 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.6 KiB/s rd, 6.9 KiB/s wr, 2 op/s; 4.8 KiB/s, 0 objects/s recovering 2024-06-14T23:23:07.534 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:07 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 12 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-06-14T23:23:07.535 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:07 smithi086 ceph-mon[97913]: osdmap e497: 8 total, 8 up, 8 in 2024-06-14T23:23:07.535 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:07 smithi086 ceph-mon[102754]: pgmap v1724: 103 pgs: 17 peering, 86 active+clean; 608 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 2.6 KiB/s rd, 6.9 KiB/s wr, 2 op/s; 4.8 KiB/s, 0 objects/s recovering 2024-06-14T23:23:07.535 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:07 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 12 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-06-14T23:23:07.535 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:07 smithi086 ceph-mon[102754]: osdmap e497: 8 total, 8 up, 8 in 2024-06-14T23:23:08.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:08 smithi086 ceph-mon[97913]: 1.0 scrub starts 2024-06-14T23:23:08.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:08 smithi086 ceph-mon[97913]: 1.0 scrub ok 2024-06-14T23:23:08.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:08 smithi086 ceph-mon[102754]: 1.0 scrub starts 2024-06-14T23:23:08.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:08 smithi086 ceph-mon[102754]: 1.0 scrub ok 2024-06-14T23:23:08.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:08 smithi137 ceph-mon[90634]: 1.0 scrub starts 2024-06-14T23:23:08.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:08 smithi137 ceph-mon[90634]: 1.0 scrub ok 2024-06-14T23:23:09.107 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.1 to 1 (10000) 2024-06-14T23:23:09.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:09 smithi086 ceph-mon[102754]: pgmap v1726: 103 pgs: 17 peering, 86 active+clean; 608 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 3.0 KiB/s rd, 7.8 KiB/s wr, 2 op/s; 5.5 KiB/s, 0 objects/s recovering 2024-06-14T23:23:09.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:09 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3726604759' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:09.122 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:09 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:09.123 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:09 smithi086 ceph-mon[97913]: pgmap v1726: 103 pgs: 17 peering, 86 active+clean; 608 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 3.0 KiB/s rd, 7.8 KiB/s wr, 2 op/s; 5.5 KiB/s, 0 objects/s recovering 2024-06-14T23:23:09.123 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:09 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3726604759' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:09.123 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:09 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:09 smithi137 ceph-mon[90634]: pgmap v1726: 103 pgs: 17 peering, 86 active+clean; 608 KiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 3.0 KiB/s rd, 7.8 KiB/s wr, 2 op/s; 5.5 KiB/s, 0 objects/s recovering 2024-06-14T23:23:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:09 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3726604759' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:09.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:09 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:09.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 1 1 2024-06-14T23:23:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:10.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:10 smithi086 ceph-mon[97913]: 1.2 deep-scrub starts 2024-06-14T23:23:10.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:10 smithi086 ceph-mon[97913]: 1.2 deep-scrub ok 2024-06-14T23:23:10.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:10 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-14T23:23:10.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:10 smithi086 ceph-mon[97913]: osdmap e498: 8 total, 8 up, 8 in 2024-06-14T23:23:10.366 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:10 smithi086 ceph-mon[102754]: 1.2 deep-scrub starts 2024-06-14T23:23:10.366 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:10 smithi086 ceph-mon[102754]: 1.2 deep-scrub ok 2024-06-14T23:23:10.366 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:10 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-14T23:23:10.366 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:10 smithi086 ceph-mon[102754]: osdmap e498: 8 total, 8 up, 8 in 2024-06-14T23:23:10.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:10 smithi137 ceph-mon[90634]: 1.2 deep-scrub starts 2024-06-14T23:23:10.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:10 smithi137 ceph-mon[90634]: 1.2 deep-scrub ok 2024-06-14T23:23:10.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:10 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-14T23:23:10.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:10 smithi137 ceph-mon[90634]: osdmap e498: 8 total, 8 up, 8 in 2024-06-14T23:23:11.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:11 smithi086 ceph-mon[97913]: pgmap v1728: 103 pgs: 17 peering, 86 active+clean; 608 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 5 op/s; 13 KiB/s, 0 objects/s recovering 2024-06-14T23:23:11.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:11 smithi086 ceph-mon[102754]: pgmap v1728: 103 pgs: 17 peering, 86 active+clean; 608 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 5 op/s; 13 KiB/s, 0 objects/s recovering 2024-06-14T23:23:11.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:11 smithi137 ceph-mon[90634]: pgmap v1728: 103 pgs: 17 peering, 86 active+clean; 608 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 5 op/s; 13 KiB/s, 0 objects/s recovering 2024-06-14T23:23:12.135 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-06-14T23:23:12.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:12 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/862191893' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:12.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:12 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:12.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:12 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/862191893' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:12.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:12 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:12.514 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:12 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/862191893' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:12.514 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:12 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-14T23:23:12.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 3 1 2024-06-14T23:23:13.126 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:13.141 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[97913]: pgmap v1729: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:13.141 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 13 pgs peering) 2024-06-14T23:23:13.142 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:23:13.142 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-06-14T23:23:13.142 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[97913]: osdmap e499: 8 total, 8 up, 8 in 2024-06-14T23:23:13.142 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:13 smithi137 ceph-mon[90634]: pgmap v1729: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:13 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 13 pgs peering) 2024-06-14T23:23:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:13 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:23:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:13 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-06-14T23:23:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:13 smithi137 ceph-mon[90634]: osdmap e499: 8 total, 8 up, 8 in 2024-06-14T23:23:13.398 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:13 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:13.405 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[102754]: pgmap v1729: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:13.405 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 13 pgs peering) 2024-06-14T23:23:13.405 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:23:13.405 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-06-14T23:23:13.405 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[102754]: osdmap e499: 8 total, 8 up, 8 in 2024-06-14T23:23:13.406 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:13 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:14.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:14 smithi086 ceph-mon[97913]: osdmap e500: 8 total, 8 up, 8 in 2024-06-14T23:23:14.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:14 smithi086 ceph-mon[102754]: osdmap e500: 8 total, 8 up, 8 in 2024-06-14T23:23:14.413 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:14 smithi137 ceph-mon[90634]: osdmap e500: 8 total, 8 up, 8 in 2024-06-14T23:23:14.970 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:23:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:23:14] "GET /metrics HTTP/1.1" 200 34272 "" "Prometheus/2.43.0" 2024-06-14T23:23:15.226 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:15 smithi086 ceph-mon[97913]: pgmap v1732: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:15.226 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:15 smithi086 ceph-mon[102754]: pgmap v1732: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:15.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:15 smithi137 ceph-mon[90634]: pgmap v1732: 103 pgs: 103 active+clean; 582 KiB data, 8.2 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:16.164 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.3 to 1 (10000) 2024-06-14T23:23:16.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:16 smithi137 ceph-mon[90634]: 1.d deep-scrub starts 2024-06-14T23:23:16.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:16 smithi137 ceph-mon[90634]: 1.d deep-scrub ok 2024-06-14T23:23:16.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:16 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1618744822' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:16.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:16 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:16.529 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:16 smithi086 ceph-mon[97913]: 1.d deep-scrub starts 2024-06-14T23:23:16.529 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:16 smithi086 ceph-mon[97913]: 1.d deep-scrub ok 2024-06-14T23:23:16.529 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:16 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1618744822' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:16.529 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:16 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:16.529 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:16 smithi086 ceph-mon[102754]: 1.d deep-scrub starts 2024-06-14T23:23:16.529 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:16 smithi086 ceph-mon[102754]: 1.d deep-scrub ok 2024-06-14T23:23:16.530 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:16 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1618744822' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:16.530 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:16 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:16.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 3 1 2024-06-14T23:23:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:17.336 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:17 smithi086 ceph-mon[97913]: pgmap v1733: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:17 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:23:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:17 smithi086 ceph-mon[97913]: osdmap e501: 8 total, 8 up, 8 in 2024-06-14T23:23:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:17 smithi086 ceph-mon[102754]: pgmap v1733: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:17 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:23:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:17 smithi086 ceph-mon[102754]: osdmap e501: 8 total, 8 up, 8 in 2024-06-14T23:23:17.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:17 smithi137 ceph-mon[90634]: pgmap v1733: 103 pgs: 103 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:17.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:17 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:23:17.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:17 smithi137 ceph-mon[90634]: osdmap e501: 8 total, 8 up, 8 in 2024-06-14T23:23:18.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:18 smithi086 ceph-mon[97913]: osdmap e502: 8 total, 8 up, 8 in 2024-06-14T23:23:18.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:18 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:18.462 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:18 smithi086 ceph-mon[102754]: osdmap e502: 8 total, 8 up, 8 in 2024-06-14T23:23:18.462 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:18 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:18 smithi137 ceph-mon[90634]: osdmap e502: 8 total, 8 up, 8 in 2024-06-14T23:23:18.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:18 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:19.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:19 smithi137 ceph-mon[90634]: pgmap v1736: 103 pgs: 3 remapped+peering, 100 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:19.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:19 smithi137 ceph-mon[90634]: osdmap e503: 8 total, 8 up, 8 in 2024-06-14T23:23:19.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:19 smithi086 ceph-mon[97913]: pgmap v1736: 103 pgs: 3 remapped+peering, 100 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:19.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:19 smithi086 ceph-mon[97913]: osdmap e503: 8 total, 8 up, 8 in 2024-06-14T23:23:19.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:19 smithi086 ceph-mon[102754]: pgmap v1736: 103 pgs: 3 remapped+peering, 100 active+clean; 582 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:19.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:19 smithi086 ceph-mon[102754]: osdmap e503: 8 total, 8 up, 8 in 2024-06-14T23:23:20.195 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-06-14T23:23:20.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:20 smithi137 ceph-mon[90634]: osdmap e504: 8 total, 8 up, 8 in 2024-06-14T23:23:20.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:20 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2046175883' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:20.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:20 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:20.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:20 smithi086 ceph-mon[102754]: osdmap e504: 8 total, 8 up, 8 in 2024-06-14T23:23:20.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:20 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2046175883' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:20.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:20 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:20.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:20 smithi086 ceph-mon[97913]: osdmap e504: 8 total, 8 up, 8 in 2024-06-14T23:23:20.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:20 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2046175883' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:20.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:20 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-14T23:23:21.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 4 1 2024-06-14T23:23:21.310 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:21.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[97913]: pgmap v1739: 103 pgs: 4 activating+undersized+remapped, 1 activating+remapped, 3 remapped+peering, 95 active+clean; 595 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 KiB/s wr, 4 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:23:21.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-14T23:23:21.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:23:21.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[97913]: osdmap e505: 8 total, 8 up, 8 in 2024-06-14T23:23:21.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[97913]: 1.1d deep-scrub starts 2024-06-14T23:23:21.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[97913]: 1.1d deep-scrub ok 2024-06-14T23:23:21.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[102754]: pgmap v1739: 103 pgs: 4 activating+undersized+remapped, 1 activating+remapped, 3 remapped+peering, 95 active+clean; 595 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 KiB/s wr, 4 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:23:21.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-14T23:23:21.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:23:21.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[102754]: osdmap e505: 8 total, 8 up, 8 in 2024-06-14T23:23:21.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[102754]: 1.1d deep-scrub starts 2024-06-14T23:23:21.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:21 smithi086 ceph-mon[102754]: 1.1d deep-scrub ok 2024-06-14T23:23:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:21 smithi137 ceph-mon[90634]: pgmap v1739: 103 pgs: 4 activating+undersized+remapped, 1 activating+remapped, 3 remapped+peering, 95 active+clean; 595 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 KiB/s wr, 4 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-06-14T23:23:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:21 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-06-14T23:23:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:21 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-14T23:23:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:21 smithi137 ceph-mon[90634]: osdmap e505: 8 total, 8 up, 8 in 2024-06-14T23:23:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:21 smithi137 ceph-mon[90634]: 1.1d deep-scrub starts 2024-06-14T23:23:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:21 smithi137 ceph-mon[90634]: 1.1d deep-scrub ok 2024-06-14T23:23:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:22 smithi137 ceph-mon[90634]: osdmap e506: 8 total, 8 up, 8 in 2024-06-14T23:23:22.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:22 smithi137 ceph-mon[90634]: pgmap v1742: 103 pgs: 9 peering, 3 activating+undersized+remapped, 3 remapped+peering, 88 active+clean; 621 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 15 KiB/s rd, 41 KiB/s wr, 12 op/s; 109 KiB/s, 0 objects/s recovering 2024-06-14T23:23:22.552 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:22 smithi086 ceph-mon[97913]: osdmap e506: 8 total, 8 up, 8 in 2024-06-14T23:23:22.552 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:22 smithi086 ceph-mon[97913]: pgmap v1742: 103 pgs: 9 peering, 3 activating+undersized+remapped, 3 remapped+peering, 88 active+clean; 621 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 15 KiB/s rd, 41 KiB/s wr, 12 op/s; 109 KiB/s, 0 objects/s recovering 2024-06-14T23:23:22.552 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:22 smithi086 ceph-mon[102754]: osdmap e506: 8 total, 8 up, 8 in 2024-06-14T23:23:22.552 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:22 smithi086 ceph-mon[102754]: pgmap v1742: 103 pgs: 9 peering, 3 activating+undersized+remapped, 3 remapped+peering, 88 active+clean; 621 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 15 KiB/s rd, 41 KiB/s wr, 12 op/s; 109 KiB/s, 0 objects/s recovering 2024-06-14T23:23:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.4 to 1 (10000) 2024-06-14T23:23:23.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:23 smithi086 ceph-mon[97913]: 1.1e scrub starts 2024-06-14T23:23:23.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:23 smithi086 ceph-mon[97913]: 1.1e scrub ok 2024-06-14T23:23:23.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:23 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1928181713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:23:23.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:23 smithi086 ceph-mon[102754]: 1.1e scrub starts 2024-06-14T23:23:23.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:23 smithi086 ceph-mon[102754]: 1.1e scrub ok 2024-06-14T23:23:23.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:23 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1928181713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:23:23.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:23 smithi137 ceph-mon[90634]: 1.1e scrub starts 2024-06-14T23:23:23.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:23 smithi137 ceph-mon[90634]: 1.1e scrub ok 2024-06-14T23:23:23.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:23 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1928181713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:23:23.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 4 1 2024-06-14T23:23:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:24.637 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1928181713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:23:24.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[97913]: osdmap e507: 8 total, 8 up, 8 in 2024-06-14T23:23:24.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[97913]: pgmap v1744: 103 pgs: 9 peering, 94 active+clean; 621 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 8.6 KiB/s rd, 23 KiB/s wr, 6 op/s; 84 KiB/s, 0 objects/s recovering 2024-06-14T23:23:24.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[97913]: 1.9 scrub starts 2024-06-14T23:23:24.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[97913]: 1.9 scrub ok 2024-06-14T23:23:24.638 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1928181713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:23:24.638 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[102754]: osdmap e507: 8 total, 8 up, 8 in 2024-06-14T23:23:24.639 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[102754]: pgmap v1744: 103 pgs: 9 peering, 94 active+clean; 621 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 8.6 KiB/s rd, 23 KiB/s wr, 6 op/s; 84 KiB/s, 0 objects/s recovering 2024-06-14T23:23:24.639 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[102754]: 1.9 scrub starts 2024-06-14T23:23:24.639 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-mon[102754]: 1.9 scrub ok 2024-06-14T23:23:24.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:24 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1928181713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:23:24.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:24 smithi137 ceph-mon[90634]: osdmap e507: 8 total, 8 up, 8 in 2024-06-14T23:23:24.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:24 smithi137 ceph-mon[90634]: pgmap v1744: 103 pgs: 9 peering, 94 active+clean; 621 KiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 8.6 KiB/s rd, 23 KiB/s wr, 6 op/s; 84 KiB/s, 0 objects/s recovering 2024-06-14T23:23:24.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:24 smithi137 ceph-mon[90634]: 1.9 scrub starts 2024-06-14T23:23:24.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:24 smithi137 ceph-mon[90634]: 1.9 scrub ok 2024-06-14T23:23:25.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:23:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:23:24] "GET /metrics HTTP/1.1" 200 34272 "" "Prometheus/2.43.0" 2024-06-14T23:23:26.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:26 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-06-14T23:23:26.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:26 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:23:26.262 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:26 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-06-14T23:23:26.262 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:26 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:23:26.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:26 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-06-14T23:23:26.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:26 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:23:27.033 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-06-14T23:23:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:27 smithi086 ceph-mon[97913]: pgmap v1745: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-06-14T23:23:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:27 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2982593060' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:23:27.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:27 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:23:27.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:27 smithi086 ceph-mon[102754]: pgmap v1745: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-06-14T23:23:27.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:27 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2982593060' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:23:27.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:27 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:23:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:27 smithi137 ceph-mon[90634]: pgmap v1745: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-06-14T23:23:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:27 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2982593060' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:23:27.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:27 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-14T23:23:27.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 7 1 2024-06-14T23:23:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:28.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:28 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:23:28.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:28 smithi086 ceph-mon[97913]: osdmap e508: 8 total, 8 up, 8 in 2024-06-14T23:23:28.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:28 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:28.303 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:28 smithi086 ceph-mon[97913]: osdmap e509: 8 total, 8 up, 8 in 2024-06-14T23:23:28.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:28 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:23:28.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:28 smithi086 ceph-mon[102754]: osdmap e508: 8 total, 8 up, 8 in 2024-06-14T23:23:28.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:28 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:28.303 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:28 smithi086 ceph-mon[102754]: osdmap e509: 8 total, 8 up, 8 in 2024-06-14T23:23:28.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:28 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-06-14T23:23:28.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:28 smithi137 ceph-mon[90634]: osdmap e508: 8 total, 8 up, 8 in 2024-06-14T23:23:28.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:28 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:28.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:28 smithi137 ceph-mon[90634]: osdmap e509: 8 total, 8 up, 8 in 2024-06-14T23:23:29.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:29 smithi086 ceph-mon[97913]: pgmap v1747: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:29.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:29 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:29.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:29 smithi086 ceph-mon[102754]: pgmap v1747: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:29.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:29 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:29.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:29 smithi137 ceph-mon[90634]: pgmap v1747: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:29.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:29 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:23:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.7 to 1 (10000) 2024-06-14T23:23:31.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:31 smithi137 ceph-mon[90634]: pgmap v1749: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:31.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:31 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3424208187' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:23:31.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:31 smithi086 ceph-mon[97913]: pgmap v1749: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:31.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:31 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3424208187' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:23:31.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:31 smithi086 ceph-mon[102754]: pgmap v1749: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:31.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:31 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3424208187' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:23:32.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 7 1 2024-06-14T23:23:32.423 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:32.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3424208187' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:23:32.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:32 smithi137 ceph-mon[90634]: osdmap e510: 8 total, 8 up, 8 in 2024-06-14T23:23:32.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3424208187' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:23:32.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:32 smithi086 ceph-mon[97913]: osdmap e510: 8 total, 8 up, 8 in 2024-06-14T23:23:32.546 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3424208187' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:23:32.546 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:32 smithi086 ceph-mon[102754]: osdmap e510: 8 total, 8 up, 8 in 2024-06-14T23:23:33.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:33 smithi137 ceph-mon[90634]: 1.1b scrub starts 2024-06-14T23:23:33.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:33 smithi137 ceph-mon[90634]: 1.1b scrub ok 2024-06-14T23:23:33.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:33 smithi137 ceph-mon[90634]: pgmap v1751: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:33.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:33 smithi086 ceph-mon[97913]: 1.1b scrub starts 2024-06-14T23:23:33.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:33 smithi086 ceph-mon[97913]: 1.1b scrub ok 2024-06-14T23:23:33.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:33 smithi086 ceph-mon[97913]: pgmap v1751: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:33.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:33 smithi086 ceph-mon[102754]: 1.1b scrub starts 2024-06-14T23:23:33.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:33 smithi086 ceph-mon[102754]: 1.1b scrub ok 2024-06-14T23:23:33.589 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:33 smithi086 ceph-mon[102754]: pgmap v1751: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:35.126 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-06-14T23:23:35.127 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:23:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:23:34] "GET /metrics HTTP/1.1" 200 34269 "" "Prometheus/2.43.0" 2024-06-14T23:23:35.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:35 smithi137 ceph-mon[90634]: pgmap v1752: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:35.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:35 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2227501388' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:23:35.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:35 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:23:35.530 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:35 smithi086 ceph-mon[97913]: pgmap v1752: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:35.530 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:35 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2227501388' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:23:35.530 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:35 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:23:35.531 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:35 smithi086 ceph-mon[102754]: pgmap v1752: 103 pgs: 103 active+clean; 582 KiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:35.531 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:35 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2227501388' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:23:35.531 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:35 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-14T23:23:35.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 6 1 2024-06-14T23:23:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:36 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:23:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:36 smithi086 ceph-mon[97913]: osdmap e511: 8 total, 8 up, 8 in 2024-06-14T23:23:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:36 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:23:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:36 smithi086 ceph-mon[102754]: osdmap e511: 8 total, 8 up, 8 in 2024-06-14T23:23:36.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:36 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-06-14T23:23:36.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:36 smithi137 ceph-mon[90634]: osdmap e511: 8 total, 8 up, 8 in 2024-06-14T23:23:37.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:37 smithi137 ceph-mon[90634]: pgmap v1754: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:37.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:37 smithi137 ceph-mon[90634]: osdmap e512: 8 total, 8 up, 8 in 2024-06-14T23:23:37.497 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:37 smithi086 ceph-mon[97913]: pgmap v1754: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:37.497 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:37 smithi086 ceph-mon[97913]: osdmap e512: 8 total, 8 up, 8 in 2024-06-14T23:23:37.497 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:37 smithi086 ceph-mon[102754]: pgmap v1754: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-06-14T23:23:37.497 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:37 smithi086 ceph-mon[102754]: osdmap e512: 8 total, 8 up, 8 in 2024-06-14T23:23:39.144 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.6 to 1 (10000) 2024-06-14T23:23:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:39 smithi137 ceph-mon[90634]: pgmap v1756: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:39 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3719143043' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:23:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:39 smithi137 ceph-mon[90634]: 1.10 scrub starts 2024-06-14T23:23:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:39 smithi137 ceph-mon[90634]: 1.10 scrub ok 2024-06-14T23:23:39.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:39 smithi086 ceph-mon[97913]: pgmap v1756: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:39.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:39 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3719143043' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:23:39.547 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:39 smithi086 ceph-mon[97913]: 1.10 scrub starts 2024-06-14T23:23:39.547 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:39 smithi086 ceph-mon[97913]: 1.10 scrub ok 2024-06-14T23:23:39.547 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:39 smithi086 ceph-mon[102754]: pgmap v1756: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:39.547 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:39 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3719143043' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:23:39.547 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:39 smithi086 ceph-mon[102754]: 1.10 scrub starts 2024-06-14T23:23:39.547 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:39 smithi086 ceph-mon[102754]: 1.10 scrub ok 2024-06-14T23:23:39.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 6 1 2024-06-14T23:23:40.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:40 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3719143043' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:23:40.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:40 smithi086 ceph-mon[97913]: osdmap e513: 8 total, 8 up, 8 in 2024-06-14T23:23:40.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:40 smithi086 ceph-mon[97913]: 1.9 scrub starts 2024-06-14T23:23:40.200 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:40 smithi086 ceph-mon[97913]: 1.9 scrub ok 2024-06-14T23:23:40.200 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:40 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3719143043' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:23:40.200 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:40 smithi086 ceph-mon[102754]: osdmap e513: 8 total, 8 up, 8 in 2024-06-14T23:23:40.200 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:40 smithi086 ceph-mon[102754]: 1.9 scrub starts 2024-06-14T23:23:40.200 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:40 smithi086 ceph-mon[102754]: 1.9 scrub ok 2024-06-14T23:23:40.219 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:40.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:40 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3719143043' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:23:40.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:40 smithi137 ceph-mon[90634]: osdmap e513: 8 total, 8 up, 8 in 2024-06-14T23:23:40.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:40 smithi137 ceph-mon[90634]: 1.9 scrub starts 2024-06-14T23:23:40.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:40 smithi137 ceph-mon[90634]: 1.9 scrub ok 2024-06-14T23:23:41.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:41 smithi086 ceph-mon[97913]: pgmap v1758: 103 pgs: 103 active+clean; 608 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 5 op/s; 13 KiB/s, 0 objects/s recovering 2024-06-14T23:23:41.463 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:41 smithi086 ceph-mon[102754]: pgmap v1758: 103 pgs: 103 active+clean; 608 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 5 op/s; 13 KiB/s, 0 objects/s recovering 2024-06-14T23:23:41.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:41 smithi137 ceph-mon[90634]: pgmap v1758: 103 pgs: 103 active+clean; 608 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 7.0 KiB/s rd, 18 KiB/s wr, 5 op/s; 13 KiB/s, 0 objects/s recovering 2024-06-14T23:23:43.161 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-06-14T23:23:43.494 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:43 smithi086 ceph-mon[97913]: pgmap v1759: 103 pgs: 103 active+clean; 608 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 6.2 KiB/s rd, 16 KiB/s wr, 4 op/s; 12 KiB/s, 0 objects/s recovering 2024-06-14T23:23:43.495 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:43 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1483623247' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:23:43.495 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:43 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:23:43.495 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:43.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:43 smithi086 ceph-mon[102754]: pgmap v1759: 103 pgs: 103 active+clean; 608 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 6.2 KiB/s rd, 16 KiB/s wr, 4 op/s; 12 KiB/s, 0 objects/s recovering 2024-06-14T23:23:43.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:43 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1483623247' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:23:43.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:43 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:23:43.495 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:43.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:43 smithi137 ceph-mon[90634]: pgmap v1759: 103 pgs: 103 active+clean; 608 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 6.2 KiB/s rd, 16 KiB/s wr, 4 op/s; 12 KiB/s, 0 objects/s recovering 2024-06-14T23:23:43.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:43 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1483623247' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:23:43.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:43 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-14T23:23:43.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:43.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 2 1 2024-06-14T23:23:44.225 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:44 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:23:44.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:44 smithi086 ceph-mon[97913]: osdmap e514: 8 total, 8 up, 8 in 2024-06-14T23:23:44.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:44 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:23:44.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:44 smithi086 ceph-mon[102754]: osdmap e514: 8 total, 8 up, 8 in 2024-06-14T23:23:44.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:44 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-06-14T23:23:44.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:44 smithi137 ceph-mon[90634]: osdmap e514: 8 total, 8 up, 8 in 2024-06-14T23:23:45.076 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:23:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:23:44] "GET /metrics HTTP/1.1" 200 34277 "" "Prometheus/2.43.0" 2024-06-14T23:23:45.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[97913]: pgmap v1761: 103 pgs: 103 active+clean; 608 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 5.4 KiB/s rd, 14 KiB/s wr, 4 op/s; 10 KiB/s, 0 objects/s recovering 2024-06-14T23:23:45.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[97913]: osdmap e515: 8 total, 8 up, 8 in 2024-06-14T23:23:45.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-06-14T23:23:45.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-06-14T23:23:45.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 0]}]: dispatch 2024-06-14T23:23:45.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[102754]: pgmap v1761: 103 pgs: 103 active+clean; 608 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 5.4 KiB/s rd, 14 KiB/s wr, 4 op/s; 10 KiB/s, 0 objects/s recovering 2024-06-14T23:23:45.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[102754]: osdmap e515: 8 total, 8 up, 8 in 2024-06-14T23:23:45.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-06-14T23:23:45.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-06-14T23:23:45.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:45 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 0]}]: dispatch 2024-06-14T23:23:45.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:45 smithi137 ceph-mon[90634]: pgmap v1761: 103 pgs: 103 active+clean; 608 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 5.4 KiB/s rd, 14 KiB/s wr, 4 op/s; 10 KiB/s, 0 objects/s recovering 2024-06-14T23:23:45.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:45 smithi137 ceph-mon[90634]: osdmap e515: 8 total, 8 up, 8 in 2024-06-14T23:23:45.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-06-14T23:23:45.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-06-14T23:23:45.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:45 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 0]}]: dispatch 2024-06-14T23:23:46.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:46 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-06-14T23:23:46.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:46 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-06-14T23:23:46.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:46 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 0]}]': finished 2024-06-14T23:23:46.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:46 smithi137 ceph-mon[90634]: osdmap e516: 8 total, 8 up, 8 in 2024-06-14T23:23:46.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:46 smithi137 ceph-mon[90634]: pgmap v1764: 103 pgs: 1 peering, 102 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:46.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:46 smithi137 ceph-mon[90634]: osdmap e517: 8 total, 8 up, 8 in 2024-06-14T23:23:46.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-06-14T23:23:46.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-06-14T23:23:46.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 0]}]': finished 2024-06-14T23:23:46.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[97913]: osdmap e516: 8 total, 8 up, 8 in 2024-06-14T23:23:46.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[97913]: pgmap v1764: 103 pgs: 1 peering, 102 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:46.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[97913]: osdmap e517: 8 total, 8 up, 8 in 2024-06-14T23:23:46.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-06-14T23:23:46.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-06-14T23:23:46.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 0]}]': finished 2024-06-14T23:23:46.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[102754]: osdmap e516: 8 total, 8 up, 8 in 2024-06-14T23:23:46.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[102754]: pgmap v1764: 103 pgs: 1 peering, 102 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:46.588 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:46 smithi086 ceph-mon[102754]: osdmap e517: 8 total, 8 up, 8 in 2024-06-14T23:23:47.192 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.2 to 1 (10000) 2024-06-14T23:23:47.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2200717690' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:23:47.578 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2200717690' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:23:47.578 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2200717690' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:23:47.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 2 1 2024-06-14T23:23:48.238 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:48.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:48 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2200717690' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:23:48.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:48 smithi086 ceph-mon[97913]: osdmap e518: 8 total, 8 up, 8 in 2024-06-14T23:23:48.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:48 smithi086 ceph-mon[97913]: pgmap v1767: 103 pgs: 1 peering, 102 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:48.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:48 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2200717690' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:23:48.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:48 smithi086 ceph-mon[102754]: osdmap e518: 8 total, 8 up, 8 in 2024-06-14T23:23:48.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:48 smithi086 ceph-mon[102754]: pgmap v1767: 103 pgs: 1 peering, 102 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:48.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:48 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2200717690' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:23:48.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:48 smithi137 ceph-mon[90634]: osdmap e518: 8 total, 8 up, 8 in 2024-06-14T23:23:48.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:48 smithi137 ceph-mon[90634]: pgmap v1767: 103 pgs: 1 peering, 102 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:50.982 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-06-14T23:23:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:50 smithi137 ceph-mon[90634]: pgmap v1768: 103 pgs: 1 peering, 102 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:50 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/320767559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:23:51.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:50 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:23:51.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:50 smithi086 ceph-mon[97913]: pgmap v1768: 103 pgs: 1 peering, 102 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:51.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:50 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/320767559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:23:51.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:50 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:23:51.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:50 smithi086 ceph-mon[102754]: pgmap v1768: 103 pgs: 1 peering, 102 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:51.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:50 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/320767559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:23:51.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:50 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-14T23:23:51.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd reweight 5 1 2024-06-14T23:23:52.186 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:52.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:51 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:23:52.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:51 smithi137 ceph-mon[90634]: osdmap e519: 8 total, 8 up, 8 in 2024-06-14T23:23:52.309 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:51 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:23:52.309 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:51 smithi086 ceph-mon[97913]: osdmap e519: 8 total, 8 up, 8 in 2024-06-14T23:23:52.309 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:51 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-14T23:23:52.309 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:51 smithi086 ceph-mon[102754]: osdmap e519: 8 total, 8 up, 8 in 2024-06-14T23:23:53.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:52 smithi137 ceph-mon[90634]: pgmap v1770: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:53.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:52 smithi086 ceph-mon[97913]: pgmap v1770: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:53.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:52 smithi086 ceph-mon[102754]: pgmap v1770: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:55.001 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.5 to 1 (10000) 2024-06-14T23:23:55.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:23:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:23:54] "GET /metrics HTTP/1.1" 200 34277 "" "Prometheus/2.43.0" 2024-06-14T23:23:55.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:54 smithi086 ceph-mon[97913]: pgmap v1771: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:55.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:54 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1891122340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:23:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:54 smithi086 ceph-mon[102754]: pgmap v1771: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:55.338 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:54 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1891122340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:23:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:55 smithi137 ceph-mon[90634]: pgmap v1771: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:55.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:55 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1891122340' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:23:55.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd primary-affinity 5 1 2024-06-14T23:23:55.997 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:56.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:56 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1891122340' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:23:56.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:56 smithi086 ceph-mon[97913]: osdmap e520: 8 total, 8 up, 8 in 2024-06-14T23:23:56.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:56 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1891122340' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:23:56.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:56 smithi086 ceph-mon[102754]: osdmap e520: 8 total, 8 up, 8 in 2024-06-14T23:23:56.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:56 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1891122340' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:23:56.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:56 smithi137 ceph-mon[90634]: osdmap e520: 8 total, 8 up, 8 in 2024-06-14T23:23:57.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:57 smithi086 ceph-mon[97913]: pgmap v1773: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:57.293 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:57 smithi086 ceph-mon[102754]: pgmap v1773: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:57.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:57 smithi137 ceph-mon[90634]: pgmap v1773: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:58.017 INFO:teuthology.orchestra.run.smithi086.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-06-14T23:23:58.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:58 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:58.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:58 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3448527972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:23:58.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:58 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:23:58.301 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:58 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:58.301 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:58 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3448527972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:23:58.301 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:58 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:23:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:58 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:23:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:58 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3448527972' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:23:58.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:58 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-14T23:23:58.724 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-06-14T23:23:58.724 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-06-14T23:23:58.724 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-06-14T23:23:58.724 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-06-14T23:23:58.724 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-14T23:23:58.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-14T23:23:58.984 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:23:59.275 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:59 smithi086 ceph-mon[102754]: pgmap v1774: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:59.275 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:59 smithi086 ceph-mon[102754]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:23:59.275 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:23:59 smithi086 ceph-mon[102754]: osdmap e521: 8 total, 8 up, 8 in 2024-06-14T23:23:59.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:59 smithi086 ceph-mon[97913]: pgmap v1774: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:59.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:59 smithi086 ceph-mon[97913]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:23:59.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:23:59 smithi086 ceph-mon[97913]: osdmap e521: 8 total, 8 up, 8 in 2024-06-14T23:23:59.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:59 smithi137 ceph-mon[90634]: pgmap v1774: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:23:59.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:59 smithi137 ceph-mon[90634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-06-14T23:23:59.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:23:59 smithi137 ceph-mon[90634]: osdmap e521: 8 total, 8 up, 8 in 2024-06-14T23:24:00.241 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:00 smithi086 ceph-mon[97913]: osdmap e522: 8 total, 8 up, 8 in 2024-06-14T23:24:00.241 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:00 smithi086 ceph-mon[102754]: osdmap e522: 8 total, 8 up, 8 in 2024-06-14T23:24:00.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:00 smithi137 ceph-mon[90634]: osdmap e522: 8 total, 8 up, 8 in 2024-06-14T23:24:00.941 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:24:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":522,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T23:23:59.012328+0000","last_up_change":"2024-06-14T23:22:49.985237+0000","last_in_change":"2024-06-14T23:22:23.447950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":25,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"288","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"285","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-14T22:33:31.126149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-06-14T22:34:44.677867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-06-14T22:34:45.034624+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":83,"up_from":91,"up_thru":516,"down_at":84,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6803","nonce":2971024428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6805","nonce":2971024428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6809","nonce":2971024428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6807","nonce":2971024428}]},"public_addr":"172.21.15.86:6803/2971024428","cluster_addr":"172.21.15.86:6805/2971024428","heartbeat_back_addr":"172.21.15.86:6809/2971024428","heartbeat_front_addr":"172.21.15.86:6807/2971024428","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":112,"up_from":116,"up_thru":504,"down_at":113,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6811","nonce":1358489517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6813","nonce":1358489517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6817","nonce":1358489517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6815","nonce":1358489517}]},"public_addr":"172.21.15.86:6811/1358489517","cluster_addr":"172.21.15.86:6813/1358489517","heartbeat_back_addr":"172.21.15.86:6817/1358489517","heartbeat_front_addr":"172.21.15.86:6815/1358489517","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":133,"last_clean_end":314,"up_from":322,"up_thru":516,"down_at":315,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3308282667},{"type":"v1","addr":"172.21.15.86:6819","nonce":3308282667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3308282667},{"type":"v1","addr":"172.21.15.86:6821","nonce":3308282667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3308282667},{"type":"v1","addr":"172.21.15.86:6825","nonce":3308282667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3308282667},{"type":"v1","addr":"172.21.15.86:6823","nonce":3308282667}]},"public_addr":"172.21.15.86:6819/3308282667","cluster_addr":"172.21.15.86:6821/3308282667","heartbeat_back_addr":"172.21.15.86:6825/3308282667","heartbeat_front_addr":"172.21.15.86:6823/3308282667","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":134,"up_from":138,"up_thru":505,"down_at":135,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6827","nonce":1034049285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6829","nonce":1034049285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6833","nonce":1034049285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6831","nonce":1034049285}]},"public_addr":"172.21.15.86:6827/1034049285","cluster_addr":"172.21.15.86:6829/1034049285","heartbeat_back_addr":"172.21.15.86:6833/1034049285","heartbeat_front_addr":"172.21.15.86:6831/1034049285","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":516,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":466,"up_from":492,"up_thru":521,"down_at":467,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":4252012775},{"type":"v1","addr":"172.21.15.137:6809","nonce":4252012775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":4252012775},{"type":"v1","addr":"172.21.15.137:6811","nonce":4252012775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":4252012775},{"type":"v1","addr":"172.21.15.137:6815","nonce":4252012775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":4252012775},{"type":"v1","addr":"172.21.15.137:6813","nonce":4252012775}]},"public_addr":"172.21.15.137:6809/4252012775","cluster_addr":"172.21.15.137:6811/4252012775","heartbeat_back_addr":"172.21.15.137:6815/4252012775","heartbeat_front_addr":"172.21.15.137:6813/4252012775","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":263,"up_from":272,"up_thru":514,"down_at":263,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2783214843},{"type":"v1","addr":"172.21.15.137:6817","nonce":2783214843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2783214843},{"type":"v1","addr":"172.21.15.137:6819","nonce":2783214843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2783214843},{"type":"v1","addr":"172.21.15.137:6823","nonce":2783214843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2783214843},{"type":"v1","addr":"172.21.15.137:6821","nonce":2783214843}]},"public_addr":"172.21.15.137:6817/2783214843","cluster_addr":"172.21.15.137:6819/2783214843","heartbeat_back_addr":"172.21.15.137:6823/2783214843","heartbeat_front_addr":"172.21.15.137:6821/2783214843","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":511,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":83},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":112},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":314},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":134},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":466},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":262},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:0/3481534205":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/1167860215":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/3024645077":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/2298288322":"2024-06-15T22:39:42.195791+0000","172.21.15.86:6801/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/418538598":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/358698000":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/1759721543":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2254669699":"2024-06-15T22:37:41.915228+0000","172.21.15.137:6834/3389028913":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/1328684620":"2024-06-15T22:39:42.195791+0000","172.21.15.137:0/2407056636":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/2914169147":"2024-06-15T22:38:44.349280+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3724463773":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/648695835":"2024-06-15T22:39:42.195791+0000","172.21.15.137:6835/3389028913":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/3471286695":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/2741142752":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6801/2254669699":"2024-06-15T22:37:41.915228+0000","172.21.15.137:0/2665974068":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/3094754408":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/3167284010":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/1921939101":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6801/1921939101":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/3941467574":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/1470436259":"2024-06-15T22:39:42.195791+0000","172.21.15.86:6800/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3107951276":"2024-06-15T22:37:41.915228+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T23:24:01.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:01 smithi086 ceph-mon[97913]: pgmap v1777: 103 pgs: 17 peering, 86 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:01.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:01 smithi086 ceph-mon[97913]: Health check failed: Reduced data availability: 6 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-06-14T23:24:01.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:01 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1155139780' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T23:24:01.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:01 smithi086 ceph-mon[102754]: pgmap v1777: 103 pgs: 17 peering, 86 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:01.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:01 smithi086 ceph-mon[102754]: Health check failed: Reduced data availability: 6 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-06-14T23:24:01.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:01 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1155139780' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T23:24:01.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:01 smithi137 ceph-mon[90634]: pgmap v1777: 103 pgs: 17 peering, 86 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:01.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:01 smithi137 ceph-mon[90634]: Health check failed: Reduced data availability: 6 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-06-14T23:24:01.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:01 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1155139780' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T23:24:01.643 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-14T23:24:01.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-14T23:24:02.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:02 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:24:02.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:02 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:24:02.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:02 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-14T23:24:02.742 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:03.131 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:03 smithi086 ceph-mon[97913]: pgmap v1778: 103 pgs: 17 peering, 86 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:03.131 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:03 smithi086 ceph-mon[102754]: pgmap v1778: 103 pgs: 17 peering, 86 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:03.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:03 smithi137 ceph-mon[90634]: pgmap v1778: 103 pgs: 17 peering, 86 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:04.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:04 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:04.229 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:04 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:04.436 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:04 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:04.688 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:24:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":522,"fsid":"44b2848a-2a9d-11ef-bc9d-c7b262605968","created":"2024-06-14T22:28:02.984269+0000","modified":"2024-06-14T23:23:59.012328+0000","last_up_change":"2024-06-14T23:22:49.985237+0000","last_in_change":"2024-06-14T23:22:23.447950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":25,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-14T22:31:01.536068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"288","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"285","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-14T22:33:31.126149+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-06-14T22:34:44.677867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-06-14T22:34:45.034624+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"db19a6c7-4c6f-4c95-aa2f-3ddc1e121270","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":83,"up_from":91,"up_thru":516,"down_at":84,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6803","nonce":2971024428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6805","nonce":2971024428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6809","nonce":2971024428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2971024428},{"type":"v1","addr":"172.21.15.86:6807","nonce":2971024428}]},"public_addr":"172.21.15.86:6803/2971024428","cluster_addr":"172.21.15.86:6805/2971024428","heartbeat_back_addr":"172.21.15.86:6809/2971024428","heartbeat_front_addr":"172.21.15.86:6807/2971024428","state":["exists","up"]},{"osd":1,"uuid":"f021e24a-e7b4-46f0-94ed-4db138b60d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":112,"up_from":116,"up_thru":504,"down_at":113,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6811","nonce":1358489517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6813","nonce":1358489517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6817","nonce":1358489517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1358489517},{"type":"v1","addr":"172.21.15.86:6815","nonce":1358489517}]},"public_addr":"172.21.15.86:6811/1358489517","cluster_addr":"172.21.15.86:6813/1358489517","heartbeat_back_addr":"172.21.15.86:6817/1358489517","heartbeat_front_addr":"172.21.15.86:6815/1358489517","state":["exists","up"]},{"osd":2,"uuid":"65dbd353-d1a7-4f6d-8112-b77fbc51f9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":133,"last_clean_end":314,"up_from":322,"up_thru":516,"down_at":315,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3308282667},{"type":"v1","addr":"172.21.15.86:6819","nonce":3308282667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3308282667},{"type":"v1","addr":"172.21.15.86:6821","nonce":3308282667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":3308282667},{"type":"v1","addr":"172.21.15.86:6825","nonce":3308282667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3308282667},{"type":"v1","addr":"172.21.15.86:6823","nonce":3308282667}]},"public_addr":"172.21.15.86:6819/3308282667","cluster_addr":"172.21.15.86:6821/3308282667","heartbeat_back_addr":"172.21.15.86:6825/3308282667","heartbeat_front_addr":"172.21.15.86:6823/3308282667","state":["exists","up"]},{"osd":3,"uuid":"5e141877-8275-4b3a-bb90-277e99a725d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":134,"up_from":138,"up_thru":505,"down_at":135,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6827","nonce":1034049285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6829","nonce":1034049285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6833","nonce":1034049285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1034049285},{"type":"v1","addr":"172.21.15.86:6831","nonce":1034049285}]},"public_addr":"172.21.15.86:6827/1034049285","cluster_addr":"172.21.15.86:6829/1034049285","heartbeat_back_addr":"172.21.15.86:6833/1034049285","heartbeat_front_addr":"172.21.15.86:6831/1034049285","state":["exists","up"]},{"osd":4,"uuid":"4f0bd061-11f3-4d2c-9395-65c3ae5489f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":516,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6801","nonce":3721662531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6803","nonce":3721662531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6807","nonce":3721662531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3721662531},{"type":"v1","addr":"172.21.15.137:6805","nonce":3721662531}]},"public_addr":"172.21.15.137:6801/3721662531","cluster_addr":"172.21.15.137:6803/3721662531","heartbeat_back_addr":"172.21.15.137:6807/3721662531","heartbeat_front_addr":"172.21.15.137:6805/3721662531","state":["exists","up"]},{"osd":5,"uuid":"c27edac8-f0e3-4b4c-a1e6-9e57165aea80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":466,"up_from":492,"up_thru":521,"down_at":467,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":4252012775},{"type":"v1","addr":"172.21.15.137:6809","nonce":4252012775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":4252012775},{"type":"v1","addr":"172.21.15.137:6811","nonce":4252012775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":4252012775},{"type":"v1","addr":"172.21.15.137:6815","nonce":4252012775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":4252012775},{"type":"v1","addr":"172.21.15.137:6813","nonce":4252012775}]},"public_addr":"172.21.15.137:6809/4252012775","cluster_addr":"172.21.15.137:6811/4252012775","heartbeat_back_addr":"172.21.15.137:6815/4252012775","heartbeat_front_addr":"172.21.15.137:6813/4252012775","state":["exists","up"]},{"osd":6,"uuid":"44e654f2-c297-49dd-94c0-9d0ef90bffe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":263,"up_from":272,"up_thru":514,"down_at":263,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":2783214843},{"type":"v1","addr":"172.21.15.137:6817","nonce":2783214843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":2783214843},{"type":"v1","addr":"172.21.15.137:6819","nonce":2783214843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":2783214843},{"type":"v1","addr":"172.21.15.137:6823","nonce":2783214843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":2783214843},{"type":"v1","addr":"172.21.15.137:6821","nonce":2783214843}]},"public_addr":"172.21.15.137:6817/2783214843","cluster_addr":"172.21.15.137:6819/2783214843","heartbeat_back_addr":"172.21.15.137:6823/2783214843","heartbeat_front_addr":"172.21.15.137:6821/2783214843","state":["exists","up"]},{"osd":7,"uuid":"f867732b-d642-4650-a57e-b01280b47e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":511,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6825","nonce":298867786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6827","nonce":298867786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6831","nonce":298867786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":298867786},{"type":"v1","addr":"172.21.15.137:6829","nonce":298867786}]},"public_addr":"172.21.15.137:6825/298867786","cluster_addr":"172.21.15.137:6827/298867786","heartbeat_back_addr":"172.21.15.137:6831/298867786","heartbeat_front_addr":"172.21.15.137:6829/298867786","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:11.219694+0000","dead_epoch":83},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:35.276986+0000","dead_epoch":112},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:30:59.272012+0000","dead_epoch":314},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:24.285444+0000","dead_epoch":134},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:31:45.927328+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:09.586220+0000","dead_epoch":466},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:33.228392+0000","dead_epoch":262},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-14T22:32:57.320995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:0/3481534205":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/1167860215":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/3024645077":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/2298288322":"2024-06-15T22:39:42.195791+0000","172.21.15.86:6801/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/418538598":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/358698000":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/1759721543":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/2254669699":"2024-06-15T22:37:41.915228+0000","172.21.15.137:6834/3389028913":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/1328684620":"2024-06-15T22:39:42.195791+0000","172.21.15.137:0/2407056636":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/2914169147":"2024-06-15T22:38:44.349280+0000","172.21.15.86:6801/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/3724463773":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/648695835":"2024-06-15T22:39:42.195791+0000","172.21.15.137:6835/3389028913":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/3471286695":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/1554099440":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/2713243150":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/2741142752":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6801/2254669699":"2024-06-15T22:37:41.915228+0000","172.21.15.137:0/2665974068":"2024-06-15T22:38:44.349280+0000","172.21.15.137:0/3094754408":"2024-06-15T22:38:44.349280+0000","172.21.15.86:0/3167284010":"2024-06-15T22:37:41.915228+0000","172.21.15.86:0/2027707012":"2024-06-15T22:28:23.452282+0000","172.21.15.86:6801/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3161947015":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/1226112395":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6800/1921939101":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/2504812592":"2024-06-15T22:28:53.238347+0000","172.21.15.86:6801/1921939101":"2024-06-15T22:39:42.195791+0000","172.21.15.86:0/3941467574":"2024-06-15T22:33:27.518416+0000","172.21.15.86:0/2139637654":"2024-06-15T22:28:23.452282+0000","172.21.15.86:0/1470436259":"2024-06-15T22:39:42.195791+0000","172.21.15.86:6800/3866885055":"2024-06-15T22:33:27.518416+0000","172.21.15.86:6800/3390625982":"2024-06-15T22:28:53.238347+0000","172.21.15.86:0/3107951276":"2024-06-15T22:37:41.915228+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-14T23:24:05.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:04 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:24:04] "GET /metrics HTTP/1.1" 200 34255 "" "Prometheus/2.43.0" 2024-06-14T23:24:05.408 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:05 smithi137 ceph-mon[90634]: pgmap v1779: 103 pgs: 17 peering, 86 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:05.408 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:05 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2566868332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T23:24:05.511 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:05 smithi086 ceph-mon[97913]: pgmap v1779: 103 pgs: 17 peering, 86 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:05.511 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:05 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2566868332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T23:24:05.511 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:05 smithi086 ceph-mon[102754]: pgmap v1779: 103 pgs: 17 peering, 86 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:05.511 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:05 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2566868332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-14T23:24:05.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-14T23:24:05.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-14T23:24:05.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-14T23:24:05.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-14T23:24:05.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-14T23:24:05.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-14T23:24:05.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-14T23:24:05.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-14T23:24:06.573 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:06.573 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:06.576 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:06.578 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:06.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:06.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:06 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:06.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:06 smithi086 ceph-mon[97913]: pgmap v1780: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:06.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:06.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:06 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:06.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:06 smithi086 ceph-mon[102754]: pgmap v1780: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:06.966 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:06.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:06.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:06 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:06.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:06 smithi137 ceph-mon[90634]: pgmap v1780: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:07.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:07 smithi086 ceph-mon[97913]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 15 pgs peering) 2024-06-14T23:24:07.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:07 smithi086 ceph-mon[97913]: Cluster is now healthy 2024-06-14T23:24:07.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:07 smithi086 ceph-mon[102754]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 15 pgs peering) 2024-06-14T23:24:07.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:07 smithi086 ceph-mon[102754]: Cluster is now healthy 2024-06-14T23:24:07.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:07 smithi137 ceph-mon[90634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 15 pgs peering) 2024-06-14T23:24:07.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:07 smithi137 ceph-mon[90634]: Cluster is now healthy 2024-06-14T23:24:08.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:08 smithi086 ceph-mon[97913]: pgmap v1781: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:08.714 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:08 smithi086 ceph-mon[102754]: pgmap v1781: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:08.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:08 smithi137 ceph-mon[90634]: pgmap v1781: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:09.736 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:09.736 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:11.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:10 smithi137 ceph-mon[90634]: pgmap v1782: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:11.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:10 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:11.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:10 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:11.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:10 smithi086 ceph-mon[102754]: pgmap v1782: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:11.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:10 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:11.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:10 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:11.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:10 smithi086 ceph-mon[97913]: pgmap v1782: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:11.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:10 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:11.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:10 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:13.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:12 smithi086 ceph-mon[102754]: pgmap v1783: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:13.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:13.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:13.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:12 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:24:13.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:12 smithi086 ceph-mon[97913]: pgmap v1783: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:13.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:13.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:13.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:12 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:24:13.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:12 smithi137 ceph-mon[90634]: pgmap v1783: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:13.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:13.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:13.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:12 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:24:13.718 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:14.309 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:14 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:14.309 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:14 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:14.319 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:14 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:15.122 INFO:teuthology.orchestra.run.smithi086.stdout:2113123909651 2024-06-14T23:24:15.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-14T23:24:15.130 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:14 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:24:14] "GET /metrics HTTP/1.1" 200 34256 "" "Prometheus/2.43.0" 2024-06-14T23:24:15.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:15 smithi137 ceph-mon[90634]: pgmap v1784: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:15.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:15 smithi086 ceph-mon[97913]: pgmap v1784: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:15.586 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:15 smithi086 ceph-mon[102754]: pgmap v1784: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:17.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:17 smithi086 ceph-mon[97913]: pgmap v1785: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:17.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:17 smithi086 ceph-mon[102754]: pgmap v1785: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:17.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:17 smithi137 ceph-mon[90634]: pgmap v1785: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:18.476 INFO:teuthology.orchestra.run.smithi086.stdout:1382979469468 2024-06-14T23:24:18.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-14T23:24:19.440 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:19 smithi086 ceph-mon[97913]: pgmap v1786: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:19.441 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:19 smithi086 ceph-mon[102754]: pgmap v1786: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:19.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:19 smithi137 ceph-mon[90634]: pgmap v1786: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:21.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:21 smithi137 ceph-mon[90634]: pgmap v1787: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:21.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:21 smithi086 ceph-mon[97913]: pgmap v1787: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:21.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:21 smithi086 ceph-mon[102754]: pgmap v1787: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:21.644 INFO:teuthology.orchestra.run.smithi086.stdout:592705487214 2024-06-14T23:24:21.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-14T23:24:22.350 INFO:teuthology.orchestra.run.smithi086.stdout:231928234604 2024-06-14T23:24:22.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-14T23:24:23.415 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:23 smithi086 ceph-mon[97913]: pgmap v1788: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:23.415 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:23 smithi086 ceph-mon[102754]: pgmap v1788: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:23.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:23 smithi137 ceph-mon[90634]: pgmap v1788: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:23.840 INFO:teuthology.orchestra.run.smithi086.stdout:1168231104710 2024-06-14T23:24:23.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-14T23:24:24.651 INFO:teuthology.orchestra.run.smithi086.stdout:498216206757 2024-06-14T23:24:24.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-14T23:24:24.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:24 smithi086 ceph-mon[97913]: pgmap v1789: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:24.832 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:24 smithi086 ceph-mon[102754]: pgmap v1789: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:24.996 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:24 smithi137 ceph-mon[90634]: pgmap v1789: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:25.069 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:25.087 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:24 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:24:24] "GET /metrics HTTP/1.1" 200 34256 "" "Prometheus/2.43.0" 2024-06-14T23:24:25.308 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:25.309 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:26.251 INFO:teuthology.orchestra.run.smithi086.stdout:128849019516 2024-06-14T23:24:26.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-14T23:24:27.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:26 smithi086 ceph-mon[97913]: pgmap v1790: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:27.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:26 smithi086 ceph-mon[102754]: pgmap v1790: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:27.245 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:26 smithi137 ceph-mon[90634]: pgmap v1790: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:28.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:27 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:24:28.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:27 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:24:28.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:27 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:24:28.413 INFO:teuthology.orchestra.run.smithi086.stdout:390842024414 2024-06-14T23:24:28.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-14T23:24:29.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:28 smithi137 ceph-mon[90634]: pgmap v1791: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:29.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:28 smithi086 ceph-mon[97913]: pgmap v1791: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:29.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:28 smithi086 ceph-mon[102754]: pgmap v1791: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:31.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:30 smithi086 ceph-mon[97913]: pgmap v1792: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:31.087 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:30 smithi086 ceph-mon[102754]: pgmap v1792: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:31.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:30 smithi137 ceph-mon[90634]: pgmap v1792: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:32.701 INFO:teuthology.orchestra.run.smithi086.stdout:1168231104712 2024-06-14T23:24:33.038 INFO:teuthology.orchestra.run.smithi086.stdout:1382979469471 2024-06-14T23:24:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:32 smithi086 ceph-mon[97913]: pgmap v1793: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:33.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:32 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/17738300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T23:24:33.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:32 smithi086 ceph-mon[102754]: pgmap v1793: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:33.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:32 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/17738300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T23:24:33.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:32 smithi137 ceph-mon[90634]: pgmap v1793: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:33.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:32 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/17738300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-14T23:24:33.921 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1382979469468 got 1382979469471 for osd.2 2024-06-14T23:24:33.921 DEBUG:teuthology.parallel:result is None 2024-06-14T23:24:34.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:34 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2401426356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T23:24:34.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:34 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2401426356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T23:24:34.412 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1168231104710 got 1168231104712 for osd.6 2024-06-14T23:24:34.413 DEBUG:teuthology.parallel:result is None 2024-06-14T23:24:34.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:34 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2401426356' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-14T23:24:34.759 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:34.760 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:34.761 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:34.983 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:24:34] "GET /metrics HTTP/1.1" 200 34257 "" "Prometheus/2.43.0" 2024-06-14T23:24:35.302 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:35 smithi086 ceph-mon[97913]: pgmap v1794: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:35.302 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:35 smithi086 ceph-mon[102754]: pgmap v1794: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:35.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:35 smithi137 ceph-mon[90634]: pgmap v1794: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:35.678 INFO:teuthology.orchestra.run.smithi086.stdout:231928234606 2024-06-14T23:24:36.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:36 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/2575074734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T23:24:36.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:36 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/2575074734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T23:24:36.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:36 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/2575074734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-14T23:24:36.498 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234604 got 231928234606 for osd.7 2024-06-14T23:24:36.498 DEBUG:teuthology.parallel:result is None 2024-06-14T23:24:37.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:37 smithi086 ceph-mon[97913]: pgmap v1795: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:37.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:37 smithi086 ceph-mon[102754]: pgmap v1795: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:37.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:37 smithi137 ceph-mon[90634]: pgmap v1795: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:38.618 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:38.619 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:39.232 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[102754]: pgmap v1796: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:39.232 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[97913]: pgmap v1796: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:24:39.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:39 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:39 smithi137 ceph-mon[90634]: pgmap v1796: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:39 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:39 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:39 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-14T23:24:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:39 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-14T23:24:39.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:39 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' 2024-06-14T23:24:41.311 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:41 smithi086 ceph-mon[102754]: pgmap v1797: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:41.311 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:41 smithi086 ceph-mon[97913]: pgmap v1797: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:41.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:41 smithi137 ceph-mon[90634]: pgmap v1797: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:43.298 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:43 smithi137 ceph-mon[90634]: pgmap v1798: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:43.298 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:43 smithi137 ceph-mon[90634]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:24:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:43 smithi086 ceph-mon[97913]: pgmap v1798: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:43.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:43 smithi086 ceph-mon[97913]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:24:43.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:43 smithi086 ceph-mon[102754]: pgmap v1798: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:43.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:43 smithi086 ceph-mon[102754]: from='mgr.25130 172.21.15.86:0/2720355033' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-14T23:24:44.988 INFO:teuthology.orchestra.run.smithi086.stdout:498216206763 2024-06-14T23:24:44.988 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:44 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:24:44] "GET /metrics HTTP/1.1" 200 34260 "" "Prometheus/2.43.0" 2024-06-14T23:24:45.313 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:45 smithi086 ceph-mon[102754]: pgmap v1799: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:45.314 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1250422133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T23:24:45.314 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:45 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/687573778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T23:24:45.314 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:45 smithi086 ceph-mon[97913]: pgmap v1799: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:45.314 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1250422133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T23:24:45.314 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:45 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/687573778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T23:24:45.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:45 smithi137 ceph-mon[90634]: pgmap v1799: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:45.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1250422133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-14T23:24:45.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:45 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/687573778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-14T23:24:45.584 INFO:teuthology.orchestra.run.smithi086.stdout:390842024417 2024-06-14T23:24:46.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:46 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/4067507893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T23:24:46.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:46 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/4067507893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T23:24:46.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:46 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/4067507893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-14T23:24:46.503 INFO:tasks.cephadm.ceph_manager.ceph:need seq 390842024414 got 390842024417 for osd.0 2024-06-14T23:24:46.503 DEBUG:teuthology.parallel:result is None 2024-06-14T23:24:46.643 INFO:teuthology.orchestra.run.smithi086.stdout:2113123909657 2024-06-14T23:24:47.141 INFO:tasks.cephadm.ceph_manager.ceph:need seq 498216206757 got 498216206763 for osd.1 2024-06-14T23:24:47.141 DEBUG:teuthology.parallel:result is None 2024-06-14T23:24:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:47 smithi086 ceph-mon[97913]: pgmap v1800: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:47.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:47 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/3417595217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T23:24:47.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:47 smithi086 ceph-mon[102754]: pgmap v1800: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:47.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:47 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/3417595217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T23:24:47.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:47 smithi137 ceph-mon[90634]: pgmap v1800: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:47.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:47 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/3417595217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-14T23:24:47.796 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2113123909651 got 2113123909657 for osd.5 2024-06-14T23:24:47.796 DEBUG:teuthology.parallel:result is None 2024-06-14T23:24:47.909 INFO:teuthology.orchestra.run.smithi086.stdout:128849019520 2024-06-14T23:24:48.660 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019516 got 128849019520 for osd.4 2024-06-14T23:24:48.660 DEBUG:teuthology.parallel:result is None 2024-06-14T23:24:49.077 INFO:teuthology.orchestra.run.smithi086.stdout:592705487219 2024-06-14T23:24:49.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:49 smithi086 ceph-mon[102754]: pgmap v1801: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:49.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:49 smithi086 ceph-mon[97913]: pgmap v1801: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:49.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:49 smithi137 ceph-mon[90634]: pgmap v1801: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:49.719 INFO:tasks.cephadm.ceph_manager.ceph:need seq 592705487214 got 592705487219 for osd.3 2024-06-14T23:24:49.720 DEBUG:teuthology.parallel:result is None 2024-06-14T23:24:49.720 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-14T23:24:49.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:24:49.975 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:50.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:50 smithi086 ceph-mon[97913]: from='client.? 172.21.15.86:0/1112330641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T23:24:50.304 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:50 smithi086 ceph-mon[102754]: from='client.? 172.21.15.86:0/1112330641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T23:24:50.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:50 smithi137 ceph-mon[90634]: from='client.? 172.21.15.86:0/1112330641' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-14T23:24:51.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:51 smithi086 ceph-mon[97913]: pgmap v1802: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:51.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:51 smithi086 ceph-mon[102754]: pgmap v1802: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:51.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:51 smithi137 ceph-mon[90634]: pgmap v1802: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:52.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:24:52.086 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:24:52.821 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1803,"stamp":"2024-06-14T23:24:51.936314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":591710,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5048,"ondisk_log_size":5048,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8246352,"kb_used_data":55764,"kb_used_omap":45,"kb_used_meta":8190418,"kb_avail":741714864,"statfs":{"total":767960285184,"available":759516020736,"internally_reserved":0,"allocated":57102336,"data_stored":35057726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46341,"internal_metadata":8386988795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003009"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":800,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023850+0000","last_change":"2024-06-14T23:23:20.288570+0000","last_active":"2024-06-14T23:23:58.023850+0000","last_peered":"2024-06-14T23:23:58.023850+0000","last_clean":"2024-06-14T23:23:58.023850+0000","last_became_active":"2024-06-14T23:23:20.194560+0000","last_became_peered":"2024-06-14T23:23:20.194560+0000","last_unstale":"2024-06-14T23:23:58.023850+0000","last_undegraded":"2024-06-14T23:23:58.023850+0000","last_fullsized":"2024-06-14T23:23:58.023850+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:20.288454+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:23:20.288454+0000","last_clean_scrub_stamp":"2024-06-14T23:23:20.288454+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:53.462606+0000","scrub_duration":0.0015503959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":257,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.022024+0000","last_change":"2024-06-14T23:23:06.104075+0000","last_active":"2024-06-14T23:23:58.022024+0000","last_peered":"2024-06-14T23:23:58.022024+0000","last_clean":"2024-06-14T23:23:58.022024+0000","last_became_active":"2024-06-14T23:23:06.103680+0000","last_became_peered":"2024-06-14T23:23:06.103680+0000","last_unstale":"2024-06-14T23:23:58.022024+0000","last_undegraded":"2024-06-14T23:23:58.022024+0000","last_fullsized":"2024-06-14T23:23:58.022024+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:17:16.172095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":496,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024564+0000","last_change":"2024-06-14T23:23:46.536793+0000","last_active":"2024-06-14T23:23:58.024564+0000","last_peered":"2024-06-14T23:23:58.024564+0000","last_clean":"2024-06-14T23:23:58.024564+0000","last_became_active":"2024-06-14T23:23:46.536459+0000","last_became_peered":"2024-06-14T23:23:46.536459+0000","last_unstale":"2024-06-14T23:23:58.024564+0000","last_undegraded":"2024-06-14T23:23:58.024564+0000","last_fullsized":"2024-06-14T23:23:58.024564+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:20:12.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":112,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019452+0000","last_change":"2024-06-14T23:23:21.191748+0000","last_active":"2024-06-14T23:23:58.019452+0000","last_peered":"2024-06-14T23:23:58.019452+0000","last_clean":"2024-06-14T23:23:58.019452+0000","last_became_active":"2024-06-14T23:23:21.191436+0000","last_became_peered":"2024-06-14T23:23:21.191436+0000","last_unstale":"2024-06-14T23:23:58.019452+0000","last_undegraded":"2024-06-14T23:23:58.019452+0000","last_fullsized":"2024-06-14T23:23:58.019452+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:46:29.102290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":118,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018943+0000","last_change":"2024-06-14T23:23:21.190170+0000","last_active":"2024-06-14T23:23:58.018943+0000","last_peered":"2024-06-14T23:23:58.018943+0000","last_clean":"2024-06-14T23:23:58.018943+0000","last_became_active":"2024-06-14T23:23:21.189878+0000","last_became_peered":"2024-06-14T23:23:21.189878+0000","last_unstale":"2024-06-14T23:23:58.018943+0000","last_undegraded":"2024-06-14T23:23:58.018943+0000","last_fullsized":"2024-06-14T23:23:58.018943+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:02:02.826758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":778,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.016965+0000","last_change":"2024-06-14T23:23:36.146519+0000","last_active":"2024-06-14T23:23:58.016965+0000","last_peered":"2024-06-14T23:23:58.016965+0000","last_clean":"2024-06-14T23:23:58.016965+0000","last_became_active":"2024-06-14T23:23:36.146150+0000","last_became_peered":"2024-06-14T23:23:36.146150+0000","last_unstale":"2024-06-14T23:23:58.016965+0000","last_undegraded":"2024-06-14T23:23:58.016965+0000","last_fullsized":"2024-06-14T23:23:58.016965+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:14:16.297860+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:14:16.297860+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:46:35.959931+0000","scrub_duration":0.0039154719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":366,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.022418+0000","last_change":"2024-06-14T23:23:59.022418+0000","last_active":"2024-06-14T23:23:59.022418+0000","last_peered":"2024-06-14T23:23:59.022418+0000","last_clean":"2024-06-14T23:23:59.022418+0000","last_became_active":"2024-06-14T23:23:59.022069+0000","last_became_peered":"2024-06-14T23:23:59.022069+0000","last_unstale":"2024-06-14T23:23:59.022418+0000","last_undegraded":"2024-06-14T23:23:59.022418+0000","last_fullsized":"2024-06-14T23:23:59.022418+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:00.688565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":508,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026423+0000","last_change":"2024-06-14T23:23:28.038120+0000","last_active":"2024-06-14T23:23:58.026423+0000","last_peered":"2024-06-14T23:23:58.026423+0000","last_clean":"2024-06-14T23:23:58.026423+0000","last_became_active":"2024-06-14T23:23:28.037605+0000","last_became_peered":"2024-06-14T23:23:28.037605+0000","last_unstale":"2024-06-14T23:23:58.026423+0000","last_undegraded":"2024-06-14T23:23:58.026423+0000","last_fullsized":"2024-06-14T23:23:58.026423+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:40:18.816673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":464,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019752+0000","last_change":"2024-06-14T23:23:22.971293+0000","last_active":"2024-06-14T23:23:58.019752+0000","last_peered":"2024-06-14T23:23:58.019752+0000","last_clean":"2024-06-14T23:23:58.019752+0000","last_became_active":"2024-06-14T23:23:20.199775+0000","last_became_peered":"2024-06-14T23:23:20.199775+0000","last_unstale":"2024-06-14T23:23:58.019752+0000","last_undegraded":"2024-06-14T23:23:58.019752+0000","last_fullsized":"2024-06-14T23:23:58.019752+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:22.971203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:22.971203+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:26:16.160391+0000","scrub_duration":0.001713518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":440,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.025204+0000","last_change":"2024-06-14T23:23:59.025204+0000","last_active":"2024-06-14T23:23:59.025204+0000","last_peered":"2024-06-14T23:23:59.025204+0000","last_clean":"2024-06-14T23:23:59.025204+0000","last_became_active":"2024-06-14T23:23:59.023607+0000","last_became_peered":"2024-06-14T23:23:59.023607+0000","last_unstale":"2024-06-14T23:23:59.025204+0000","last_undegraded":"2024-06-14T23:23:59.025204+0000","last_fullsized":"2024-06-14T23:23:59.025204+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:27:51.272020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":517,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026552+0000","last_change":"2024-06-14T23:23:28.031970+0000","last_active":"2024-06-14T23:23:58.026552+0000","last_peered":"2024-06-14T23:23:58.026552+0000","last_clean":"2024-06-14T23:23:58.026552+0000","last_became_active":"2024-06-14T23:23:28.031625+0000","last_became_peered":"2024-06-14T23:23:28.031625+0000","last_unstale":"2024-06-14T23:23:58.026552+0000","last_undegraded":"2024-06-14T23:23:58.026552+0000","last_fullsized":"2024-06-14T23:23:58.026552+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:08:44.108966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":149,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021736+0000","last_change":"2024-06-14T23:17:22.888832+0000","last_active":"2024-06-14T23:23:58.021736+0000","last_peered":"2024-06-14T23:23:58.021736+0000","last_clean":"2024-06-14T23:23:58.021736+0000","last_became_active":"2024-06-14T23:17:22.888525+0000","last_became_peered":"2024-06-14T23:17:22.888525+0000","last_unstale":"2024-06-14T23:23:58.021736+0000","last_undegraded":"2024-06-14T23:23:58.021736+0000","last_fullsized":"2024-06-14T23:23:58.021736+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:36.414213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":122,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.014675+0000","last_change":"2024-06-14T23:23:46.990897+0000","last_active":"2024-06-14T23:23:58.014675+0000","last_peered":"2024-06-14T23:23:58.014675+0000","last_clean":"2024-06-14T23:23:58.014675+0000","last_became_active":"2024-06-14T23:23:46.990507+0000","last_became_peered":"2024-06-14T23:23:46.990507+0000","last_unstale":"2024-06-14T23:23:58.014675+0000","last_undegraded":"2024-06-14T23:23:58.014675+0000","last_fullsized":"2024-06-14T23:23:58.014675+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:31:46.522707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":741,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018785+0000","last_change":"2024-06-14T23:23:21.191578+0000","last_active":"2024-06-14T23:23:58.018785+0000","last_peered":"2024-06-14T23:23:58.018785+0000","last_clean":"2024-06-14T23:23:58.018785+0000","last_became_active":"2024-06-14T23:23:21.191188+0000","last_became_peered":"2024-06-14T23:23:21.191188+0000","last_unstale":"2024-06-14T23:23:58.018785+0000","last_undegraded":"2024-06-14T23:23:58.018785+0000","last_fullsized":"2024-06-14T23:23:58.018785+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:26.447181+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":441,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026518+0000","last_change":"2024-06-14T23:23:28.035129+0000","last_active":"2024-06-14T23:23:58.026518+0000","last_peered":"2024-06-14T23:23:58.026518+0000","last_clean":"2024-06-14T23:23:58.026518+0000","last_became_active":"2024-06-14T23:23:28.034629+0000","last_became_peered":"2024-06-14T23:23:28.034629+0000","last_unstale":"2024-06-14T23:23:58.026518+0000","last_undegraded":"2024-06-14T23:23:58.026518+0000","last_fullsized":"2024-06-14T23:23:58.026518+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:37.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":58,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019436+0000","last_change":"2024-06-14T23:23:21.291179+0000","last_active":"2024-06-14T23:23:58.019436+0000","last_peered":"2024-06-14T23:23:58.019436+0000","last_clean":"2024-06-14T23:23:58.019436+0000","last_became_active":"2024-06-14T23:23:21.290706+0000","last_became_peered":"2024-06-14T23:23:21.290706+0000","last_unstale":"2024-06-14T23:23:58.019436+0000","last_undegraded":"2024-06-14T23:23:58.019436+0000","last_fullsized":"2024-06-14T23:23:58.019436+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:42:06.026436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":879,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.027604+0000","last_change":"2024-06-14T23:23:59.027604+0000","last_active":"2024-06-14T23:23:59.027604+0000","last_peered":"2024-06-14T23:23:59.027604+0000","last_clean":"2024-06-14T23:23:59.027604+0000","last_became_active":"2024-06-14T23:23:59.027183+0000","last_became_peered":"2024-06-14T23:23:59.027183+0000","last_unstale":"2024-06-14T23:23:59.027604+0000","last_undegraded":"2024-06-14T23:23:59.027604+0000","last_fullsized":"2024-06-14T23:23:59.027604+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:48:40.059953+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":51,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.016996+0000","last_change":"2024-06-14T23:22:23.531169+0000","last_active":"2024-06-14T23:23:59.016996+0000","last_peered":"2024-06-14T23:23:59.016996+0000","last_clean":"2024-06-14T23:23:59.016996+0000","last_became_active":"2024-06-14T23:22:23.530840+0000","last_became_peered":"2024-06-14T23:22:23.530840+0000","last_unstale":"2024-06-14T23:23:59.016996+0000","last_undegraded":"2024-06-14T23:23:59.016996+0000","last_fullsized":"2024-06-14T23:23:59.016996+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:13:04.472958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":359,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.020678+0000","last_change":"2024-06-14T23:23:17.169294+0000","last_active":"2024-06-14T23:23:59.020678+0000","last_peered":"2024-06-14T23:23:59.020678+0000","last_clean":"2024-06-14T23:23:59.020678+0000","last_became_active":"2024-06-14T23:23:17.168995+0000","last_became_peered":"2024-06-14T23:23:17.168995+0000","last_unstale":"2024-06-14T23:23:59.020678+0000","last_undegraded":"2024-06-14T23:23:59.020678+0000","last_fullsized":"2024-06-14T23:23:59.020678+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:43:01.161977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":833,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.026517+0000","last_change":"2024-06-14T23:23:59.026517+0000","last_active":"2024-06-14T23:23:59.026517+0000","last_peered":"2024-06-14T23:23:59.026517+0000","last_clean":"2024-06-14T23:23:59.026517+0000","last_became_active":"2024-06-14T23:23:59.025650+0000","last_became_peered":"2024-06-14T23:23:59.025650+0000","last_unstale":"2024-06-14T23:23:59.026517+0000","last_undegraded":"2024-06-14T23:23:59.026517+0000","last_fullsized":"2024-06-14T23:23:59.026517+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:31.405872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:31.405872+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:04:19.311760+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":221,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018719+0000","last_change":"2024-06-14T23:23:21.291096+0000","last_active":"2024-06-14T23:23:58.018719+0000","last_peered":"2024-06-14T23:23:58.018719+0000","last_clean":"2024-06-14T23:23:58.018719+0000","last_became_active":"2024-06-14T23:23:21.290798+0000","last_became_peered":"2024-06-14T23:23:21.290798+0000","last_unstale":"2024-06-14T23:23:58.018719+0000","last_undegraded":"2024-06-14T23:23:58.018719+0000","last_fullsized":"2024-06-14T23:23:58.018719+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:31:41.895801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":514,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025244+0000","last_change":"2024-06-14T23:22:52.005874+0000","last_active":"2024-06-14T23:23:58.025244+0000","last_peered":"2024-06-14T23:23:58.025244+0000","last_clean":"2024-06-14T23:23:58.025244+0000","last_became_active":"2024-06-14T23:22:52.005570+0000","last_became_peered":"2024-06-14T23:22:52.005570+0000","last_unstale":"2024-06-14T23:23:58.025244+0000","last_undegraded":"2024-06-14T23:23:58.025244+0000","last_fullsized":"2024-06-14T23:23:58.025244+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:09:16.899262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":806,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025797+0000","last_change":"2024-06-14T23:23:28.041830+0000","last_active":"2024-06-14T23:23:58.025797+0000","last_peered":"2024-06-14T23:23:58.025797+0000","last_clean":"2024-06-14T23:23:58.025797+0000","last_became_active":"2024-06-14T23:23:28.039425+0000","last_became_peered":"2024-06-14T23:23:28.039425+0000","last_unstale":"2024-06-14T23:23:58.025797+0000","last_undegraded":"2024-06-14T23:23:58.025797+0000","last_fullsized":"2024-06-14T23:23:58.025797+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:08.026179+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019684+0000","last_change":"2024-06-14T23:23:17.163583+0000","last_active":"2024-06-14T23:23:58.019684+0000","last_peered":"2024-06-14T23:23:58.019684+0000","last_clean":"2024-06-14T23:23:58.019684+0000","last_became_active":"2024-06-14T23:23:17.163254+0000","last_became_peered":"2024-06-14T23:23:17.163254+0000","last_unstale":"2024-06-14T23:23:58.019684+0000","last_undegraded":"2024-06-14T23:23:58.019684+0000","last_fullsized":"2024-06-14T23:23:58.019684+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:50.673160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":298,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021932+0000","last_change":"2024-06-14T23:23:17.170000+0000","last_active":"2024-06-14T23:23:58.021932+0000","last_peered":"2024-06-14T23:23:58.021932+0000","last_clean":"2024-06-14T23:23:58.021932+0000","last_became_active":"2024-06-14T23:23:17.169688+0000","last_became_peered":"2024-06-14T23:23:17.169688+0000","last_unstale":"2024-06-14T23:23:58.021932+0000","last_undegraded":"2024-06-14T23:23:58.021932+0000","last_fullsized":"2024-06-14T23:23:58.021932+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:28.426330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":865,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019138+0000","last_change":"2024-06-14T23:22:52.005620+0000","last_active":"2024-06-14T23:23:58.019138+0000","last_peered":"2024-06-14T23:23:58.019138+0000","last_clean":"2024-06-14T23:23:58.019138+0000","last_became_active":"2024-06-14T23:22:52.004963+0000","last_became_peered":"2024-06-14T23:22:52.004963+0000","last_unstale":"2024-06-14T23:23:58.019138+0000","last_undegraded":"2024-06-14T23:23:58.019138+0000","last_fullsized":"2024-06-14T23:23:58.019138+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:45:05.867553+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":393,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.026755+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:23:59.026755+0000","last_peered":"2024-06-14T23:23:59.026755+0000","last_clean":"2024-06-14T23:23:59.026755+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:23:59.026755+0000","last_undegraded":"2024-06-14T23:23:59.026755+0000","last_fullsized":"2024-06-14T23:23:59.026755+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":320,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.024532+0000","last_change":"2024-06-14T23:23:59.024532+0000","last_active":"2024-06-14T23:23:59.024532+0000","last_peered":"2024-06-14T23:23:59.024532+0000","last_clean":"2024-06-14T23:23:59.024532+0000","last_became_active":"2024-06-14T23:23:59.024216+0000","last_became_peered":"2024-06-14T23:23:59.024216+0000","last_unstale":"2024-06-14T23:23:59.024532+0000","last_undegraded":"2024-06-14T23:23:59.024532+0000","last_fullsized":"2024-06-14T23:23:59.024532+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:36:36.377058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":680,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.014568+0000","last_change":"2024-06-14T23:22:15.024503+0000","last_active":"2024-06-14T23:23:58.014568+0000","last_peered":"2024-06-14T23:23:58.014568+0000","last_clean":"2024-06-14T23:23:58.014568+0000","last_became_active":"2024-06-14T23:20:36.537993+0000","last_became_peered":"2024-06-14T23:20:36.537993+0000","last_unstale":"2024-06-14T23:23:58.014568+0000","last_undegraded":"2024-06-14T23:23:58.014568+0000","last_fullsized":"2024-06-14T23:23:58.014568+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:15.024379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:15.024379+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:41:14.672203+0000","scrub_duration":0.013254276000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":128,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018118+0000","last_change":"2024-06-14T23:23:36.142633+0000","last_active":"2024-06-14T23:23:58.018118+0000","last_peered":"2024-06-14T23:23:58.018118+0000","last_clean":"2024-06-14T23:23:58.018118+0000","last_became_active":"2024-06-14T23:23:36.142302+0000","last_became_peered":"2024-06-14T23:23:36.142302+0000","last_unstale":"2024-06-14T23:23:58.018118+0000","last_undegraded":"2024-06-14T23:23:58.018118+0000","last_fullsized":"2024-06-14T23:23:58.018118+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:19.659336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":561,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023990+0000","last_change":"2024-06-14T23:23:13.145733+0000","last_active":"2024-06-14T23:23:58.023990+0000","last_peered":"2024-06-14T23:23:58.023990+0000","last_clean":"2024-06-14T23:23:58.023990+0000","last_became_active":"2024-06-14T23:23:13.145456+0000","last_became_peered":"2024-06-14T23:23:13.145456+0000","last_unstale":"2024-06-14T23:23:58.023990+0000","last_undegraded":"2024-06-14T23:23:58.023990+0000","last_fullsized":"2024-06-14T23:23:58.023990+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:17.692823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":630,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.028342+0000","last_change":"2024-06-14T23:23:59.028342+0000","last_active":"2024-06-14T23:23:59.028342+0000","last_peered":"2024-06-14T23:23:59.028342+0000","last_clean":"2024-06-14T23:23:59.028342+0000","last_became_active":"2024-06-14T23:23:59.028020+0000","last_became_peered":"2024-06-14T23:23:59.028020+0000","last_unstale":"2024-06-14T23:23:59.028342+0000","last_undegraded":"2024-06-14T23:23:59.028342+0000","last_fullsized":"2024-06-14T23:23:59.028342+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:59.004461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:59.004461+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:58.866191+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":366,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026225+0000","last_change":"2024-06-14T23:23:28.040402+0000","last_active":"2024-06-14T23:23:58.026225+0000","last_peered":"2024-06-14T23:23:58.026225+0000","last_clean":"2024-06-14T23:23:58.026225+0000","last_became_active":"2024-06-14T23:23:28.038550+0000","last_became_peered":"2024-06-14T23:23:28.038550+0000","last_unstale":"2024-06-14T23:23:58.026225+0000","last_undegraded":"2024-06-14T23:23:58.026225+0000","last_fullsized":"2024-06-14T23:23:58.026225+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:13:39.022225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":458,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026636+0000","last_change":"2024-06-14T23:23:28.040648+0000","last_active":"2024-06-14T23:23:58.026636+0000","last_peered":"2024-06-14T23:23:58.026636+0000","last_clean":"2024-06-14T23:23:58.026636+0000","last_became_active":"2024-06-14T23:23:28.039483+0000","last_became_peered":"2024-06-14T23:23:28.039483+0000","last_unstale":"2024-06-14T23:23:58.026636+0000","last_undegraded":"2024-06-14T23:23:58.026636+0000","last_fullsized":"2024-06-14T23:23:58.026636+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:24:52.826818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":617,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017492+0000","last_change":"2024-06-14T23:23:20.195514+0000","last_active":"2024-06-14T23:23:59.017492+0000","last_peered":"2024-06-14T23:23:59.017492+0000","last_clean":"2024-06-14T23:23:59.017492+0000","last_became_active":"2024-06-14T23:23:20.195194+0000","last_became_peered":"2024-06-14T23:23:20.195194+0000","last_unstale":"2024-06-14T23:23:59.017492+0000","last_undegraded":"2024-06-14T23:23:59.017492+0000","last_fullsized":"2024-06-14T23:23:59.017492+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.634078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.634078+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:57:24.404193+0000","scrub_duration":0.00160432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":344,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025440+0000","last_change":"2024-06-14T23:23:28.040993+0000","last_active":"2024-06-14T23:23:58.025440+0000","last_peered":"2024-06-14T23:23:58.025440+0000","last_clean":"2024-06-14T23:23:58.025440+0000","last_became_active":"2024-06-14T23:23:28.039242+0000","last_became_peered":"2024-06-14T23:23:28.039242+0000","last_unstale":"2024-06-14T23:23:58.025440+0000","last_undegraded":"2024-06-14T23:23:58.025440+0000","last_fullsized":"2024-06-14T23:23:58.025440+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:26.643804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":514,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024949+0000","last_change":"2024-06-14T23:20:33.516210+0000","last_active":"2024-06-14T23:23:58.024949+0000","last_peered":"2024-06-14T23:23:58.024949+0000","last_clean":"2024-06-14T23:23:58.024949+0000","last_became_active":"2024-06-14T23:20:33.515892+0000","last_became_peered":"2024-06-14T23:20:33.515892+0000","last_unstale":"2024-06-14T23:23:58.024949+0000","last_undegraded":"2024-06-14T23:23:58.024949+0000","last_fullsized":"2024-06-14T23:23:58.024949+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:52.415552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":127,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.020757+0000","last_change":"2024-06-14T23:22:23.550370+0000","last_active":"2024-06-14T23:23:59.020757+0000","last_peered":"2024-06-14T23:23:59.020757+0000","last_clean":"2024-06-14T23:23:59.020757+0000","last_became_active":"2024-06-14T23:22:23.520680+0000","last_became_peered":"2024-06-14T23:22:23.520680+0000","last_unstale":"2024-06-14T23:23:59.020757+0000","last_undegraded":"2024-06-14T23:23:59.020757+0000","last_fullsized":"2024-06-14T23:23:59.020757+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:00:36.472602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":548,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019467+0000","last_change":"2024-06-14T23:23:20.194637+0000","last_active":"2024-06-14T23:23:58.019467+0000","last_peered":"2024-06-14T23:23:58.019467+0000","last_clean":"2024-06-14T23:23:58.019467+0000","last_became_active":"2024-06-14T23:23:20.194088+0000","last_became_peered":"2024-06-14T23:23:20.194088+0000","last_unstale":"2024-06-14T23:23:58.019467+0000","last_undegraded":"2024-06-14T23:23:58.019467+0000","last_fullsized":"2024-06-14T23:23:58.019467+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:52.977636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:52.977636+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:57:19.372132+0000","scrub_duration":0.001529084,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019525+0000","last_change":"2024-06-14T23:23:17.168929+0000","last_active":"2024-06-14T23:23:58.019525+0000","last_peered":"2024-06-14T23:23:58.019525+0000","last_clean":"2024-06-14T23:23:58.019525+0000","last_became_active":"2024-06-14T23:23:17.168615+0000","last_became_peered":"2024-06-14T23:23:17.168615+0000","last_unstale":"2024-06-14T23:23:58.019525+0000","last_undegraded":"2024-06-14T23:23:58.019525+0000","last_fullsized":"2024-06-14T23:23:58.019525+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:22:02.752839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017285+0000","last_change":"2024-06-14T23:22:23.535725+0000","last_active":"2024-06-14T23:23:59.017285+0000","last_peered":"2024-06-14T23:23:59.017285+0000","last_clean":"2024-06-14T23:23:59.017285+0000","last_became_active":"2024-06-14T23:22:23.535450+0000","last_became_peered":"2024-06-14T23:22:23.535450+0000","last_unstale":"2024-06-14T23:23:59.017285+0000","last_undegraded":"2024-06-14T23:23:59.017285+0000","last_fullsized":"2024-06-14T23:23:59.017285+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:44:55.548227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":343,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019968+0000","last_change":"2024-06-14T23:23:21.190343+0000","last_active":"2024-06-14T23:23:58.019968+0000","last_peered":"2024-06-14T23:23:58.019968+0000","last_clean":"2024-06-14T23:23:58.019968+0000","last_became_active":"2024-06-14T23:23:21.190029+0000","last_became_peered":"2024-06-14T23:23:21.190029+0000","last_unstale":"2024-06-14T23:23:58.019968+0000","last_undegraded":"2024-06-14T23:23:58.019968+0000","last_fullsized":"2024-06-14T23:23:58.019968+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:19:29.959380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":817,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.027003+0000","last_change":"2024-06-14T23:23:28.042678+0000","last_active":"2024-06-14T23:23:58.027003+0000","last_peered":"2024-06-14T23:23:58.027003+0000","last_clean":"2024-06-14T23:23:58.027003+0000","last_became_active":"2024-06-14T23:23:28.041664+0000","last_became_peered":"2024-06-14T23:23:28.041664+0000","last_unstale":"2024-06-14T23:23:58.027003+0000","last_undegraded":"2024-06-14T23:23:58.027003+0000","last_fullsized":"2024-06-14T23:23:58.027003+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:51.748589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:51.748589+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:28:36.624812+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":335,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021582+0000","last_change":"2024-06-14T23:22:51.003878+0000","last_active":"2024-06-14T23:23:58.021582+0000","last_peered":"2024-06-14T23:23:58.021582+0000","last_clean":"2024-06-14T23:23:58.021582+0000","last_became_active":"2024-06-14T23:22:51.003543+0000","last_became_peered":"2024-06-14T23:22:51.003543+0000","last_unstale":"2024-06-14T23:23:58.021582+0000","last_undegraded":"2024-06-14T23:23:58.021582+0000","last_fullsized":"2024-06-14T23:23:58.021582+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:49.194146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":515,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019153+0000","last_change":"2024-06-14T23:17:22.890776+0000","last_active":"2024-06-14T23:23:58.019153+0000","last_peered":"2024-06-14T23:23:58.019153+0000","last_clean":"2024-06-14T23:23:58.019153+0000","last_became_active":"2024-06-14T23:17:22.890463+0000","last_became_peered":"2024-06-14T23:17:22.890463+0000","last_unstale":"2024-06-14T23:23:58.019153+0000","last_undegraded":"2024-06-14T23:23:58.019153+0000","last_fullsized":"2024-06-14T23:23:58.019153+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:45:42.223422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":118,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023479+0000","last_change":"2024-06-14T23:23:13.144735+0000","last_active":"2024-06-14T23:23:58.023479+0000","last_peered":"2024-06-14T23:23:58.023479+0000","last_clean":"2024-06-14T23:23:58.023479+0000","last_became_active":"2024-06-14T23:23:13.144365+0000","last_became_peered":"2024-06-14T23:23:13.144365+0000","last_unstale":"2024-06-14T23:23:58.023479+0000","last_undegraded":"2024-06-14T23:23:58.023479+0000","last_fullsized":"2024-06-14T23:23:58.023479+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.814540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":881,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.022930+0000","last_change":"2024-06-14T23:23:20.196260+0000","last_active":"2024-06-14T23:23:58.022930+0000","last_peered":"2024-06-14T23:23:58.022930+0000","last_clean":"2024-06-14T23:23:58.022930+0000","last_became_active":"2024-06-14T23:23:20.195740+0000","last_became_peered":"2024-06-14T23:23:20.195740+0000","last_unstale":"2024-06-14T23:23:58.022930+0000","last_undegraded":"2024-06-14T23:23:58.022930+0000","last_fullsized":"2024-06-14T23:23:58.022930+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:26:18.594571+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":62,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017598+0000","last_change":"2024-06-14T23:22:52.002567+0000","last_active":"2024-06-14T23:23:59.017598+0000","last_peered":"2024-06-14T23:23:59.017598+0000","last_clean":"2024-06-14T23:23:59.017598+0000","last_became_active":"2024-06-14T23:22:52.002401+0000","last_became_peered":"2024-06-14T23:22:52.002401+0000","last_unstale":"2024-06-14T23:23:59.017598+0000","last_undegraded":"2024-06-14T23:23:59.017598+0000","last_fullsized":"2024-06-14T23:23:59.017598+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:28:51.732409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":185,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.018317+0000","last_change":"2024-06-14T23:23:13.143752+0000","last_active":"2024-06-14T23:23:59.018317+0000","last_peered":"2024-06-14T23:23:59.018317+0000","last_clean":"2024-06-14T23:23:59.018317+0000","last_became_active":"2024-06-14T23:23:13.143216+0000","last_became_peered":"2024-06-14T23:23:13.143216+0000","last_unstale":"2024-06-14T23:23:59.018317+0000","last_undegraded":"2024-06-14T23:23:59.018317+0000","last_fullsized":"2024-06-14T23:23:59.018317+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:48:43.394464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":459,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.028767+0000","last_change":"2024-06-14T23:23:59.028767+0000","last_active":"2024-06-14T23:23:59.028767+0000","last_peered":"2024-06-14T23:23:59.028767+0000","last_clean":"2024-06-14T23:23:59.028767+0000","last_became_active":"2024-06-14T23:23:59.028466+0000","last_became_peered":"2024-06-14T23:23:59.028466+0000","last_unstale":"2024-06-14T23:23:59.028767+0000","last_undegraded":"2024-06-14T23:23:59.028767+0000","last_fullsized":"2024-06-14T23:23:59.028767+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:03:52.013585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":858,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025063+0000","last_change":"2024-06-14T23:23:20.196205+0000","last_active":"2024-06-14T23:23:58.025063+0000","last_peered":"2024-06-14T23:23:58.025063+0000","last_clean":"2024-06-14T23:23:58.025063+0000","last_became_active":"2024-06-14T23:23:20.195770+0000","last_became_peered":"2024-06-14T23:23:20.195770+0000","last_unstale":"2024-06-14T23:23:58.025063+0000","last_undegraded":"2024-06-14T23:23:58.025063+0000","last_fullsized":"2024-06-14T23:23:58.025063+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:24.335823+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":98,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024100+0000","last_change":"2024-06-14T23:23:17.167178+0000","last_active":"2024-06-14T23:23:58.024100+0000","last_peered":"2024-06-14T23:23:58.024100+0000","last_clean":"2024-06-14T23:23:58.024100+0000","last_became_active":"2024-06-14T23:23:17.166697+0000","last_became_peered":"2024-06-14T23:23:17.166697+0000","last_unstale":"2024-06-14T23:23:58.024100+0000","last_undegraded":"2024-06-14T23:23:58.024100+0000","last_fullsized":"2024-06-14T23:23:58.024100+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:26.764829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":381,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026748+0000","last_change":"2024-06-14T23:23:28.041548+0000","last_active":"2024-06-14T23:23:58.026748+0000","last_peered":"2024-06-14T23:23:58.026748+0000","last_clean":"2024-06-14T23:23:58.026748+0000","last_became_active":"2024-06-14T23:23:28.040596+0000","last_became_peered":"2024-06-14T23:23:28.040596+0000","last_unstale":"2024-06-14T23:23:58.026748+0000","last_undegraded":"2024-06-14T23:23:58.026748+0000","last_fullsized":"2024-06-14T23:23:58.026748+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:13:28.178465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":405,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.025146+0000","last_change":"2024-06-14T23:23:59.025146+0000","last_active":"2024-06-14T23:23:59.025146+0000","last_peered":"2024-06-14T23:23:59.025146+0000","last_clean":"2024-06-14T23:23:59.025146+0000","last_became_active":"2024-06-14T23:23:59.024719+0000","last_became_peered":"2024-06-14T23:23:59.024719+0000","last_unstale":"2024-06-14T23:23:59.025146+0000","last_undegraded":"2024-06-14T23:23:59.025146+0000","last_fullsized":"2024-06-14T23:23:59.025146+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:18.741182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":858,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.023421+0000","last_change":"2024-06-14T23:23:59.023421+0000","last_active":"2024-06-14T23:23:59.023421+0000","last_peered":"2024-06-14T23:23:59.023421+0000","last_clean":"2024-06-14T23:23:59.023421+0000","last_became_active":"2024-06-14T23:23:59.023052+0000","last_became_peered":"2024-06-14T23:23:59.023052+0000","last_unstale":"2024-06-14T23:23:59.023421+0000","last_undegraded":"2024-06-14T23:23:59.023421+0000","last_fullsized":"2024-06-14T23:23:59.023421+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.522124+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:18:05.624868+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":62,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017699+0000","last_change":"2024-06-14T23:22:52.005158+0000","last_active":"2024-06-14T23:23:59.017699+0000","last_peered":"2024-06-14T23:23:59.017699+0000","last_clean":"2024-06-14T23:23:59.017699+0000","last_became_active":"2024-06-14T23:22:52.004707+0000","last_became_peered":"2024-06-14T23:22:52.004707+0000","last_unstale":"2024-06-14T23:23:59.017699+0000","last_undegraded":"2024-06-14T23:23:59.017699+0000","last_fullsized":"2024-06-14T23:23:59.017699+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:12:50.379251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":89,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023246+0000","last_change":"2024-06-14T23:23:13.141787+0000","last_active":"2024-06-14T23:23:58.023246+0000","last_peered":"2024-06-14T23:23:58.023246+0000","last_clean":"2024-06-14T23:23:58.023246+0000","last_became_active":"2024-06-14T23:23:13.141524+0000","last_became_peered":"2024-06-14T23:23:13.141524+0000","last_unstale":"2024-06-14T23:23:58.023246+0000","last_undegraded":"2024-06-14T23:23:58.023246+0000","last_fullsized":"2024-06-14T23:23:58.023246+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:02:49.929637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":114,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.018630+0000","last_change":"2024-06-14T23:23:17.165494+0000","last_active":"2024-06-14T23:23:59.018630+0000","last_peered":"2024-06-14T23:23:59.018630+0000","last_clean":"2024-06-14T23:23:59.018630+0000","last_became_active":"2024-06-14T23:23:17.165132+0000","last_became_peered":"2024-06-14T23:23:17.165132+0000","last_unstale":"2024-06-14T23:23:59.018630+0000","last_undegraded":"2024-06-14T23:23:59.018630+0000","last_fullsized":"2024-06-14T23:23:59.018630+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:02:54.197396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":652,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.016326+0000","last_change":"2024-06-14T23:23:36.143094+0000","last_active":"2024-06-14T23:23:58.016326+0000","last_peered":"2024-06-14T23:23:58.016326+0000","last_clean":"2024-06-14T23:23:58.016326+0000","last_became_active":"2024-06-14T23:23:36.142685+0000","last_became_peered":"2024-06-14T23:23:36.142685+0000","last_unstale":"2024-06-14T23:23:58.016326+0000","last_undegraded":"2024-06-14T23:23:58.016326+0000","last_fullsized":"2024-06-14T23:23:58.016326+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:08.655367+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:23:08.655367+0000","last_clean_scrub_stamp":"2024-06-14T23:23:08.655367+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:48:48.456845+0000","scrub_duration":0.0087017850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":392,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021647+0000","last_change":"2024-06-14T23:23:46.452498+0000","last_active":"2024-06-14T23:23:58.021647+0000","last_peered":"2024-06-14T23:23:58.021647+0000","last_clean":"2024-06-14T23:23:58.021647+0000","last_became_active":"2024-06-14T23:23:46.452203+0000","last_became_peered":"2024-06-14T23:23:46.452203+0000","last_unstale":"2024-06-14T23:23:58.021647+0000","last_undegraded":"2024-06-14T23:23:58.021647+0000","last_fullsized":"2024-06-14T23:23:58.021647+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:51:38.638988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":305,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.020312+0000","last_change":"2024-06-14T23:23:17.170011+0000","last_active":"2024-06-14T23:23:59.020312+0000","last_peered":"2024-06-14T23:23:59.020312+0000","last_clean":"2024-06-14T23:23:59.020312+0000","last_became_active":"2024-06-14T23:23:17.169676+0000","last_became_peered":"2024-06-14T23:23:17.169676+0000","last_unstale":"2024-06-14T23:23:59.020312+0000","last_undegraded":"2024-06-14T23:23:59.020312+0000","last_fullsized":"2024-06-14T23:23:59.020312+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:15.492853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":374,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.017480+0000","last_change":"2024-06-14T23:23:36.146686+0000","last_active":"2024-06-14T23:23:58.017480+0000","last_peered":"2024-06-14T23:23:58.017480+0000","last_clean":"2024-06-14T23:23:58.017480+0000","last_became_active":"2024-06-14T23:23:36.145912+0000","last_became_peered":"2024-06-14T23:23:36.145912+0000","last_unstale":"2024-06-14T23:23:58.017480+0000","last_undegraded":"2024-06-14T23:23:58.017480+0000","last_fullsized":"2024-06-14T23:23:58.017480+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:49.288111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":617,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017324+0000","last_change":"2024-06-14T23:22:26.522180+0000","last_active":"2024-06-14T23:23:59.017324+0000","last_peered":"2024-06-14T23:23:59.017324+0000","last_clean":"2024-06-14T23:23:59.017324+0000","last_became_active":"2024-06-14T23:22:26.521598+0000","last_became_peered":"2024-06-14T23:22:26.521598+0000","last_unstale":"2024-06-14T23:23:59.017324+0000","last_undegraded":"2024-06-14T23:23:59.017324+0000","last_fullsized":"2024-06-14T23:23:59.017324+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.735114+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:17:53.735114+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.735114+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:12:25.561031+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":437,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023885+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:23:58.023885+0000","last_peered":"2024-06-14T23:23:58.023885+0000","last_clean":"2024-06-14T23:23:58.023885+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:23:58.023885+0000","last_undegraded":"2024-06-14T23:23:58.023885+0000","last_fullsized":"2024-06-14T23:23:58.023885+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":332,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018987+0000","last_change":"2024-06-14T23:23:21.191266+0000","last_active":"2024-06-14T23:23:58.018987+0000","last_peered":"2024-06-14T23:23:58.018987+0000","last_clean":"2024-06-14T23:23:58.018987+0000","last_became_active":"2024-06-14T23:23:21.191075+0000","last_became_peered":"2024-06-14T23:23:21.191075+0000","last_unstale":"2024-06-14T23:23:58.018987+0000","last_undegraded":"2024-06-14T23:23:58.018987+0000","last_fullsized":"2024-06-14T23:23:58.018987+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:56:12.169553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":446,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.024349+0000","last_change":"2024-06-14T23:23:59.024349+0000","last_active":"2024-06-14T23:23:59.024349+0000","last_peered":"2024-06-14T23:23:59.024349+0000","last_clean":"2024-06-14T23:23:59.024349+0000","last_became_active":"2024-06-14T23:23:59.023980+0000","last_became_peered":"2024-06-14T23:23:59.023980+0000","last_unstale":"2024-06-14T23:23:59.024349+0000","last_undegraded":"2024-06-14T23:23:59.024349+0000","last_fullsized":"2024-06-14T23:23:59.024349+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:52:43.929377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":759,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.017578+0000","last_change":"2024-06-14T23:23:36.144792+0000","last_active":"2024-06-14T23:23:58.017578+0000","last_peered":"2024-06-14T23:23:58.017578+0000","last_clean":"2024-06-14T23:23:58.017578+0000","last_became_active":"2024-06-14T23:23:36.144221+0000","last_became_peered":"2024-06-14T23:23:36.144221+0000","last_unstale":"2024-06-14T23:23:58.017578+0000","last_undegraded":"2024-06-14T23:23:58.017578+0000","last_fullsized":"2024-06-14T23:23:58.017578+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:06.661195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:06.661195+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:45:27.980586+0000","scrub_duration":0.002049375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":51,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017610+0000","last_change":"2024-06-14T23:22:23.532985+0000","last_active":"2024-06-14T23:23:59.017610+0000","last_peered":"2024-06-14T23:23:59.017610+0000","last_clean":"2024-06-14T23:23:59.017610+0000","last_became_active":"2024-06-14T23:22:23.532685+0000","last_became_peered":"2024-06-14T23:22:23.532685+0000","last_unstale":"2024-06-14T23:23:59.017610+0000","last_undegraded":"2024-06-14T23:23:59.017610+0000","last_fullsized":"2024-06-14T23:23:59.017610+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:52:43.046719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019612+0000","last_change":"2024-06-14T23:23:17.170036+0000","last_active":"2024-06-14T23:23:58.019612+0000","last_peered":"2024-06-14T23:23:58.019612+0000","last_clean":"2024-06-14T23:23:58.019612+0000","last_became_active":"2024-06-14T23:23:17.169744+0000","last_became_peered":"2024-06-14T23:23:17.169744+0000","last_unstale":"2024-06-14T23:23:58.019612+0000","last_undegraded":"2024-06-14T23:23:58.019612+0000","last_fullsized":"2024-06-14T23:23:58.019612+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:37:18.976998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":794,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.022869+0000","last_change":"2024-06-14T23:23:59.022869+0000","last_active":"2024-06-14T23:23:59.022869+0000","last_peered":"2024-06-14T23:23:59.022869+0000","last_clean":"2024-06-14T23:23:59.022869+0000","last_became_active":"2024-06-14T23:23:59.022524+0000","last_became_peered":"2024-06-14T23:23:59.022524+0000","last_unstale":"2024-06-14T23:23:59.022869+0000","last_undegraded":"2024-06-14T23:23:59.022869+0000","last_fullsized":"2024-06-14T23:23:59.022869+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:51.104492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:51.104492+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:55.736048+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":537,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024903+0000","last_change":"2024-06-14T23:23:28.041159+0000","last_active":"2024-06-14T23:23:58.024903+0000","last_peered":"2024-06-14T23:23:58.024903+0000","last_clean":"2024-06-14T23:23:58.024903+0000","last_became_active":"2024-06-14T23:23:28.039520+0000","last_became_peered":"2024-06-14T23:23:28.039520+0000","last_unstale":"2024-06-14T23:23:58.024903+0000","last_undegraded":"2024-06-14T23:23:58.024903+0000","last_fullsized":"2024-06-14T23:23:58.024903+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:15:12.820828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":381,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025679+0000","last_change":"2024-06-14T23:23:28.042229+0000","last_active":"2024-06-14T23:23:58.025679+0000","last_peered":"2024-06-14T23:23:58.025679+0000","last_clean":"2024-06-14T23:23:58.025679+0000","last_became_active":"2024-06-14T23:23:28.041807+0000","last_became_peered":"2024-06-14T23:23:28.041807+0000","last_unstale":"2024-06-14T23:23:58.025679+0000","last_undegraded":"2024-06-14T23:23:58.025679+0000","last_fullsized":"2024-06-14T23:23:58.025679+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:21:33.795242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":824,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017047+0000","last_change":"2024-06-14T23:23:13.140635+0000","last_active":"2024-06-14T23:23:59.017047+0000","last_peered":"2024-06-14T23:23:59.017047+0000","last_clean":"2024-06-14T23:23:59.017047+0000","last_became_active":"2024-06-14T23:23:13.140207+0000","last_became_peered":"2024-06-14T23:23:13.140207+0000","last_unstale":"2024-06-14T23:23:59.017047+0000","last_undegraded":"2024-06-14T23:23:59.017047+0000","last_fullsized":"2024-06-14T23:23:59.017047+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:03:18.870266+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":400,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.027091+0000","last_change":"2024-06-14T23:23:28.037425+0000","last_active":"2024-06-14T23:23:58.027091+0000","last_peered":"2024-06-14T23:23:58.027091+0000","last_clean":"2024-06-14T23:23:58.027091+0000","last_became_active":"2024-06-14T23:23:28.035090+0000","last_became_peered":"2024-06-14T23:23:28.035090+0000","last_unstale":"2024-06-14T23:23:58.027091+0000","last_undegraded":"2024-06-14T23:23:58.027091+0000","last_fullsized":"2024-06-14T23:23:58.027091+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:09:52.542795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":379,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024532+0000","last_change":"2024-06-14T23:23:44.237001+0000","last_active":"2024-06-14T23:23:58.024532+0000","last_peered":"2024-06-14T23:23:58.024532+0000","last_clean":"2024-06-14T23:23:58.024532+0000","last_became_active":"2024-06-14T23:23:44.236684+0000","last_became_peered":"2024-06-14T23:23:44.236684+0000","last_unstale":"2024-06-14T23:23:58.024532+0000","last_undegraded":"2024-06-14T23:23:58.024532+0000","last_fullsized":"2024-06-14T23:23:58.024532+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:45.200924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":785,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.031047+0000","last_change":"2024-06-14T23:23:59.031047+0000","last_active":"2024-06-14T23:23:59.031047+0000","last_peered":"2024-06-14T23:23:59.031047+0000","last_clean":"2024-06-14T23:23:59.031047+0000","last_became_active":"2024-06-14T23:23:59.030716+0000","last_became_peered":"2024-06-14T23:23:59.030716+0000","last_unstale":"2024-06-14T23:23:59.031047+0000","last_undegraded":"2024-06-14T23:23:59.031047+0000","last_fullsized":"2024-06-14T23:23:59.031047+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:53.058787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:53.058787+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:48:33.858182+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":417,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026133+0000","last_change":"2024-06-14T23:23:28.039267+0000","last_active":"2024-06-14T23:23:58.026133+0000","last_peered":"2024-06-14T23:23:58.026133+0000","last_clean":"2024-06-14T23:23:58.026133+0000","last_became_active":"2024-06-14T23:23:28.038714+0000","last_became_peered":"2024-06-14T23:23:28.038714+0000","last_unstale":"2024-06-14T23:23:58.026133+0000","last_undegraded":"2024-06-14T23:23:58.026133+0000","last_fullsized":"2024-06-14T23:23:58.026133+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:57:23.263278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":198,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018891+0000","last_change":"2024-06-14T23:23:21.192300+0000","last_active":"2024-06-14T23:23:58.018891+0000","last_peered":"2024-06-14T23:23:58.018891+0000","last_clean":"2024-06-14T23:23:58.018891+0000","last_became_active":"2024-06-14T23:23:21.192008+0000","last_became_peered":"2024-06-14T23:23:21.192008+0000","last_unstale":"2024-06-14T23:23:58.018891+0000","last_undegraded":"2024-06-14T23:23:58.018891+0000","last_fullsized":"2024-06-14T23:23:58.018891+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:26:32.088220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":577,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.016737+0000","last_change":"2024-06-14T23:23:39.243862+0000","last_active":"2024-06-14T23:23:58.016737+0000","last_peered":"2024-06-14T23:23:58.016737+0000","last_clean":"2024-06-14T23:23:58.016737+0000","last_became_active":"2024-06-14T23:23:36.141570+0000","last_became_peered":"2024-06-14T23:23:36.141570+0000","last_unstale":"2024-06-14T23:23:58.016737+0000","last_undegraded":"2024-06-14T23:23:58.016737+0000","last_fullsized":"2024-06-14T23:23:58.016737+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:39.243791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:39.243791+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:53:31.026410+0000","scrub_duration":0.0032923150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":336,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021796+0000","last_change":"2024-06-14T23:22:23.626905+0000","last_active":"2024-06-14T23:23:58.021796+0000","last_peered":"2024-06-14T23:23:58.021796+0000","last_clean":"2024-06-14T23:23:58.021796+0000","last_became_active":"2024-06-14T23:22:23.602799+0000","last_became_peered":"2024-06-14T23:22:23.602799+0000","last_unstale":"2024-06-14T23:23:58.021796+0000","last_undegraded":"2024-06-14T23:23:58.021796+0000","last_fullsized":"2024-06-14T23:23:58.021796+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:02:31.842714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":381,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.031020+0000","last_change":"2024-06-14T23:23:59.031020+0000","last_active":"2024-06-14T23:23:59.031020+0000","last_peered":"2024-06-14T23:23:59.031020+0000","last_clean":"2024-06-14T23:23:59.031020+0000","last_became_active":"2024-06-14T23:23:59.030650+0000","last_became_peered":"2024-06-14T23:23:59.030650+0000","last_unstale":"2024-06-14T23:23:59.031020+0000","last_undegraded":"2024-06-14T23:23:59.031020+0000","last_fullsized":"2024-06-14T23:23:59.031020+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:58:28.125967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":621,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024681+0000","last_change":"2024-06-14T23:23:28.031929+0000","last_active":"2024-06-14T23:23:58.024681+0000","last_peered":"2024-06-14T23:23:58.024681+0000","last_clean":"2024-06-14T23:23:58.024681+0000","last_became_active":"2024-06-14T23:23:28.031584+0000","last_became_peered":"2024-06-14T23:23:28.031584+0000","last_unstale":"2024-06-14T23:23:58.024681+0000","last_undegraded":"2024-06-14T23:23:58.024681+0000","last_fullsized":"2024-06-14T23:23:58.024681+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:12.943827+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:15:12.943827+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:21:39.406089+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":490,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024346+0000","last_change":"2024-06-14T23:22:23.534414+0000","last_active":"2024-06-14T23:23:58.024346+0000","last_peered":"2024-06-14T23:23:58.024346+0000","last_clean":"2024-06-14T23:23:58.024346+0000","last_became_active":"2024-06-14T23:22:23.534050+0000","last_became_peered":"2024-06-14T23:22:23.534050+0000","last_unstale":"2024-06-14T23:23:58.024346+0000","last_undegraded":"2024-06-14T23:23:58.024346+0000","last_fullsized":"2024-06-14T23:23:58.024346+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:06:51.839502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":109,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019041+0000","last_change":"2024-06-14T23:23:36.144429+0000","last_active":"2024-06-14T23:23:58.019041+0000","last_peered":"2024-06-14T23:23:58.019041+0000","last_clean":"2024-06-14T23:23:58.019041+0000","last_became_active":"2024-06-14T23:23:36.143977+0000","last_became_peered":"2024-06-14T23:23:36.143977+0000","last_unstale":"2024-06-14T23:23:58.019041+0000","last_undegraded":"2024-06-14T23:23:58.019041+0000","last_fullsized":"2024-06-14T23:23:58.019041+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:20.036203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":818,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025150+0000","last_change":"2024-06-14T23:22:51.010054+0000","last_active":"2024-06-14T23:23:58.025150+0000","last_peered":"2024-06-14T23:23:58.025150+0000","last_clean":"2024-06-14T23:23:58.025150+0000","last_became_active":"2024-06-14T23:22:51.009335+0000","last_became_peered":"2024-06-14T23:22:51.009335+0000","last_unstale":"2024-06-14T23:23:58.025150+0000","last_undegraded":"2024-06-14T23:23:58.025150+0000","last_fullsized":"2024-06-14T23:23:58.025150+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:07:08.001347+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":394,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025587+0000","last_change":"2024-06-14T23:23:28.035153+0000","last_active":"2024-06-14T23:23:58.025587+0000","last_peered":"2024-06-14T23:23:58.025587+0000","last_clean":"2024-06-14T23:23:58.025587+0000","last_became_active":"2024-06-14T23:23:28.034861+0000","last_became_peered":"2024-06-14T23:23:28.034861+0000","last_unstale":"2024-06-14T23:23:58.025587+0000","last_undegraded":"2024-06-14T23:23:58.025587+0000","last_fullsized":"2024-06-14T23:23:58.025587+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:44.007355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":513,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019005+0000","last_change":"2024-06-14T23:22:23.551854+0000","last_active":"2024-06-14T23:23:58.019005+0000","last_peered":"2024-06-14T23:23:58.019005+0000","last_clean":"2024-06-14T23:23:58.019005+0000","last_became_active":"2024-06-14T23:22:23.536550+0000","last_became_peered":"2024-06-14T23:22:23.536550+0000","last_unstale":"2024-06-14T23:23:58.019005+0000","last_undegraded":"2024-06-14T23:23:58.019005+0000","last_fullsized":"2024-06-14T23:23:58.019005+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:52.734655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":843,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019382+0000","last_change":"2024-06-14T23:23:20.199876+0000","last_active":"2024-06-14T23:23:58.019382+0000","last_peered":"2024-06-14T23:23:58.019382+0000","last_clean":"2024-06-14T23:23:58.019382+0000","last_became_active":"2024-06-14T23:23:20.199523+0000","last_became_peered":"2024-06-14T23:23:20.199523+0000","last_unstale":"2024-06-14T23:23:58.019382+0000","last_undegraded":"2024-06-14T23:23:58.019382+0000","last_fullsized":"2024-06-14T23:23:58.019382+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:22:24.933402+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:22:24.933402+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:35:39.661035+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":394,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.028446+0000","last_change":"2024-06-14T23:23:59.028446+0000","last_active":"2024-06-14T23:23:59.028446+0000","last_peered":"2024-06-14T23:23:59.028446+0000","last_clean":"2024-06-14T23:23:59.028446+0000","last_became_active":"2024-06-14T23:23:59.027914+0000","last_became_peered":"2024-06-14T23:23:59.027914+0000","last_unstale":"2024-06-14T23:23:59.028446+0000","last_undegraded":"2024-06-14T23:23:59.028446+0000","last_fullsized":"2024-06-14T23:23:59.028446+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:32:27.725711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":512,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018518+0000","last_change":"2024-06-14T23:17:22.904229+0000","last_active":"2024-06-14T23:23:58.018518+0000","last_peered":"2024-06-14T23:23:58.018518+0000","last_clean":"2024-06-14T23:23:58.018518+0000","last_became_active":"2024-06-14T23:17:22.896077+0000","last_became_peered":"2024-06-14T23:17:22.896077+0000","last_unstale":"2024-06-14T23:23:58.018518+0000","last_undegraded":"2024-06-14T23:23:58.018518+0000","last_fullsized":"2024-06-14T23:23:58.018518+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:05.630800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":679,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019067+0000","last_change":"2024-06-14T23:23:36.146430+0000","last_active":"2024-06-14T23:23:58.019067+0000","last_peered":"2024-06-14T23:23:58.019067+0000","last_clean":"2024-06-14T23:23:58.019067+0000","last_became_active":"2024-06-14T23:23:36.146108+0000","last_became_peered":"2024-06-14T23:23:36.146108+0000","last_unstale":"2024-06-14T23:23:58.019067+0000","last_undegraded":"2024-06-14T23:23:58.019067+0000","last_fullsized":"2024-06-14T23:23:58.019067+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:15.211071+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:23:15.211071+0000","last_clean_scrub_stamp":"2024-06-14T23:23:15.211071+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:38:09.477585+0000","scrub_duration":0.001336177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":545,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.022474+0000","last_change":"2024-06-14T23:22:51.008383+0000","last_active":"2024-06-14T23:23:58.022474+0000","last_peered":"2024-06-14T23:23:58.022474+0000","last_clean":"2024-06-14T23:23:58.022474+0000","last_became_active":"2024-06-14T23:22:51.007961+0000","last_became_peered":"2024-06-14T23:22:51.007961+0000","last_unstale":"2024-06-14T23:23:58.022474+0000","last_undegraded":"2024-06-14T23:23:58.022474+0000","last_fullsized":"2024-06-14T23:23:58.022474+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:37:32.001034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019311+0000","last_change":"2024-06-14T23:23:17.164437+0000","last_active":"2024-06-14T23:23:58.019311+0000","last_peered":"2024-06-14T23:23:58.019311+0000","last_clean":"2024-06-14T23:23:58.019311+0000","last_became_active":"2024-06-14T23:23:17.164146+0000","last_became_peered":"2024-06-14T23:23:17.164146+0000","last_unstale":"2024-06-14T23:23:58.019311+0000","last_undegraded":"2024-06-14T23:23:58.019311+0000","last_fullsized":"2024-06-14T23:23:58.019311+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:24:36.188657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":789,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.024594+0000","last_change":"2024-06-14T23:23:59.024594+0000","last_active":"2024-06-14T23:23:59.024594+0000","last_peered":"2024-06-14T23:23:59.024594+0000","last_clean":"2024-06-14T23:23:59.024594+0000","last_became_active":"2024-06-14T23:23:59.024282+0000","last_became_peered":"2024-06-14T23:23:59.024282+0000","last_unstale":"2024-06-14T23:23:59.024594+0000","last_undegraded":"2024-06-14T23:23:59.024594+0000","last_fullsized":"2024-06-14T23:23:59.024594+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:31.102737+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:22:31.102737+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:46:07.794469+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":266,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.016936+0000","last_change":"2024-06-14T23:23:36.144650+0000","last_active":"2024-06-14T23:23:58.016936+0000","last_peered":"2024-06-14T23:23:58.016936+0000","last_clean":"2024-06-14T23:23:58.016936+0000","last_became_active":"2024-06-14T23:23:36.144172+0000","last_became_peered":"2024-06-14T23:23:36.144172+0000","last_unstale":"2024-06-14T23:23:58.016936+0000","last_undegraded":"2024-06-14T23:23:58.016936+0000","last_fullsized":"2024-06-14T23:23:58.016936+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:23.575240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":297,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021857+0000","last_change":"2024-06-14T23:23:17.170253+0000","last_active":"2024-06-14T23:23:58.021857+0000","last_peered":"2024-06-14T23:23:58.021857+0000","last_clean":"2024-06-14T23:23:58.021857+0000","last_became_active":"2024-06-14T23:23:17.169885+0000","last_became_peered":"2024-06-14T23:23:17.169885+0000","last_unstale":"2024-06-14T23:23:58.021857+0000","last_undegraded":"2024-06-14T23:23:58.021857+0000","last_fullsized":"2024-06-14T23:23:58.021857+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:00.101912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":784,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017169+0000","last_change":"2024-06-14T23:23:20.195118+0000","last_active":"2024-06-14T23:23:59.017169+0000","last_peered":"2024-06-14T23:23:59.017169+0000","last_clean":"2024-06-14T23:23:59.017169+0000","last_became_active":"2024-06-14T23:23:20.194666+0000","last_became_peered":"2024-06-14T23:23:20.194666+0000","last_unstale":"2024-06-14T23:23:59.017169+0000","last_undegraded":"2024-06-14T23:23:59.017169+0000","last_fullsized":"2024-06-14T23:23:59.017169+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:59:52.906430+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":132,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.020413+0000","last_change":"2024-06-14T23:23:17.169352+0000","last_active":"2024-06-14T23:23:59.020413+0000","last_peered":"2024-06-14T23:23:59.020413+0000","last_clean":"2024-06-14T23:23:59.020413+0000","last_became_active":"2024-06-14T23:23:17.169050+0000","last_became_peered":"2024-06-14T23:23:17.169050+0000","last_unstale":"2024-06-14T23:23:59.020413+0000","last_undegraded":"2024-06-14T23:23:59.020413+0000","last_fullsized":"2024-06-14T23:23:59.020413+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:46.429724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":394,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.028731+0000","last_change":"2024-06-14T23:23:59.028731+0000","last_active":"2024-06-14T23:23:59.028731+0000","last_peered":"2024-06-14T23:23:59.028731+0000","last_clean":"2024-06-14T23:23:59.028731+0000","last_became_active":"2024-06-14T23:23:59.028427+0000","last_became_peered":"2024-06-14T23:23:59.028427+0000","last_unstale":"2024-06-14T23:23:59.028731+0000","last_undegraded":"2024-06-14T23:23:59.028731+0000","last_fullsized":"2024-06-14T23:23:59.028731+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:54:34.343018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":753,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.023584+0000","last_change":"2024-06-14T23:23:59.023584+0000","last_active":"2024-06-14T23:23:59.023584+0000","last_peered":"2024-06-14T23:23:59.023584+0000","last_clean":"2024-06-14T23:23:59.023584+0000","last_became_active":"2024-06-14T23:23:59.023280+0000","last_became_peered":"2024-06-14T23:23:59.023280+0000","last_unstale":"2024-06-14T23:23:59.023584+0000","last_undegraded":"2024-06-14T23:23:59.023584+0000","last_fullsized":"2024-06-14T23:23:59.023584+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:38.210830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:38.210830+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:39.146677+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":564,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.017405+0000","last_change":"2024-06-14T23:20:33.536774+0000","last_active":"2024-06-14T23:23:58.017405+0000","last_peered":"2024-06-14T23:23:58.017405+0000","last_clean":"2024-06-14T23:23:58.017405+0000","last_became_active":"2024-06-14T23:20:33.515459+0000","last_became_peered":"2024-06-14T23:20:33.515459+0000","last_unstale":"2024-06-14T23:23:58.017405+0000","last_undegraded":"2024-06-14T23:23:58.017405+0000","last_fullsized":"2024-06-14T23:23:58.017405+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:39.163824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":228,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.026748+0000","last_change":"2024-06-14T23:23:21.291083+0000","last_active":"2024-06-14T23:23:59.026748+0000","last_peered":"2024-06-14T23:23:59.026748+0000","last_clean":"2024-06-14T23:23:59.026748+0000","last_became_active":"2024-06-14T23:23:21.290740+0000","last_became_peered":"2024-06-14T23:23:21.290740+0000","last_unstale":"2024-06-14T23:23:59.026748+0000","last_undegraded":"2024-06-14T23:23:59.026748+0000","last_fullsized":"2024-06-14T23:23:59.026748+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:56:41.113340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":959,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021486+0000","last_change":"2024-06-14T23:23:20.199638+0000","last_active":"2024-06-14T23:23:58.021486+0000","last_peered":"2024-06-14T23:23:58.021486+0000","last_clean":"2024-06-14T23:23:58.021486+0000","last_became_active":"2024-06-14T23:23:20.199266+0000","last_became_peered":"2024-06-14T23:23:20.199266+0000","last_unstale":"2024-06-14T23:23:58.021486+0000","last_undegraded":"2024-06-14T23:23:58.021486+0000","last_fullsized":"2024-06-14T23:23:58.021486+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:47:05.889769+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1342,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28245,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234610,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509832,"kb_used_data":7032,"kb_used_omap":1,"kb_used_meta":1502782,"kb_avail":92235320,"statfs":{"total":95995035648,"available":94448967680,"internally_reserved":0,"allocated":7200768,"data_stored":4426885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1538849112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.67100000000000004,"15min":0.92000000000000004},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.68899999999999995,"5min":4.899,"15min":12.263999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.88100000000000001,"15min":1.054},"min":{"1min":0.32600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.70499999999999996,"5min":3.8889999999999998,"15min":6.5149999999999997},"last":0.63600000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:24:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.70499999999999996,"15min":0.998},"min":{"1min":0.375,"5min":0.25900000000000001,"15min":0.25600000000000001},"max":{"1min":0.72999999999999998,"5min":3.98,"15min":6.0229999999999997},"last":0.75},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.67900000000000005,"15min":0.999},"min":{"1min":0.30199999999999999,"5min":0.29599999999999999,"15min":0.27700000000000002},"max":{"1min":0.625,"5min":4.8710000000000004,"15min":12.295},"last":0.69199999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.71899999999999997,"15min":1.028},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.27000000000000002},"max":{"1min":0.71799999999999997,"5min":4.6079999999999997,"15min":10.539},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.79400000000000004,"15min":1.056},"min":{"1min":0.48099999999999998,"5min":0.34000000000000002,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":4.6980000000000004,"15min":10.56},"last":0.65400000000000003}]},{"osd":3,"last update":"Fri Jun 14 23:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.70899999999999996,"15min":1.087},"min":{"1min":0.41999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69599999999999995,"5min":4.5499999999999998,"15min":17.991},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.749,"15min":1.145},"min":{"1min":0.42099999999999999,"5min":0.313,"15min":0.29399999999999998},"max":{"1min":0.71099999999999997,"5min":3.9489999999999998,"15min":17.824999999999999},"last":0.67800000000000005}]},{"osd":4,"last update":"Fri Jun 14 23:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.81699999999999995,"15min":1.204},"min":{"1min":0.30499999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.65200000000000002,"5min":5.6710000000000003,"15min":6.1539999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.83899999999999997,"15min":1.1930000000000001},"min":{"1min":0.33500000000000002,"5min":0.27700000000000002,"15min":0.22700000000000001},"max":{"1min":0.67100000000000004,"5min":4.6699999999999999,"15min":10.499000000000001},"last":0.47499999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.503},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.73199999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":1.0389999999999999,"15min":1.508},"min":{"1min":0.379,"5min":0.309,"15min":0.309},"max":{"1min":0.70399999999999996,"5min":5.2779999999999996,"15min":12.317},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.96899999999999997,"15min":1.4410000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.75900000000000001,"5min":6.9509999999999996,"15min":10.087999999999999},"last":0.71599999999999997}]}]},{"osd":6,"up_from":272,"seq":1168231104716,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963892,"kb_used_data":7076,"kb_used_omap":7,"kb_used_meta":956792,"kb_avail":92781260,"statfs":{"total":95995035648,"available":95008010240,"internally_reserved":0,"allocated":7245824,"data_stored":4491408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":979755450},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":1.024,"15min":1.0429999999999999},"min":{"1min":0.29799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.66900000000000004,"5min":5.923,"15min":6.109},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.071,"15min":1.1399999999999999},"min":{"1min":0.42799999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.69099999999999995,"5min":6.0620000000000003,"15min":14.148999999999999},"last":0.60999999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.81899999999999995,"15min":0.98499999999999999},"min":{"1min":0.34000000000000002,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":0.59899999999999998,"5min":7.0970000000000004,"15min":7.0970000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.94699999999999995,"15min":1.0900000000000001},"min":{"1min":0.442,"5min":0.26800000000000002,"15min":0.248},"max":{"1min":0.68999999999999995,"5min":5.9980000000000002,"15min":14.086},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Jun 14 23:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.83999999999999997,"15min":1.153},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.249},"max":{"1min":0.749,"5min":7.8019999999999996,"15min":29.940999999999999},"last":0.434},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.96399999999999997,"15min":1.242},"min":{"1min":0.35499999999999998,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":7.8609999999999998,"15min":30.135999999999999},"last":0.59099999999999997}]},{"osd":3,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":1.0509999999999999,"15min":1.1259999999999999},"min":{"1min":0.40000000000000002,"5min":0.29899999999999999,"15min":0.26200000000000001},"max":{"1min":0.67600000000000005,"5min":6.0339999999999998,"15min":6.0839999999999996},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.98299999999999998,"15min":1.2290000000000001},"min":{"1min":0.40999999999999998,"5min":0.312,"15min":0.28499999999999998},"max":{"1min":0.80200000000000005,"5min":7.8319999999999999,"15min":8.4740000000000002},"last":0.53200000000000003}]},{"osd":4,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.90600000000000003,"15min":1.242},"min":{"1min":0.28000000000000003,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.66200000000000003,"5min":6.4089999999999998,"15min":11.010999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":1.1399999999999999,"15min":1.347},"min":{"1min":0.35399999999999998,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":0.73199999999999998,"5min":8.7550000000000008,"15min":14.199999999999999},"last":0.46500000000000002}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.48399999999999999}]},{"osd":7,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":1.002,"15min":1.458},"min":{"1min":0.371,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.84299999999999997,"5min":7.7720000000000002,"15min":10.016999999999999},"last":0.623},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.90500000000000003,"15min":1.4470000000000001},"min":{"1min":0.38,"5min":0.29199999999999998,"15min":0.26000000000000001},"max":{"1min":0.872,"5min":6.9240000000000004,"15min":10.906000000000001},"last":0.5}]}]},{"osd":5,"up_from":492,"seq":2113123909658,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384688,"kb_used_data":7008,"kb_used_omap":7,"kb_used_meta":377656,"kb_avail":93360464,"statfs":{"total":95995035648,"available":95601115136,"internally_reserved":0,"allocated":7176192,"data_stored":4422230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7909,"internal_metadata":386720027},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.496},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.626}]},{"osd":1,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51100000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.52400000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.44}]},{"osd":6,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.60299999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.65900000000000003}]}]},{"osd":4,"up_from":30,"seq":128849019521,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1494908,"kb_used_data":6764,"kb_used_omap":1,"kb_used_meta":1488126,"kb_avail":92250244,"statfs":{"total":95995035648,"available":94464249856,"internally_reserved":0,"allocated":6926336,"data_stored":4163862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1523841368},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":1.242,"15min":1.022},"min":{"1min":0.30199999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.77300000000000002,"5min":21.262,"15min":21.262},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":1.248,"15min":1.0700000000000001},"min":{"1min":0.32100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.72299999999999998,"5min":21.491,"15min":21.491},"last":0.501}]},{"osd":1,"last update":"Fri Jun 14 23:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.91300000000000003,"15min":0.92800000000000005},"min":{"1min":0.441,"5min":0.28100000000000003,"15min":0.26400000000000001},"max":{"1min":0.75800000000000001,"5min":6.0179999999999998,"15min":6.6790000000000003},"last":0.621},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.98299999999999998,"15min":0.96799999999999997},"min":{"1min":0.40500000000000003,"5min":0.27500000000000002,"15min":0.249},"max":{"1min":0.78200000000000003,"5min":6.2169999999999996,"15min":6.9770000000000003},"last":0.60699999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.93200000000000005,"15min":0.97599999999999998},"min":{"1min":0.36599999999999999,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.79300000000000004,"5min":6.0469999999999997,"15min":6.7110000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.93999999999999995,"15min":1.018},"min":{"1min":0.47999999999999998,"5min":0.34499999999999997,"15min":0.27200000000000002},"max":{"1min":0.71999999999999997,"5min":6.0750000000000002,"15min":6.6550000000000002},"last":0.51900000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":1.0880000000000001,"15min":1.1739999999999999},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.27500000000000002},"max":{"1min":0.80100000000000005,"5min":7.2009999999999996,"15min":44.389000000000003},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":1.125,"15min":1.2250000000000001},"min":{"1min":0.45900000000000002,"5min":0.29799999999999999,"15min":0.29599999999999999},"max":{"1min":0.76900000000000002,"5min":6.1950000000000003,"15min":46.101999999999997},"last":0.55800000000000005}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.46700000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:24:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":1.49,"15min":1.849},"min":{"1min":0.38200000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.79600000000000004,"5min":8.1669999999999998,"15min":122.88500000000001},"last":0.438},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.5,"15min":1.851},"min":{"1min":0.40899999999999997,"5min":0.28100000000000003,"15min":0.25700000000000001},"max":{"1min":0.76000000000000001,"5min":14.263999999999999,"15min":122.21299999999999},"last":0.54000000000000004}]},{"osd":7,"last update":"Fri Jun 14 23:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0229999999999999,"15min":1.4319999999999999},"min":{"1min":0.38300000000000001,"5min":0.27000000000000002,"15min":0.255},"max":{"1min":0.78200000000000003,"5min":6.3789999999999996,"15min":11.430999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.625,"5min":1.135,"15min":1.427},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76500000000000001,"5min":7.1420000000000003,"15min":8.5340000000000007},"last":0.57599999999999996}]}]},{"osd":3,"up_from":138,"seq":592705487220,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1278124,"kb_used_data":7068,"kb_used_omap":4,"kb_used_meta":1271035,"kb_avail":92467028,"statfs":{"total":95995035648,"available":94686236672,"internally_reserved":0,"allocated":7237632,"data_stored":4486995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":1301540318},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.46500000000000002,"15min":0.51400000000000001},"min":{"1min":0.189,"5min":0.154,"15min":0.154},"max":{"1min":0.48899999999999999,"5min":2.3919999999999999,"15min":25.190000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.70899999999999996,"15min":0.71399999999999997},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.65300000000000002,"5min":9.6940000000000008,"15min":25.652000000000001},"last":0.35999999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.622,"15min":0.746},"min":{"1min":0.28100000000000003,"5min":0.23000000000000001,"15min":0.20200000000000001},"max":{"1min":0.60099999999999998,"5min":5.5010000000000003,"15min":25.754000000000001},"last":0.318},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.65800000000000003,"15min":0.79500000000000004},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":0.57699999999999996,"5min":3.956,"15min":27.992999999999999},"last":0.40400000000000003}]},{"osd":2,"last update":"Fri Jun 14 23:23:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.56499999999999995,"15min":0.65400000000000003},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.184},"max":{"1min":0.58299999999999996,"5min":5.9189999999999996,"15min":5.9189999999999996},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.55900000000000005,"15min":0.61699999999999999},"min":{"1min":0.26700000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.63,"5min":2.6269999999999998,"15min":3.5550000000000002},"last":0.41499999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.89100000000000001,"15min":1.1970000000000001},"min":{"1min":0.29899999999999999,"5min":0.28999999999999998,"15min":0.27800000000000002},"max":{"1min":0.84599999999999997,"5min":6.5110000000000001,"15min":25.094000000000001},"last":0.498},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.91200000000000003,"15min":1.1679999999999999},"min":{"1min":0.41699999999999998,"5min":0.29999999999999999,"15min":0.27600000000000002},"max":{"1min":0.79900000000000004,"5min":8.2349999999999994,"15min":25.5},"last":0.48699999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.46000000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.153,"15min":1.448},"min":{"1min":0.38200000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82499999999999996,"5min":5.907,"15min":25.869},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.0760000000000001,"15min":1.321},"min":{"1min":0.378,"5min":0.32000000000000001,"15min":0.30499999999999999},"max":{"1min":0.80100000000000005,"5min":9.9309999999999992,"15min":25.143000000000001},"last":0.47499999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.88300000000000001,"15min":1.2450000000000001},"min":{"1min":0.42599999999999999,"5min":0.33600000000000002,"15min":0.28799999999999998},"max":{"1min":0.70899999999999996,"5min":6.085,"15min":44.887999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.877,"15min":1.319},"min":{"1min":0.435,"5min":0.37,"15min":0.36099999999999999},"max":{"1min":0.83799999999999997,"5min":5.0679999999999996,"15min":45.835000000000001},"last":0.55200000000000005}]}]},{"osd":2,"up_from":322,"seq":1382979469475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571056,"kb_used_data":7008,"kb_used_omap":10,"kb_used_meta":564021,"kb_avail":93174096,"statfs":{"total":95995035648,"available":95410274304,"internally_reserved":0,"allocated":7176192,"data_stored":4422217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":577558054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":1.0609999999999999,"15min":0.63100000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.98399999999999999,"5min":40.081000000000003,"15min":40.081000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":1.123,"15min":0.68899999999999995},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.21299999999999999},"max":{"1min":0.56999999999999995,"5min":40.106000000000002,"15min":40.106000000000002},"last":0.442}]},{"osd":1,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.44700000000000001,"15min":0.44500000000000001},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.82099999999999995,"5min":2.7770000000000001,"15min":4.04},"last":0.312},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.46899999999999997,"15min":0.50800000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.192},"max":{"1min":0.48999999999999999,"5min":3.2370000000000001,"15min":6.9889999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.54100000000000004,"15min":0.57199999999999995},"min":{"1min":0.23300000000000001,"5min":0.19,"15min":0.19},"max":{"1min":1.2709999999999999,"5min":3.319,"15min":3.7519999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":0.90300000000000002,"5min":2.8460000000000001,"15min":5.5890000000000004},"last":0.38300000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.81799999999999995,"15min":1.008},"min":{"1min":0.47899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.82399999999999995,"5min":4.1820000000000004,"15min":6.9820000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.74099999999999999,"15min":0.90400000000000003},"min":{"1min":0.39800000000000002,"5min":0.29199999999999998,"15min":0.26400000000000001},"max":{"1min":1.3049999999999999,"5min":3.2909999999999999,"15min":5.0759999999999996},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.504},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.629}]},{"osd":6,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.90600000000000003,"15min":1.0369999999999999},"min":{"1min":0.41899999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.998,"5min":7.0960000000000001,"15min":7.0960000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.875,"15min":1.02},"min":{"1min":0.43099999999999999,"5min":0.32000000000000001,"15min":0.31900000000000001},"max":{"1min":1.3320000000000001,"5min":4.1849999999999996,"15min":4.6950000000000003},"last":0.55800000000000005}]},{"osd":7,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.79100000000000004,"15min":0.92900000000000005},"min":{"1min":0.45100000000000001,"5min":0.377,"15min":0.32500000000000001},"max":{"1min":0.88900000000000001,"5min":4.4960000000000004,"15min":5},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.752,"15min":1.258},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.86299999999999999,"5min":3.8809999999999998,"15min":49.552999999999997},"last":0.57799999999999996}]}]},{"osd":1,"up_from":116,"seq":498216206764,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":791268,"kb_used_data":6744,"kb_used_omap":5,"kb_used_meta":784506,"kb_avail":92953884,"statfs":{"total":95995035648,"available":95184777216,"internally_reserved":0,"allocated":6905856,"data_stored":4159225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":803334424},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.83699999999999997,"15min":0.57799999999999996},"min":{"1min":0.21099999999999999,"5min":0.184,"15min":0.16200000000000001},"max":{"1min":0.88500000000000001,"5min":24.329999999999998,"15min":24.329999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.81999999999999995,"15min":0.56499999999999995},"min":{"1min":0.218,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":0.78600000000000003,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.29099999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:23:45 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.499,"15min":0.51100000000000001},"min":{"1min":0.24099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.751,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.52500000000000002,"15min":0.47999999999999998},"min":{"1min":0.26100000000000001,"5min":0.20799999999999999,"15min":0.188},"max":{"1min":0.83799999999999997,"5min":2.524,"15min":4.2240000000000002},"last":0.34999999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.61499999999999999,"15min":0.59599999999999997},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.20499999999999999},"max":{"1min":0.58799999999999997,"5min":4.8380000000000001,"15min":7.0739999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.59099999999999997,"15min":0.61399999999999999},"min":{"1min":0.26800000000000002,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":0.97799999999999998,"5min":3.0179999999999998,"15min":10.541},"last":0.376}]},{"osd":4,"last update":"Fri Jun 14 23:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.98599999999999999,"15min":1.155},"min":{"1min":0.39100000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73299999999999998,"5min":4.806,"15min":43.959000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.90800000000000003,"15min":1.0780000000000001},"min":{"1min":0.39500000000000002,"5min":0.28899999999999998,"15min":0.28000000000000003},"max":{"1min":0.93200000000000005,"5min":4.7130000000000001,"15min":44.006999999999998},"last":0.50800000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.52200000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.038,"15min":1.262},"min":{"1min":0.40500000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77200000000000002,"5min":4.7720000000000002,"15min":44.241},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.96899999999999997,"15min":1.337},"min":{"1min":0.47099999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88500000000000001,"5min":4.431,"15min":45.627000000000002},"last":0.46000000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.85999999999999999,"15min":1.169},"min":{"1min":0.45500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.94399999999999995,"5min":4.2389999999999999,"15min":44.667000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.86599999999999999,"15min":1.1579999999999999},"min":{"1min":0.47099999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92600000000000005,"5min":4.258,"15min":44.679000000000002},"last":0.57099999999999995}]}]},{"osd":0,"up_from":91,"seq":390842024419,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1252584,"kb_used_data":7064,"kb_used_omap":5,"kb_used_meta":1245498,"kb_avail":92492568,"statfs":{"total":95995035648,"available":94712389632,"internally_reserved":0,"allocated":7233536,"data_stored":4484904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":1275390042},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:24:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.64000000000000001,"15min":0.47799999999999998},"min":{"1min":0.20200000000000001,"5min":0.191,"15min":0.17899999999999999},"max":{"1min":0.73599999999999999,"5min":11.733000000000001,"15min":11.733000000000001},"last":0.375},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.61799999999999999,"15min":0.48699999999999999},"min":{"1min":0.21099999999999999,"5min":0.20899999999999999,"15min":0.19500000000000001},"max":{"1min":0.79300000000000004,"5min":4.3769999999999998,"15min":4.3769999999999998},"last":0.216}]},{"osd":2,"last update":"Fri Jun 14 23:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.55800000000000005,"15min":0.53800000000000003},"min":{"1min":0.22900000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.58799999999999997,"5min":4.3109999999999999,"15min":4.3109999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.53800000000000003,"15min":0.52200000000000002},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.20100000000000001},"max":{"1min":0.70599999999999996,"5min":2.7469999999999999,"15min":3.2509999999999999},"last":0.28799999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.73199999999999998,"15min":0.63500000000000001},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.61499999999999999,"5min":10.65,"15min":10.65},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.55500000000000005,"15min":0.61299999999999999},"min":{"1min":0.29499999999999998,"5min":0.252,"15min":0.217},"max":{"1min":0.82499999999999996,"5min":2.5,"15min":7.4279999999999999},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.93000000000000005,"15min":1.0589999999999999},"min":{"1min":0.39800000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0649999999999999,"5min":7.4009999999999998,"15min":24.170999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.996,"15min":1.0760000000000001},"min":{"1min":0.36199999999999999,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":0.877,"5min":4.5540000000000003,"15min":24.204000000000001},"last":0.51400000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.53400000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:24:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.538,"15min":1.3460000000000001},"min":{"1min":0.45200000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76500000000000001,"5min":36.478000000000002,"15min":36.478000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.516,"15min":1.3600000000000001},"min":{"1min":0.46100000000000002,"5min":0.39300000000000002,"15min":0.33700000000000002},"max":{"1min":1.024,"5min":36.494,"15min":36.494},"last":0.498}]},{"osd":7,"last update":"Fri Jun 14 23:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.90900000000000003,"15min":1.1339999999999999},"min":{"1min":0.43099999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71199999999999997,"5min":4.3920000000000003,"15min":24.131},"last":0.433},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.87,"15min":1.1519999999999999},"min":{"1min":0.46800000000000003,"5min":0.38500000000000001,"15min":0.313},"max":{"1min":1.0229999999999999,"5min":3.6200000000000001,"15min":24.257000000000001},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6205,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:24:52.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-14T23:24:53.081 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/mon.a/config 2024-06-14T23:24:53.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:53 smithi086 ceph-mon[97913]: pgmap v1803: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:53.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:53 smithi086 ceph-mon[97913]: from='client.47657 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:24:53.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:53 smithi086 ceph-mon[102754]: pgmap v1803: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:53.329 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:53 smithi086 ceph-mon[102754]: from='client.47657 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:24:53.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:53 smithi137 ceph-mon[90634]: pgmap v1803: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:53.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:53 smithi137 ceph-mon[90634]: from='client.47657 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:24:55.053 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:54 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: ::ffff:172.21.15.137 - - [14/Jun/2024:23:24:54] "GET /metrics HTTP/1.1" 200 34260 "" "Prometheus/2.43.0" 2024-06-14T23:24:55.076 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-06-14T23:24:55.078 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-06-14T23:24:55.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:55 smithi086 ceph-mon[97913]: pgmap v1804: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:55.337 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:55 smithi086 ceph-mon[102754]: pgmap v1804: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:55.495 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:55 smithi137 ceph-mon[90634]: pgmap v1804: 103 pgs: 103 active+clean; 582 KiB data, 7.9 GiB used, 707 GiB / 715 GiB avail 2024-06-14T23:24:55.892 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1804,"stamp":"2024-06-14T23:24:53.936718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":591710,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5048,"ondisk_log_size":5048,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8246352,"kb_used_data":55764,"kb_used_omap":45,"kb_used_meta":8190418,"kb_avail":741714864,"statfs":{"total":767960285184,"available":759516020736,"internally_reserved":0,"allocated":57102336,"data_stored":35057726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46341,"internal_metadata":8386988795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002849"},"pg_stats":[{"pgid":"1.1d","version":"77'161","reported_seq":800,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023850+0000","last_change":"2024-06-14T23:23:20.288570+0000","last_active":"2024-06-14T23:23:58.023850+0000","last_peered":"2024-06-14T23:23:58.023850+0000","last_clean":"2024-06-14T23:23:58.023850+0000","last_became_active":"2024-06-14T23:23:20.194560+0000","last_became_peered":"2024-06-14T23:23:20.194560+0000","last_unstale":"2024-06-14T23:23:58.023850+0000","last_undegraded":"2024-06-14T23:23:58.023850+0000","last_fullsized":"2024-06-14T23:23:58.023850+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:20.288454+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:23:20.288454+0000","last_clean_scrub_stamp":"2024-06-14T23:23:20.288454+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:46:53.462606+0000","scrub_duration":0.0015503959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"68'4","reported_seq":257,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.022024+0000","last_change":"2024-06-14T23:23:06.104075+0000","last_active":"2024-06-14T23:23:58.022024+0000","last_peered":"2024-06-14T23:23:58.022024+0000","last_clean":"2024-06-14T23:23:58.022024+0000","last_became_active":"2024-06-14T23:23:06.103680+0000","last_became_peered":"2024-06-14T23:23:06.103680+0000","last_unstale":"2024-06-14T23:23:58.022024+0000","last_undegraded":"2024-06-14T23:23:58.022024+0000","last_fullsized":"2024-06-14T23:23:58.022024+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:17:16.172095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":496,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024564+0000","last_change":"2024-06-14T23:23:46.536793+0000","last_active":"2024-06-14T23:23:58.024564+0000","last_peered":"2024-06-14T23:23:58.024564+0000","last_clean":"2024-06-14T23:23:58.024564+0000","last_became_active":"2024-06-14T23:23:46.536459+0000","last_became_peered":"2024-06-14T23:23:46.536459+0000","last_unstale":"2024-06-14T23:23:58.024564+0000","last_undegraded":"2024-06-14T23:23:58.024564+0000","last_fullsized":"2024-06-14T23:23:58.024564+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:20:12.239490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":112,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019452+0000","last_change":"2024-06-14T23:23:21.191748+0000","last_active":"2024-06-14T23:23:58.019452+0000","last_peered":"2024-06-14T23:23:58.019452+0000","last_clean":"2024-06-14T23:23:58.019452+0000","last_became_active":"2024-06-14T23:23:21.191436+0000","last_became_peered":"2024-06-14T23:23:21.191436+0000","last_unstale":"2024-06-14T23:23:58.019452+0000","last_undegraded":"2024-06-14T23:23:58.019452+0000","last_fullsized":"2024-06-14T23:23:58.019452+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:46:29.102290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":118,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018943+0000","last_change":"2024-06-14T23:23:21.190170+0000","last_active":"2024-06-14T23:23:58.018943+0000","last_peered":"2024-06-14T23:23:58.018943+0000","last_clean":"2024-06-14T23:23:58.018943+0000","last_became_active":"2024-06-14T23:23:21.189878+0000","last_became_peered":"2024-06-14T23:23:21.189878+0000","last_unstale":"2024-06-14T23:23:58.018943+0000","last_undegraded":"2024-06-14T23:23:58.018943+0000","last_fullsized":"2024-06-14T23:23:58.018943+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:02:02.826758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"267'162","reported_seq":778,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.016965+0000","last_change":"2024-06-14T23:23:36.146519+0000","last_active":"2024-06-14T23:23:58.016965+0000","last_peered":"2024-06-14T23:23:58.016965+0000","last_clean":"2024-06-14T23:23:58.016965+0000","last_became_active":"2024-06-14T23:23:36.146150+0000","last_became_peered":"2024-06-14T23:23:36.146150+0000","last_unstale":"2024-06-14T23:23:58.016965+0000","last_undegraded":"2024-06-14T23:23:58.016965+0000","last_fullsized":"2024-06-14T23:23:58.016965+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":5,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:14:16.297860+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:14:16.297860+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:46:35.959931+0000","scrub_duration":0.0039154719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"68'2","reported_seq":366,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.022418+0000","last_change":"2024-06-14T23:23:59.022418+0000","last_active":"2024-06-14T23:23:59.022418+0000","last_peered":"2024-06-14T23:23:59.022418+0000","last_clean":"2024-06-14T23:23:59.022418+0000","last_became_active":"2024-06-14T23:23:59.022069+0000","last_became_peered":"2024-06-14T23:23:59.022069+0000","last_unstale":"2024-06-14T23:23:59.022418+0000","last_undegraded":"2024-06-14T23:23:59.022418+0000","last_fullsized":"2024-06-14T23:23:59.022418+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:29:00.688565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":508,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026423+0000","last_change":"2024-06-14T23:23:28.038120+0000","last_active":"2024-06-14T23:23:58.026423+0000","last_peered":"2024-06-14T23:23:58.026423+0000","last_clean":"2024-06-14T23:23:58.026423+0000","last_became_active":"2024-06-14T23:23:28.037605+0000","last_became_peered":"2024-06-14T23:23:28.037605+0000","last_unstale":"2024-06-14T23:23:58.026423+0000","last_undegraded":"2024-06-14T23:23:58.026423+0000","last_fullsized":"2024-06-14T23:23:58.026423+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:40:18.816673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"77'161","reported_seq":464,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019752+0000","last_change":"2024-06-14T23:23:22.971293+0000","last_active":"2024-06-14T23:23:58.019752+0000","last_peered":"2024-06-14T23:23:58.019752+0000","last_clean":"2024-06-14T23:23:58.019752+0000","last_became_active":"2024-06-14T23:23:20.199775+0000","last_became_peered":"2024-06-14T23:23:20.199775+0000","last_unstale":"2024-06-14T23:23:58.019752+0000","last_undegraded":"2024-06-14T23:23:58.019752+0000","last_fullsized":"2024-06-14T23:23:58.019752+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:22.971203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:22.971203+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:26:16.160391+0000","scrub_duration":0.001713518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"68'2","reported_seq":440,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.025204+0000","last_change":"2024-06-14T23:23:59.025204+0000","last_active":"2024-06-14T23:23:59.025204+0000","last_peered":"2024-06-14T23:23:59.025204+0000","last_clean":"2024-06-14T23:23:59.025204+0000","last_became_active":"2024-06-14T23:23:59.023607+0000","last_became_peered":"2024-06-14T23:23:59.023607+0000","last_unstale":"2024-06-14T23:23:59.025204+0000","last_undegraded":"2024-06-14T23:23:59.025204+0000","last_fullsized":"2024-06-14T23:23:59.025204+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:27:51.272020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":517,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026552+0000","last_change":"2024-06-14T23:23:28.031970+0000","last_active":"2024-06-14T23:23:58.026552+0000","last_peered":"2024-06-14T23:23:58.026552+0000","last_clean":"2024-06-14T23:23:58.026552+0000","last_became_active":"2024-06-14T23:23:28.031625+0000","last_became_peered":"2024-06-14T23:23:28.031625+0000","last_unstale":"2024-06-14T23:23:58.026552+0000","last_undegraded":"2024-06-14T23:23:58.026552+0000","last_fullsized":"2024-06-14T23:23:58.026552+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:08:44.108966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"68'1","reported_seq":149,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021736+0000","last_change":"2024-06-14T23:17:22.888832+0000","last_active":"2024-06-14T23:23:58.021736+0000","last_peered":"2024-06-14T23:23:58.021736+0000","last_clean":"2024-06-14T23:23:58.021736+0000","last_became_active":"2024-06-14T23:17:22.888525+0000","last_became_peered":"2024-06-14T23:17:22.888525+0000","last_unstale":"2024-06-14T23:23:58.021736+0000","last_undegraded":"2024-06-14T23:23:58.021736+0000","last_fullsized":"2024-06-14T23:23:58.021736+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:32:36.414213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":122,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.014675+0000","last_change":"2024-06-14T23:23:46.990897+0000","last_active":"2024-06-14T23:23:58.014675+0000","last_peered":"2024-06-14T23:23:58.014675+0000","last_clean":"2024-06-14T23:23:58.014675+0000","last_became_active":"2024-06-14T23:23:46.990507+0000","last_became_peered":"2024-06-14T23:23:46.990507+0000","last_unstale":"2024-06-14T23:23:58.014675+0000","last_undegraded":"2024-06-14T23:23:58.014675+0000","last_fullsized":"2024-06-14T23:23:58.014675+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:31:46.522707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"77'161","reported_seq":741,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018785+0000","last_change":"2024-06-14T23:23:21.191578+0000","last_active":"2024-06-14T23:23:58.018785+0000","last_peered":"2024-06-14T23:23:58.018785+0000","last_clean":"2024-06-14T23:23:58.018785+0000","last_became_active":"2024-06-14T23:23:21.191188+0000","last_became_peered":"2024-06-14T23:23:21.191188+0000","last_unstale":"2024-06-14T23:23:58.018785+0000","last_undegraded":"2024-06-14T23:23:58.018785+0000","last_fullsized":"2024-06-14T23:23:58.018785+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.294457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.294457+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:06:26.447181+0000","scrub_duration":0.0032619889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"68'1","reported_seq":441,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026518+0000","last_change":"2024-06-14T23:23:28.035129+0000","last_active":"2024-06-14T23:23:58.026518+0000","last_peered":"2024-06-14T23:23:58.026518+0000","last_clean":"2024-06-14T23:23:58.026518+0000","last_became_active":"2024-06-14T23:23:28.034629+0000","last_became_peered":"2024-06-14T23:23:28.034629+0000","last_unstale":"2024-06-14T23:23:58.026518+0000","last_undegraded":"2024-06-14T23:23:58.026518+0000","last_fullsized":"2024-06-14T23:23:58.026518+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:37.038650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":58,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019436+0000","last_change":"2024-06-14T23:23:21.291179+0000","last_active":"2024-06-14T23:23:58.019436+0000","last_peered":"2024-06-14T23:23:58.019436+0000","last_clean":"2024-06-14T23:23:58.019436+0000","last_became_active":"2024-06-14T23:23:21.290706+0000","last_became_peered":"2024-06-14T23:23:21.290706+0000","last_unstale":"2024-06-14T23:23:58.019436+0000","last_undegraded":"2024-06-14T23:23:58.019436+0000","last_fullsized":"2024-06-14T23:23:58.019436+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:42:06.026436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.18","version":"77'161","reported_seq":879,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.027604+0000","last_change":"2024-06-14T23:23:59.027604+0000","last_active":"2024-06-14T23:23:59.027604+0000","last_peered":"2024-06-14T23:23:59.027604+0000","last_clean":"2024-06-14T23:23:59.027604+0000","last_became_active":"2024-06-14T23:23:59.027183+0000","last_became_peered":"2024-06-14T23:23:59.027183+0000","last_unstale":"2024-06-14T23:23:59.027604+0000","last_undegraded":"2024-06-14T23:23:59.027604+0000","last_fullsized":"2024-06-14T23:23:59.027604+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.212494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.212494+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:48:40.059953+0000","scrub_duration":0.0040473590000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":51,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.016996+0000","last_change":"2024-06-14T23:22:23.531169+0000","last_active":"2024-06-14T23:23:59.016996+0000","last_peered":"2024-06-14T23:23:59.016996+0000","last_clean":"2024-06-14T23:23:59.016996+0000","last_became_active":"2024-06-14T23:22:23.530840+0000","last_became_peered":"2024-06-14T23:22:23.530840+0000","last_unstale":"2024-06-14T23:23:59.016996+0000","last_undegraded":"2024-06-14T23:23:59.016996+0000","last_fullsized":"2024-06-14T23:23:59.016996+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:13:04.472958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":359,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.020678+0000","last_change":"2024-06-14T23:23:17.169294+0000","last_active":"2024-06-14T23:23:59.020678+0000","last_peered":"2024-06-14T23:23:59.020678+0000","last_clean":"2024-06-14T23:23:59.020678+0000","last_became_active":"2024-06-14T23:23:17.168995+0000","last_became_peered":"2024-06-14T23:23:17.168995+0000","last_unstale":"2024-06-14T23:23:59.020678+0000","last_undegraded":"2024-06-14T23:23:59.020678+0000","last_fullsized":"2024-06-14T23:23:59.020678+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:43:01.161977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"77'161","reported_seq":833,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.026517+0000","last_change":"2024-06-14T23:23:59.026517+0000","last_active":"2024-06-14T23:23:59.026517+0000","last_peered":"2024-06-14T23:23:59.026517+0000","last_clean":"2024-06-14T23:23:59.026517+0000","last_became_active":"2024-06-14T23:23:59.025650+0000","last_became_peered":"2024-06-14T23:23:59.025650+0000","last_unstale":"2024-06-14T23:23:59.026517+0000","last_undegraded":"2024-06-14T23:23:59.026517+0000","last_fullsized":"2024-06-14T23:23:59.026517+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:31.405872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:31.405872+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:04:19.311760+0000","scrub_duration":0.0024420650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":221,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018719+0000","last_change":"2024-06-14T23:23:21.291096+0000","last_active":"2024-06-14T23:23:58.018719+0000","last_peered":"2024-06-14T23:23:58.018719+0000","last_clean":"2024-06-14T23:23:58.018719+0000","last_became_active":"2024-06-14T23:23:21.290798+0000","last_became_peered":"2024-06-14T23:23:21.290798+0000","last_unstale":"2024-06-14T23:23:58.018719+0000","last_undegraded":"2024-06-14T23:23:58.018719+0000","last_fullsized":"2024-06-14T23:23:58.018719+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:31:41.895801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":514,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025244+0000","last_change":"2024-06-14T23:22:52.005874+0000","last_active":"2024-06-14T23:23:58.025244+0000","last_peered":"2024-06-14T23:23:58.025244+0000","last_clean":"2024-06-14T23:23:58.025244+0000","last_became_active":"2024-06-14T23:22:52.005570+0000","last_became_peered":"2024-06-14T23:22:52.005570+0000","last_unstale":"2024-06-14T23:23:58.025244+0000","last_undegraded":"2024-06-14T23:23:58.025244+0000","last_fullsized":"2024-06-14T23:23:58.025244+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:09:16.899262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"77'161","reported_seq":806,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025797+0000","last_change":"2024-06-14T23:23:28.041830+0000","last_active":"2024-06-14T23:23:58.025797+0000","last_peered":"2024-06-14T23:23:58.025797+0000","last_clean":"2024-06-14T23:23:58.025797+0000","last_became_active":"2024-06-14T23:23:28.039425+0000","last_became_peered":"2024-06-14T23:23:28.039425+0000","last_unstale":"2024-06-14T23:23:58.025797+0000","last_undegraded":"2024-06-14T23:23:58.025797+0000","last_fullsized":"2024-06-14T23:23:58.025797+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.072813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.072813+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:34:08.026179+0000","scrub_duration":0.0065527240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019684+0000","last_change":"2024-06-14T23:23:17.163583+0000","last_active":"2024-06-14T23:23:58.019684+0000","last_peered":"2024-06-14T23:23:58.019684+0000","last_clean":"2024-06-14T23:23:58.019684+0000","last_became_active":"2024-06-14T23:23:17.163254+0000","last_became_peered":"2024-06-14T23:23:17.163254+0000","last_unstale":"2024-06-14T23:23:58.019684+0000","last_undegraded":"2024-06-14T23:23:58.019684+0000","last_fullsized":"2024-06-14T23:23:58.019684+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:36:50.673160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":298,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021932+0000","last_change":"2024-06-14T23:23:17.170000+0000","last_active":"2024-06-14T23:23:58.021932+0000","last_peered":"2024-06-14T23:23:58.021932+0000","last_clean":"2024-06-14T23:23:58.021932+0000","last_became_active":"2024-06-14T23:23:17.169688+0000","last_became_peered":"2024-06-14T23:23:17.169688+0000","last_unstale":"2024-06-14T23:23:58.021932+0000","last_undegraded":"2024-06-14T23:23:58.021932+0000","last_fullsized":"2024-06-14T23:23:58.021932+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:26:28.426330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"77'161","reported_seq":865,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019138+0000","last_change":"2024-06-14T23:22:52.005620+0000","last_active":"2024-06-14T23:23:58.019138+0000","last_peered":"2024-06-14T23:23:58.019138+0000","last_clean":"2024-06-14T23:23:58.019138+0000","last_became_active":"2024-06-14T23:22:52.004963+0000","last_became_peered":"2024-06-14T23:22:52.004963+0000","last_unstale":"2024-06-14T23:23:58.019138+0000","last_undegraded":"2024-06-14T23:23:58.019138+0000","last_fullsized":"2024-06-14T23:23:58.019138+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.180554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.180554+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:45:05.867553+0000","scrub_duration":0.0020557959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":393,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.026755+0000","last_change":"2024-06-14T23:07:58.027242+0000","last_active":"2024-06-14T23:23:59.026755+0000","last_peered":"2024-06-14T23:23:59.026755+0000","last_clean":"2024-06-14T23:23:59.026755+0000","last_became_active":"2024-06-14T23:07:58.026965+0000","last_became_peered":"2024-06-14T23:07:58.026965+0000","last_unstale":"2024-06-14T23:23:59.026755+0000","last_undegraded":"2024-06-14T23:23:59.026755+0000","last_fullsized":"2024-06-14T23:23:59.026755+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:34:29.270607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":320,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.024532+0000","last_change":"2024-06-14T23:23:59.024532+0000","last_active":"2024-06-14T23:23:59.024532+0000","last_peered":"2024-06-14T23:23:59.024532+0000","last_clean":"2024-06-14T23:23:59.024532+0000","last_became_active":"2024-06-14T23:23:59.024216+0000","last_became_peered":"2024-06-14T23:23:59.024216+0000","last_unstale":"2024-06-14T23:23:59.024532+0000","last_undegraded":"2024-06-14T23:23:59.024532+0000","last_fullsized":"2024-06-14T23:23:59.024532+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:36:36.377058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"77'161","reported_seq":680,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.014568+0000","last_change":"2024-06-14T23:22:15.024503+0000","last_active":"2024-06-14T23:23:58.014568+0000","last_peered":"2024-06-14T23:23:58.014568+0000","last_clean":"2024-06-14T23:23:58.014568+0000","last_became_active":"2024-06-14T23:20:36.537993+0000","last_became_peered":"2024-06-14T23:20:36.537993+0000","last_unstale":"2024-06-14T23:23:58.014568+0000","last_undegraded":"2024-06-14T23:23:58.014568+0000","last_fullsized":"2024-06-14T23:23:58.014568+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:15.024379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:15.024379+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:41:14.672203+0000","scrub_duration":0.013254276000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"68'3","reported_seq":128,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018118+0000","last_change":"2024-06-14T23:23:36.142633+0000","last_active":"2024-06-14T23:23:58.018118+0000","last_peered":"2024-06-14T23:23:58.018118+0000","last_clean":"2024-06-14T23:23:58.018118+0000","last_became_active":"2024-06-14T23:23:36.142302+0000","last_became_peered":"2024-06-14T23:23:36.142302+0000","last_unstale":"2024-06-14T23:23:58.018118+0000","last_undegraded":"2024-06-14T23:23:58.018118+0000","last_fullsized":"2024-06-14T23:23:58.018118+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:19.659336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":561,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023990+0000","last_change":"2024-06-14T23:23:13.145733+0000","last_active":"2024-06-14T23:23:58.023990+0000","last_peered":"2024-06-14T23:23:58.023990+0000","last_clean":"2024-06-14T23:23:58.023990+0000","last_became_active":"2024-06-14T23:23:13.145456+0000","last_became_peered":"2024-06-14T23:23:13.145456+0000","last_unstale":"2024-06-14T23:23:58.023990+0000","last_undegraded":"2024-06-14T23:23:58.023990+0000","last_fullsized":"2024-06-14T23:23:58.023990+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:35:17.692823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"77'161","reported_seq":630,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.028342+0000","last_change":"2024-06-14T23:23:59.028342+0000","last_active":"2024-06-14T23:23:59.028342+0000","last_peered":"2024-06-14T23:23:59.028342+0000","last_clean":"2024-06-14T23:23:59.028342+0000","last_became_active":"2024-06-14T23:23:59.028020+0000","last_became_peered":"2024-06-14T23:23:59.028020+0000","last_unstale":"2024-06-14T23:23:59.028342+0000","last_undegraded":"2024-06-14T23:23:59.028342+0000","last_fullsized":"2024-06-14T23:23:59.028342+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:59.004461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:59.004461+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:17:58.866191+0000","scrub_duration":0.0026353430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'22","reported_seq":366,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026225+0000","last_change":"2024-06-14T23:23:28.040402+0000","last_active":"2024-06-14T23:23:58.026225+0000","last_peered":"2024-06-14T23:23:58.026225+0000","last_clean":"2024-06-14T23:23:58.026225+0000","last_became_active":"2024-06-14T23:23:28.038550+0000","last_became_peered":"2024-06-14T23:23:28.038550+0000","last_unstale":"2024-06-14T23:23:58.026225+0000","last_undegraded":"2024-06-14T23:23:58.026225+0000","last_fullsized":"2024-06-14T23:23:58.026225+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:13:39.022225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":458,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026636+0000","last_change":"2024-06-14T23:23:28.040648+0000","last_active":"2024-06-14T23:23:58.026636+0000","last_peered":"2024-06-14T23:23:58.026636+0000","last_clean":"2024-06-14T23:23:58.026636+0000","last_became_active":"2024-06-14T23:23:28.039483+0000","last_became_peered":"2024-06-14T23:23:28.039483+0000","last_unstale":"2024-06-14T23:23:58.026636+0000","last_undegraded":"2024-06-14T23:23:58.026636+0000","last_fullsized":"2024-06-14T23:23:58.026636+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:24:52.826818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"77'161","reported_seq":617,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017492+0000","last_change":"2024-06-14T23:23:20.195514+0000","last_active":"2024-06-14T23:23:59.017492+0000","last_peered":"2024-06-14T23:23:59.017492+0000","last_clean":"2024-06-14T23:23:59.017492+0000","last_became_active":"2024-06-14T23:23:20.195194+0000","last_became_peered":"2024-06-14T23:23:20.195194+0000","last_unstale":"2024-06-14T23:23:59.017492+0000","last_undegraded":"2024-06-14T23:23:59.017492+0000","last_fullsized":"2024-06-14T23:23:59.017492+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":285,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.634078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.634078+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:57:24.404193+0000","scrub_duration":0.00160432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":344,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025440+0000","last_change":"2024-06-14T23:23:28.040993+0000","last_active":"2024-06-14T23:23:58.025440+0000","last_peered":"2024-06-14T23:23:58.025440+0000","last_clean":"2024-06-14T23:23:58.025440+0000","last_became_active":"2024-06-14T23:23:28.039242+0000","last_became_peered":"2024-06-14T23:23:28.039242+0000","last_unstale":"2024-06-14T23:23:58.025440+0000","last_undegraded":"2024-06-14T23:23:58.025440+0000","last_fullsized":"2024-06-14T23:23:58.025440+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:28:26.643804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":514,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024949+0000","last_change":"2024-06-14T23:20:33.516210+0000","last_active":"2024-06-14T23:23:58.024949+0000","last_peered":"2024-06-14T23:23:58.024949+0000","last_clean":"2024-06-14T23:23:58.024949+0000","last_became_active":"2024-06-14T23:20:33.515892+0000","last_became_peered":"2024-06-14T23:20:33.515892+0000","last_unstale":"2024-06-14T23:23:58.024949+0000","last_undegraded":"2024-06-14T23:23:58.024949+0000","last_fullsized":"2024-06-14T23:23:58.024949+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:23:52.415552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":127,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.020757+0000","last_change":"2024-06-14T23:22:23.550370+0000","last_active":"2024-06-14T23:23:59.020757+0000","last_peered":"2024-06-14T23:23:59.020757+0000","last_clean":"2024-06-14T23:23:59.020757+0000","last_became_active":"2024-06-14T23:22:23.520680+0000","last_became_peered":"2024-06-14T23:22:23.520680+0000","last_unstale":"2024-06-14T23:23:59.020757+0000","last_undegraded":"2024-06-14T23:23:59.020757+0000","last_fullsized":"2024-06-14T23:23:59.020757+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:00:36.472602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"77'161","reported_seq":548,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019467+0000","last_change":"2024-06-14T23:23:20.194637+0000","last_active":"2024-06-14T23:23:58.019467+0000","last_peered":"2024-06-14T23:23:58.019467+0000","last_clean":"2024-06-14T23:23:58.019467+0000","last_became_active":"2024-06-14T23:23:20.194088+0000","last_became_peered":"2024-06-14T23:23:20.194088+0000","last_unstale":"2024-06-14T23:23:58.019467+0000","last_undegraded":"2024-06-14T23:23:58.019467+0000","last_fullsized":"2024-06-14T23:23:58.019467+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:52.977636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:52.977636+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:57:19.372132+0000","scrub_duration":0.001529084,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019525+0000","last_change":"2024-06-14T23:23:17.168929+0000","last_active":"2024-06-14T23:23:58.019525+0000","last_peered":"2024-06-14T23:23:58.019525+0000","last_clean":"2024-06-14T23:23:58.019525+0000","last_became_active":"2024-06-14T23:23:17.168615+0000","last_became_peered":"2024-06-14T23:23:17.168615+0000","last_unstale":"2024-06-14T23:23:58.019525+0000","last_undegraded":"2024-06-14T23:23:58.019525+0000","last_fullsized":"2024-06-14T23:23:58.019525+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:22:02.752839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017285+0000","last_change":"2024-06-14T23:22:23.535725+0000","last_active":"2024-06-14T23:23:59.017285+0000","last_peered":"2024-06-14T23:23:59.017285+0000","last_clean":"2024-06-14T23:23:59.017285+0000","last_became_active":"2024-06-14T23:22:23.535450+0000","last_became_peered":"2024-06-14T23:22:23.535450+0000","last_unstale":"2024-06-14T23:23:59.017285+0000","last_undegraded":"2024-06-14T23:23:59.017285+0000","last_fullsized":"2024-06-14T23:23:59.017285+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:44:55.548227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'2","reported_seq":343,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019968+0000","last_change":"2024-06-14T23:23:21.190343+0000","last_active":"2024-06-14T23:23:58.019968+0000","last_peered":"2024-06-14T23:23:58.019968+0000","last_clean":"2024-06-14T23:23:58.019968+0000","last_became_active":"2024-06-14T23:23:21.190029+0000","last_became_peered":"2024-06-14T23:23:21.190029+0000","last_unstale":"2024-06-14T23:23:58.019968+0000","last_undegraded":"2024-06-14T23:23:58.019968+0000","last_fullsized":"2024-06-14T23:23:58.019968+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:19:29.959380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"77'161","reported_seq":817,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.027003+0000","last_change":"2024-06-14T23:23:28.042678+0000","last_active":"2024-06-14T23:23:58.027003+0000","last_peered":"2024-06-14T23:23:58.027003+0000","last_clean":"2024-06-14T23:23:58.027003+0000","last_became_active":"2024-06-14T23:23:28.041664+0000","last_became_peered":"2024-06-14T23:23:28.041664+0000","last_unstale":"2024-06-14T23:23:58.027003+0000","last_undegraded":"2024-06-14T23:23:58.027003+0000","last_fullsized":"2024-06-14T23:23:58.027003+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:51.748589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:51.748589+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:28:36.624812+0000","scrub_duration":0.0050803430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":335,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021582+0000","last_change":"2024-06-14T23:22:51.003878+0000","last_active":"2024-06-14T23:23:58.021582+0000","last_peered":"2024-06-14T23:23:58.021582+0000","last_clean":"2024-06-14T23:23:58.021582+0000","last_became_active":"2024-06-14T23:22:51.003543+0000","last_became_peered":"2024-06-14T23:22:51.003543+0000","last_unstale":"2024-06-14T23:23:58.021582+0000","last_undegraded":"2024-06-14T23:23:58.021582+0000","last_fullsized":"2024-06-14T23:23:58.021582+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:49.194146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":515,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019153+0000","last_change":"2024-06-14T23:17:22.890776+0000","last_active":"2024-06-14T23:23:58.019153+0000","last_peered":"2024-06-14T23:23:58.019153+0000","last_clean":"2024-06-14T23:23:58.019153+0000","last_became_active":"2024-06-14T23:17:22.890463+0000","last_became_peered":"2024-06-14T23:17:22.890463+0000","last_unstale":"2024-06-14T23:23:58.019153+0000","last_undegraded":"2024-06-14T23:23:58.019153+0000","last_fullsized":"2024-06-14T23:23:58.019153+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:45:42.223422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":118,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023479+0000","last_change":"2024-06-14T23:23:13.144735+0000","last_active":"2024-06-14T23:23:58.023479+0000","last_peered":"2024-06-14T23:23:58.023479+0000","last_clean":"2024-06-14T23:23:58.023479+0000","last_became_active":"2024-06-14T23:23:13.144365+0000","last_became_peered":"2024-06-14T23:23:13.144365+0000","last_unstale":"2024-06-14T23:23:58.023479+0000","last_undegraded":"2024-06-14T23:23:58.023479+0000","last_fullsized":"2024-06-14T23:23:58.023479+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:41:59.814540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'161","reported_seq":881,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.022930+0000","last_change":"2024-06-14T23:23:20.196260+0000","last_active":"2024-06-14T23:23:58.022930+0000","last_peered":"2024-06-14T23:23:58.022930+0000","last_clean":"2024-06-14T23:23:58.022930+0000","last_became_active":"2024-06-14T23:23:20.195740+0000","last_became_peered":"2024-06-14T23:23:20.195740+0000","last_unstale":"2024-06-14T23:23:58.022930+0000","last_undegraded":"2024-06-14T23:23:58.022930+0000","last_fullsized":"2024-06-14T23:23:58.022930+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:55.324136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:55.324136+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:26:18.594571+0000","scrub_duration":0.0032030190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":62,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017598+0000","last_change":"2024-06-14T23:22:52.002567+0000","last_active":"2024-06-14T23:23:59.017598+0000","last_peered":"2024-06-14T23:23:59.017598+0000","last_clean":"2024-06-14T23:23:59.017598+0000","last_became_active":"2024-06-14T23:22:52.002401+0000","last_became_peered":"2024-06-14T23:22:52.002401+0000","last_unstale":"2024-06-14T23:23:59.017598+0000","last_undegraded":"2024-06-14T23:23:59.017598+0000","last_fullsized":"2024-06-14T23:23:59.017598+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:28:51.732409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":185,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.018317+0000","last_change":"2024-06-14T23:23:13.143752+0000","last_active":"2024-06-14T23:23:59.018317+0000","last_peered":"2024-06-14T23:23:59.018317+0000","last_clean":"2024-06-14T23:23:59.018317+0000","last_became_active":"2024-06-14T23:23:13.143216+0000","last_became_peered":"2024-06-14T23:23:13.143216+0000","last_unstale":"2024-06-14T23:23:59.018317+0000","last_undegraded":"2024-06-14T23:23:59.018317+0000","last_fullsized":"2024-06-14T23:23:59.018317+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:48:43.394464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":459,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.028767+0000","last_change":"2024-06-14T23:23:59.028767+0000","last_active":"2024-06-14T23:23:59.028767+0000","last_peered":"2024-06-14T23:23:59.028767+0000","last_clean":"2024-06-14T23:23:59.028767+0000","last_became_active":"2024-06-14T23:23:59.028466+0000","last_became_peered":"2024-06-14T23:23:59.028466+0000","last_unstale":"2024-06-14T23:23:59.028767+0000","last_undegraded":"2024-06-14T23:23:59.028767+0000","last_fullsized":"2024-06-14T23:23:59.028767+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:03:52.013585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"77'161","reported_seq":858,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025063+0000","last_change":"2024-06-14T23:23:20.196205+0000","last_active":"2024-06-14T23:23:58.025063+0000","last_peered":"2024-06-14T23:23:58.025063+0000","last_clean":"2024-06-14T23:23:58.025063+0000","last_became_active":"2024-06-14T23:23:20.195770+0000","last_became_peered":"2024-06-14T23:23:20.195770+0000","last_unstale":"2024-06-14T23:23:58.025063+0000","last_undegraded":"2024-06-14T23:23:58.025063+0000","last_fullsized":"2024-06-14T23:23:58.025063+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:53.304763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:53.304763+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:15:24.335823+0000","scrub_duration":0.0029370009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":98,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024100+0000","last_change":"2024-06-14T23:23:17.167178+0000","last_active":"2024-06-14T23:23:58.024100+0000","last_peered":"2024-06-14T23:23:58.024100+0000","last_clean":"2024-06-14T23:23:58.024100+0000","last_became_active":"2024-06-14T23:23:17.166697+0000","last_became_peered":"2024-06-14T23:23:17.166697+0000","last_unstale":"2024-06-14T23:23:58.024100+0000","last_undegraded":"2024-06-14T23:23:58.024100+0000","last_fullsized":"2024-06-14T23:23:58.024100+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:17:26.764829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":381,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026748+0000","last_change":"2024-06-14T23:23:28.041548+0000","last_active":"2024-06-14T23:23:58.026748+0000","last_peered":"2024-06-14T23:23:58.026748+0000","last_clean":"2024-06-14T23:23:58.026748+0000","last_became_active":"2024-06-14T23:23:28.040596+0000","last_became_peered":"2024-06-14T23:23:28.040596+0000","last_unstale":"2024-06-14T23:23:58.026748+0000","last_undegraded":"2024-06-14T23:23:58.026748+0000","last_fullsized":"2024-06-14T23:23:58.026748+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T00:13:28.178465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":405,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.025146+0000","last_change":"2024-06-14T23:23:59.025146+0000","last_active":"2024-06-14T23:23:59.025146+0000","last_peered":"2024-06-14T23:23:59.025146+0000","last_clean":"2024-06-14T23:23:59.025146+0000","last_became_active":"2024-06-14T23:23:59.024719+0000","last_became_peered":"2024-06-14T23:23:59.024719+0000","last_unstale":"2024-06-14T23:23:59.025146+0000","last_undegraded":"2024-06-14T23:23:59.025146+0000","last_fullsized":"2024-06-14T23:23:59.025146+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:27:18.741182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"77'161","reported_seq":858,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.023421+0000","last_change":"2024-06-14T23:23:59.023421+0000","last_active":"2024-06-14T23:23:59.023421+0000","last_peered":"2024-06-14T23:23:59.023421+0000","last_clean":"2024-06-14T23:23:59.023421+0000","last_became_active":"2024-06-14T23:23:59.023052+0000","last_became_peered":"2024-06-14T23:23:59.023052+0000","last_unstale":"2024-06-14T23:23:59.023421+0000","last_undegraded":"2024-06-14T23:23:59.023421+0000","last_fullsized":"2024-06-14T23:23:59.023421+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.522124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.522124+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:18:05.624868+0000","scrub_duration":0.0041198459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":62,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017699+0000","last_change":"2024-06-14T23:22:52.005158+0000","last_active":"2024-06-14T23:23:59.017699+0000","last_peered":"2024-06-14T23:23:59.017699+0000","last_clean":"2024-06-14T23:23:59.017699+0000","last_became_active":"2024-06-14T23:22:52.004707+0000","last_became_peered":"2024-06-14T23:22:52.004707+0000","last_unstale":"2024-06-14T23:23:59.017699+0000","last_undegraded":"2024-06-14T23:23:59.017699+0000","last_fullsized":"2024-06-14T23:23:59.017699+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:12:50.379251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":89,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023246+0000","last_change":"2024-06-14T23:23:13.141787+0000","last_active":"2024-06-14T23:23:58.023246+0000","last_peered":"2024-06-14T23:23:58.023246+0000","last_clean":"2024-06-14T23:23:58.023246+0000","last_became_active":"2024-06-14T23:23:13.141524+0000","last_became_peered":"2024-06-14T23:23:13.141524+0000","last_unstale":"2024-06-14T23:23:58.023246+0000","last_undegraded":"2024-06-14T23:23:58.023246+0000","last_fullsized":"2024-06-14T23:23:58.023246+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:02:49.929637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":114,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.018630+0000","last_change":"2024-06-14T23:23:17.165494+0000","last_active":"2024-06-14T23:23:59.018630+0000","last_peered":"2024-06-14T23:23:59.018630+0000","last_clean":"2024-06-14T23:23:59.018630+0000","last_became_active":"2024-06-14T23:23:17.165132+0000","last_became_peered":"2024-06-14T23:23:17.165132+0000","last_unstale":"2024-06-14T23:23:59.018630+0000","last_undegraded":"2024-06-14T23:23:59.018630+0000","last_fullsized":"2024-06-14T23:23:59.018630+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:02:54.197396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"77'161","reported_seq":652,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.016326+0000","last_change":"2024-06-14T23:23:36.143094+0000","last_active":"2024-06-14T23:23:58.016326+0000","last_peered":"2024-06-14T23:23:58.016326+0000","last_clean":"2024-06-14T23:23:58.016326+0000","last_became_active":"2024-06-14T23:23:36.142685+0000","last_became_peered":"2024-06-14T23:23:36.142685+0000","last_unstale":"2024-06-14T23:23:58.016326+0000","last_undegraded":"2024-06-14T23:23:58.016326+0000","last_fullsized":"2024-06-14T23:23:58.016326+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:08.655367+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:23:08.655367+0000","last_clean_scrub_stamp":"2024-06-14T23:23:08.655367+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:48:48.456845+0000","scrub_duration":0.0087017850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":392,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021647+0000","last_change":"2024-06-14T23:23:46.452498+0000","last_active":"2024-06-14T23:23:58.021647+0000","last_peered":"2024-06-14T23:23:58.021647+0000","last_clean":"2024-06-14T23:23:58.021647+0000","last_became_active":"2024-06-14T23:23:46.452203+0000","last_became_peered":"2024-06-14T23:23:46.452203+0000","last_unstale":"2024-06-14T23:23:58.021647+0000","last_undegraded":"2024-06-14T23:23:58.021647+0000","last_fullsized":"2024-06-14T23:23:58.021647+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:51:38.638988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":305,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.020312+0000","last_change":"2024-06-14T23:23:17.170011+0000","last_active":"2024-06-14T23:23:59.020312+0000","last_peered":"2024-06-14T23:23:59.020312+0000","last_clean":"2024-06-14T23:23:59.020312+0000","last_became_active":"2024-06-14T23:23:17.169676+0000","last_became_peered":"2024-06-14T23:23:17.169676+0000","last_unstale":"2024-06-14T23:23:59.020312+0000","last_undegraded":"2024-06-14T23:23:59.020312+0000","last_fullsized":"2024-06-14T23:23:59.020312+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:59:15.492853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":374,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.017480+0000","last_change":"2024-06-14T23:23:36.146686+0000","last_active":"2024-06-14T23:23:58.017480+0000","last_peered":"2024-06-14T23:23:58.017480+0000","last_clean":"2024-06-14T23:23:58.017480+0000","last_became_active":"2024-06-14T23:23:36.145912+0000","last_became_peered":"2024-06-14T23:23:36.145912+0000","last_unstale":"2024-06-14T23:23:58.017480+0000","last_undegraded":"2024-06-14T23:23:58.017480+0000","last_fullsized":"2024-06-14T23:23:58.017480+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:24:49.288111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"77'161","reported_seq":617,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017324+0000","last_change":"2024-06-14T23:22:26.522180+0000","last_active":"2024-06-14T23:23:59.017324+0000","last_peered":"2024-06-14T23:23:59.017324+0000","last_clean":"2024-06-14T23:23:59.017324+0000","last_became_active":"2024-06-14T23:22:26.521598+0000","last_became_peered":"2024-06-14T23:22:26.521598+0000","last_unstale":"2024-06-14T23:23:59.017324+0000","last_undegraded":"2024-06-14T23:23:59.017324+0000","last_fullsized":"2024-06-14T23:23:59.017324+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:17:53.735114+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:17:53.735114+0000","last_clean_scrub_stamp":"2024-06-14T23:17:53.735114+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:12:25.561031+0000","scrub_duration":0.0046746460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":437,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.023885+0000","last_change":"2024-06-14T23:07:58.029493+0000","last_active":"2024-06-14T23:23:58.023885+0000","last_peered":"2024-06-14T23:23:58.023885+0000","last_clean":"2024-06-14T23:23:58.023885+0000","last_became_active":"2024-06-14T23:07:58.029225+0000","last_became_peered":"2024-06-14T23:07:58.029225+0000","last_unstale":"2024-06-14T23:23:58.023885+0000","last_undegraded":"2024-06-14T23:23:58.023885+0000","last_fullsized":"2024-06-14T23:23:58.023885+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:56:22.644035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":332,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018987+0000","last_change":"2024-06-14T23:23:21.191266+0000","last_active":"2024-06-14T23:23:58.018987+0000","last_peered":"2024-06-14T23:23:58.018987+0000","last_clean":"2024-06-14T23:23:58.018987+0000","last_became_active":"2024-06-14T23:23:21.191075+0000","last_became_peered":"2024-06-14T23:23:21.191075+0000","last_unstale":"2024-06-14T23:23:58.018987+0000","last_undegraded":"2024-06-14T23:23:58.018987+0000","last_fullsized":"2024-06-14T23:23:58.018987+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:56:12.169553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":446,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.024349+0000","last_change":"2024-06-14T23:23:59.024349+0000","last_active":"2024-06-14T23:23:59.024349+0000","last_peered":"2024-06-14T23:23:59.024349+0000","last_clean":"2024-06-14T23:23:59.024349+0000","last_became_active":"2024-06-14T23:23:59.023980+0000","last_became_peered":"2024-06-14T23:23:59.023980+0000","last_unstale":"2024-06-14T23:23:59.024349+0000","last_undegraded":"2024-06-14T23:23:59.024349+0000","last_fullsized":"2024-06-14T23:23:59.024349+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:33:31.873832+0000","last_clean_scrub_stamp":"2024-06-14T22:33:31.873832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:52:43.929377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'161","reported_seq":759,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.017578+0000","last_change":"2024-06-14T23:23:36.144792+0000","last_active":"2024-06-14T23:23:58.017578+0000","last_peered":"2024-06-14T23:23:58.017578+0000","last_clean":"2024-06-14T23:23:58.017578+0000","last_became_active":"2024-06-14T23:23:36.144221+0000","last_became_peered":"2024-06-14T23:23:36.144221+0000","last_unstale":"2024-06-14T23:23:58.017578+0000","last_undegraded":"2024-06-14T23:23:58.017578+0000","last_fullsized":"2024-06-14T23:23:58.017578+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:06.661195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:06.661195+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T10:45:27.980586+0000","scrub_duration":0.002049375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":51,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017610+0000","last_change":"2024-06-14T23:22:23.532985+0000","last_active":"2024-06-14T23:23:59.017610+0000","last_peered":"2024-06-14T23:23:59.017610+0000","last_clean":"2024-06-14T23:23:59.017610+0000","last_became_active":"2024-06-14T23:22:23.532685+0000","last_became_peered":"2024-06-14T23:22:23.532685+0000","last_unstale":"2024-06-14T23:23:59.017610+0000","last_undegraded":"2024-06-14T23:23:59.017610+0000","last_fullsized":"2024-06-14T23:23:59.017610+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:52:43.046719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019612+0000","last_change":"2024-06-14T23:23:17.170036+0000","last_active":"2024-06-14T23:23:58.019612+0000","last_peered":"2024-06-14T23:23:58.019612+0000","last_clean":"2024-06-14T23:23:58.019612+0000","last_became_active":"2024-06-14T23:23:17.169744+0000","last_became_peered":"2024-06-14T23:23:17.169744+0000","last_unstale":"2024-06-14T23:23:58.019612+0000","last_undegraded":"2024-06-14T23:23:58.019612+0000","last_fullsized":"2024-06-14T23:23:58.019612+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:37:18.976998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'161","reported_seq":794,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.022869+0000","last_change":"2024-06-14T23:23:59.022869+0000","last_active":"2024-06-14T23:23:59.022869+0000","last_peered":"2024-06-14T23:23:59.022869+0000","last_clean":"2024-06-14T23:23:59.022869+0000","last_became_active":"2024-06-14T23:23:59.022524+0000","last_became_peered":"2024-06-14T23:23:59.022524+0000","last_unstale":"2024-06-14T23:23:59.022869+0000","last_undegraded":"2024-06-14T23:23:59.022869+0000","last_fullsized":"2024-06-14T23:23:59.022869+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:51.104492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:51.104492+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:32:55.736048+0000","scrub_duration":0.0037175730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":537,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024903+0000","last_change":"2024-06-14T23:23:28.041159+0000","last_active":"2024-06-14T23:23:58.024903+0000","last_peered":"2024-06-14T23:23:58.024903+0000","last_clean":"2024-06-14T23:23:58.024903+0000","last_became_active":"2024-06-14T23:23:28.039520+0000","last_became_peered":"2024-06-14T23:23:28.039520+0000","last_unstale":"2024-06-14T23:23:58.024903+0000","last_undegraded":"2024-06-14T23:23:58.024903+0000","last_fullsized":"2024-06-14T23:23:58.024903+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:15:12.820828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":381,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025679+0000","last_change":"2024-06-14T23:23:28.042229+0000","last_active":"2024-06-14T23:23:58.025679+0000","last_peered":"2024-06-14T23:23:58.025679+0000","last_clean":"2024-06-14T23:23:58.025679+0000","last_became_active":"2024-06-14T23:23:28.041807+0000","last_became_peered":"2024-06-14T23:23:28.041807+0000","last_unstale":"2024-06-14T23:23:58.025679+0000","last_undegraded":"2024-06-14T23:23:58.025679+0000","last_fullsized":"2024-06-14T23:23:58.025679+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:21:33.795242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'161","reported_seq":824,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017047+0000","last_change":"2024-06-14T23:23:13.140635+0000","last_active":"2024-06-14T23:23:59.017047+0000","last_peered":"2024-06-14T23:23:59.017047+0000","last_clean":"2024-06-14T23:23:59.017047+0000","last_became_active":"2024-06-14T23:23:13.140207+0000","last_became_peered":"2024-06-14T23:23:13.140207+0000","last_unstale":"2024-06-14T23:23:59.017047+0000","last_undegraded":"2024-06-14T23:23:59.017047+0000","last_fullsized":"2024-06-14T23:23:59.017047+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:57.265045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:57.265045+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:03:18.870266+0000","scrub_duration":0.0044480530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":400,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.027091+0000","last_change":"2024-06-14T23:23:28.037425+0000","last_active":"2024-06-14T23:23:58.027091+0000","last_peered":"2024-06-14T23:23:58.027091+0000","last_clean":"2024-06-14T23:23:58.027091+0000","last_became_active":"2024-06-14T23:23:28.035090+0000","last_became_peered":"2024-06-14T23:23:28.035090+0000","last_unstale":"2024-06-14T23:23:58.027091+0000","last_undegraded":"2024-06-14T23:23:58.027091+0000","last_fullsized":"2024-06-14T23:23:58.027091+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:09:52.542795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":379,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024532+0000","last_change":"2024-06-14T23:23:44.237001+0000","last_active":"2024-06-14T23:23:58.024532+0000","last_peered":"2024-06-14T23:23:58.024532+0000","last_clean":"2024-06-14T23:23:58.024532+0000","last_became_active":"2024-06-14T23:23:44.236684+0000","last_became_peered":"2024-06-14T23:23:44.236684+0000","last_unstale":"2024-06-14T23:23:58.024532+0000","last_undegraded":"2024-06-14T23:23:58.024532+0000","last_fullsized":"2024-06-14T23:23:58.024532+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:45.200924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'161","reported_seq":785,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.031047+0000","last_change":"2024-06-14T23:23:59.031047+0000","last_active":"2024-06-14T23:23:59.031047+0000","last_peered":"2024-06-14T23:23:59.031047+0000","last_clean":"2024-06-14T23:23:59.031047+0000","last_became_active":"2024-06-14T23:23:59.030716+0000","last_became_peered":"2024-06-14T23:23:59.030716+0000","last_unstale":"2024-06-14T23:23:59.031047+0000","last_undegraded":"2024-06-14T23:23:59.031047+0000","last_fullsized":"2024-06-14T23:23:59.031047+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:53.058787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:22:53.058787+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:48:33.858182+0000","scrub_duration":0.0034892389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":417,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.026133+0000","last_change":"2024-06-14T23:23:28.039267+0000","last_active":"2024-06-14T23:23:58.026133+0000","last_peered":"2024-06-14T23:23:58.026133+0000","last_clean":"2024-06-14T23:23:58.026133+0000","last_became_active":"2024-06-14T23:23:28.038714+0000","last_became_peered":"2024-06-14T23:23:28.038714+0000","last_unstale":"2024-06-14T23:23:58.026133+0000","last_undegraded":"2024-06-14T23:23:58.026133+0000","last_fullsized":"2024-06-14T23:23:58.026133+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:57:23.263278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"68'2","reported_seq":198,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018891+0000","last_change":"2024-06-14T23:23:21.192300+0000","last_active":"2024-06-14T23:23:58.018891+0000","last_peered":"2024-06-14T23:23:58.018891+0000","last_clean":"2024-06-14T23:23:58.018891+0000","last_became_active":"2024-06-14T23:23:21.192008+0000","last_became_peered":"2024-06-14T23:23:21.192008+0000","last_unstale":"2024-06-14T23:23:58.018891+0000","last_undegraded":"2024-06-14T23:23:58.018891+0000","last_fullsized":"2024-06-14T23:23:58.018891+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:26:32.088220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"77'161","reported_seq":577,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.016737+0000","last_change":"2024-06-14T23:23:39.243862+0000","last_active":"2024-06-14T23:23:58.016737+0000","last_peered":"2024-06-14T23:23:58.016737+0000","last_clean":"2024-06-14T23:23:58.016737+0000","last_became_active":"2024-06-14T23:23:36.141570+0000","last_became_peered":"2024-06-14T23:23:36.141570+0000","last_unstale":"2024-06-14T23:23:58.016737+0000","last_undegraded":"2024-06-14T23:23:58.016737+0000","last_fullsized":"2024-06-14T23:23:58.016737+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:39.243791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:39.243791+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:53:31.026410+0000","scrub_duration":0.0032923150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":336,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021796+0000","last_change":"2024-06-14T23:22:23.626905+0000","last_active":"2024-06-14T23:23:58.021796+0000","last_peered":"2024-06-14T23:23:58.021796+0000","last_clean":"2024-06-14T23:23:58.021796+0000","last_became_active":"2024-06-14T23:22:23.602799+0000","last_became_peered":"2024-06-14T23:22:23.602799+0000","last_unstale":"2024-06-14T23:23:58.021796+0000","last_undegraded":"2024-06-14T23:23:58.021796+0000","last_fullsized":"2024-06-14T23:23:58.021796+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:02:31.842714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":381,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.031020+0000","last_change":"2024-06-14T23:23:59.031020+0000","last_active":"2024-06-14T23:23:59.031020+0000","last_peered":"2024-06-14T23:23:59.031020+0000","last_clean":"2024-06-14T23:23:59.031020+0000","last_became_active":"2024-06-14T23:23:59.030650+0000","last_became_peered":"2024-06-14T23:23:59.030650+0000","last_unstale":"2024-06-14T23:23:59.031020+0000","last_undegraded":"2024-06-14T23:23:59.031020+0000","last_fullsized":"2024-06-14T23:23:59.031020+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:58:28.125967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"77'161","reported_seq":621,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024681+0000","last_change":"2024-06-14T23:23:28.031929+0000","last_active":"2024-06-14T23:23:58.024681+0000","last_peered":"2024-06-14T23:23:58.024681+0000","last_clean":"2024-06-14T23:23:58.024681+0000","last_became_active":"2024-06-14T23:23:28.031584+0000","last_became_peered":"2024-06-14T23:23:28.031584+0000","last_unstale":"2024-06-14T23:23:58.024681+0000","last_undegraded":"2024-06-14T23:23:58.024681+0000","last_fullsized":"2024-06-14T23:23:58.024681+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:15:12.943827+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:09:17.382389+0000","last_clean_scrub_stamp":"2024-06-14T23:15:12.943827+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:21:39.406089+0000","scrub_duration":0.0043807899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":490,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.024346+0000","last_change":"2024-06-14T23:22:23.534414+0000","last_active":"2024-06-14T23:23:58.024346+0000","last_peered":"2024-06-14T23:23:58.024346+0000","last_clean":"2024-06-14T23:23:58.024346+0000","last_became_active":"2024-06-14T23:22:23.534050+0000","last_became_peered":"2024-06-14T23:22:23.534050+0000","last_unstale":"2024-06-14T23:23:58.024346+0000","last_undegraded":"2024-06-14T23:23:58.024346+0000","last_fullsized":"2024-06-14T23:23:58.024346+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:06:51.839502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'1","reported_seq":109,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019041+0000","last_change":"2024-06-14T23:23:36.144429+0000","last_active":"2024-06-14T23:23:58.019041+0000","last_peered":"2024-06-14T23:23:58.019041+0000","last_clean":"2024-06-14T23:23:58.019041+0000","last_became_active":"2024-06-14T23:23:36.143977+0000","last_became_peered":"2024-06-14T23:23:36.143977+0000","last_unstale":"2024-06-14T23:23:58.019041+0000","last_undegraded":"2024-06-14T23:23:58.019041+0000","last_fullsized":"2024-06-14T23:23:58.019041+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:15:20.036203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"77'161","reported_seq":818,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025150+0000","last_change":"2024-06-14T23:22:51.010054+0000","last_active":"2024-06-14T23:23:58.025150+0000","last_peered":"2024-06-14T23:23:58.025150+0000","last_clean":"2024-06-14T23:23:58.025150+0000","last_became_active":"2024-06-14T23:22:51.009335+0000","last_became_peered":"2024-06-14T23:22:51.009335+0000","last_unstale":"2024-06-14T23:23:58.025150+0000","last_undegraded":"2024-06-14T23:23:58.025150+0000","last_fullsized":"2024-06-14T23:23:58.025150+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:09:52.265744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:09:52.265744+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:07:08.001347+0000","scrub_duration":0.0013205999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":394,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.025587+0000","last_change":"2024-06-14T23:23:28.035153+0000","last_active":"2024-06-14T23:23:58.025587+0000","last_peered":"2024-06-14T23:23:58.025587+0000","last_clean":"2024-06-14T23:23:58.025587+0000","last_became_active":"2024-06-14T23:23:28.034861+0000","last_became_peered":"2024-06-14T23:23:28.034861+0000","last_unstale":"2024-06-14T23:23:58.025587+0000","last_undegraded":"2024-06-14T23:23:58.025587+0000","last_fullsized":"2024-06-14T23:23:58.025587+0000","mapping_epoch":508,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T04:45:44.007355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'2","reported_seq":513,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019005+0000","last_change":"2024-06-14T23:22:23.551854+0000","last_active":"2024-06-14T23:23:58.019005+0000","last_peered":"2024-06-14T23:23:58.019005+0000","last_clean":"2024-06-14T23:23:58.019005+0000","last_became_active":"2024-06-14T23:22:23.536550+0000","last_became_peered":"2024-06-14T23:22:23.536550+0000","last_unstale":"2024-06-14T23:23:58.019005+0000","last_undegraded":"2024-06-14T23:23:58.019005+0000","last_fullsized":"2024-06-14T23:23:58.019005+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T07:42:52.734655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"267'162","reported_seq":843,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019382+0000","last_change":"2024-06-14T23:23:20.199876+0000","last_active":"2024-06-14T23:23:58.019382+0000","last_peered":"2024-06-14T23:23:58.019382+0000","last_clean":"2024-06-14T23:23:58.019382+0000","last_became_active":"2024-06-14T23:23:20.199523+0000","last_became_peered":"2024-06-14T23:23:20.199523+0000","last_unstale":"2024-06-14T23:23:58.019382+0000","last_undegraded":"2024-06-14T23:23:58.019382+0000","last_fullsized":"2024-06-14T23:23:58.019382+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":4,"last_scrub":"267'162","last_scrub_stamp":"2024-06-14T23:22:24.933402+0000","last_deep_scrub":"267'162","last_deep_scrub_stamp":"2024-06-14T23:07:16.188207+0000","last_clean_scrub_stamp":"2024-06-14T23:22:24.933402+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:35:39.661035+0000","scrub_duration":0.182211177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":394,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.028446+0000","last_change":"2024-06-14T23:23:59.028446+0000","last_active":"2024-06-14T23:23:59.028446+0000","last_peered":"2024-06-14T23:23:59.028446+0000","last_clean":"2024-06-14T23:23:59.028446+0000","last_became_active":"2024-06-14T23:23:59.027914+0000","last_became_peered":"2024-06-14T23:23:59.027914+0000","last_unstale":"2024-06-14T23:23:59.028446+0000","last_undegraded":"2024-06-14T23:23:59.028446+0000","last_fullsized":"2024-06-14T23:23:59.028446+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:32:27.725711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'1","reported_seq":512,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.018518+0000","last_change":"2024-06-14T23:17:22.904229+0000","last_active":"2024-06-14T23:23:58.018518+0000","last_peered":"2024-06-14T23:23:58.018518+0000","last_clean":"2024-06-14T23:23:58.018518+0000","last_became_active":"2024-06-14T23:17:22.896077+0000","last_became_peered":"2024-06-14T23:17:22.896077+0000","last_unstale":"2024-06-14T23:23:58.018518+0000","last_undegraded":"2024-06-14T23:23:58.018518+0000","last_fullsized":"2024-06-14T23:23:58.018518+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:29:05.630800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"77'161","reported_seq":679,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019067+0000","last_change":"2024-06-14T23:23:36.146430+0000","last_active":"2024-06-14T23:23:58.019067+0000","last_peered":"2024-06-14T23:23:58.019067+0000","last_clean":"2024-06-14T23:23:58.019067+0000","last_became_active":"2024-06-14T23:23:36.146108+0000","last_became_peered":"2024-06-14T23:23:36.146108+0000","last_unstale":"2024-06-14T23:23:58.019067+0000","last_undegraded":"2024-06-14T23:23:58.019067+0000","last_fullsized":"2024-06-14T23:23:58.019067+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:15.211071+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:23:15.211071+0000","last_clean_scrub_stamp":"2024-06-14T23:23:15.211071+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:38:09.477585+0000","scrub_duration":0.001336177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"68'8","reported_seq":545,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.022474+0000","last_change":"2024-06-14T23:22:51.008383+0000","last_active":"2024-06-14T23:23:58.022474+0000","last_peered":"2024-06-14T23:23:58.022474+0000","last_clean":"2024-06-14T23:23:58.022474+0000","last_became_active":"2024-06-14T23:22:51.007961+0000","last_became_peered":"2024-06-14T23:22:51.007961+0000","last_unstale":"2024-06-14T23:23:58.022474+0000","last_undegraded":"2024-06-14T23:23:58.022474+0000","last_fullsized":"2024-06-14T23:23:58.022474+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T22:37:32.001034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.019311+0000","last_change":"2024-06-14T23:23:17.164437+0000","last_active":"2024-06-14T23:23:58.019311+0000","last_peered":"2024-06-14T23:23:58.019311+0000","last_clean":"2024-06-14T23:23:58.019311+0000","last_became_active":"2024-06-14T23:23:17.164146+0000","last_became_peered":"2024-06-14T23:23:17.164146+0000","last_unstale":"2024-06-14T23:23:58.019311+0000","last_undegraded":"2024-06-14T23:23:58.019311+0000","last_fullsized":"2024-06-14T23:23:58.019311+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:24:36.188657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"77'161","reported_seq":789,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.024594+0000","last_change":"2024-06-14T23:23:59.024594+0000","last_active":"2024-06-14T23:23:59.024594+0000","last_peered":"2024-06-14T23:23:59.024594+0000","last_clean":"2024-06-14T23:23:59.024594+0000","last_became_active":"2024-06-14T23:23:59.024282+0000","last_became_peered":"2024-06-14T23:23:59.024282+0000","last_unstale":"2024-06-14T23:23:59.024594+0000","last_undegraded":"2024-06-14T23:23:59.024594+0000","last_fullsized":"2024-06-14T23:23:59.024594+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:22:31.102737+0000","last_deep_scrub":"77'161","last_deep_scrub_stamp":"2024-06-14T23:07:16.093566+0000","last_clean_scrub_stamp":"2024-06-14T23:22:31.102737+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T06:46:07.794469+0000","scrub_duration":0.0053580870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":266,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.016936+0000","last_change":"2024-06-14T23:23:36.144650+0000","last_active":"2024-06-14T23:23:58.016936+0000","last_peered":"2024-06-14T23:23:58.016936+0000","last_clean":"2024-06-14T23:23:58.016936+0000","last_became_active":"2024-06-14T23:23:36.144172+0000","last_became_peered":"2024-06-14T23:23:36.144172+0000","last_unstale":"2024-06-14T23:23:58.016936+0000","last_undegraded":"2024-06-14T23:23:58.016936+0000","last_fullsized":"2024-06-14T23:23:58.016936+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T02:38:23.575240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":297,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021857+0000","last_change":"2024-06-14T23:23:17.170253+0000","last_active":"2024-06-14T23:23:58.021857+0000","last_peered":"2024-06-14T23:23:58.021857+0000","last_clean":"2024-06-14T23:23:58.021857+0000","last_became_active":"2024-06-14T23:23:17.169885+0000","last_became_peered":"2024-06-14T23:23:17.169885+0000","last_unstale":"2024-06-14T23:23:58.021857+0000","last_undegraded":"2024-06-14T23:23:58.021857+0000","last_fullsized":"2024-06-14T23:23:58.021857+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-15T23:47:00.101912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'161","reported_seq":784,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.017169+0000","last_change":"2024-06-14T23:23:20.195118+0000","last_active":"2024-06-14T23:23:59.017169+0000","last_peered":"2024-06-14T23:23:59.017169+0000","last_clean":"2024-06-14T23:23:59.017169+0000","last_became_active":"2024-06-14T23:23:20.194666+0000","last_became_peered":"2024-06-14T23:23:20.194666+0000","last_unstale":"2024-06-14T23:23:59.017169+0000","last_undegraded":"2024-06-14T23:23:59.017169+0000","last_fullsized":"2024-06-14T23:23:59.017169+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:10:01.315202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:10:01.315202+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T05:59:52.906430+0000","scrub_duration":0.0022960770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":132,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.020413+0000","last_change":"2024-06-14T23:23:17.169352+0000","last_active":"2024-06-14T23:23:59.020413+0000","last_peered":"2024-06-14T23:23:59.020413+0000","last_clean":"2024-06-14T23:23:59.020413+0000","last_became_active":"2024-06-14T23:23:17.169050+0000","last_became_peered":"2024-06-14T23:23:17.169050+0000","last_unstale":"2024-06-14T23:23:59.020413+0000","last_undegraded":"2024-06-14T23:23:59.020413+0000","last_fullsized":"2024-06-14T23:23:59.020413+0000","mapping_epoch":501,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T03:03:46.429724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":394,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.028731+0000","last_change":"2024-06-14T23:23:59.028731+0000","last_active":"2024-06-14T23:23:59.028731+0000","last_peered":"2024-06-14T23:23:59.028731+0000","last_clean":"2024-06-14T23:23:59.028731+0000","last_became_active":"2024-06-14T23:23:59.028427+0000","last_became_peered":"2024-06-14T23:23:59.028427+0000","last_unstale":"2024-06-14T23:23:59.028731+0000","last_undegraded":"2024-06-14T23:23:59.028731+0000","last_fullsized":"2024-06-14T23:23:59.028731+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T01:54:34.343018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"77'161","reported_seq":753,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.023584+0000","last_change":"2024-06-14T23:23:59.023584+0000","last_active":"2024-06-14T23:23:59.023584+0000","last_peered":"2024-06-14T23:23:59.023584+0000","last_clean":"2024-06-14T23:23:59.023584+0000","last_became_active":"2024-06-14T23:23:59.023280+0000","last_became_peered":"2024-06-14T23:23:59.023280+0000","last_unstale":"2024-06-14T23:23:59.023584+0000","last_undegraded":"2024-06-14T23:23:59.023584+0000","last_fullsized":"2024-06-14T23:23:59.023584+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:23:38.210830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:23:38.210830+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:26:39.146677+0000","scrub_duration":0.001674523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'2","reported_seq":564,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.017405+0000","last_change":"2024-06-14T23:20:33.536774+0000","last_active":"2024-06-14T23:23:58.017405+0000","last_peered":"2024-06-14T23:23:58.017405+0000","last_clean":"2024-06-14T23:23:58.017405+0000","last_became_active":"2024-06-14T23:20:33.515459+0000","last_became_peered":"2024-06-14T23:20:33.515459+0000","last_unstale":"2024-06-14T23:23:58.017405+0000","last_undegraded":"2024-06-14T23:23:58.017405+0000","last_fullsized":"2024-06-14T23:23:58.017405+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:44.787356+0000","last_clean_scrub_stamp":"2024-06-14T22:34:44.787356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:39:39.163824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":228,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-06-14T23:23:59.026748+0000","last_change":"2024-06-14T23:23:21.291083+0000","last_active":"2024-06-14T23:23:59.026748+0000","last_peered":"2024-06-14T23:23:59.026748+0000","last_clean":"2024-06-14T23:23:59.026748+0000","last_became_active":"2024-06-14T23:23:21.290740+0000","last_became_peered":"2024-06-14T23:23:21.290740+0000","last_unstale":"2024-06-14T23:23:59.026748+0000","last_undegraded":"2024-06-14T23:23:59.026748+0000","last_fullsized":"2024-06-14T23:23:59.026748+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:34:45.793132+0000","last_clean_scrub_stamp":"2024-06-14T22:34:45.793132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:56:41.113340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"77'161","reported_seq":959,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-06-14T23:23:58.021486+0000","last_change":"2024-06-14T23:23:20.199638+0000","last_active":"2024-06-14T23:23:58.021486+0000","last_peered":"2024-06-14T23:23:58.021486+0000","last_clean":"2024-06-14T23:23:58.021486+0000","last_became_active":"2024-06-14T23:23:20.199266+0000","last_became_peered":"2024-06-14T23:23:20.199266+0000","last_unstale":"2024-06-14T23:23:58.021486+0000","last_undegraded":"2024-06-14T23:23:58.021486+0000","last_fullsized":"2024-06-14T23:23:58.021486+0000","mapping_epoch":504,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'161","last_scrub_stamp":"2024-06-14T23:07:12.090035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-14T22:31:01.536459+0000","last_clean_scrub_stamp":"2024-06-14T23:07:12.090035+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T09:47:05.889769+0000","scrub_duration":0.0065626970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5542,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1342,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28245,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4993,"ondisk_log_size":4993,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234610,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509832,"kb_used_data":7032,"kb_used_omap":1,"kb_used_meta":1502782,"kb_avail":92235320,"statfs":{"total":95995035648,"available":94448967680,"internally_reserved":0,"allocated":7200768,"data_stored":4426885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1538849112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.67100000000000004,"15min":0.92000000000000004},"min":{"1min":0.23999999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.68899999999999995,"5min":4.899,"15min":12.263999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.88100000000000001,"15min":1.054},"min":{"1min":0.32600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.70499999999999996,"5min":3.8889999999999998,"15min":6.5149999999999997},"last":0.63600000000000001}]},{"osd":1,"last update":"Fri Jun 14 23:24:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.70499999999999996,"15min":0.998},"min":{"1min":0.375,"5min":0.25900000000000001,"15min":0.25600000000000001},"max":{"1min":0.72999999999999998,"5min":3.98,"15min":6.0229999999999997},"last":0.75},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.67900000000000005,"15min":0.999},"min":{"1min":0.30199999999999999,"5min":0.29599999999999999,"15min":0.27700000000000002},"max":{"1min":0.625,"5min":4.8710000000000004,"15min":12.295},"last":0.69199999999999995}]},{"osd":2,"last update":"Fri Jun 14 23:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.71899999999999997,"15min":1.028},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.27000000000000002},"max":{"1min":0.71799999999999997,"5min":4.6079999999999997,"15min":10.539},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.79400000000000004,"15min":1.056},"min":{"1min":0.48099999999999998,"5min":0.34000000000000002,"15min":0.26000000000000001},"max":{"1min":0.69299999999999995,"5min":4.6980000000000004,"15min":10.56},"last":0.65400000000000003}]},{"osd":3,"last update":"Fri Jun 14 23:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.70899999999999996,"15min":1.087},"min":{"1min":0.41999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69599999999999995,"5min":4.5499999999999998,"15min":17.991},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.749,"15min":1.145},"min":{"1min":0.42099999999999999,"5min":0.313,"15min":0.29399999999999998},"max":{"1min":0.71099999999999997,"5min":3.9489999999999998,"15min":17.824999999999999},"last":0.67800000000000005}]},{"osd":4,"last update":"Fri Jun 14 23:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.81699999999999995,"15min":1.204},"min":{"1min":0.30499999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.65200000000000002,"5min":5.6710000000000003,"15min":6.1539999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.83899999999999997,"15min":1.1930000000000001},"min":{"1min":0.33500000000000002,"5min":0.27700000000000002,"15min":0.22700000000000001},"max":{"1min":0.67100000000000004,"5min":4.6699999999999999,"15min":10.499000000000001},"last":0.47499999999999998}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.503},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.73199999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":1.0389999999999999,"15min":1.508},"min":{"1min":0.379,"5min":0.309,"15min":0.309},"max":{"1min":0.70399999999999996,"5min":5.2779999999999996,"15min":12.317},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.96899999999999997,"15min":1.4410000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.75900000000000001,"5min":6.9509999999999996,"15min":10.087999999999999},"last":0.71599999999999997}]}]},{"osd":6,"up_from":272,"seq":1168231104717,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963892,"kb_used_data":7076,"kb_used_omap":7,"kb_used_meta":956792,"kb_avail":92781260,"statfs":{"total":95995035648,"available":95008010240,"internally_reserved":0,"allocated":7245824,"data_stored":4491408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7750,"internal_metadata":979755450},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":1.024,"15min":1.0429999999999999},"min":{"1min":0.29799999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.66900000000000004,"5min":5.923,"15min":6.109},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.071,"15min":1.1399999999999999},"min":{"1min":0.42799999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.69099999999999995,"5min":6.0620000000000003,"15min":14.148999999999999},"last":0.54100000000000004}]},{"osd":1,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.81899999999999995,"15min":0.98499999999999999},"min":{"1min":0.34000000000000002,"5min":0.26500000000000001,"15min":0.25800000000000001},"max":{"1min":0.59899999999999998,"5min":7.0970000000000004,"15min":7.0970000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.94699999999999995,"15min":1.0900000000000001},"min":{"1min":0.442,"5min":0.26800000000000002,"15min":0.248},"max":{"1min":0.68999999999999995,"5min":5.9980000000000002,"15min":14.086},"last":0.48199999999999998}]},{"osd":2,"last update":"Fri Jun 14 23:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.83999999999999997,"15min":1.153},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.249},"max":{"1min":0.749,"5min":7.8019999999999996,"15min":29.940999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.96399999999999997,"15min":1.242},"min":{"1min":0.35499999999999998,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":7.8609999999999998,"15min":30.135999999999999},"last":0.63800000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":1.0509999999999999,"15min":1.1259999999999999},"min":{"1min":0.40000000000000002,"5min":0.29899999999999999,"15min":0.26200000000000001},"max":{"1min":0.67600000000000005,"5min":6.0339999999999998,"15min":6.0839999999999996},"last":0.629},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.98299999999999998,"15min":1.2290000000000001},"min":{"1min":0.40999999999999998,"5min":0.312,"15min":0.28499999999999998},"max":{"1min":0.80200000000000005,"5min":7.8319999999999999,"15min":8.4740000000000002},"last":0.65600000000000003}]},{"osd":4,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.90600000000000003,"15min":1.242},"min":{"1min":0.28000000000000003,"5min":0.24399999999999999,"15min":0.21299999999999999},"max":{"1min":0.66200000000000003,"5min":6.4089999999999998,"15min":11.010999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":1.1399999999999999,"15min":1.347},"min":{"1min":0.35399999999999998,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":0.73199999999999998,"5min":8.7550000000000008,"15min":14.199999999999999},"last":0.58999999999999997}]},{"osd":5,"last update":"Fri Jun 14 23:24:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.32300000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70399999999999996,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":1.002,"15min":1.458},"min":{"1min":0.371,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.84299999999999997,"5min":7.7720000000000002,"15min":10.016999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.90500000000000003,"15min":1.4470000000000001},"min":{"1min":0.38,"5min":0.29199999999999998,"15min":0.26000000000000001},"max":{"1min":0.872,"5min":6.9240000000000004,"15min":10.906000000000001},"last":0.60299999999999998}]}]},{"osd":5,"up_from":492,"seq":2113123909658,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384688,"kb_used_data":7008,"kb_used_omap":7,"kb_used_meta":377656,"kb_avail":93360464,"statfs":{"total":95995035648,"available":95601115136,"internally_reserved":0,"allocated":7176192,"data_stored":4422230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7909,"internal_metadata":386720027},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.496},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.626}]},{"osd":1,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51100000000000001}]},{"osd":3,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.52400000000000002}]},{"osd":4,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.44}]},{"osd":6,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.60299999999999998}]},{"osd":7,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.65900000000000003}]}]},{"osd":4,"up_from":30,"seq":128849019522,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1494908,"kb_used_data":6764,"kb_used_omap":1,"kb_used_meta":1488126,"kb_avail":92250244,"statfs":{"total":95995035648,"available":94464249856,"internally_reserved":0,"allocated":6926336,"data_stored":4163862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1523841368},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":1.242,"15min":1.022},"min":{"1min":0.30199999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.77300000000000002,"5min":21.262,"15min":21.262},"last":0.497},{"interface":"front","average":{"1min":0.52800000000000002,"5min":1.248,"15min":1.0700000000000001},"min":{"1min":0.32100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.72299999999999998,"5min":21.491,"15min":21.491},"last":0.53100000000000003}]},{"osd":1,"last update":"Fri Jun 14 23:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.91300000000000003,"15min":0.92800000000000005},"min":{"1min":0.441,"5min":0.28100000000000003,"15min":0.26400000000000001},"max":{"1min":0.75800000000000001,"5min":6.0179999999999998,"15min":6.6790000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.98299999999999998,"15min":0.96799999999999997},"min":{"1min":0.40500000000000003,"5min":0.27500000000000002,"15min":0.249},"max":{"1min":0.78200000000000003,"5min":6.2169999999999996,"15min":6.9770000000000003},"last":0.59299999999999997}]},{"osd":2,"last update":"Fri Jun 14 23:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.93200000000000005,"15min":0.97599999999999998},"min":{"1min":0.36599999999999999,"5min":0.27200000000000002,"15min":0.25700000000000001},"max":{"1min":0.79300000000000004,"5min":6.0469999999999997,"15min":6.7110000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.93999999999999995,"15min":1.018},"min":{"1min":0.47999999999999998,"5min":0.34499999999999997,"15min":0.27200000000000002},"max":{"1min":0.71999999999999997,"5min":6.0750000000000002,"15min":6.6550000000000002},"last":0.51900000000000002}]},{"osd":3,"last update":"Fri Jun 14 23:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":1.0880000000000001,"15min":1.1739999999999999},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.27500000000000002},"max":{"1min":0.80100000000000005,"5min":7.2009999999999996,"15min":44.389000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":1.125,"15min":1.2250000000000001},"min":{"1min":0.45900000000000002,"5min":0.29799999999999999,"15min":0.29599999999999999},"max":{"1min":0.76900000000000002,"5min":6.1950000000000003,"15min":46.101999999999997},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.378},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.46500000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:24:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":1.49,"15min":1.849},"min":{"1min":0.38200000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.79600000000000004,"5min":8.1669999999999998,"15min":122.88500000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.5,"15min":1.851},"min":{"1min":0.40899999999999997,"5min":0.28100000000000003,"15min":0.25700000000000001},"max":{"1min":0.76000000000000001,"5min":14.263999999999999,"15min":122.21299999999999},"last":0.40400000000000003}]},{"osd":7,"last update":"Fri Jun 14 23:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.0229999999999999,"15min":1.4319999999999999},"min":{"1min":0.38300000000000001,"5min":0.27000000000000002,"15min":0.255},"max":{"1min":0.78200000000000003,"5min":6.3789999999999996,"15min":11.430999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.625,"5min":1.135,"15min":1.427},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76500000000000001,"5min":7.1420000000000003,"15min":8.5340000000000007},"last":0.48399999999999999}]}]},{"osd":3,"up_from":138,"seq":592705487221,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1278124,"kb_used_data":7068,"kb_used_omap":4,"kb_used_meta":1271035,"kb_avail":92467028,"statfs":{"total":95995035648,"available":94686236672,"internally_reserved":0,"allocated":7237632,"data_stored":4486995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4642,"internal_metadata":1301540318},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.46500000000000002,"15min":0.51400000000000001},"min":{"1min":0.189,"5min":0.154,"15min":0.154},"max":{"1min":0.48899999999999999,"5min":2.3919999999999999,"15min":25.190000000000001},"last":0.251},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.70899999999999996,"15min":0.71399999999999997},"min":{"1min":0.20300000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.65300000000000002,"5min":9.6940000000000008,"15min":25.652000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Fri Jun 14 23:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.622,"15min":0.746},"min":{"1min":0.28100000000000003,"5min":0.23000000000000001,"15min":0.20200000000000001},"max":{"1min":0.60099999999999998,"5min":5.5010000000000003,"15min":25.754000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.65800000000000003,"15min":0.79500000000000004},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.249},"max":{"1min":0.57699999999999996,"5min":3.956,"15min":27.992999999999999},"last":0.34100000000000003}]},{"osd":2,"last update":"Fri Jun 14 23:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.504,"15min":0.61899999999999999},"min":{"1min":0.26600000000000001,"5min":0.24199999999999999,"15min":0.184},"max":{"1min":0.57599999999999996,"5min":2.7650000000000001,"15min":5.9189999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.54300000000000004,"15min":0.59799999999999998},"min":{"1min":0.29599999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.98899999999999999,"5min":2.6269999999999998,"15min":3.5550000000000002},"last":0.32800000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.89100000000000001,"15min":1.1970000000000001},"min":{"1min":0.29899999999999999,"5min":0.28999999999999998,"15min":0.27800000000000002},"max":{"1min":0.84599999999999997,"5min":6.5110000000000001,"15min":25.094000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.91200000000000003,"15min":1.1679999999999999},"min":{"1min":0.41699999999999998,"5min":0.29999999999999999,"15min":0.27600000000000002},"max":{"1min":0.79900000000000004,"5min":8.2349999999999994,"15min":25.5},"last":0.56200000000000006}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Jun 14 23:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.153,"15min":1.448},"min":{"1min":0.38200000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82499999999999996,"5min":5.907,"15min":25.869},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.0760000000000001,"15min":1.321},"min":{"1min":0.378,"5min":0.32000000000000001,"15min":0.30499999999999999},"max":{"1min":0.80100000000000005,"5min":9.9309999999999992,"15min":25.143000000000001},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.88300000000000001,"15min":1.2450000000000001},"min":{"1min":0.42599999999999999,"5min":0.33600000000000002,"15min":0.28799999999999998},"max":{"1min":0.70899999999999996,"5min":6.085,"15min":44.887999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.877,"15min":1.319},"min":{"1min":0.435,"5min":0.37,"15min":0.36099999999999999},"max":{"1min":0.83799999999999997,"5min":5.0679999999999996,"15min":45.835000000000001},"last":0.66400000000000003}]}]},{"osd":2,"up_from":322,"seq":1382979469476,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571056,"kb_used_data":7008,"kb_used_omap":10,"kb_used_meta":564021,"kb_avail":93174096,"statfs":{"total":95995035648,"available":95410274304,"internally_reserved":0,"allocated":7176192,"data_stored":4422217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10714,"internal_metadata":577558054},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":1.0609999999999999,"15min":0.63100000000000001},"min":{"1min":0.215,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.98399999999999999,"5min":40.081000000000003,"15min":40.081000000000003},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":1.123,"15min":0.68899999999999995},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.21299999999999999},"max":{"1min":0.56999999999999995,"5min":40.106000000000002,"15min":40.106000000000002},"last":0.36399999999999999}]},{"osd":1,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.44700000000000001,"15min":0.44500000000000001},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.82099999999999995,"5min":2.7770000000000001,"15min":4.04},"last":0.309},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.46899999999999997,"15min":0.50800000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.192},"max":{"1min":0.48999999999999999,"5min":3.2370000000000001,"15min":6.9889999999999999},"last":0.28199999999999997}]},{"osd":3,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.54100000000000004,"15min":0.57199999999999995},"min":{"1min":0.23300000000000001,"5min":0.19,"15min":0.19},"max":{"1min":1.2709999999999999,"5min":3.319,"15min":3.7519999999999998},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.21199999999999999},"max":{"1min":0.90300000000000002,"5min":2.8460000000000001,"15min":5.5890000000000004},"last":0.34699999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.81799999999999995,"15min":1.008},"min":{"1min":0.47899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.82399999999999995,"5min":4.1820000000000004,"15min":6.9820000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.74099999999999999,"15min":0.90400000000000003},"min":{"1min":0.39800000000000002,"5min":0.29199999999999998,"15min":0.26400000000000001},"max":{"1min":1.3049999999999999,"5min":3.2909999999999999,"15min":5.0759999999999996},"last":0.48599999999999999}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.47699999999999998}]},{"osd":6,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.90600000000000003,"15min":1.0369999999999999},"min":{"1min":0.41899999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.998,"5min":7.0960000000000001,"15min":7.0960000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.875,"15min":1.02},"min":{"1min":0.43099999999999999,"5min":0.32000000000000001,"15min":0.31900000000000001},"max":{"1min":1.3320000000000001,"5min":4.1849999999999996,"15min":4.6950000000000003},"last":0.46800000000000003}]},{"osd":7,"last update":"Fri Jun 14 23:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.79100000000000004,"15min":0.92900000000000005},"min":{"1min":0.45100000000000001,"5min":0.377,"15min":0.32500000000000001},"max":{"1min":0.88900000000000001,"5min":4.4960000000000004,"15min":5},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.752,"15min":1.258},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.86299999999999999,"5min":3.8809999999999998,"15min":49.552999999999997},"last":0.501}]}]},{"osd":1,"up_from":116,"seq":498216206765,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":791268,"kb_used_data":6744,"kb_used_omap":5,"kb_used_meta":784506,"kb_avail":92953884,"statfs":{"total":95995035648,"available":95184777216,"internally_reserved":0,"allocated":6905856,"data_stored":4159225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5864,"internal_metadata":803334424},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 14 23:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.83699999999999997,"15min":0.57799999999999996},"min":{"1min":0.21099999999999999,"5min":0.184,"15min":0.16200000000000001},"max":{"1min":0.88500000000000001,"5min":24.329999999999998,"15min":24.329999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.81999999999999995,"15min":0.56499999999999995},"min":{"1min":0.218,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":0.78600000000000003,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Fri Jun 14 23:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.24199999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.505,"5min":4.1120000000000001,"15min":4.1120000000000001},"last":0.505},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.48999999999999999,"15min":0.47299999999999998},"min":{"1min":0.27400000000000002,"5min":0.20799999999999999,"15min":0.188},"max":{"1min":0.48499999999999999,"5min":2.524,"15min":4.2240000000000002},"last":0.435}]},{"osd":3,"last update":"Fri Jun 14 23:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.61499999999999999,"15min":0.59599999999999997},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.20499999999999999},"max":{"1min":0.58799999999999997,"5min":4.8380000000000001,"15min":7.0739999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.59099999999999997,"15min":0.61399999999999999},"min":{"1min":0.26800000000000002,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":0.97799999999999998,"5min":3.0179999999999998,"15min":10.541},"last":0.47499999999999998}]},{"osd":4,"last update":"Fri Jun 14 23:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.98599999999999999,"15min":1.155},"min":{"1min":0.39100000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73299999999999998,"5min":4.806,"15min":43.959000000000003},"last":0.625},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.90800000000000003,"15min":1.0780000000000001},"min":{"1min":0.39500000000000002,"5min":0.28899999999999998,"15min":0.28000000000000003},"max":{"1min":0.93200000000000005,"5min":4.7130000000000001,"15min":44.006999999999998},"last":0.70699999999999996}]},{"osd":5,"last update":"Fri Jun 14 23:23:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri Jun 14 23:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.038,"15min":1.262},"min":{"1min":0.40500000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77200000000000002,"5min":4.7720000000000002,"15min":44.241},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.96899999999999997,"15min":1.337},"min":{"1min":0.47099999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88500000000000001,"5min":4.431,"15min":45.627000000000002},"last":0.77700000000000002}]},{"osd":7,"last update":"Fri Jun 14 23:24:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.85999999999999999,"15min":1.169},"min":{"1min":0.45500000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.94399999999999995,"5min":4.2389999999999999,"15min":44.667000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.86599999999999999,"15min":1.1579999999999999},"min":{"1min":0.47099999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92600000000000005,"5min":4.258,"15min":44.679000000000002},"last":0.83799999999999997}]}]},{"osd":0,"up_from":91,"seq":390842024419,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1252584,"kb_used_data":7064,"kb_used_omap":5,"kb_used_meta":1245498,"kb_avail":92492568,"statfs":{"total":95995035648,"available":94712389632,"internally_reserved":0,"allocated":7233536,"data_stored":4484904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6054,"internal_metadata":1275390042},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 14 23:24:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.64000000000000001,"15min":0.47799999999999998},"min":{"1min":0.20200000000000001,"5min":0.191,"15min":0.17899999999999999},"max":{"1min":0.73599999999999999,"5min":11.733000000000001,"15min":11.733000000000001},"last":0.375},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.61799999999999999,"15min":0.48699999999999999},"min":{"1min":0.21099999999999999,"5min":0.20899999999999999,"15min":0.19500000000000001},"max":{"1min":0.79300000000000004,"5min":4.3769999999999998,"15min":4.3769999999999998},"last":0.216}]},{"osd":2,"last update":"Fri Jun 14 23:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.55800000000000005,"15min":0.53800000000000003},"min":{"1min":0.22900000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.58799999999999997,"5min":4.3109999999999999,"15min":4.3109999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.53800000000000003,"15min":0.52200000000000002},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.20100000000000001},"max":{"1min":0.70599999999999996,"5min":2.7469999999999999,"15min":3.2509999999999999},"last":0.28799999999999998}]},{"osd":3,"last update":"Fri Jun 14 23:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.73199999999999998,"15min":0.63500000000000001},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.61499999999999999,"5min":10.65,"15min":10.65},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.55500000000000005,"15min":0.61299999999999999},"min":{"1min":0.29499999999999998,"5min":0.252,"15min":0.217},"max":{"1min":0.82499999999999996,"5min":2.5,"15min":7.4279999999999999},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Jun 14 23:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.93000000000000005,"15min":1.0589999999999999},"min":{"1min":0.39800000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0649999999999999,"5min":7.4009999999999998,"15min":24.170999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.996,"15min":1.0760000000000001},"min":{"1min":0.36199999999999999,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":0.877,"5min":4.5540000000000003,"15min":24.204000000000001},"last":0.51400000000000001}]},{"osd":5,"last update":"Fri Jun 14 23:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.53400000000000003}]},{"osd":6,"last update":"Fri Jun 14 23:24:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.538,"15min":1.3460000000000001},"min":{"1min":0.45200000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.76500000000000001,"5min":36.478000000000002,"15min":36.478000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.516,"15min":1.3600000000000001},"min":{"1min":0.46100000000000002,"5min":0.39300000000000002,"15min":0.33700000000000002},"max":{"1min":1.024,"5min":36.494,"15min":36.494},"last":0.498}]},{"osd":7,"last update":"Fri Jun 14 23:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.90900000000000003,"15min":1.1339999999999999},"min":{"1min":0.43099999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71199999999999997,"5min":4.3920000000000003,"15min":24.131},"last":0.433},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.87,"15min":1.1519999999999999},"min":{"1min":0.46800000000000003,"5min":0.38500000000000001,"15min":0.313},"max":{"1min":1.0229999999999999,"5min":3.6200000000000001,"15min":24.257000000000001},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4350,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4160,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4546,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2938,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6205,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-14T23:24:55.897 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-14T23:24:55.897 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_906f73e382c3916fe5507b8227cc56e63a20794f/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_906f73e382c3916fe5507b8227cc56e63a20794f/qa/tasks/workunit.py", line 345, in _run_tests remote.run(logger=log.getChild(role), File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://github.com/cbodley/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0' 2024-06-14T23:24:56.072 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=740e16f69a91470992e9bffea96fe812 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_906f73e382c3916fe5507b8227cc56e63a20794f/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_cbodley_ceph_906f73e382c3916fe5507b8227cc56e63a20794f/qa/tasks/workunit.py", line 345, in _run_tests remote.run(logger=log.getChild(role), File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://github.com/cbodley/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0' 2024-06-14T23:24:56.075 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-14T23:24:56.085 INFO:tasks.cephadm:Teardown begin 2024-06-14T23:24:56.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-14T23:24:56.135 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-mon[97913]: from='client.47663 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:24:56.135 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-mon[102754]: from='client.47663 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:24:56.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-14T23:24:56.171 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-14T23:24:56.171 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-14T23:24:56.193 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-14T23:24:56.228 INFO:tasks.cephadm:Stopping all daemons... 2024-06-14T23:24:56.228 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-06-14T23:24:56.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a 2024-06-14T23:24:56.496 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:56 smithi137 ceph-mon[90634]: from='client.47663 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-14T23:24:56.557 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:56 smithi086 systemd[1]: Stopping Ceph mon.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:24:56.837 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:56] ENGINE Bus STOPPING 2024-06-14T23:24:56.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:24:56.554+0000 7fe01b9e6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:24:56.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a[97869]: 2024-06-14T23:24:56.554+0000 7fe01b9e6640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-06-14T23:24:56.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:56 smithi086 podman[371412]: 2024-06-14 23:24:56.59556835 +0000 UTC m=+0.167340090 container died 0fe78616d3a277d695dd048c54f444bcc0f0ca52558a870e0dccbb4a29247d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-14T23:24:57.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:57 smithi086 podman[371412]: 2024-06-14 23:24:57.018522534 +0000 UTC m=+0.590294273 container cleanup 0fe78616d3a277d695dd048c54f444bcc0f0ca52558a870e0dccbb4a29247d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-14T23:24:57.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:57 smithi086 bash[371412]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a 2024-06-14T23:24:57.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T23:24:57.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:56] ENGINE Bus STOPPED 2024-06-14T23:24:57.337 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:56] ENGINE Bus STARTING 2024-06-14T23:24:57.338 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:56] ENGINE Serving on http://:::9283 2024-06-14T23:24:57.338 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:56 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:56] ENGINE Bus STARTED 2024-06-14T23:24:57.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Jun 14 23:24:57 smithi086 podman[371429]: 2024-06-14 23:24:57.361659535 +0000 UTC m=+0.765769790 container remove 0fe78616d3a277d695dd048c54f444bcc0f0ca52558a870e0dccbb4a29247d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-14T23:24:57.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.a.service' 2024-06-14T23:24:57.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:24:57.632 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-06-14T23:24:57.632 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-06-14T23:24:57.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.c 2024-06-14T23:24:57.953 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:57 smithi086 systemd[1]: Stopping Ceph mon.c for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:24:58.286 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:57 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c[102728]: 2024-06-14T23:24:57.950+0000 7f1d1c37b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:24:58.287 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:57 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c[102728]: 2024-06-14T23:24:57.950+0000 7f1d1c37b640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-06-14T23:24:58.287 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:58 smithi086 podman[371554]: 2024-06-14 23:24:58.063504461 +0000 UTC m=+0.213386707 container died 1c4e71d265f00f50c1d0b7e3a7f4f1c3dfeb1ab75de74b98ded9b1b91b7dd71b (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.vendor=CentOS) 2024-06-14T23:24:58.287 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:57 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:57] ENGINE Bus STOPPING 2024-06-14T23:24:58.287 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:57 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-14T23:24:58.287 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:57 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:57] ENGINE Bus STOPPED 2024-06-14T23:24:58.288 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:57 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:57] ENGINE Bus STARTING 2024-06-14T23:24:58.288 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:58 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:58] ENGINE Serving on http://:::9283 2024-06-14T23:24:58.288 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:24:58 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: [14/Jun/2024:23:24:58] ENGINE Bus STARTED 2024-06-14T23:24:58.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:58 smithi086 podman[371554]: 2024-06-14 23:24:58.489954304 +0000 UTC m=+0.639836550 container cleanup 1c4e71d265f00f50c1d0b7e3a7f4f1c3dfeb1ab75de74b98ded9b1b91b7dd71b (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:24:58.587 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:58 smithi086 bash[371554]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c 2024-06-14T23:24:58.967 INFO:journalctl@ceph.mon.c.smithi086.stdout:Jun 14 23:24:58 smithi086 podman[371582]: 2024-06-14 23:24:58.724623765 +0000 UTC m=+0.662283887 container remove 1c4e71d265f00f50c1d0b7e3a7f4f1c3dfeb1ab75de74b98ded9b1b91b7dd71b (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-14T23:24:58.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.c.service' 2024-06-14T23:24:59.009 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:24:59.010 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-06-14T23:24:59.010 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-06-14T23:24:59.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.b 2024-06-14T23:24:59.345 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:59 smithi137 systemd[1]: Stopping Ceph mon.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:24:59.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b[90612]: 2024-06-14T23:24:59.341+0000 7f8ba92b4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:24:59.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b[90612]: 2024-06-14T23:24:59.341+0000 7f8ba92b4640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-06-14T23:24:59.746 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:24:59 smithi137 podman[241517]: 2024-06-14 23:24:59.566817452 +0000 UTC m=+0.355647203 container died 74ce1e795ab2fef3fa15149bb35f25a1ca9aa8f16b8098d2abcadfc30b9b4ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-14T23:25:00.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:25:00 smithi137 podman[241517]: 2024-06-14 23:25:00.072546598 +0000 UTC m=+0.861376347 container cleanup 74ce1e795ab2fef3fa15149bb35f25a1ca9aa8f16b8098d2abcadfc30b9b4ae6 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, maintainer=Guillaume Abrioux ) 2024-06-14T23:25:00.246 INFO:journalctl@ceph.mon.b.smithi137.stdout:Jun 14 23:25:00 smithi137 bash[241517]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mon-b 2024-06-14T23:25:00.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mon.b.service' 2024-06-14T23:25:00.585 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:25:00.585 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-06-14T23:25:00.585 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-06-14T23:25:00.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y 2024-06-14T23:25:00.899 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:25:00 smithi086 systemd[1]: Stopping Ceph mgr.y for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:25:01.222 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:25:00 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T23:25:00.896+0000 7f49455c1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:25:01.223 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:25:00 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y[88792]: 2024-06-14T23:25:00.896+0000 7f49455c1640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-06-14T23:25:01.223 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:25:00 smithi086 podman[371708]: 2024-06-14 23:25:00.92775526 +0000 UTC m=+0.147777839 container died edafb7b2b182f1c704baab1d447c3cd128aff25ced52c1a50ce6c194e1b85cd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True) 2024-06-14T23:25:01.587 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:25:01 smithi086 podman[371708]: 2024-06-14 23:25:01.385816166 +0000 UTC m=+0.605838810 container cleanup edafb7b2b182f1c704baab1d447c3cd128aff25ced52c1a50ce6c194e1b85cd1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.35.3, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:25:01.587 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Jun 14 23:25:01 smithi086 bash[371708]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-y 2024-06-14T23:25:01.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.y.service' 2024-06-14T23:25:01.912 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:25:01.912 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-06-14T23:25:01.912 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-06-14T23:25:01.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.x 2024-06-14T23:25:02.245 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 23:25:01 smithi137 systemd[1]: Stopping Ceph mgr.x for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:25:02.495 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 23:25:02 smithi137 podman[241649]: 2024-06-14 23:25:02.242881005 +0000 UTC m=+0.139147333 container died 086e4706077c6f7bf553ce7380ebdf262b12e5dc3233dcb5bedb3e748149cfb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:25:02.889 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 23:25:02 smithi137 podman[241649]: 2024-06-14 23:25:02.677335566 +0000 UTC m=+0.573601893 container cleanup 086e4706077c6f7bf553ce7380ebdf262b12e5dc3233dcb5bedb3e748149cfb1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x, ceph=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T23:25:02.889 INFO:journalctl@ceph.mgr.x.smithi137.stdout:Jun 14 23:25:02 smithi137 bash[241649]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-mgr-x 2024-06-14T23:25:03.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@mgr.x.service' 2024-06-14T23:25:03.137 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:25:03.138 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-06-14T23:25:03.138 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-14T23:25:03.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.0 2024-06-14T23:25:03.587 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:03 smithi086 systemd[1]: Stopping Ceph osd.0 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:25:04.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:03 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[112881]: 2024-06-14T23:25:03.709+0000 7f321d7a9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:25:04.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:03 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[112881]: 2024-06-14T23:25:03.709+0000 7f321d7a9640 -1 osd.0 522 *** Got signal Terminated *** 2024-06-14T23:25:04.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:03 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0[112881]: 2024-06-14T23:25:03.709+0000 7f321d7a9640 -1 osd.0 522 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:25:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:25:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:25:05.239Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:25:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:25:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:25:05.240Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:25:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:25:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:25:05.240Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:25:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:25:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:25:05.241Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:25:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:25:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:25:05.241Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:25:09.337 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:09 smithi086 podman[371840]: 2024-06-14 23:25:09.141569761 +0000 UTC m=+5.813900534 container died 62732bd1f8028eee8a271d30772f3307e979d37d7611e1c18be7d4e18ea67b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T23:25:09.837 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:09 smithi086 podman[371840]: 2024-06-14 23:25:09.673906586 +0000 UTC m=+6.346237358 container cleanup 62732bd1f8028eee8a271d30772f3307e979d37d7611e1c18be7d4e18ea67b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-14T23:25:09.837 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:09 smithi086 bash[371840]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0 2024-06-14T23:25:10.337 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:09 smithi086 podman[371862]: 2024-06-14 23:25:09.900419739 +0000 UTC m=+0.753362114 container remove 62732bd1f8028eee8a271d30772f3307e979d37d7611e1c18be7d4e18ea67b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-06-14T23:25:10.639 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:10 smithi086 podman[371957]: 2024-06-14 23:25:10.242986345 +0000 UTC m=+0.017020517 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T23:25:10.639 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:10 smithi086 podman[371957]: 2024-06-14 23:25:10.351328683 +0000 UTC m=+0.125362837 container create 13dcea6a51ceb3be6a63a5f8a6dc704381fcc7fdced4a0648b7fe9dd120163d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.35.3) 2024-06-14T23:25:10.639 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:10 smithi086 podman[371957]: 2024-06-14 23:25:10.637444476 +0000 UTC m=+0.411478641 container init 13dcea6a51ceb3be6a63a5f8a6dc704381fcc7fdced4a0648b7fe9dd120163d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-14T23:25:11.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:10 smithi086 podman[371957]: 2024-06-14 23:25:10.643468326 +0000 UTC m=+0.417502479 container start 13dcea6a51ceb3be6a63a5f8a6dc704381fcc7fdced4a0648b7fe9dd120163d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T23:25:11.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:10 smithi086 podman[371957]: 2024-06-14 23:25:10.795697799 +0000 UTC m=+0.569731960 container attach 13dcea6a51ceb3be6a63a5f8a6dc704381fcc7fdced4a0648b7fe9dd120163d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T23:25:12.293 INFO:journalctl@ceph.osd.0.smithi086.stdout:Jun 14 23:25:11 smithi086 podman[371957]: 2024-06-14 23:25:11.921822072 +0000 UTC m=+1.695856224 container died 13dcea6a51ceb3be6a63a5f8a6dc704381fcc7fdced4a0648b7fe9dd120163d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, ceph=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-06-14T23:25:12.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.0.service' 2024-06-14T23:25:12.720 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:25:12.720 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-14T23:25:12.721 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-14T23:25:12.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.1 2024-06-14T23:25:13.065 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:12 smithi086 systemd[1]: Stopping Ceph osd.1 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:25:13.337 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[123167]: 2024-06-14T23:25:13.062+0000 7f25c679f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:25:13.337 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[123167]: 2024-06-14T23:25:13.062+0000 7f25c679f640 -1 osd.1 522 *** Got signal Terminated *** 2024-06-14T23:25:13.337 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:13 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1[123167]: 2024-06-14T23:25:13.062+0000 7f25c679f640 -1 osd.1 522 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:25:18.987 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:18 smithi086 podman[372085]: 2024-06-14 23:25:18.677623803 +0000 UTC m=+5.732834590 container died 50c27a83b12af885134b5ddae3a9dd317998a06a7c1c3565f5096b9d3eecc5d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-14T23:25:19.337 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:19 smithi086 podman[372085]: 2024-06-14 23:25:19.135284673 +0000 UTC m=+6.190495459 container cleanup 50c27a83b12af885134b5ddae3a9dd317998a06a7c1c3565f5096b9d3eecc5d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-14T23:25:19.337 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:19 smithi086 bash[372085]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1 2024-06-14T23:25:19.764 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:19 smithi086 podman[372107]: 2024-06-14 23:25:19.445090402 +0000 UTC m=+0.763102922 container remove 50c27a83b12af885134b5ddae3a9dd317998a06a7c1c3565f5096b9d3eecc5d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-06-14T23:25:20.087 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:19 smithi086 podman[372201]: 2024-06-14 23:25:19.762598096 +0000 UTC m=+0.114299909 container create 55f8a30227f61d94e8dad496b61134f4c6a2f5a21cbf2f9b692b7174afdf4c82 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528) 2024-06-14T23:25:20.087 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:19 smithi086 podman[372201]: 2024-06-14 23:25:19.66546358 +0000 UTC m=+0.017165362 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T23:25:20.087 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:20 smithi086 podman[372201]: 2024-06-14 23:25:20.006972579 +0000 UTC m=+0.358674362 container init 55f8a30227f61d94e8dad496b61134f4c6a2f5a21cbf2f9b692b7174afdf4c82 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.license=GPLv2) 2024-06-14T23:25:20.087 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:20 smithi086 podman[372201]: 2024-06-14 23:25:20.012511309 +0000 UTC m=+0.364213086 container start 55f8a30227f61d94e8dad496b61134f4c6a2f5a21cbf2f9b692b7174afdf4c82 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:25:20.587 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:20 smithi086 podman[372201]: 2024-06-14 23:25:20.125490744 +0000 UTC m=+0.477192543 container attach 55f8a30227f61d94e8dad496b61134f4c6a2f5a21cbf2f9b692b7174afdf4c82 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095) 2024-06-14T23:25:20.587 INFO:journalctl@ceph.osd.1.smithi086.stdout:Jun 14 23:25:20 smithi086 podman[372201]: 2024-06-14 23:25:20.269248061 +0000 UTC m=+0.620949835 container died 55f8a30227f61d94e8dad496b61134f4c6a2f5a21cbf2f9b692b7174afdf4c82 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3) 2024-06-14T23:25:20.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.1.service' 2024-06-14T23:25:20.935 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:25:20.936 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-14T23:25:20.936 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-14T23:25:20.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.2 2024-06-14T23:25:21.293 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:21 smithi086 systemd[1]: Stopping Ceph osd.2 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:25:21.587 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[270968]: 2024-06-14T23:25:21.290+0000 7f28f108c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:25:21.587 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[270968]: 2024-06-14T23:25:21.290+0000 7f28f108c640 -1 osd.2 522 *** Got signal Terminated *** 2024-06-14T23:25:21.587 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:21 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2[270968]: 2024-06-14T23:25:21.290+0000 7f28f108c640 -1 osd.2 522 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:25:27.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:26 smithi086 podman[372330]: 2024-06-14 23:25:26.77492926 +0000 UTC m=+5.622046880 container died 8d3db12678a64641df0dd43eeccbf68efc095493c8138d732bf37d3ad7857178 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3) 2024-06-14T23:25:27.336 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:27 smithi086 podman[372330]: 2024-06-14 23:25:27.254383187 +0000 UTC m=+6.101500804 container cleanup 8d3db12678a64641df0dd43eeccbf68efc095493c8138d732bf37d3ad7857178 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.vendor=CentOS) 2024-06-14T23:25:27.337 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:27 smithi086 bash[372330]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2 2024-06-14T23:25:27.836 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:27 smithi086 podman[372351]: 2024-06-14 23:25:27.572755192 +0000 UTC m=+0.799590251 container remove 8d3db12678a64641df0dd43eeccbf68efc095493c8138d732bf37d3ad7857178 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, ceph=True, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-14T23:25:28.194 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:27 smithi086 podman[372448]: 2024-06-14 23:25:27.829437897 +0000 UTC m=+0.017064371 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T23:25:28.195 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:27 smithi086 podman[372448]: 2024-06-14 23:25:27.940237837 +0000 UTC m=+0.127864313 container create 490412f197f0b8c0500b544fe5a2a992f339fffa98dd7522c1bba42887a45d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-14T23:25:28.195 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:28 smithi086 podman[372448]: 2024-06-14 23:25:28.192638063 +0000 UTC m=+0.380264541 container init 490412f197f0b8c0500b544fe5a2a992f339fffa98dd7522c1bba42887a45d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-06-14T23:25:28.463 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:28 smithi086 podman[372448]: 2024-06-14 23:25:28.198703762 +0000 UTC m=+0.386330234 container start 490412f197f0b8c0500b544fe5a2a992f339fffa98dd7522c1bba42887a45d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3) 2024-06-14T23:25:28.463 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:28 smithi086 podman[372448]: 2024-06-14 23:25:28.309601498 +0000 UTC m=+0.497227996 container attach 490412f197f0b8c0500b544fe5a2a992f339fffa98dd7522c1bba42887a45d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-06-14T23:25:28.815 INFO:journalctl@ceph.osd.2.smithi086.stdout:Jun 14 23:25:28 smithi086 podman[372496]: 2024-06-14 23:25:28.48684803 +0000 UTC m=+0.018800497 container died 490412f197f0b8c0500b544fe5a2a992f339fffa98dd7522c1bba42887a45d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-2-deactivate, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-06-14T23:25:29.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.2.service' 2024-06-14T23:25:29.308 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:25:29.308 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-14T23:25:29.309 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-14T23:25:29.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.3 2024-06-14T23:25:29.495 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:29 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:29.149+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:29.587 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:29 smithi086 systemd[1]: Stopping Ceph osd.3 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:25:30.087 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T23:25:29.660+0000 7f61b2a01640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:25:30.087 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T23:25:29.660+0000 7f61b2a01640 -1 osd.3 522 *** Got signal Terminated *** 2024-06-14T23:25:30.087 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:29 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T23:25:29.660+0000 7f61b2a01640 -1 osd.3 522 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:25:30.495 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:30.173+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:30.995 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:30 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:30.510+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:31.337 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:31 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T23:25:31.078+0000 7f61ab00d640 -1 osd.3 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.381532+0000 front 2024-06-14T23:25:05.381605+0000 (oldest deadline 2024-06-14T23:25:30.681696+0000) 2024-06-14T23:25:31.474 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:31 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:31.148+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:31.745 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:31 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:31.472+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:32.337 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:32 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T23:25:32.059+0000 7f61ab00d640 -1 osd.3 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.381532+0000 front 2024-06-14T23:25:05.381605+0000 (oldest deadline 2024-06-14T23:25:30.681696+0000) 2024-06-14T23:25:32.438 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:32 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:32.181+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:32.745 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:32 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:32.450+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:32.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:32 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:32.435+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:33.337 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:33 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T23:25:33.074+0000 7f61ab00d640 -1 osd.3 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.381532+0000 front 2024-06-14T23:25:05.381605+0000 (oldest deadline 2024-06-14T23:25:30.681696+0000) 2024-06-14T23:25:33.476 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:33 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:33.211+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:33.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:33 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:33.474+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:33.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:33 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:33.479+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:34.495 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:34.225+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:34.496 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:34.473+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:34.496 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:34.332+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:34.586 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:34 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3[137906]: 2024-06-14T23:25:34.117+0000 7f61ab00d640 -1 osd.3 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.381532+0000 front 2024-06-14T23:25:05.381605+0000 (oldest deadline 2024-06-14T23:25:30.681696+0000) 2024-06-14T23:25:34.995 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:34 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:34.522+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:35.495 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:35 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:35.212+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:35.496 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:35 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:35.423+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:35.496 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:35 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:35.343+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:35.587 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:35 smithi086 podman[372590]: 2024-06-14 23:25:35.299339856 +0000 UTC m=+5.771113598 container died 47b006e23b2a60d7adea74948de7e51ac45bd6657fc24c6adb76e41138487488 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-14T23:25:35.995 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:35 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:35.500+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:36.042 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:35 smithi086 podman[372590]: 2024-06-14 23:25:35.755503978 +0000 UTC m=+6.227277766 container cleanup 47b006e23b2a60d7adea74948de7e51ac45bd6657fc24c6adb76e41138487488 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.35.3) 2024-06-14T23:25:36.042 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:35 smithi086 bash[372590]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3 2024-06-14T23:25:36.337 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:36 smithi086 podman[372610]: 2024-06-14 23:25:36.039994615 +0000 UTC m=+0.739633736 container remove 47b006e23b2a60d7adea74948de7e51ac45bd6657fc24c6adb76e41138487488 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-14T23:25:36.495 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:36.234+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:36.496 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:36.411+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:36.496 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:36.324+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:36.497 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:36 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:36.473+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:36.645 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:36 smithi086 podman[372705]: 2024-06-14 23:25:36.273907143 +0000 UTC m=+0.017674666 image pull e7282a3bffba6450e7bbc61351625ccbdc874ba3abbccb32d6c2163005000667 quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0 2024-06-14T23:25:36.645 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:36 smithi086 podman[372705]: 2024-06-14 23:25:36.399110852 +0000 UTC m=+0.142878472 container create 540b536c53e59e9e6c524462d633143db5e69f8340b0dc8ff560edb18c47b0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528) 2024-06-14T23:25:36.901 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:36 smithi086 podman[372705]: 2024-06-14 23:25:36.643529355 +0000 UTC m=+0.387296879 container init 540b536c53e59e9e6c524462d633143db5e69f8340b0dc8ff560edb18c47b0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True) 2024-06-14T23:25:36.902 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:36 smithi086 podman[372705]: 2024-06-14 23:25:36.649499382 +0000 UTC m=+0.393266893 container start 540b536c53e59e9e6c524462d633143db5e69f8340b0dc8ff560edb18c47b0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True) 2024-06-14T23:25:36.902 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:36 smithi086 podman[372705]: 2024-06-14 23:25:36.734785604 +0000 UTC m=+0.478553125 container attach 540b536c53e59e9e6c524462d633143db5e69f8340b0dc8ff560edb18c47b0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528) 2024-06-14T23:25:37.240 INFO:journalctl@ceph.osd.3.smithi086.stdout:Jun 14 23:25:36 smithi086 podman[372749]: 2024-06-14 23:25:36.928641877 +0000 UTC m=+0.018557607 container died 540b536c53e59e9e6c524462d633143db5e69f8340b0dc8ff560edb18c47b0d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:86fcceac5e55a93f3e03f800f2d89bf9c8b6526203c40d1d5c5822005a1b39f0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-3-deactivate, RELEASE=wip-yuri5-testing-2024-06-08-0902-reef-6f4b095, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-14T23:25:37.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.3.service' 2024-06-14T23:25:37.569 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:25:37.569 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-14T23:25:37.570 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-14T23:25:37.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.4 2024-06-14T23:25:37.599 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:37.257+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:37.600 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:37.447+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:37.600 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:37.290+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:37.601 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:37.475+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:37.900 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:37 smithi137 systemd[1]: Stopping Ceph osd.4 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:25:38.245 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:37.898+0000 7feddb166640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:25:38.246 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:37.898+0000 7feddb166640 -1 osd.4 522 *** Got signal Terminated *** 2024-06-14T23:25:38.246 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:37 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:37.898+0000 7feddb166640 -1 osd.4 522 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:25:38.745 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:38.245+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:38.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:38.462+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:38.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:38.272+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:38.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:38 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:38.493+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:39.487 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:39.257+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:39.487 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:39.426+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:39.488 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:39.227+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:39.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:39.485+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:39.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:39 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:39.485+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:40.450 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:40.226+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:40.451 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:40.195+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:40.745 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:40.460+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:40.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:40.448+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:40.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:40 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:40.448+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:41.482 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:41.412+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:41.483 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:41.213+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:41.483 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:41.213+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:15.044042+0000 front 2024-06-14T23:25:15.044016+0000 (oldest deadline 2024-06-14T23:25:40.343862+0000) 2024-06-14T23:25:41.483 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:41.204+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:41.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:41.481+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:41.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:41 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:41.481+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:42.483 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:42.233+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:04.443074+0000 front 2024-06-14T23:25:04.443245+0000 (oldest deadline 2024-06-14T23:25:29.143034+0000) 2024-06-14T23:25:42.484 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4[38337]: 2024-06-14T23:25:42.233+0000 7fedd4f79640 -1 osd.4 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:15.044042+0000 front 2024-06-14T23:25:15.044016+0000 (oldest deadline 2024-06-14T23:25:40.343862+0000) 2024-06-14T23:25:42.484 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:42.367+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:42.484 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:42.367+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.497180+0000 front 2024-06-14T23:25:16.497198+0000 (oldest deadline 2024-06-14T23:25:41.797059+0000) 2024-06-14T23:25:42.485 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:42.210+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:42.485 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:42.210+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:42.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:42.481+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:42.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:42 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:42.481+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:43.464 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:43.370+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:43.465 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:43.370+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.497180+0000 front 2024-06-14T23:25:16.497198+0000 (oldest deadline 2024-06-14T23:25:41.797059+0000) 2024-06-14T23:25:43.465 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:43.184+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:43.465 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:43.184+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:43.745 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:43 smithi137 podman[241793]: 2024-06-14 23:25:43.462542872 +0000 UTC m=+5.684241052 container died 4b2729030303cb8c8c7c2cce791ba9dea3a39aa60074cce7bf13cbdc85cb7e03 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-14T23:25:43.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:43.509+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:43.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:43 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:43.509+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:44.073 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:43 smithi137 podman[241793]: 2024-06-14 23:25:43.845115364 +0000 UTC m=+6.066813544 container cleanup 4b2729030303cb8c8c7c2cce791ba9dea3a39aa60074cce7bf13cbdc85cb7e03 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-14T23:25:44.073 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:43 smithi137 bash[241793]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4 2024-06-14T23:25:44.390 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:44 smithi137 podman[241814]: 2024-06-14 23:25:44.071553748 +0000 UTC m=+0.605523046 container remove 4b2729030303cb8c8c7c2cce791ba9dea3a39aa60074cce7bf13cbdc85cb7e03 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0) 2024-06-14T23:25:44.390 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:44.190+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:44.390 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:44.190+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:44.726 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:44.396+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:44.727 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:44.396+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.497180+0000 front 2024-06-14T23:25:16.497198+0000 (oldest deadline 2024-06-14T23:25:41.797059+0000) 2024-06-14T23:25:44.727 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:44.500+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:44.727 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:44 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:44.500+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:44.727 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:44 smithi137 podman[241907]: 2024-06-14 23:25:44.291443341 +0000 UTC m=+0.016991593 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:25:44.728 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:44 smithi137 podman[241907]: 2024-06-14 23:25:44.389164074 +0000 UTC m=+0.114712326 container create 3631b9c6d7f245709b8549b6f2d60722569790c91799c2629c984aeab87d00f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4-deactivate, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-14T23:25:44.728 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:44 smithi137 podman[241907]: 2024-06-14 23:25:44.62527271 +0000 UTC m=+0.350820967 container init 3631b9c6d7f245709b8549b6f2d60722569790c91799c2629c984aeab87d00f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4-deactivate, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-14T23:25:44.728 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:44 smithi137 podman[241907]: 2024-06-14 23:25:44.630498135 +0000 UTC m=+0.356046391 container start 3631b9c6d7f245709b8549b6f2d60722569790c91799c2629c984aeab87d00f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-14T23:25:44.728 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:44 smithi137 podman[241907]: 2024-06-14 23:25:44.725107387 +0000 UTC m=+0.450655647 container attach 3631b9c6d7f245709b8549b6f2d60722569790c91799c2629c984aeab87d00f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-14T23:25:44.995 INFO:journalctl@ceph.osd.4.smithi137.stdout:Jun 14 23:25:44 smithi137 podman[241907]: 2024-06-14 23:25:44.924384756 +0000 UTC m=+0.649933011 container died 3631b9c6d7f245709b8549b6f2d60722569790c91799c2629c984aeab87d00f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD) 2024-06-14T23:25:45.495 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:45.398+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:45.495 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:45.398+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.497180+0000 front 2024-06-14T23:25:16.497198+0000 (oldest deadline 2024-06-14T23:25:41.797059+0000) 2024-06-14T23:25:45.496 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:45.180+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:45.496 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:45.180+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:45.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.4.service' 2024-06-14T23:25:45.605 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:25:45.606 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-14T23:25:45.606 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-14T23:25:45.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.5 2024-06-14T23:25:45.945 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:45.509+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:45.945 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:45.509+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:45.945 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:45 smithi137 systemd[1]: Stopping Ceph osd.5 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:25:45.946 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:45.941+0000 7fb67f279640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:25:45.946 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:45.941+0000 7fb67f279640 -1 osd.5 522 *** Got signal Terminated *** 2024-06-14T23:25:45.946 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:45 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:45.941+0000 7fb67f279640 -1 osd.5 522 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:25:46.495 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:46.368+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:46.495 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:46.368+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.497180+0000 front 2024-06-14T23:25:16.497198+0000 (oldest deadline 2024-06-14T23:25:41.797059+0000) 2024-06-14T23:25:46.496 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:46.219+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:46.496 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:46.219+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:46.995 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:46.512+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:46.995 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:46 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:46.512+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:47.526 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:47.335+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:47.527 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:47.335+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.497180+0000 front 2024-06-14T23:25:16.497198+0000 (oldest deadline 2024-06-14T23:25:41.797059+0000) 2024-06-14T23:25:47.527 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:47.259+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:47.527 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:47.259+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:47.995 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:47.523+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:47.996 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:47 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:47.523+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:48.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:48.300+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:48.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:48.300+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:48.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:48.337+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:48.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:48.337+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.497180+0000 front 2024-06-14T23:25:16.497198+0000 (oldest deadline 2024-06-14T23:25:41.797059+0000) 2024-06-14T23:25:48.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:48.484+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:48.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:48.484+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:48.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:48 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:48.484+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:49.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:49.337+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:49.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:49.337+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:49.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:49.381+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:49.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:49.381+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.497180+0000 front 2024-06-14T23:25:16.497198+0000 (oldest deadline 2024-06-14T23:25:41.797059+0000) 2024-06-14T23:25:49.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:49.496+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:49.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:49.496+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:49.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:49 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:49.496+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:50.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:50.381+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:50.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:50.381+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:50.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:50.381+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:26.048771+0000 front 2024-06-14T23:25:26.048722+0000 (oldest deadline 2024-06-14T23:25:50.148659+0000) 2024-06-14T23:25:50.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:50.359+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:05.995664+0000 front 2024-06-14T23:25:05.995644+0000 (oldest deadline 2024-06-14T23:25:29.495391+0000) 2024-06-14T23:25:50.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:50.359+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.497180+0000 front 2024-06-14T23:25:16.497198+0000 (oldest deadline 2024-06-14T23:25:41.797059+0000) 2024-06-14T23:25:50.746 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5[238967]: 2024-06-14T23:25:50.359+0000 7fb67888b640 -1 osd.5 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:24.097907+0000 front 2024-06-14T23:25:24.097948+0000 (oldest deadline 2024-06-14T23:25:49.898362+0000) 2024-06-14T23:25:50.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:50.536+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:50.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:50.536+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:50.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:50 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:50.536+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:51.744 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:51 smithi137 podman[242035]: 2024-06-14 23:25:51.437084055 +0000 UTC m=+5.607563834 container died c89b91667200ed2f7ec6750ce19a7e8561708e2c83d2ca45967f093bf0e5563d (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-14T23:25:51.744 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:51.378+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:51.744 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:51.378+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:51.744 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:51.378+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:26.048771+0000 front 2024-06-14T23:25:26.048722+0000 (oldest deadline 2024-06-14T23:25:50.148659+0000) 2024-06-14T23:25:51.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:51.519+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:51.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:51.519+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:51.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:51 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:51.519+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:51.995 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:51 smithi137 podman[242035]: 2024-06-14 23:25:51.823004603 +0000 UTC m=+5.993484382 container cleanup c89b91667200ed2f7ec6750ce19a7e8561708e2c83d2ca45967f093bf0e5563d (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-06-14T23:25:51.995 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:51 smithi137 bash[242035]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5 2024-06-14T23:25:52.385 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:52 smithi137 podman[242055]: 2024-06-14 23:25:52.099283791 +0000 UTC m=+0.663480274 container remove c89b91667200ed2f7ec6750ce19a7e8561708e2c83d2ca45967f093bf0e5563d (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD) 2024-06-14T23:25:52.679 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:52.535+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:52.680 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:52.535+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:52.680 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:52.535+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:52.680 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:52 smithi137 podman[242148]: 2024-06-14 23:25:52.339890845 +0000 UTC m=+0.017292497 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:25:52.680 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:52 smithi137 podman[242148]: 2024-06-14 23:25:52.450484104 +0000 UTC m=+0.127885753 container create 4b5907fe5ffa2921cd856369a8ee86e95ba2037330253ee6cba9571f0814641a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240612) 2024-06-14T23:25:52.680 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:52 smithi137 podman[242148]: 2024-06-14 23:25:52.678019511 +0000 UTC m=+0.355421170 container init 4b5907fe5ffa2921cd856369a8ee86e95ba2037330253ee6cba9571f0814641a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-14T23:25:52.681 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:52.382+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:52.681 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:52.382+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:52.681 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:52 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:52.382+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:26.048771+0000 front 2024-06-14T23:25:26.048722+0000 (oldest deadline 2024-06-14T23:25:50.148659+0000) 2024-06-14T23:25:52.937 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:52 smithi137 podman[242148]: 2024-06-14 23:25:52.683585032 +0000 UTC m=+0.360986683 container start 4b5907fe5ffa2921cd856369a8ee86e95ba2037330253ee6cba9571f0814641a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True) 2024-06-14T23:25:52.937 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:52 smithi137 podman[242148]: 2024-06-14 23:25:52.786181307 +0000 UTC m=+0.463582985 container attach 4b5907fe5ffa2921cd856369a8ee86e95ba2037330253ee6cba9571f0814641a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-14T23:25:52.937 INFO:journalctl@ceph.osd.5.smithi137.stdout:Jun 14 23:25:52 smithi137 podman[242148]: 2024-06-14 23:25:52.935231155 +0000 UTC m=+0.612632810 container died 4b5907fe5ffa2921cd856369a8ee86e95ba2037330253ee6cba9571f0814641a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2) 2024-06-14T23:25:53.574 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:53.365+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:53.575 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:53.365+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:53.575 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:53.365+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:26.048771+0000 front 2024-06-14T23:25:26.048722+0000 (oldest deadline 2024-06-14T23:25:50.148659+0000) 2024-06-14T23:25:53.575 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:53.508+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:53.575 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:53.508+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:53.575 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:53 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:53.508+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:53.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.5.service' 2024-06-14T23:25:53.635 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:25:53.635 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-14T23:25:53.635 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-06-14T23:25:53.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.6 2024-06-14T23:25:53.995 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:53 smithi137 systemd[1]: Stopping Ceph osd.6 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:25:54.327 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:54.052+0000 7ff3f5211640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:25:54.327 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:54.052+0000 7ff3f5211640 -1 osd.6 522 *** Got signal Terminated *** 2024-06-14T23:25:54.327 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:54.052+0000 7ff3f5211640 -1 osd.6 522 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:25:54.327 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:54.324+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:54.327 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:54.324+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:54.328 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:54.324+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:26.048771+0000 front 2024-06-14T23:25:26.048722+0000 (oldest deadline 2024-06-14T23:25:50.148659+0000) 2024-06-14T23:25:54.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:54.459+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:54.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:54.459+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:54.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:54 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:54.459+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:55.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:55 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:55.449+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:55.745 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:55 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:55.449+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:55.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:55 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:55.449+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:55.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:55 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:55.357+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:55.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:55 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:55.357+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:55.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:55 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:55.357+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:26.048771+0000 front 2024-06-14T23:25:26.048722+0000 (oldest deadline 2024-06-14T23:25:50.148659+0000) 2024-06-14T23:25:56.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:56.362+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:56.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:56.362+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:56.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:56.362+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:26.048771+0000 front 2024-06-14T23:25:26.048722+0000 (oldest deadline 2024-06-14T23:25:50.148659+0000) 2024-06-14T23:25:56.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:56.429+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:56.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:56.429+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:56.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:56.429+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:56.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:56 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:56.429+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:25:57.745 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:57.352+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:57.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:57.352+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:57.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:57.352+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:26.048771+0000 front 2024-06-14T23:25:26.048722+0000 (oldest deadline 2024-06-14T23:25:50.148659+0000) 2024-06-14T23:25:57.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:57.380+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:57.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:57.380+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:57.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:57.380+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:57.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:57 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:57.380+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:25:58.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:58.396+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:08.646348+0000 front 2024-06-14T23:25:08.646401+0000 (oldest deadline 2024-06-14T23:25:33.946233+0000) 2024-06-14T23:25:58.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:58.396+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:14.447156+0000 front 2024-06-14T23:25:14.447144+0000 (oldest deadline 2024-06-14T23:25:42.047486+0000) 2024-06-14T23:25:58.746 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6[162058]: 2024-06-14T23:25:58.396+0000 7ff3f0827640 -1 osd.6 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:26.048771+0000 front 2024-06-14T23:25:26.048722+0000 (oldest deadline 2024-06-14T23:25:50.148659+0000) 2024-06-14T23:25:58.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:58.346+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:58.746 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:58.346+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:58.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:58.346+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:58.747 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:58 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:58.346+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:25:59.647 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:59.310+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:25:59.647 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:59.310+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:25:59.647 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:59.310+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:25:59.648 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:25:59 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:25:59.310+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:25:59.927 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:25:59 smithi137 podman[242275]: 2024-06-14 23:25:59.673555388 +0000 UTC m=+5.812788887 container died b31f6edf3d3c82e1db7f0bf5d932fece97d6d88da144e3640c43a4848b87b4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-14T23:26:00.245 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:26:00 smithi137 podman[242275]: 2024-06-14 23:26:00.025266779 +0000 UTC m=+6.164500277 container cleanup b31f6edf3d3c82e1db7f0bf5d932fece97d6d88da144e3640c43a4848b87b4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, GIT_CLEAN=True) 2024-06-14T23:26:00.245 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:26:00 smithi137 bash[242275]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6 2024-06-14T23:26:00.687 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:26:00 smithi137 podman[242296]: 2024-06-14 23:26:00.34356469 +0000 UTC m=+0.668142683 container remove b31f6edf3d3c82e1db7f0bf5d932fece97d6d88da144e3640c43a4848b87b4f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD) 2024-06-14T23:26:00.688 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:00 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:00.316+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:26:00.688 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:00 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:00.316+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:26:00.688 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:00 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:00.316+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:26:00.688 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:00 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:00.316+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:26:00.995 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:26:00 smithi137 podman[242401]: 2024-06-14 23:26:00.59006976 +0000 UTC m=+0.019700065 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:26:00.996 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:26:00 smithi137 podman[242401]: 2024-06-14 23:26:00.711390942 +0000 UTC m=+0.141021230 container create e2892b640d632c7ddaca015c4501b5ca35386e652b7fb2930e7be6f1be3a304a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, ceph=True) 2024-06-14T23:26:00.996 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:26:00 smithi137 podman[242401]: 2024-06-14 23:26:00.91370194 +0000 UTC m=+0.343332229 container init e2892b640d632c7ddaca015c4501b5ca35386e652b7fb2930e7be6f1be3a304a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612) 2024-06-14T23:26:00.996 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:26:00 smithi137 podman[242401]: 2024-06-14 23:26:00.91911866 +0000 UTC m=+0.348748954 container start e2892b640d632c7ddaca015c4501b5ca35386e652b7fb2930e7be6f1be3a304a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, GIT_CLEAN=True) 2024-06-14T23:26:01.273 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:26:00 smithi137 podman[242401]: 2024-06-14 23:26:00.996924823 +0000 UTC m=+0.426555122 container attach e2892b640d632c7ddaca015c4501b5ca35386e652b7fb2930e7be6f1be3a304a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-14T23:26:01.274 INFO:journalctl@ceph.osd.6.smithi137.stdout:Jun 14 23:26:01 smithi137 podman[242401]: 2024-06-14 23:26:01.167690095 +0000 UTC m=+0.597320386 container died e2892b640d632c7ddaca015c4501b5ca35386e652b7fb2930e7be6f1be3a304a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-14T23:26:01.727 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:01 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:01.270+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:26:01.727 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:01 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:01.270+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:26:01.727 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:01 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:01.270+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:26:01.727 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:01 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:01.270+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:26:01.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.6.service' 2024-06-14T23:26:01.788 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:26:01.788 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-06-14T23:26:01.788 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-06-14T23:26:01.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.7 2024-06-14T23:26:01.995 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:01 smithi137 systemd[1]: Stopping Ceph osd.7 for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:26:02.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:02 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:02.114+0000 7fec3a556640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-14T23:26:02.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:02 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:02.114+0000 7fec3a556640 -1 osd.7 522 *** Got signal Terminated *** 2024-06-14T23:26:02.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:02 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:02.114+0000 7fec3a556640 -1 osd.7 522 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-14T23:26:02.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:02 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:02.225+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:26:02.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:02 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:02.225+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:26:02.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:02 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:02.225+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:26:02.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:02 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:02.225+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:26:03.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:03.236+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:26:03.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:03.236+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:26:03.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:03.236+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:26:03.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:03 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:03.236+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:26:04.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:04 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:04.217+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:26:04.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:04 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:04.217+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:26:04.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:04 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:04.217+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:26:04.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:04 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:04.217+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:26:05.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:05.228+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:26:05.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:05.228+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:26:05.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:05.228+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:26:05.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:05.228+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:26:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:05.240Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:26:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:05.240Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:26:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:05.240Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:26:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:05.240Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:26:05.496 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:05 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:05.242Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.86:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.86:8765: connect: connection refused" 2024-06-14T23:26:06.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:06 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:06.203+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6806 osd.0 since back 2024-06-14T23:25:07.698734+0000 front 2024-06-14T23:25:07.698752+0000 (oldest deadline 2024-06-14T23:25:31.798668+0000) 2024-06-14T23:26:06.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:06 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:06.203+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6814 osd.1 since back 2024-06-14T23:25:16.499552+0000 front 2024-06-14T23:25:16.499561+0000 (oldest deadline 2024-06-14T23:25:39.399479+0000) 2024-06-14T23:26:06.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:06 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:06.203+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6822 osd.2 since back 2024-06-14T23:25:25.200752+0000 front 2024-06-14T23:25:25.200816+0000 (oldest deadline 2024-06-14T23:25:48.100838+0000) 2024-06-14T23:26:06.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:06 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:06.203+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.86:6830 osd.3 since back 2024-06-14T23:25:34.402742+0000 front 2024-06-14T23:25:34.402709+0000 (oldest deadline 2024-06-14T23:25:56.102599+0000) 2024-06-14T23:26:06.496 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:06 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7[56831]: 2024-06-14T23:26:06.203+0000 7fec3536b640 -1 osd.7 522 heartbeat_check: no reply from 172.21.15.137:6804 osd.4 since back 2024-06-14T23:25:39.603277+0000 front 2024-06-14T23:25:39.603307+0000 (oldest deadline 2024-06-14T23:26:05.503309+0000) 2024-06-14T23:26:07.866 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:07 smithi137 podman[242530]: 2024-06-14 23:26:07.565148724 +0000 UTC m=+5.549465926 container died a9ca11c000f28bf5408a5e21e01bca2164010aa1206340ebcb569ef049c10676 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-06-14T23:26:08.206 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:07 smithi137 podman[242530]: 2024-06-14 23:26:07.961614195 +0000 UTC m=+5.945931396 container cleanup a9ca11c000f28bf5408a5e21e01bca2164010aa1206340ebcb569ef049c10676 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T23:26:08.206 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:07 smithi137 bash[242530]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7 2024-06-14T23:26:08.495 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:08 smithi137 podman[242554]: 2024-06-14 23:26:08.204453564 +0000 UTC m=+0.633129148 container remove a9ca11c000f28bf5408a5e21e01bca2164010aa1206340ebcb569ef049c10676 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-14T23:26:08.820 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:08 smithi137 podman[242648]: 2024-06-14 23:26:08.423690512 +0000 UTC m=+0.017425233 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-14T23:26:08.820 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:08 smithi137 podman[242648]: 2024-06-14 23:26:08.521881901 +0000 UTC m=+0.115616604 container create 3da18773cae17dc38fa551388b9b73683807e273a369723877af2df1d0f6e033 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-14T23:26:08.820 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:08 smithi137 podman[242648]: 2024-06-14 23:26:08.724392697 +0000 UTC m=+0.318127399 container init 3da18773cae17dc38fa551388b9b73683807e273a369723877af2df1d0f6e033 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, ceph=True) 2024-06-14T23:26:08.820 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:08 smithi137 podman[242648]: 2024-06-14 23:26:08.730555909 +0000 UTC m=+0.324290612 container start 3da18773cae17dc38fa551388b9b73683807e273a369723877af2df1d0f6e033 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-06-14T23:26:08.820 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:08 smithi137 podman[242648]: 2024-06-14 23:26:08.818161144 +0000 UTC m=+0.411895860 container attach 3da18773cae17dc38fa551388b9b73683807e273a369723877af2df1d0f6e033 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-14T23:26:09.245 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:08 smithi137 podman[242648]: 2024-06-14 23:26:08.979543146 +0000 UTC m=+0.573277850 container died 3da18773cae17dc38fa551388b9b73683807e273a369723877af2df1d0f6e033 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-06-14T23:26:09.542 INFO:journalctl@ceph.osd.7.smithi137.stdout:Jun 14 23:26:09 smithi137 podman[242691]: 2024-06-14 23:26:09.527429831 +0000 UTC m=+0.537420546 container remove 3da18773cae17dc38fa551388b9b73683807e273a369723877af2df1d0f6e033 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-14T23:26:09.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@osd.7.service' 2024-06-14T23:26:09.585 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:26:09.585 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-06-14T23:26:09.586 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-06-14T23:26:09.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@prometheus.a 2024-06-14T23:26:09.935 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 systemd[1]: Stopping Ceph prometheus.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:26:10.238 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.933Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-14T23:26:10.238 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.933Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-06-14T23:26:10.238 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.933Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-06-14T23:26:10.238 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.933Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-14T23:26:10.238 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.933Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-06-14T23:26:10.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.933Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-06-14T23:26:10.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.933Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-06-14T23:26:10.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.933Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-06-14T23:26:10.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.941Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-06-14T23:26:10.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.944Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-06-14T23:26:10.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.945Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-06-14T23:26:10.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a[80251]: ts=2024-06-14T23:26:09.945Z caller=main.go:1147 level=info msg="See you next time!" 2024-06-14T23:26:10.239 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:09 smithi137 podman[242776]: 2024-06-14 23:26:09.953875685 +0000 UTC m=+0.152986407 container died 1604035f49dfb23dfcea6ed42b510e46166e35aaf5accf39bea89b2ca6c06089 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T23:26:10.495 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:10 smithi137 podman[242776]: 2024-06-14 23:26:10.35511619 +0000 UTC m=+0.554226910 container cleanup 1604035f49dfb23dfcea6ed42b510e46166e35aaf5accf39bea89b2ca6c06089 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-14T23:26:10.495 INFO:journalctl@ceph.prometheus.a.smithi137.stdout:Jun 14 23:26:10 smithi137 bash[242776]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-prometheus-a 2024-06-14T23:26:10.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@prometheus.a.service' 2024-06-14T23:26:10.766 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-14T23:26:10.766 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-06-14T23:26:10.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-14T23:26:12.587 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:12 smithi086 systemd[1]: Stopping Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:26:12.999 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:12 smithi086 ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a[87756]: ts=2024-06-14T23:26:12.675Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-06-14T23:26:13.000 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:12 smithi086 podman[373133]: 2024-06-14 23:26:12.726193496 +0000 UTC m=+0.193336454 container died 1c9917382faac8a5d9d505bb607b6203aaeea0114aa2bc6fe83e757e75c03f30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T23:26:13.337 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:13 smithi086 podman[373133]: 2024-06-14 23:26:13.205593669 +0000 UTC m=+0.672736623 container cleanup 1c9917382faac8a5d9d505bb607b6203aaeea0114aa2bc6fe83e757e75c03f30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T23:26:13.337 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:13 smithi086 bash[373133]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a 2024-06-14T23:26:13.836 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:13 smithi086 podman[373152]: 2024-06-14 23:26:13.507000754 +0000 UTC m=+0.780606988 container remove 1c9917382faac8a5d9d505bb607b6203aaeea0114aa2bc6fe83e757e75c03f30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-14T23:26:13.837 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:13 smithi086 podman[373152]: 2024-06-14 23:26:13.749981728 +0000 UTC m=+1.023588004 volume remove 2b944fe13be593311291ac67446c7b2832416ba066ab53ff48b15955c3d44bf6 2024-06-14T23:26:14.336 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:13 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-06-14T23:26:14.337 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:13 smithi086 systemd[1]: Stopped Ceph alertmanager.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T23:26:14.337 INFO:journalctl@ceph.alertmanager.a.smithi086.stdout:Jun 14 23:26:13 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@alertmanager.a.service: Consumed 4.310s CPU time. 2024-06-14T23:26:14.337 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 23:26:14 smithi086 systemd[1]: Stopping Ceph node-exporter.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:26:14.629 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 23:26:14 smithi086 podman[373266]: 2024-06-14 23:26:14.340578847 +0000 UTC m=+0.121301965 container died d1de50e6087955ce1dabe399095a42d9ebac98b2290c1643769a421f423b99d6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T23:26:14.996 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 23:26:14 smithi086 podman[373266]: 2024-06-14 23:26:14.718601973 +0000 UTC m=+0.499325137 container cleanup d1de50e6087955ce1dabe399095a42d9ebac98b2290c1643769a421f423b99d6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T23:26:14.996 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 23:26:14 smithi086 bash[373266]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a 2024-06-14T23:26:15.325 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 23:26:14 smithi086 podman[373285]: 2024-06-14 23:26:14.994438192 +0000 UTC m=+0.654777613 container remove d1de50e6087955ce1dabe399095a42d9ebac98b2290c1643769a421f423b99d6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-14T23:26:15.325 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 23:26:14 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-06-14T23:26:15.325 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 23:26:15 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-06-14T23:26:15.326 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 23:26:15 smithi086 systemd[1]: Stopped Ceph node-exporter.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T23:26:15.326 INFO:journalctl@ceph.node-exporter.a.smithi086.stdout:Jun 14 23:26:15 smithi086 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.a.service: Consumed 7.814s CPU time. 2024-06-14T23:26:26.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-14T23:26:28.746 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:26:28 smithi137 systemd[1]: Stopping Ceph grafana.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:26:29.002 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:26:28 smithi137 podman[243196]: 2024-06-14 23:26:28.746645525 +0000 UTC m=+0.110856521 container died 12b0617a16e548189d76b0fdaeb60cfc07cf125520d1955aa756668e62f52a31 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, name=ubi8, maintainer=Paul Cuzner , architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, com.redhat.component=ubi8-container, vcs-type=git, version=8.5) 2024-06-14T23:26:29.385 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:26:29 smithi137 podman[243196]: 2024-06-14 23:26:29.149566526 +0000 UTC m=+0.513777528 container cleanup 12b0617a16e548189d76b0fdaeb60cfc07cf125520d1955aa756668e62f52a31 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, name=ubi8, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, release=239.1651231664, vendor=Red Hat, Inc., vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , architecture=x86_64, io.openshift.tags=base rhel8, version=8.5) 2024-06-14T23:26:29.386 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:26:29 smithi137 bash[243196]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a 2024-06-14T23:26:29.692 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:26:29 smithi137 podman[243214]: 2024-06-14 23:26:29.384105907 +0000 UTC m=+0.635887631 container remove 12b0617a16e548189d76b0fdaeb60cfc07cf125520d1955aa756668e62f52a31 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-grafana-a, vendor=Red Hat, Inc., release=239.1651231664, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , architecture=x86_64, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8) 2024-06-14T23:26:29.692 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:26:29 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-06-14T23:26:29.693 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:26:29 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-06-14T23:26:29.693 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:26:29 smithi137 systemd[1]: Stopped Ceph grafana.a for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T23:26:29.693 INFO:journalctl@ceph.grafana.a.smithi137.stdout:Jun 14 23:26:29 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@grafana.a.service: Consumed 33.467s CPU time. 2024-06-14T23:26:29.944 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 23:26:29 smithi137 systemd[1]: Stopping Ceph node-exporter.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968... 2024-06-14T23:26:30.197 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 23:26:29 smithi137 podman[243328]: 2024-06-14 23:26:29.94223801 +0000 UTC m=+0.110027351 container died b8c1542255f2b228a5cdb45bade4bab608b7a34f9b299291291e96755d7cbc98 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T23:26:30.498 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 23:26:30 smithi137 podman[243328]: 2024-06-14 23:26:30.311636092 +0000 UTC m=+0.479425445 container cleanup b8c1542255f2b228a5cdb45bade4bab608b7a34f9b299291291e96755d7cbc98 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T23:26:30.498 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 23:26:30 smithi137 bash[243328]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b 2024-06-14T23:26:30.872 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 23:26:30 smithi137 podman[243344]: 2024-06-14 23:26:30.537723446 +0000 UTC m=+0.599496038 container remove b8c1542255f2b228a5cdb45bade4bab608b7a34f9b299291291e96755d7cbc98 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-14T23:26:30.872 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 23:26:30 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-06-14T23:26:30.873 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 23:26:30 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-06-14T23:26:30.873 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 23:26:30 smithi137 systemd[1]: Stopped Ceph node-exporter.b for 44b2848a-2a9d-11ef-bc9d-c7b262605968. 2024-06-14T23:26:30.873 INFO:journalctl@ceph.node-exporter.b.smithi137.stdout:Jun 14 23:26:30 smithi137 systemd[1]: ceph-44b2848a-2a9d-11ef-bc9d-c7b262605968@node-exporter.b.service: Consumed 7.518s CPU time. 2024-06-14T23:26:34.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-14T23:26:34.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-14T23:26:34.662 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-14T23:26:34.664 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954/remote/smithi086/crash 2024-06-14T23:26:34.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/crash -- . 2024-06-14T23:26:34.725 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-14T23:26:34.725 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-06-14T23:26:34.727 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954/remote/smithi137/crash 2024-06-14T23:26:34.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/crash -- . 2024-06-14T23:26:34.768 INFO:teuthology.orchestra.run.smithi137.stderr:tar: /var/lib/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-14T23:26:34.769 INFO:teuthology.orchestra.run.smithi137.stderr:tar: Error is not recoverable: exiting now 2024-06-14T23:26:34.770 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-14T23:26:34.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-06-14T23:26:34.798 INFO:teuthology.orchestra.run.smithi086.stderr:grep: /var/log/ceph/44b2848a-2a9d-11ef-bc9d-c7b262605968/ceph.log: No such file or directory 2024-06-14T23:26:34.800 INFO:tasks.cephadm:Compressing logs... 2024-06-14T23:26:34.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-14T23:26:34.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-14T23:26:34.877 INFO:teuthology.orchestra.run.smithi137.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-14T23:26:34.898 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-14T23:38:56.949 INFO:tasks.cephadm:Archiving logs... 2024-06-14T23:38:56.950 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954/remote/smithi086/log 2024-06-14T23:38:56.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-14T23:40:18.167 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954/remote/smithi137/log 2024-06-14T23:40:18.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-14T23:41:42.365 INFO:tasks.cephadm:Removing cluster... 2024-06-14T23:41:42.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 --force 2024-06-14T23:41:43.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44b2848a-2a9d-11ef-bc9d-c7b262605968 --force 2024-06-14T23:41:44.771 INFO:tasks.cephadm:Removing cephadm ... 2024-06-14T23:41:44.772 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-14T23:41:44.791 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-14T23:41:44.807 INFO:tasks.cephadm:Teardown complete 2024-06-14T23:41:44.808 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-14T23:41:44.818 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-14T23:41:44.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-14T23:41:44.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-14T23:41:44.944 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-14T23:41:44.953 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-14T23:41:44.953 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-14T23:41:44.956 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-14T23:41:44.970 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-06-14T23:41:44.971 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-06-14T23:41:44.990 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-14T23:41:44.990 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-06-14T23:41:44.990 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv01.front.sepia.ceph.com 3 8 377 253 +1358us[+1346us] +/- 54ms 2024-06-14T23:41:44.990 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 58 -987us[ -987us] +/- 61ms 2024-06-14T23:41:44.991 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -1116us[-1116us] +/- 85ms 2024-06-14T23:41:44.991 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:41:44.998 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-14T23:41:44.999 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-06-14T23:41:44.999 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv01.front.sepia.ceph.com 3 7 377 126 +1383us[+1382us] +/- 56ms 2024-06-14T23:41:44.999 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 -986us[ -986us] +/- 62ms 2024-06-14T23:41:44.999 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -1069us[-1069us] +/- 85ms 2024-06-14T23:41:44.999 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:41:45.000 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-14T23:41:45.009 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-14T23:41:45.010 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-14T23:41:45.030 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-14T23:41:45.121 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-14T23:41:45.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-14T23:41:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:41:45.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-14T23:41:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:41:45.277 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-14T23:41:45.287 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-14T23:41:45.322 INFO:teuthology.task.internal:Duration was 4855.882634 seconds 2024-06-14T23:41:45.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-14T23:41:45.355 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-14T23:41:45.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-14T23:41:45.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-14T23:41:45.417 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-14T23:41:45.440 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-14T23:41:45.799 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-14T23:41:45.799 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-06-14T23:41:45.799 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-14T23:41:45.824 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-06-14T23:41:45.825 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-14T23:41:45.853 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-14T23:41:45.853 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-14T23:41:45.867 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-14T23:41:45.929 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-14T23:41:45.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-14T23:41:45.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-14T23:41:47.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-14T23:41:47.539 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-14T23:41:47.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-14T23:41:47.575 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-14T23:41:47.630 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-14T23:41:47.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-14T23:41:47.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-14T23:41:47.671 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-06-14T23:41:47.672 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-06-14T23:41:47.695 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-14T23:41:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:41:47.750 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-14T23:41:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:41:47.765 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-14T23:41:47.775 INFO:teuthology.task.internal:Transferring archived files... 2024-06-14T23:41:47.776 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954/remote/smithi086 2024-06-14T23:41:47.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-14T23:41:47.865 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-14_18:07:03-upgrade:quincy-x-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7755954/remote/smithi137 2024-06-14T23:41:47.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-14T23:41:47.910 INFO:teuthology.task.internal:Removing archive directory... 2024-06-14T23:41:47.911 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-14T23:41:47.913 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-14T23:41:47.961 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-14T23:41:47.973 INFO:teuthology.task.internal:Not uploading archives. 2024-06-14T23:41:47.973 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-14T23:41:48.031 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-14T23:41:48.032 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-14T23:41:48.034 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-14T23:41:48.049 INFO:teuthology.orchestra.run.smithi137.stdout: 266045 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 14 23:41 /home/ubuntu/cephtest 2024-06-14T23:41:48.053 INFO:teuthology.orchestra.run.smithi086.stdout: 262165 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 14 23:41 /home/ubuntu/cephtest 2024-06-14T23:41:48.053 INFO:teuthology.orchestra.run.smithi086.stdout: 426335 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 14 23:22 /home/ubuntu/cephtest/mnt.0 2024-06-14T23:41:48.053 INFO:teuthology.orchestra.run.smithi086.stdout: 426336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 14 23:22 /home/ubuntu/cephtest/mnt.0/client.0 2024-06-14T23:41:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:41:48.054 INFO:teuthology.orchestra.run.smithi086.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-06-14T23:41:48.055 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-14T23:41:48.062 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-14T23:41:48.074 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-14T23:41:48.125 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi086 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-14T23:41:48.128 INFO:teuthology.run:Summary data: description: upgrade:quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} duration: 4855.882633924484 failure_reason: 'Command failed on smithi086 with status 128: ''rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://github.com/cbodley/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=740e16f69a91470992e9bffea96fe812 status: fail success: false 2024-06-14T23:41:48.129 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-14T23:41:48.224 INFO:teuthology.run:FAIL